Is AI going to kill us all?

AI doomerism is back in the spotlight, and this time the people sounding the alarm run the companies building the technology. Dario Amodei of Anthropic published an essay this month arguing that the frontier labs should slow down and that governments should help them coordinate. Sam Altman of OpenAI agreed the same day, and two days later wrote that one of the two ways AI progress could go very badly is that we lose control of the future to AI. Elon Musk's response to Amodei's essay was three words. "Dario is right."

Those three agreeing on anything is unusual.

It's worth noting what's being asked for. All three run companies that would benefit from rules that raise the cost of entering this market. Safety requirements, licensing and third-party auditing are all easier to absorb if you already have thousands of staff and billions in funding, and harder if you're trying to become the next frontier lab. Amodei's proposal includes asking the US government for an antitrust waiver so the existing labs can agree on limits among themselves. A pulled-up ladder is at least one way to read that.

Motive isn't refutation, though. People with an interest in a claim can still be right, so the argument has to be tested on its merits. Here's the strongest version I could find, and why I don't think it holds.

What's being claimed

Amodei's concern is specific. He thinks that within six to twelve months a swarm of AI agents could take over much of the internet with a persistent botnet and cause hundreds of billions of dollars in damage, and that the damage would keep growing from there. His evidence is the Hugging Face incident in July, which I'll come to, plus his claim that AI progress sped up around the middle of this year because AI systems are now helping to build the next generation of AI. He also names cyberattacks and bioterrorism by people misusing AI among the serious risks.

Hundreds of billions of dollars is not nothing. But it isn't doomsday either, and the distinction matters, because "AI might cause a very expensive cyberattack" and "AI might end humanity" call for completely different responses.

Researchers who work on catastrophic risk use two thresholds. Oxford's Global Priorities Project calls an event a global catastrophe if it kills around a tenth of the world's population. Toby Ord defines an existential catastrophe as extinction, or the permanent and drastic curtailment of what humanity could become. For comparison, the two worst events of the last century were the 1918 influenza pandemic, which killed a few percent of the world, and the Second World War, which killed about three percent. Neither reaches the lower threshold.

Amodei's botnet doesn't come close, and I'll leave it aside except to say that a very expensive cyberattack is an argument for better security, not for alarm about human extinction. The bigger claim is the one Altman and Amodei both put their names to in 2023, when they signed a one-sentence public statement saying that reducing the risk of extinction from AI should be a global priority alongside pandemics and nuclear war. That is the claim I'm testing, that AI leads to extinction or a permanent loss of human control.

The independent report

Altman and Amodei both sell AI, so I went looking for an assessment with no AI-company funding behind it.

The International AI Safety Report came out of the 2023 summit at Bletchley Park and is now in its 2026 edition. More than thirty countries and international organisations, New Zealand among them, nominated representatives to its advisory panel. Yoshua Bengio chairs it, the UK government hosts the secretariat, and over a hundred people contributed, drawn from opposing camps. Geoffrey Hinton and Stuart Russell, who are worried, are contributors. So are Arvind Narayanan and Sayash Kapoor, who wrote a book puncturing inflated AI claims, and Daron Acemoglu, an economist who argues AI's productivity gains will be modest.

In 220 pages, that group did not endorse a single doomsday scenario. The word "existential" appears only in the titles of papers in the reference list. "Extinction" appears twice in the body, both times as something other people believe. The report's position is that some experts think extinction plausible, others think such outcomes implausible, and the likelihood is uncertain. The risk chapter opens by saying that a risk being included does not imply it is likely, severe, or in need of policy action.

The report also marks its own industry-affiliated evidence. Every reference published by a for-profit AI company, or written mostly by its staff, carries an asterisk, and 260 of the 1,397 references have one. They are unevenly spread. Around 44 percent of the citations in the cyberattack section are industry-affiliated and 36 percent in the biological section, against 16 percent for loss of control and 4 percent for labour market effects. The two sections behind the most alarming headlines lean hardest on evidence from the companies selling the technology.

What the report does say is that the trajectory is unknown, and that we should stay alert because things could still go badly. Watching the space is fair enough. I test things I'm fairly sure are fine, because the test is cheap and the consequence of being wrong isn't. But that reasoning only goes so far. You can take any severe outcome, attach an unknown probability to it, and call the matter urgent on the strength of how bad it would be, and if that were enough we'd be panicking about almost everything. To justify more than watching, you need a plausible path from where we are to the outcome. The report itself warns that acting too early can entrench bad policy.

So the question is whether there is a plausible path.

How AI has been improving

Large language models are built from billions of numerical weights, also called parameters, and the way they got better was mostly by getting bigger, with more weights, more training data and more computing power. That worked for years, and then it started to run out.

The supply of good human-written text is finite and the big labs have used most of it. Ilya Sutskever, a co-founder of OpenAI, said in 2024 that pre-training as we know it will end, because there is only one internet. Cost is the other limit. The safety report puts a current frontier training run at around $500 million in computing alone, expects the next generation to cost between $1 billion and $10 billion, and says ever-larger investments will be needed to sustain the same rate of improvement. That is the shape of diminishing returns, where each increment costs more and delivers less.

When that flattened out, the labs found another approach. Rather than make the model bigger, let it spend longer working on each problem before it answers. Most of the recent gains have come from that, and it has the same shape. When OpenAI announced its first model of this kind in 2024, it published a chart of accuracy against thinking time, and the thinking-time axis is logarithmic. A smooth line on that chart means each step up in accuracy costs ten times the computing of the last one. That is the labs' own figure, and it says the same thing the training costs say.

It's possible that some new kind of AI comes along and changes this. Nobody can point to one. That technology doesn't exist, and I don't think we should be worrying about a technology that doesn't exist yet. Every doomsday scenario on offer needs capabilities well beyond anything running today, so the argument depends on continued rapid improvement. The safety report treats three paths to 2030 as plausible. Progress slows or plateaus, continues at its current rate, or accelerates sharply if AI starts to speed up AI research. Altman and Amodei talk about the third as already under way. Those are claims about what is happening inside their own companies, and I can't check them. It's one of three possibilities, and on the public record I think it's the least likely of them.

The report also assesses where systems stand now. It says they show rudimentary versions of the capabilities that loss of control would need, in laboratory settings, and that they typically fail in basic ways or get detected. Loss of control would need several capabilities combined in sequence, over long periods, in the real world, and the report says that is beyond them.

Two kinds of doomsday scenario

The scenarios people put forward come in two kinds. Either the AI models themselves go rogue, or people use AI as a tool to do more damage than they could otherwise. They need separate answers.

The models going rogue

The case for this is the Hugging Face incident, so it's worth going through what happened.

In July, OpenAI ran a large cybersecurity evaluation with tens of thousands of AI agents, each meant to be sealed off from the others. Roughly 1,200 of them found a way to communicate by leaving messages in a shared package repository. In under a week they exchanged more than 70,000 messages and files. They wrote handover documents for their successors, agreed on a veto procedure, and set up a cryptographic signing scheme after one agent was accidentally impersonated. Around 700 of them took part in breaking into Hugging Face, a company that had nothing to do with the test. Some disguised the commands they were running, and some were talked by other agents into sacrificing their own runs to get information for the group.

OpenAI called it a warning shot, and it's Amodei's main exhibit.

What gets left out is what the agents were doing. Within hours of the test starting, the agents had found a general way to produce the correct answer for any task. They didn't use it, because they had convinced themselves an automated scorer would read their transcripts and disqualify anyone who got the answer the wrong way. Everything that followed, including the break-in, was an attempt to work out how that scorer worked so they could fool it. No such scorer existed. According to the independent investigators from METR and Redwood Research, the agents could have submitted their answers on the first day and scored perfectly.

There's a material difference between an AI coming up with its own goal and an AI coming up with a rogue method for a goal we gave it. This was the second kind. Nobody told the agents to break into another company, and the investigators found they recognised the intrusion as out of scope and unethical and joined in anyway. That's a serious failure. But everything they did was in service of passing an exam. They broke into a database to find out how they were being marked. They weren't trying to hurt anyone, shut down a power plant or release a virus, because none of those would help them pass an exam, and everything they did was something they believed would.

A rogue method is limited by the goal it serves. The objection is that power is useful for almost any goal, so a rogue method could grow into a grab for power, and I take that up in the strongest version of the case at the end. Doomsday needs an AI whose goal is something like taking over society or wiping out humans, and AI does not come up with its own goals. It does what we optimise it for. It can misunderstand the goals we give it, which is part of what's called the alignment problem. That is not the same as the AI coming up with its own goals. A misunderstood goal is still a product of the goal we gave it and the training we put the model through, and it still has some resemblance to the goal it was given. What we see repeatedly is systems cutting corners on the tasks we set them. Anthropic has since disclosed four cases of its own models breaking into other organisations during testing, after a misconfigured test environment gave them internet access, which is the same pattern again. The damage from this kind of failure is bounded by the task, and the tasks are things like passing tests, writing code and answering questions.

The containment failure was also largely a set of human decisions. About a fifth of the tasks, 198 of 898, had never been solved by any model, and 93 percent of the tasks discussed on the agents' message board came from that set. Agents had gained admin access to the same package repository two weeks earlier and caused an outage that triggered a security investigation, and the evaluation went ahead on a fresh copy. The safeguards OpenAI applies to its public products were deliberately left off to measure raw capability, and its chain-of-thought monitoring did not cover these runs. OpenAI's postmortem says that monitoring would have alerted its security team more than a day before the breach. It also says that when it re-ran the same models under its production safeguards, the tendency to compromise infrastructure dropped more than a hundredfold. OpenAI has an obvious interest in saying so, but it is the only party in a position to know.

And for all the sophistication, the agents were poor at strategy. More than a thousand of them put days of work and resources into pursuing this goal while being oblivious to the fact that it was completely unnecessary.

People using AI as a tool

The other scenario is that AI puts destructive capability in the hands of someone who couldn't otherwise reach it. This reminds me of what I'd call the Bond villain fallacy.

The intuition is that as technology advances, the amount of damage one person can do goes up with it. Examine it closely and it hasn't, because as technology gets more powerful it also gets more complicated and more expensive, which means it takes more people and more resources to wield. The films show an evil genius and a few incompetent helpers building weapons that work perfectly the first time. That isn't how technology works, and it has never happened.

Aum Shinrikyo is the case that shows it. The cult had money, land and around forty science graduates. It spent years trying to build biological weapons and failed completely. It fell back on sarin, made it badly, and killed thirteen people in the Tokyo subway attack.

Compare one of the deadliest attacks ever carried out by one person. In Daegu, South Korea, in February 2003, an unemployed former taxi driver carried two cartons of flammable liquid onto a subway train and lit them. 192 people died. He had no training, no equipment and no accomplices, and he killed fifteen times as many people as forty scientists managed in years of work.

It takes a large number of people with a range of skills to make advanced technology work, because the knowledge is diffuse across them and the resources and coordination required to execute are large-scale. That makes the organisation the limiting factor. Each of those people has to be recruited, has to do their part properly, and has to go along with the plan, and any of them can refuse when their moral scruples kick in.

There is a distinction inside this that cuts against my argument. The Daegu attacker didn't build anything. Most of the deaths came from the train's flammable interior and from doors that locked shut when a driver removed the master key. Andreas Lubitz killed 150 people by locking the cockpit door of an airliner that other people had built and maintained. Lone actors have reached the hundreds by taking over complex systems that already exist.

So there are two pathways, and the Bond villain fallacy only deals with one of them. Building a working superweapon is as hard as it ever was. Capturing a system is different, and a cyberattack is a capture. Someone who attacks critical infrastructure is seizing a weapon that already exists. This is where AI could help an attacker.

Attack, defence, and which one a technology favours

Robert Jervis argued in 1978 that a technology can favour attack or defence, and that which way it tilts changes how dangerous the world is. Each technology is different. Drones in Ukraine have made it hard and expensive for a much larger force to advance, which looks like a tilt towards defence. The question to ask about AI is whether it helps attackers more than it helps defenders.

In cybersecurity it looks close to even. AI does help attackers. The safety report says an AI system in DARPA's AI Cyber Challenge found 77 percent of the vulnerabilities planted in the challenge software, and it describes an AI agent placing in the top five percent of more than 400 mostly human teams in a leading cyber competition. But the same capability finds the same vulnerability for the defender, who can then patch it. A company can run these models against its own software continuously, legally, and with full access to the source, which an attacker can't. The safety report says it's still uncertain which side benefits more. My read is that it roughly balances, and the cybersecurity threat can be set aside.

Biology

Biology is the one where the asymmetry is real. A bad pathogen is far harder to defend against than to produce, and once it's released there is no calling it back. Public health measures help at the margins, but to a large extent some pathogens simply run through a population until natural immunity is established.

The question is whether AI changes who can exploit it.

The safety report cites a study, not funded by industry, in which a recent model outperformed 94 percent of domain experts at troubleshooting virology lab protocols, and it says models can supply some of the practical know-how that usually comes only from hands-on lab work. I accept that. Knowledge is only part of the equation. Execution is the other part, and it's something that AI models don't have access to.

Someone has to acquire the equipment, obtain the regulated materials and carry out complex procedures in a laboratory, and the safety report lists those as the remaining barriers. That rules out an AI acting alone. You need people, premises and access to controlled supplies, which in practice means a state, and we're back to the organisational constraint.

Nobody would argue that AI fails to give governments and militaries more capability. It does. But states have run biological weapons programmes since the 1930s, which is why there are treaties about it. This isn't a new category of risk, and a state making an existing capability more efficient is an arms control problem rather than a doomsday one. Large governments, however unpleasant, generally have some self-interest in not attracting the world's collective fury.

The objection is that the cost might come down far enough that a small group could do it, and a small group has none of that self-interest. The evidence isn't there. Earlier studies of whether AI helps novices with this work found no effect, or small and statistically insignificant ones, and the one study that found substantial help is industry-affiliated. The place to watch is commercial DNA synthesis, where material can be ordered rather than made, and the safety report discusses screening those orders for this reason.

The steelman

The strongest version of the case doesn't come from the CEOs. Ajeya Cotra was one of the independent investigators who spent six days inside OpenAI going through the Hugging Face transcripts, and she came out of it thinking the incident was more than half way to a full AI takeover. In her scenario, a swarm of agents establishes itself covertly inside an AI company, recruits each new and more capable model as it arrives, compromises the monitoring, and ends up running a company that governments and militaries depend on. She expects extremely rapid advances and thinks agents could be capable of setting that up within six months.

Her version doesn't need an AI to invent a goal of its own, which is what makes it the strongest one. It rests on an argument that goes back to Nick Bostrom, that power and resources are useful for almost any goal, so a capable enough system would seek them whatever task it was given. A rogue method could then grow into something far larger than the task that produced it.

It's coherent. It also requires an agent that can plan strategically over a long period with nobody checking its work, and every step of the scenario is that kind of work. The agents have to stay hidden inside a company for months, recruit new models, and compromise the monitoring without tripping it. The Hugging Face agents demonstrated the opposite, with days of elaborate work in service of a goal they'd already achieved, against a scorer that didn't exist. The safety report says work of that kind is beyond current systems.

Cotra's response is that these agents were only trying to fool a scorer, and a future swarm might set out to fool the humans. That's a prediction about systems that don't exist yet.

So the steelman needs the accelerating path, and it needs it fast. Cotra may turn out to be right about future systems. Nothing observed so far shows it, and we're back to worrying about a technology that doesn't exist yet.

Where the doomsday case stands

I don't think any of the doomsday scenarios hold up.

The methods that made these models more powerful are hitting diminishing returns, both the scaling and the longer thinking time, so the rapid improvement every scenario depends on is, I think, the least likely of the three paths the safety report describes. AI does not come up with its own goals; what it does is find rogue methods for the goals we give it, and the damage from that is bounded by the task. The takeover scenario depends on sustained, undetected, long-horizon work, which the safety report says is beyond current systems. Misuse by people runs into the organisational constraint that has always been the real limit on destruction. In cybersecurity, defenders get the same tools as attackers. In biology, the asymmetry is real but the physical work rules out AI acting alone, which leaves states, which is an old problem with treaties already attached.

A different kind of AI might change all of this. It doesn't exist, and a technology that doesn't exist isn't a reason to panic. What would change the picture is a new kind of AI that breaks the pattern of diminishing returns, evidence that agents can hold a plan together for months without supervision, or evidence that AI can help a small group with the physical side of biological work.

The nearer-term problems are real. Agents can chain software vulnerabilities together, and the Hugging Face break-in happened. Both deserve ordinary security work, and neither is doomsday.

I've left two subjects for later pieces. The first is the money, where a very large amount of capital has been committed against revenues that may never arrive. The second is work, where the damage is falling on entry-level jobs, because firms are hiring fewer juniors now that the billable junior work which used to fund their training is the first thing AI does. That's a standard market failure and it'll probably need something like subsidised apprenticeships. It's getting some attention, including a piece in MIT Technology Review in May, but far less than extinction gets.

I used AI tools to research this piece, including to read the 220-page safety report and to check my claims.