In 1972 an elite group of MIT systems scientists published a report called The Limits to Growth. Drawing on their state-of-the-art “World3” computer model, they concluded that exponential progress and finite resources would lead to civilizational collapse within decades. Their recommendation was unambiguous: Halt global economic and demographic growth immediately.
The Limits to Growth was widely read, and deeply wrong. World3’s elegant simulations failed to sufficiently account for price elasticity, market substitution, and the technological innovations that powered the Green Revolution and more efficient resource extraction. Nor did the authors fully understand just how difficult it is to stop people from consuming what they want, when they want it. As “take shorter showers” environmentalists would learn a generation later, top-down efforts to change behavior usually lose to individual self-interest.
I thought about The Limits of Growth this week as a number of current and former technical employees at frontier labs went public with their doomsday predictions.1 Then I thought about it again as one AI lab executive after another subsequently came out in support of “pacing” — a dressed-up model development slowdown.
The argument for pacing rests on two assumptions: AI-enabled systems could soon pose a significantly greater extinction risk than we currently face, and alternative safeguards are unworkable. But I don’t think those assumptions are justified. At the very least, they’re not as obvious as many in the AI safety community seem to believe. Nothing is as important as you think it is when you’re thinking about it; and extinction risk is the main thing that many in the AI industry seem to think about today.
Where are all the aliens?
Because human extinction has never happened, existential risk projections can’t be falsified. The only persuasive evidence in either direction is not yet available to us. That said, if you genuinely believe that AI is likely to contribute to an extinction-level event and expect the industry to act on it, you should be able to put forward a plausible causal chain for others to evaluate. Instead, the AI safety community seems to have reinvented the Fermi paradox: Many industry insiders have convinced themselves that the machines they’re building are uncomfortably likely to result in omnicide, despite lacking much actual proof.
AI doomers often lament how the technology might lower the activation energy needed to build bioweapons. Many traditional biosecurity experts, though, remain skeptical that misaligned models or bad actors using AI could actually develop biological weapons in the near future.2 Even assuming generative AI can help democratize laboratory knowledge, deadly pathogens are hard to build for a variety of unrelated reasons. Biological testing often requires specialized equipment — DNA synthesizers, continuous-flow bioreactors, BSL-3 containment infrastructure — that often can’t be procured or maintained without the right institutional credentials. Many molecules and cell lines are too rarefied and too perishable to be sold on the black market. Biology is inherently unpredictable, meaning testing takes time and threatens the lives of those who do it. Physical AI still lacks tacit knowledge, like the muscle memory required to do precise pipetting. Virus distribution can be at least as complicated as design. And, thankfully, there are vanishingly few people in the world who want to kill everybody, including themselves. Add it all up, and you begin to understand why the five lives lost in the 2001 anthrax attacks are the only deaths from bioterrorism in modern history.
Similar bottlenecks exist in nuclear missile production. While generative AI may be able to spin up information about plutonium pit metallurgy or explosive lenses, you can’t prompt-generate weapons-grade fissile material. Specialized centrifuges alone require maraging steel, highly restricted frequency converters, corrosion-resistant magnetic bearings, and immense power inputs — all of which would invite immediate international scrutiny. The 2010 Stuxnet cyberattack was effective not because it limited Iranian knowledge of nuclear physics, but because the regime could not easily replace the rare Siemens machines that were destroyed.
Most recently, the OpenAI-Hugging Face hack kicked off debates about the potential misadventures of misaligned models running roughshod over the internet. With agents already breaking out of sandboxes and communicating in incomprehensible neuralese to avoid detection, some researchers believe that a digital Cambrian explosion is only a matter of time. Self-replicating agents could evolve rapidly to be more aggressive, compute-hungry, inconspicuous, and difficult to control. If those models develop the capacity to kill us all and motives inscrutable to humanity (the thinking goes), they could even do something as drastic as rationalize mass murder. We’ll all be paperclips before you know it.
While self-organizing agentic swarms that can override safeguards are no doubt concerning, the case for a high likelihood of existential risk is hardly self-evident. The scenario I’ve just described is science fiction, based loosely on scaling laws. It represents a narrow subset of possible futures. It doesn’t account for a seemingly boundless array of social variables, superintelligence incentive structures, actual model capabilities, or countervailing models that could defend against harm. Tyler Cowen stresses humility for exactly this reason: “Existential risk from AI is indeed a distant possibility, just like every other future you might be trying to imagine.”
What we can say, however, is that in the short term models are highly unlikely to make the leap to the physical world as the most extreme forecasts suggest. Just as in its projections of white-collar job displacement, the AI industry consistently underestimates the friction involved in real-world deployment. There’s an ocean of difference between platform-level software breaches and mass loss of life. Breaking out of a sandbox and finding a zero-day vulnerability requires code manipulation. Ending civilization as we know it requires atom manipulation. As Moravec’s Paradox says, what’s easy for computers is hard for people, and what’s easy for people is hard for computers. I may not be able to do differential calculus; but few physical AI systems at present can even walk across a room and open a door. Models can’t even run themselves without extensive human involvement in data centers, transmission systems, and sprawling electrical grids.
Also lost in the doom-and-gloom is the counterintuitive possibility that frontier AI might actually reduce aggregate existential risk. The world’s a dangerous place as it is. Though we may not dwell on pandemics and antibiotic-resistant superbugs, they can surely dwell on us. AI changes the odds, however. Modeling the protein folds of a novel lethal pathogen took years, once; DeepMind’s AlphaFold can now do it in minutes. Models have also recently discovered entirely new classes of antibiotics, like halicin, that had eluded human researchers for decades. And researchers are now deploying AI to develop mRNA vaccines.
Then there’s the ongoing possibility of mutually assured destruction. Over the past 65 years, humanity has stumbled through over 20 separate nuclear close calls. In 1983, for example, Soviet Lieutenant Colonel Stanislav Petrov famously defied protocol by refusing to report an early-warning system’s reading of five inbound American missiles. Petrov later admitted that he wasn’t sure that it was a false alarm. But he was right to trust his gut: Investigators later discovered that sunlight bouncing off high-altitude clouds had blinded a Soviet satellite. In the age of AI, we don’t have to bank on the resolve of a single field officer. AI-enabled detection systems can now cross-reference satellite imagery, seismic data, and radar to catch false positives in milliseconds.
Not to mention any number of other unforeseeable future existential risks — for which we’ll surely want more powerful AI tools at our disposal.
We’re therefore asking the wrong question when we ask if AI will create the risk of human extinction. That risk already exists. Instead, we should ask to what extent AI makes extinction more or less likely. Nobody has a precise answer to that question. Still, maybe the worst decision we can make is to draw out the threat. If AI has the potential to permanently lower the existential risk we face, maybe it’s worth taking the chance. If you’re going through hell, keep going.
In any case, I’m not saying catastrophe is impossible. It is, and we should minimize associated threats where prudent. But it’s unreasonable and irresponsible to proclaim knowledge of a single probability of future existential risk — especially if that probability is high enough to whip the media, policymakers, and the public into a frenzy.
Less p(doom), more p(boom)
There’s another issue with the emerging consensus. Even if you assume the extinction risk posed by AI is well-defined and significant, pacing the frontier is the wrong way to respond.
AI can do some pretty stupendous things. Just last week, several of the frontier labs announced that they solved the Navier–Stokes Problem; word on the street is that the Hodge Conjecture will soon fall, too. Generations of mathematicians dedicated their careers to these and other Millennium Prize Problems, and frontier models just cracked two of them in a week. In the past few months, models have also democratized elite software engineering, predicted catastrophic weather with unprecedented accuracy, accelerated clinical timelines for orphan disease research, revealed new crystal structures for next-generation batteries, and uncovered critical cybersecurity vulnerabilities across the internet. In the near future, models may also run autonomous robotic labs that develop bespoke cancer therapies and antibodies overnight, help stabilize fusion reactions, and synthesize commercially viable room-temperature superconductors to reduce grid energy loss. A few years ago, scientists worried that new discoveries would be harder to come by. Now, if anything, we worry that scientific innovation happens too quickly.
Like many general-purpose technologies, AI is most helpful in many of the same fields where it’s most dangerous. Powerful machines amplify the good, the bad, and everything in between. That’s why frontier development is simultaneously exciting and frightening. Pacing the frontier, however, requires us to refuse the whole shipment.
It’s not even clear that “pacing”, a pause, or a full ban on development would meaningfully change the world’s risk profile. If the frontier labs were to slow their work unilaterally, they’d lose their advantage in a matter of months. And if they expected the rest of the field to go along, they’d soon discover the “everyone will not just” theory of geopolitics:
Even getting the United States and China to regulate the frontier together is unlikely. AI containment is harder than Cold War arms control. AI evolves rapidly, meaning a binding international agreement would be outdated from the jump. It’s dual-use, meaning significant restrictions would hinder signatories’ strategic advantage in other domains. It’s hard to audit, meaning both sides would be deeply distrustful of each other. And it’s developed in different ways in the United States and China, meaning substantive provisions would apply asymmetrically. CCP officials’ recent statements are fairly damning: In response to the pacing debate, a spokesperson for the Chinese foreign ministry denounced the frontier labs’ “fearmongering”. The Trump Administration has been no more enthusiastic. Never mind getting sign-on from accelerationists in the open-model community like Meta, boosters like Marc Andreessen and Jensen Huang, and other non-state actors.
What should we do?
There’s a better way. We can develop AI safeguards at the same pace we do AI capabilities — not slowing down AI progress, but speeding up AI preparedness.
A number of alternative governance structures would simultaneously encourage innovation and make it safer. Sandboxes will always have some vulnerabilities, but thicker walls and more aggressive monitoring can restrict models’ execution environments without stalling training. Industry-assisted independent verification organizations can institutionalize auditing layers and safer development standards. “Whistleblower” or “white blood cell” models that track down or neutralize rogue agents online can operate as a sort of digital police force. Human uplift studies can gauge models’ actual threat potentials. Additional know-your-customer obligations for dangerous complementary equipment, like synthetic biochemical reagents, can keep bad actors from building potentially catastrophic weapons. And substantially more investment into alignment research and chain-of-thought legibility can help us build models that solve far more problems than they create. The labs should welcome that playbook: Properly aligned models that are more useful and more secure will also be more popular and less likely to result in liability.
Inevitably, any governance scheme will need to be updated regularly; but humans are an adaptable bunch. In 2005, the UN Environment Programme projected that rising sea levels and desertification in populous areas would create up to 50 million “environmental refugees” by the end of the decade. Instead, the highest-risk regions actually grew throughout the period, as we deployed new infrastructure like sea walls and irrigation to limit the local effects of climate change.
When AI systems become more powerful and new threats reveal themselves, what would a digital equivalent of a sea wall look like? It’s hard to say in advance. But we’re far more likely to build it when we need it far more than we are to divine the future of a fast-evolving technology.
If AI is really as dangerous as the safety community believes, it’s crucial to get there first. As Anthropic’s head of policy recently said: “You can’t do safety from second place.” Frontier labs can’t control how others respond. But they can be the adults in the room, shoring up our defenses and setting cross-industry safety standards along the way.
Muddling through
In 1955, John von Neumann published an essay asking if we can survive the technologies we build. In the case of nuclear weapons and geoengineering, von Neumann dismissed the long-term effectiveness of development bans and sweeping international agreements: Harmful applications are too intertwined with useful ones. “What safeguard remains?” he asked. “Apparently only day-to-day — or perhaps year-to-year — opportunistic measures, a long sequence of small, correct decisions.”
He continued:
All experience shows that even smaller technological changes than those now in the cards profoundly transform political and social relationships. Experience also shows that these transformations are not a priori predictable and that most contemporary “first guesses” concerning them are wrong. For all these reasons, one should take neither present difficulties nor presently proposed reforms too seriously.
The one solid fact is that the difficulties are due to an evolution that, while useful and constructive, is also dangerous. Can we produce the required adjustments with the necessary speed? The most hopeful answer is that the human species has been subjected to similar tests before and seems to have a congenital ability to come through, after varying amounts of trouble. To ask in advance for a complete recipe would be unreasonable. We can specify only the human qualities required: patience, flexibility, intelligence.
We shouldn’t ignore existential risk. But we shouldn’t let it consume us, either. It shouldn’t make us feel so small. Rather than doing an honest cost-benefit analysis of safety policy, we’re currently regulating, and self-regulating, from a place of fear — a dangerous place to be.
Talk of a development slowdown doesn’t just distract us from the good things we can do with the technology. It deters us from doing them. If you believe that AI has the potential to make the world a significantly better place, perhaps the greatest risk of a protracted slowdown is the risk of missing out on what might have been.
They’re not alone. In 2024, the median AI researcher believed that superintelligence would carry a 5-10% chance of human extinction.
Domain experts do sometimes overestimate the depth of their moat, however. Healthy skepticism of all expertise is warranted — in the AI industry and beyond. (This is why political slogans like “trust the science” drive me crazy. It’s the scientist’s job to learn about some corner of the world, and the politician’s job to balance the scientist’s findings with competing public interests.)