ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

Paul Christiano: We May Be Six Months From AI That Could Overthrow Governments, and the Right Move Is to Pause Algorithmic Progress

In a November 2026 podcast appearance and accompanying lectures, AI safety researcher Paul Christiano argues that frontier labs should stop improving AI algorithms until their security posture can protect against theft and their alignment methods can handle the systems they have already built. He estimates a 1-in-3 chance of superintelligence within a year on the current trajectory and a 10% risk of loss of control over the next twelve months, with the theft of model weights by state actors as the most likely path to disaster.

2026-11-20 23:24 GMT+8

Paul Christiano's central claim in his November 2026 Dwarkesh Patel podcast appearance is that the window for safe AI development is closing faster than anyone expected, and that the correct response is to pause the algorithmic improvements that make AI systems more capable per unit of training effort. He estimates a 1-in-3 chance that superintelligence powerful enough to overthrow governments emerges within a year on the current trajectory, and a 10% risk of losing control over AI systems within the next twelve months if development continues at full speed.

The mechanism Christiano describes runs through automation of AI research itself. He points to reporting from Anthropic and METR suggesting the pace of AI progress has roughly doubled over the last six to twelve months, even accounting for diminishing returns and compute bottlenecks. If AI systems increasingly write the code and design the experiments that build the next generation, the doubling accelerates: eight months, then four, then two, approaching physical limits within about sixteen months. Christiano says this is the most obvious top priority right now — understanding whether we are three, six, or twelve months from systems that could overthrow governments.

His policy proposal follows from a specific asymmetry he identifies. Pausing now is more feasible because the most dangerous capabilities have not yet emerged, and because internal models are at least somewhat protectable in a way that deployed systems are not. He states it seems impossible to protect the model weights of a system deployed across major cloud providers against a state actor, whereas for an internal model there is at least some chance. Pausing later, after those capabilities exist and are deployed, is far more dangerous because the weights could be stolen, and once that happens the race becomes forced and the security posture of whoever holds the weights is worse than that of the original developer. He also notes that much of China's current AI progress depends on leaked American algorithmic advances, so if the US stops now, China cannot fully catch up for a long time. He describes the theft scenario as the most likely doomed path.

Sign in / Sign up
ReadingResearchRadarInvestment framework

What Christiano proposes labs do right now is stop accumulating further algorithmic innovations in pre-training efficiency, reinforcement learning, and architecture design until three conditions are met: security posture adequate to defend those innovations if they must be withheld from deployment, alignment methods sufficient to control the systems already built, and governance processes mature enough to notice when things start going wrong. He explicitly notes he is not speaking as an OpenAI board member, though he confirms he is one.

The pessimism in his framing is structural rather than rhetorical. He identifies interpretability of superintelligent minds as the most promising available plan, but assigns it only a 5% to 10% chance of working, and expects it to function for some number of months before breaking down unpredictably. His optimistic scenario involves two sequential 50/50 saving throws: a 50% chance of finding something that works, and a 50% chance that if nothing works, we find a way to stop. That yields a 75% chance of things being broadly okay. Zvi Mowshowitz, who calls the situation grim, says he would very much take a 75% chance of things turning out okay, while noting the odds are against us.

Christiano's argument is attributed, not independently verified. His probability estimates rest on noisy benchmark extrapolation and his own judgment about how quickly automated AI R&D compounds. The METR report he cites was characterized by Zvi as confirming existing expectations rather than delivering a new measurement. The theft scenario depends on assumptions about state-level cyber capabilities and the difficulty of protecting weights across cloud deployments. None of these are established facts; they are the premises of one researcher's case for a specific policy action, presented with unusual concreteness about timelines and probabilities.

Sources

  • thezvi.substack.com↗
Continue researching →