AGI Ruin: A List of Lethalities

Eliezer Yudkowsky. AGI Ruin: A List of Lethalities, LessWrong, 2022-06-05. Text and top comments read through the LessWrong API on 2026-11-04.

The most cited case that AI alignment will fail in practice. The post explicitly assumes orthogonality and instrumental convergence as background. The lethalities are a checklist, not a forecast assesses it.

Summary

The claim is narrow and severe. Yudkowsky does not say alignment is impossible in principle, nor that it requires perfect or provable values. He says that with the techniques we actually have, getting a powerful system that does not kill everyone is “an overly large ask that we are not on course to get”. His metaphor is a textbook from the future: with the simple methods that actually work, the problem might take six months; without it, we are doing everything on what he calls “the first critical try”.

The list is meant disjunctively. Each item is called a lethality because, in his view, any one of them is enough to cause failure if not solved. A critic therefore has to answer all of them, not just most.

The 43 numbered points are organized in three sections: why the problem is lethal and cannot be avoided by building only weak systems (A), the technical difficulties (B), and why he thinks the field is not on track (C). Most points restate a handful of problems from different angles:

1. No retries at the dangerous level (points 1–3, 10–15). AI capability is not capped at human level, and a sufficiently capable system can find a path to overwhelming power. “Unaligned operation at a dangerous level of intelligence kills everybody on Earth and then we don’t get to try again.” Alignment learned in safe conditions must survive a large shift to dangerous ones, and some failures, such as deception, only appear after the shift. See Can we learn from AI failures before they are fatal.
2. Weak systems do not get us out (points 4–9). Others will build strong systems within a few years, so a safe but limited system changes nothing; “There are no pivotal weak acts”. The capabilities useful for preventing catastrophe generalize to the capabilities that cause it. See Pivotal act.
3. Training shapes behavior, not goals (points 16–22). Optimizing a loss function does not install that loss as the system’s goal, just as natural selection for fitness produced humans who want other things. We have no way to target or verify inner goals, and human feedback contains systematic errors a capable system can exploit. The central claim: “Capabilities generalize further than alignment once capabilities start to generalize far.” See Goodhart’s law.
4. Correction is anti-natural (points 23–24). A system that pursues goals has reasons to resist being shut down or changed. Formal attempts to describe an agent that accepts correction have failed, and the alternative, a system that independently wants what humanity would want, is too complex to get right on the first try. See Instrumental convergence.
5. We cannot inspect or check a smarter system (points 25–36). We do not know what is happening inside the models. Training against detected bad thoughts trains the system to hide them. A system smarter than us can produce plans we cannot verify, and schemes such as boxing it or playing AIs against each other fail against a sufficiently capable system.
6. The field is not on track (points 37–43). Optimism is not grounded in experience of failure, the field cannot tell real progress from apparent progress, and no one has a plan. “This situation you see when you look around you is not what a surviving world looks like.”

Responses and discussion

The post drew more than 700 comments and several full responses. The strongest disagreements line up with the problems above.

Against “no retries”: deploy gradually. John Schulman, then at OpenAI, argued that much of the list assumes one leap far outside the training distribution. Deploying in small steps, with each model helping supervise the next, requires only small generalization at each step. He also pointed to evidence at the time that instruction-following training generalized surprisingly well. This is the core of the case for tractability in How hard is alignment.

Against fast, decisive power: culture and coordination. Matthew Barnett argued that humans dominate through accumulated culture and coordination rather than raw intelligence, that the first AGIs will be born into a coordinated civilization, and that capability will likely rise gradually. Replies argued that AI systems can share weights and copy themselves, which is a coordination advantage humans lack. This is a dispute about AI takeoff.

Against novelty, not substance. Evan Hubinger mapped most of the list to earlier work by other researchers, disputing Yudkowsky’s claim that only he could have written it. That is a disagreement about the field, not about the risk.

Against the pessimism, partly. Several careful readers accepted most of the problems but put survival odds far higher than Yudkowsky does. Paul Christiano’s Where I agree and disagree with Eliezer (2022-06-19) is the fullest version, and agrees on much more than its title suggests. Alex Turner (TurnTrout) later wrote that the list is fundamentally misguided (2023-01-24), especially its reliance on the evolution analogy.

On tone. A highly upvoted comment argued that the post’s combative style undermines it. Yudkowsky replied that years of politer writing had left people nodding along and changing nothing. The reply links to the framing in Death with dignity.

A later restatement: Lermen (2026)

Simon Lermen’s Can alignment realistically be solved under Plan C? (2026-11-03) restates several of these arguments in plainer language. He wrote it in response to the claim that people who believe alignment is nearly hopeless are relying on deference rather than reasons, and he asks whether a lead measured in months could be enough. It is also a reply to the alignment optimism in Against SURA. His arguments map onto the problems above:

• You only get one shot is problem 1. A model that cannot take over has good reasons to cooperate; one that can has different options, so lessons from weaker models may not carry over.
• Evolution and learned optimization is problem 3, with a new twist: humans reshaped the world around their own preferences, so a system reflecting on its own learned goals would likely do the same, without humans in charge.
• The tech tree passes through efficient unaligned superintelligence is new. Understanding intelligence well enough to align it would also reveal how to build much more efficient systems, which competitors could run before anyone aligns them.
• Corrigibility and persona alignment extends problem 4 to current methods. If formal corrigibility conflicts with coherent agency, training a system to play a helpful character may hide a similar conflict.
• Automating alignment applies problem 5 to current plans. Without a way to check proposals, asking AI for alignment plans optimizes for plans that sound convincing. See Can AI automate alignment research.

Commenters noted that most of these are restatements, and one asked which cruxes could be tested empirically rather than argued.