Yudkowsky on pivotal use

Attributed author: Eliezer Yudkowsky, account @allTheYud.

Original link: X post on pivotal use.

The text below was supplied by Eddie with this link. X could not be fetched directly when this note was created, so the wording and account attribution have not been independently verified against the live post. The original posting date is not recorded.

Argument

The passage distinguishes making one limited AI safe from preventing catastrophe caused by other developers. It argues that the protective task determines the capability level that needs to be aligned. In the author’s argument, the difficulty of safely achieving that task supports a treaty rather than a technical substitute. See Pivotal act and AI alignment.

Supplied text

You’re misunderstanding the notion of what I apparently have to call “pivotal use”. It’s the reply to the standard novice’s argument, “But I have invented a very clever way to weaken AI until it is safe; so there is no need for your treaty to prevent further escalation, for I can build a very safe AI just by <having it only prove theorems, having it only play chess, burying it on the Moon and never running it, etc>!” To this the reply is, “Your AI is now Safe But Useless, so you doing this doesn’t prevent Anthropic or OpenAI from destroying the world a month later.”

For an alignment method to matter, it must be able to prevent unaligned AGI from destroying the world a month later. Any way you could stop that from happening would be a “pivotal use” and that particular use would determine the power level of AGI you needed to align, hence the engineering difficulty of aligning it that would be greater than the difficulty of building a safe chess engine.

It’s harder to build an airplane that can fly faster than sound and carry 100,000 tons of cargo than one that can fly at 20mph and carry 100 pounds. In real engineering done by sane people, the task to be performed determines the engineering difficulties. “I can build a safe airplane!” boasts the novice, and the sane person replies, “Build a safe airplane that does what?”

“There is no need for a treaty to prevent the Earth from being destroyed by AI-builders; I can see how to build a safe AI!” boasts the novice. We reply, “How does this you building this very safe AI prevent other AGIs from destroying the world?” And the novice stares blankly and then flushes these words out of his brain, because he does not think ahead to the equilibrium; when he imagines building an AI himself and imagines that AI not destroying the world, he imagines himself having solved the problem.

The “pivotal use” is what the novice would have to claim to be able to align, if by the truth of his claim, he were to establish that no treaty to prevent Earth’s destruction would be needed.

It is a concept that is a lemma along the way to trying to explain, “You cannot align an AI that is powerful enough to prevent humanity’s ruin by unaligned AI, so we need a treaty rather than a technical solution.”

If there is some very clever way to say this in a way that will be understood by minds at the Altman level then I do not myself know it.