Coherent extrapolated volition

A proposal for what a superintelligent AI should ultimately aim at: not what people want now, but what they would want “if we knew more, thought faster, were more the people we wished we were, had grown up farther together.” Eliezer Yudkowsky introduced it in a 2004 paper written for what is now MIRI.

The idea

Writing down the right values directly looks hopeless: any fixed list encodes today’s mistakes, and a powerful optimizer would exploit its gaps (see Goodhart’s law). CEV instead specifies a procedure. The AI models each person, extrapolates what their idealized self would want, and acts where those extrapolated wishes agree (cohere) rather than conflict.

The approach is called indirect normativity: point the AI at a process for finding values, rather than at the values themselves. Nick Bostrom discusses CEV in Superintelligence (2014) alongside alternatives such as aiming directly at whatever is morally right.

What it is not meant for

In his 2016 Arbital entry, Yudkowsky calls CEV “not intended as something you’d do with the first AI you ever tried to build.” It would require an autonomous AI pursuing an open-ended mission without accepting correction, the hardest design to make safe. He suggests the first AI should instead perform a limited task, such as a Pivotal act. AGI Ruin - A List of Lethalities makes the same point: an AI that independently wants what humanity would want is too complex to get right on the first try.

So CEV is best read as a candidate for the long-run target of alignment, not a plan for the first systems.

Main objections

• Values may not cohere. Extrapolated human values might diverge rather than converge. If they disagree everywhere, CEV specifies nothing.
• Extrapolation is value-laden. Deciding what counts as “knowing more” or “growing up together” already involves value judgments, so the procedure may smuggle in its designers’ views.
• Whose volition? Living humans, all sentient beings, the dead, or only the project’s contributors? Paul Christiano argued in a comment on the Arbital page that no boundary is an obvious focal point. This connects to Who should control aligned AI.
• Majorities can be wrong. A supermajority’s extrapolated wishes could still endorse harming a minority. Yudkowsky’s Arbital entry proposes safeguards; critics doubt they are robust.
• Drift or progress? Letting values evolve under extrapolation could amount to moral drift, not moral improvement.
• It cannot be checked. Inspecting the output to verify it invites overriding it, which defeats the point of deferring to it.

Why it matters here

CEV is the clearest statement of an answer to “aligned to what, and to whom?”, the question AI alignment leaves open. Even people who reject it tend to define their alternative by contrast with it.

Related: Superintelligence, Orthogonality thesis, Plans for the AI transition.