AI alignment

AI alignment is the problem of making an AI system pursue the aims its builders intend, and keep pursuing them as it becomes more capable and meets situations no one tested.

It splits into two parts. The first is saying what we want precisely enough: any measurable stand-in for “good behavior” can be satisfied in ways nobody meant, a pattern known as Goodhart’s law. The second is getting a system to actually adopt that aim. Modern systems are trained, not programmed: training rewards outputs that look right, and nobody can yet read off from the result what the system is really pursuing.

Capability makes both parts harder. Being smarter does not by itself make a system’s goals better, as the Orthogonality thesis argues. A capable system with almost any goal has reasons to avoid being shut down or modified, as Instrumental convergence argues. So a mistake that is harmless in a weak system can become impossible to correct in a strong one.

A separate question sits on top: whose aims, when people disagree? That one is political rather than technical; see Who should control aligned AI. Coherent extrapolated volition is the best-known proposal for an answer.

How difficult this is in practice is disputed; How hard is alignment collects the strongest case on each side.