Who should control aligned AI?

If alignment succeeds, who decides what the most capable systems do, and what keeps that power accountable?

Avoiding a catastrophe in the narrow technical sense would leave this question open. Whoever controls the first very capable systems could hold extraordinary economic and political power. Making AI do what its controllers want, the technical problem of AI alignment, does nothing to ensure that what they want is good for everyone else. See AI ownership, Political bargaining power, Permanent underclass, and Institutional resilience.

In his 2026 essay Against SURA, Scott Alexander argues that if one company ends up deciding whether to hand power to a government, keep it, or distribute it, the ethics of the people in that company matter. That is a reason for ethical people to stay inside. Critics replied that this assumes builders will be in control, treats everyone else at the table as unethical, and gives democratic governments no role.