Should safety-focused labs keep building?

Should people and organizations who believe advanced AI could be catastrophic help build it?

Why it matters

The answer determines where much of the safety community’s talent and money goes, and what to ask of labs that talk openly about risk. It is downstream of the other cruxes: How hard is alignment, Can AI automate alignment research, and Can AI progress be paused.

The case for building

• If others will build it regardless, it is better for careful developers to lead.
• Safety research on the most capable systems requires access to them.
• Labs and their employees influence policy, norms, and public understanding, and insiders who warn about risk are hard to dismiss.
• A lead can be spent on safety at the critical moment, instead of handed to less careful competitors.
• If alignment works, the people in control of powerful AI will shape what follows. See Who should control aligned AI.
• A country or lab that stops alone gives up its bargaining position. Scott Alexander makes this case, and the case for lab money funding pro-regulation politics, in Against SURA.

The case against

• Participation accelerates the race: each frontier lab pushes the others to move faster.
• Commercial incentives can gradually change what people believe and how hard they push back.
• “Race only if you are one of the good guys” generalizes badly, because every racer believes it is the good guy.
• Stopping, or loudly leaving, is a costly signal that makes the warning credible.
• If careful developers cannot actually solve alignment in the time available, leading just means being first to build something unsafe.
• Insiders’ track record is weak. Daniel Kokotajlo argues that most safety staff inside labs have not done better than the baseline of quitting and speaking out.

A spectrum of positions

SURA, short for “Stop Unilaterally + Raise Awareness”, is Scott Alexander’s name for the view that safety-minded labs, their staff, and countries should stop building frontier AI even if others continue. The fallback plan, if no global pause happens, is for the most careful lab to spend its lead on alignment before others catch up; see Plans for the AI transition. In his reply to Against SURA, Zvi Mowshowitz distinguishes:

• Strong SURA: stop building regardless of others, and do not invest in a fallback plan.
• Weak SURA: push for a coordinated pause, have more people quit loudly, have labs stop alone if coordination fails, and improve the fallback plan meanwhile.
• Non-SURA: back the developers judged most careful and maximize their lead.

Commenters added middle options for a lab: stay at the frontier without releasing leading models, commit to a conditional stop, and lobby for binding regulation. “Disband or race” is a false choice. See Against SURA; The SURA debate turns on how doomed Plan C is argues that the dispute turns on how likely a rushed alignment effort is to work.

What would change minds

Evidence about whether safety-focused labs actually behave differently under pressure. Whether insiders have changed lab decisions. The answers to the upstream cruxes.

For the motives behind building, stated and unstated, see Why people build AGI despite the danger.

Related: Pivotal act, AI ownership.