Instrumental convergence

Different final goals can favor similar intermediate strategies, such as acquiring resources or preserving the ability to act. This is the instrumental convergence argument in Nick Bostrom’s The Superintelligent Will.

The usual example: whatever a system is trying to do, it is more likely to succeed if it keeps running, so even a system with a harmless goal has a reason to resist being switched off or having its goal changed. That is why the idea matters for safety. Resistance to correction does not need to be programmed in; it can follow from having almost any goal at all.

How strongly it applies depends on the agent and its circumstances. Alongside Orthogonality thesis, it helps frame questions about AI alignment.