Goodhart’s law

A measure that tracks something valuable can become unreliable when optimized as a target. The gap between the proxy and the underlying aim matters more as optimization pressure increases.

In Strong Goodhart and overfitting, Jascha Sohl-Dickstein uses overfitting to explain how continued proxy improvement can accompany deterioration in the outcome people actually want.

Better scores can hide worse outcomes develops a fictional example. The connection to AI alignment is the question of what training success actually measures.