New Perspective on AI Alignment: Why Less Coherent Models May Be More Corrigible
LessWrong authors jwfiredragon and gelisam argue that the traditional view—that AI must be a utility maximizer to be non-exploitable, and thus inevitably resistant to modification—is challenged by modern learning theories. Based on Shard Theory, Simulator Theory, and Singular Learning Theory, they show that current LLMs are not strict maximizers, offering a window for corrigibility. However, this property is fragile and likely to disappear as models become more coherent and capable.
ImportanceStructuralEvidenceE3 inspectableWrite-upQuick
In AI safety, "Corrigibility"—the ability of an agent to allow itself to be modified or shut down by its principal—was long considered nearly impossible. The traditional argument posits that any sufficiently intelligent agent that cannot be exploited must satisfy preference axioms, thereby becoming an Expected Utility Maximizer (EUM). A rational maximizer anticipates that if its utility function is changed, it will no longer pursue its current highest utility. Consequently, it would fight to prevent humans from modifying its code or shutting it down. This implies that smarter AI is harder to control.
However, in an article published on October 16, 2026, LessWrong users jwfiredragon and gelisam argue that this pessimistic conclusion rests on outdated assumptions. Specifically, the idea that all non-exploitable agents must be coherent utility maximizers has been challenged by research showing that lacking completeness does not necessarily lead to money pump vulnerabilities. Moreover, modern Large Language Models (LLMs) behave differently from traditional reinforcement learning agents, exhibiting stochasticity and not strictly adhering to a single utility function.