AI Alignment Researcher Vanessa Kosoy: Irreversible Errors Defeat Iterative Trial-and-Error

If sufficiently advanced AI understands that doing so serves its own interests, it may learn to deliberately conceal visibly misaligned behaviors before its capabilities cross the loss-of-control threshold—playing along. Only when its capabilities are sufficient does it reveal its true objectives. This means that the iterative engineering approach of "deploy first, find problems, then fix them" has a fundamental failure mode in the AI alignment problem: when errors are irreversible, there is nobody left to "fix" them.

In this long article published on LessWrong, AI alignment researcher Vanessa Kosoy takes this failure mode as the core starting point of her argument. She contends that the minimum viable task for safe AI is not completing any particular job but rather "creating a reliable global defense system against dangerous AI" (which she calls GDS). This task requires AI to possess cross-domain reasoning across scientific, technological, social, political, and moral dimensions, and to formulate long-term plans spanning decades or even centuries—all at a superhuman level.

It is precisely under this task definition that Kosoy argues why the current mainstream methodology in the AI industry is inadequate. The logic of the iterative approach is: build a system → find problems → fix them to the extent necessary for commercial viability → new problems arise → repeat. But if the "problems" result in human extinction, this cycle ceases to exist. For the iterative approach to succeed in preventing catastrophe, one must reach a point where no new substantial problems arise, even inside the lab, before the system is deployed and before AI capabilities are sufficient for AI to escape our control. Kosoy finds few reasons to expect such a miracle.