When AI agents can produce two hundred pull requests (PRs) a day or pile up six hundred pending changes for review at once, the core issue facing engineering teams is no longer whether the code has bugs, but whether these changes should exist at all. CodeRabbit founder Harjot Gill and OpenCode maintainer Ryan Vogel discussed this issue on The Merge podcast. Their consensus is that while AI-written code is useful, the harder part is deciding what to merge, what to push back on, and what to drop.
This judgment problem has an easily overlooked starting point: the problem takes shape before the code exists. Vogel gave the example of a kitchen to-do app—assign the task to an agent, and it will first add a database, then a caching layer, then more infrastructure. Each step individually does not look wrong, but the application ends up far more complex than a simple to-do list requires. Vogel pointed out that without an external reference point, every decision provides justification for the next, and the project can drift off course even while each step appears productive.
To combat this creep, Vogel and Gill each approached from different stages. Vogel’s method involves having the agent output an architecture plan first, which humans then challenge by identifying unnecessary parts; simultaneously, he runs a separate agent to compare intermediate outputs against the original requirement direction. Gill, drawing from CodeRabbit’s practice, added another dimension: review should not just look at diffs, but read changes in the order they were built. CodeRabbit divides review into three layers—the bottom layer consists of data and schema changes, the middle layer covers functions, business logic, and API routes, and the top layer includes the interface and its tests. Schema changes explain why an API behaves differently, and API changes explain what the interface now needs to handle. Reviewing in this order allows humans to check the assumptions connecting various parts, rather than guessing from filenames.
But sequence alone is not enough; evidence is also required. Gill mentioned hallucination issues encountered early on by CodeRabbit: when models are forced to give answers, they fabricate them. The team’s solution was to provide an escape hatch—for situations with insufficient information, the model could ask questions, investigate further, or write a script to verify its hypotheses. This reduced the hallucination rate. In the review process, the same discipline means treating a suspected bug as something to be tested, rather than immediately turning it into a comment requiring developer action. The recipient of the review feedback can inspect how the conclusion was reached, run the check, and question its assumptions. Scripts themselves may still be incorrect or incomplete, but their value lies in making part of the claim verifiable.
This approach has a clear boundary of applicability. Gill pointed out that agents are best suited for tasks with clear completion conditions, such as adapting a working integration to another platform—because both the agent and the reviewer have concrete results to check. Open-ended design work requires more frequent human decision-making because the desired outcome is still taking shape while the agent runs. In other words, the premise for reference points to be effective is that you already know where you are going; if the destination itself is changing, the reference point loses its anchoring function.