Qodo Blog Author Ravid Cohen: When Agent Outputs Are Unstable, Run Twice Before Optimizing Prompts
Noise in AI code review is often attributed to false positives—the model flagging issues that shouldn't be reported. However, Qodo blog author Ravid Cohen points to another type of noise: when the same agent runs twice on the same pull request (PR), it may assign different judgment labels. Their recommendation agent predicts whether a team is likely to adopt a specific review finding, but on an evaluation set of 200 PRs, 20.7% of labels soft-flipped between 'yes' and 'maybe' or 'maybe' and 'no', while 4.3% hard-flipped directly between 'yes' and 'no'. This inconsistency indicates instability in the agent's evidence gathering or decision logic, rather than merely a need to tighten judgment criteria.
Standard GEPA (an algorithm that iteratively improves prompts by analyzing agent outputs and evaluation feedback) faces a structural difficulty with such noisy agents: the agent's full execution trace is long, and the reflector can only inspect a limited number of samples at once. If lessons are extracted using only small batches, a single accidental unstable output might be misread as a general pattern. Therefore, the Cohen team adapted GEPA by adding two steps: first, run the agent twice for each training sample, allowing an LLM to compare the two full traces and output a structured analysis detailing where different evidence was collected, where different decisions were made, and what might explain these differences; second, aggregate all structured analyses across the entire training set into a single high-reasoning-effort LLM call, enabling it to find patterns across samples without processing every full trace individually.
The Cohen team argues that the core reasoning behind the effectiveness of dual-run comparison is that a single run might mask unstable behavior behind a seemingly plausible answer, whereas comparing two runs exposes where the agent gathered different evidence or reached different conclusions. Starting from a minimal prompt of only 43 words that described only the agent's goal, they performed a single wide-step optimization using approximately 120 PRs from the entire training set. Cohen explains that production prompts, already optimized by humans, leave less room for improvement by automated optimizers, whereas a minimal prompt allows the optimizer to freely restructure instructions. In offline validation, the hard flip rate dropped from 8.2% to 2.2%, and the total flip rate from about 27% to 20%.
Qodo reported the effects of this adaptation in a production environment. They conducted a 50/50 A/B test assigned by PR across all customers, covering 27,582 review runs and approximately 21,000 PRs. Among review findings actually adopted by developers, the proportion labeled 'no' by the recommendation agent (i.e., the miss rate) fell from 7.83% to 4.46%. The average cost per instance for the recommendation agent dropped from $0.24 to $0.12, and average latency decreased from 40 seconds to 30 seconds. Precision showed no statistically significant difference. Cohen emphasized that these figures apply only to the recommendation agent component and do not cover the entire code review process; quality analysis covered a subset of PRs with available attribution labels and at least one adopted finding.
Fuentes:Qodo