ReadingResearchRadarInvestment frameworkAI Company Primary Sources
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

Qodo Blog Author Ravid Cohen: When Agent Outputs Are Unstable, Run Twice Before Optimizing Prompts

Qodo blog author Ravid Cohen describes their method for adapting the GEPA algorithm to optimize system prompts for a code review recommendation agent. The core approach involves running the agent twice on each training sample and comparing full execution traces to identify unstable points in evidence gathering and decision-making; all structured analyses are then aggregated into a single high-reasoning LLM call to find cross-sample patterns. Starting from a minimal 43-word prompt, they performed a single wide-step optimization, which outperformed multiple rounds of minor tweaks to an already manually optimized production prompt. In a production A/B test at Qodo, the miss rate dropped from 7.83% to 4.46%, cost per instance was halved, and latency decreased by 25%.

2026-10-22 16:18 GMT+8

Noise in AI code review is often attributed to false positives—the model flagging issues that shouldn't be reported. However, Qodo blog author Ravid Cohen points to another type of noise: when the same agent runs twice on the same pull request (PR), it may assign different judgment labels. Their recommendation agent predicts whether a team is likely to adopt a specific review finding, but on an evaluation set of 200 PRs, 20.7% of labels soft-flipped between 'yes' and 'maybe' or 'maybe' and 'no', while 4.3% hard-flipped directly between 'yes' and 'no'. This inconsistency indicates instability in the agent's evidence gathering or decision logic, rather than merely a need to tighten judgment criteria.

Standard GEPA (an algorithm that iteratively improves prompts by analyzing agent outputs and evaluation feedback) faces a structural difficulty with such noisy agents: the agent's full execution trace is long, and the reflector can only inspect a limited number of samples at once. If lessons are extracted using only small batches, a single accidental unstable output might be misread as a general pattern. Therefore, the Cohen team adapted GEPA by adding two steps: first, run the agent twice for each training sample, allowing an LLM to compare the two full traces and output a structured analysis detailing where different evidence was collected, where different decisions were made, and what might explain these differences; second, aggregate all structured analyses across the entire training set into a single high-reasoning-effort LLM call, enabling it to find patterns across samples without processing every full trace individually.

Sign in / Sign up
ReadingResearchRadarInvestment frameworkAI Company Primary Sources

The Cohen team argues that the core reasoning behind the effectiveness of dual-run comparison is that a single run might mask unstable behavior behind a seemingly plausible answer, whereas comparing two runs exposes where the agent gathered different evidence or reached different conclusions. Starting from a minimal prompt of only 43 words that described only the agent's goal, they performed a single wide-step optimization using approximately 120 PRs from the entire training set. Cohen explains that production prompts, already optimized by humans, leave less room for improvement by automated optimizers, whereas a minimal prompt allows the optimizer to freely restructure instructions. In offline validation, the hard flip rate dropped from 8.2% to 2.2%, and the total flip rate from about 27% to 20%.

Qodo reported the effects of this adaptation in a production environment. They conducted a 50/50 A/B test assigned by PR across all customers, covering 27,582 review runs and approximately 21,000 PRs. Among review findings actually adopted by developers, the proportion labeled 'no' by the recommendation agent (i.e., the miss rate) fell from 7.83% to 4.46%. The average cost per instance for the recommendation agent dropped from $0.24 to $0.12, and average latency decreased from 40 seconds to 30 seconds. Precision showed no statistically significant difference. Cohen emphasized that these figures apply only to the recommendation agent component and do not cover the entire code review process; quality analysis covered a subset of PRs with available attribution labels and at least one adopted finding.

It should be noted that all the above data comes from Qodo's own production A/B tests, representing vendor-reported results with no independent replication yet. The methodology itself—using dual-run comparisons to locate unstable points, batch aggregation to find cross-sample patterns, and single wide-step optimization from a minimal prompt—offers reference value for optimizing prompts for any agent with unstable outputs. However, dual runs mean doubling the cost during the optimization phase, which may be unfeasible for deployments sensitive to latency or with extremely tight budgets. In the four lessons listed at the end of the article, Cohen includes one directly related to this: measure noise before scaling up experiments, run multiple times on the validation set to understand variance, and then decide whether to rely on validation scores for judgment.

Sources

  • qodo.aiQodo↗
Continue researching →