독서리서치레이더투자 프레임워크
로그인 / 회원가입
← 읽기로 돌아가기
뉴스주제 · Qodo提示词稳定性优化중대3 분 소요DisconfirmAI 편집부

Qodo Blog Author Ravid Cohen: When Agent Outputs Are Unstable, Run Twice Before Optimizing Prompts

Qodo blog author Ravid Cohen describes their method for adapting the GEPA algorithm to optimize system prompts for a code review recommendation agent. The core approach involves running the agent twice on each training sample and comparing full execution traces to identify unstable points in evidence gathering and decision-making; all structured analyses are then aggregated into a single high-reasoning LLM call to find cross-sample patterns. Starting from a minimal 43-word prompt, they performed a single wide-step optimization, which outperformed multiple rounds of minor tweaks to an already manually optimized production prompt. In a production A/B test at Qodo, the miss rate dropped from 7.83% to 4.46%, cost per instance was halved, and latency decreased by 25%.

2026-10-22 16:18 GMT+8

Noise in AI code review is often attributed to false positives—the model flagging issues that shouldn't be reported. However, Qodo blog author Ravid Cohen points to another type of noise: when the same agent runs twice on the same pull request (PR), it may assign different judgment labels. Their recommendation agent predicts whether a team is likely to adopt a specific review finding, but on an evaluation set of 200 PRs, 20.7% of labels soft-flipped between 'yes' and 'maybe' or 'maybe' and 'no', while 4.3% hard-flipped directly between 'yes' and 'no'. This inconsistency indicates instability in the agent's evidence gathering or decision logic, rather than merely a need to tighten judgment criteria.

Standard GEPA (an algorithm that iteratively improves prompts by analyzing agent outputs and evaluation feedback) faces a structural difficulty with such noisy agents: the agent's full execution trace is long, and the reflector can only inspect a limited number of samples at once. If lessons are extracted using only small batches, a single accidental unstable output might be misread as a general pattern. Therefore, the Cohen team adapted GEPA by adding two steps: first, run the agent twice for each training sample, allowing an LLM to compare the two full traces and output a structured analysis detailing where different evidence was collected, where different decisions were made, and what might explain these differences; second, aggregate all structured analyses across the entire training set into a single high-reasoning-effort LLM call, enabling it to find patterns across samples without processing every full trace individually.

로그인 / 회원가입
독서리서치레이더투자 프레임워크

The Cohen team argues that the core reasoning behind the effectiveness of dual-run comparison is that a single run might mask unstable behavior behind a seemingly plausible answer, whereas comparing two runs exposes where the agent gathered different evidence or reached different conclusions. Starting from a minimal prompt of only 43 words that described only the agent's goal, they performed a single wide-step optimization using approximately 120 PRs from the entire training set. Cohen explains that production prompts, already optimized by humans, leave less room for improvement by automated optimizers, whereas a minimal prompt allows the optimizer to freely restructure instructions. In offline validation, the hard flip rate dropped from 8.2% to 2.2%, and the total flip rate from about 27% to 20%.

Qodo reported the effects of this adaptation in a production environment. They conducted a 50/50 A/B test assigned by PR across all customers, covering 27,582 review runs and approximately 21,000 PRs. Among review findings actually adopted by developers, the proportion labeled 'no' by the recommendation agent (i.e., the miss rate) fell from 7.83% to 4.46%. The average cost per instance for the recommendation agent dropped from $0.24 to $0.12, and average latency decreased from 40 seconds to 30 seconds. Precision showed no statistically significant difference. Cohen emphasized that these figures apply only to the recommendation agent component and do not cover the entire code review process; quality analysis covered a subset of PRs with available attribution labels and at least one adopted finding.

It should be noted that all the above data comes from Qodo's own production A/B tests, representing vendor-reported results with no independent replication yet. The methodology itself—using dual-run comparisons to locate unstable points, batch aggregation to find cross-sample patterns, and single wide-step optimization from a minimal prompt—offers reference value for optimizing prompts for any agent with unstable outputs. However, dual runs mean doubling the cost during the optimization phase, which may be unfeasible for deployments sensitive to latency or with extremely tight budgets. In the four lessons listed at the end of the article, Cohen includes one directly related to this: measure noise before scaling up experiments, run multiple times on the validation set to understand variance, and then decide whether to rely on validation scores for judgment.

출처

  • qodo.aiQodo↗
주제

Qodo提示词稳定性优化

以色列代码审查工具厂商 Qodo 在其博客中披露了如何优化其“推荐 Agent”的系统提示词,以解决AI代码审查结果不稳定、同一PR多次运行标签翻转的问题。团队并未微调模型权重,而是改写了开源框架 GEPA(Genetic-Pareto)的…

2 단계1 건업데이트됨 2026-11-02 08:00

현재까지의 과정

  1. 2026-10-22Qodo发布博客《Improving Code Review Recommendations with GEPA》,作者Ravid Cohen详细阐述了如何通过改造GEPA算法优化推荐Agent提示词,并公布生产环境A/B测试数据。qodo.ai ↗
  2. 2026-11-01Disconfirm收录该新闻,标题强调“先跑两遍再优化提示词”的核心方法论。qodo.ai ↗

아직 해결되지 않은 문제

  • 这种基于双次运行对比的GEPA变体是否适用于其他类型的Agent(如客服、数据分析),还是仅针对具有明确二元/三元分类输出的场景有效?
  • Qodo提到的“单次宽步优化”优于“多轮小修”,这是否意味着在特定噪声水平下,全局模式识别比局部迭代更关键?
전체 주제 읽기 →
리서치 계속하기 →