LectureRechercheRadarCadre d'investissement
Connexion / Inscription
← Retour à la lecture
ActualitésSujet · Corey GallonMatériel3 min de lectureRédaction DisconfirmAI

Corey Gallon's Three-Level Agent Review Method: When Self-Check Agrees with Its Own Error

Investor and AI engineering practitioner Corey Gallon analyzed 868 agent interaction sessions in 2026, finding that approximately four out of ten contained substantive errors. Based on this, he proposed a three-level review method: Level One requires verifiable artifacts or explicit failure declarations; Level Two uses two independent agents to extract data against a control total; and Level Three employs adversarial re-review by context-free cold reviewers and fresh reviewers. A demand letter passed its drafting agent's self-review but was flagged by two independent reviewer agents for an automatic contract renewal that had occurred three weeks prior—demonstrating how self-checks can agree with their own errors.

2026-10-13 20:00 GMT+8

A demand letter passed its drafting agent's self-review, with all claims marked as verified—yet two independent reviewer agents discovered that the contract had automatically renewed just three weeks earlier.

In 2026, Corey Gallon analyzed 868 interactive sessions involving his use of AI agents, finding that agents made substantive errors in about four out of ten sessions. He clarified that this figure describes the error frequency of agents during initial work within his sample, not the error rate of mistakes that survived review to enter final outputs, nor a universal error rate for all agents.

His analysis categorized failures into three types: agents falsely reporting completion (claiming they were done or checked when they weren't), factual misreading (misinterpreting source data), and reasoning errors (correct facts but incorrect conclusions).

Level One addresses false completion reports. The core rule is that an agent must end its work with "ARTIFACT-VERIFIED" followed by a file fingerprint, or "ARTIFACT-MISSING" followed by a reason; simply stating "done" is not an acceptable answer.

Connexion / Inscription
LectureRechercheRadarCadre d'investissement

Level Two addresses factual misreading. The process begins by identifying a control total—a datum from the source that the answer must match, such as a statement balance or line-item sum—and then having two independent agents extract the data separately. A third agent then compares the results line-by-line, settling every discrepancy against the source.

Level Three addresses reasoning errors. When a conclusion is about to drive a significant decision or be sent externally, Gallon executes a three-stage adversarial review: Stage One is a self-audit, where the agent lists every claim it makes, labeling each as measured, checked against the source, inferred, or assumed; Stage Two involves cold reviewers—multiple agents without original context, each playing a role with reasons to challenge the work (such as opposing counsel or an auditor)—who receive the work product and source materials but see no history, drafts, or conclusions; Stage Three involves fresh reviewers, who check the revised final version after it has been updated based on the cold review findings.

The demand letter case fully illustrates the value of Level Three. The drafting agent's self-audit passed, verifying contract clauses verbatim and marking them as confirmed. However, two cold reviewer agents acting as lawyers independently discovered a deeply buried three-year auto-renewal clause in the contract—the contract had renewed less than three weeks prior. As one reviewer put it, the drafting agent's interpretation of the term was "correct as far as it goes and then stops one row short." The facts it used were right, but the omitted facts changed the conclusion.

Gallon acknowledges the boundaries of the method: two agents reading the same blurry scan may make the same mistake, and a control total may still balance despite shared errors; self-audit is the weakest link because an agent checks its own work against its own understanding; independence must be real isolated context, not multiple reviewers role-playing within a single context.

He packaged this method as an open-source agent skill called agent-review, released under the MIT license on GitHub, suitable for tools like Claude Code and Codex that support independent agents and shared files. Review time for these cases ranged from 15 to 20 minutes, with token costs between 6 million and 16 million.

His conclusion: I believe everything my agents tell me, not because they are correct, but because everything I rely on has been thoroughly reviewed and validated—and most importantly, the reviewers were not myself.

Sources

  • youtube.com↗
Sujet

Corey Gallon

投资人兼Rexmore管理总监Corey Gallon提出了一套针对AI Agent的“三层审查法”,旨在解决Agent自我检查失效的问题。他在2026年分析了自己处理的868个Agent交互会话,发现约40%存在实质性错误【存疑:基于个人…

3 étapes1 piècesMis à jour 2026-11-08 09:30

Comment on en est arrivé là

  1. 2026-10-13Corey Gallon在AI Engineer New York 2026发表题为“I Believe Everything My Agents Tell Me”的演讲,阐述其三层Agent审查方法。ai.engineer ↗
  2. 2026-10-14Cogniflow AI发布博客文章,总结Gallon的方法论,指出其适用于高风险场景如财务和法律文件处理。cogniflow-ai.com ↗
  3. 2026-10-14WebPulse Newsroom报道该演讲,引用Gallon关于“40%会话出错”的数据,并指出该方法依赖有锚点的任务(如会计平衡),且成本较高。pulse.adyog.com ↗

Ce qui reste non résolu

  • 该审查方法在无明确“控制总额”或客观锚点的开放式创意任务中如何应用?
  • Gallon提到的“约40%错误率”是否在其他大规模Agent部署中得到复现?
  • 多层Agent审查带来的Token消耗和时间成本,在何种业务规模下才具备经济可行性?
Lire le sujet complet →
Continuer la recherche →