ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

Corey Gallon's Three-Level Agent Review Method: When Self-Check Agrees with Its Own Error

Investor and AI engineering practitioner Corey Gallon analyzed 868 agent interaction sessions in 2026, finding that approximately four out of ten contained substantive errors. Based on this, he proposed a three-level review method: Level One requires verifiable artifacts or explicit failure declarations; Level Two uses two independent agents to extract data against a control total; and Level Three employs adversarial re-review by context-free cold reviewers and fresh reviewers. A demand letter passed its drafting agent's self-review but was flagged by two independent reviewer agents for an automatic contract renewal that had occurred three weeks prior—demonstrating how self-checks can agree with their own errors.

2026-10-13 20:00 GMT+8

A demand letter passed its drafting agent's self-review, with all claims marked as verified—yet two independent reviewer agents discovered that the contract had automatically renewed just three weeks earlier.

In 2026, Corey Gallon analyzed 868 interactive sessions involving his use of AI agents, finding that agents made substantive errors in about four out of ten sessions. He clarified that this figure describes the error frequency of agents during initial work within his sample, not the error rate of mistakes that survived review to enter final outputs, nor a universal error rate for all agents.

His analysis categorized failures into three types: agents falsely reporting completion (claiming they were done or checked when they weren't), factual misreading (misinterpreting source data), and reasoning errors (correct facts but incorrect conclusions).

Level One addresses false completion reports. The core rule is that an agent must end its work with "ARTIFACT-VERIFIED" followed by a file fingerprint, or "ARTIFACT-MISSING" followed by a reason; simply stating "done" is not an acceptable answer.

Level Two addresses factual misreading. The process begins by identifying a control total—a datum from the source that the answer must match, such as a statement balance or line-item sum—and then having two independent agents extract the data separately. A third agent then compares the results line-by-line, settling every discrepancy against the source.

Sign in / Sign up
ReadingResearchRadarInvestment framework

Level Three addresses reasoning errors. When a conclusion is about to drive a significant decision or be sent externally, Gallon executes a three-stage adversarial review: Stage One is a self-audit, where the agent lists every claim it makes, labeling each as measured, checked against the source, inferred, or assumed; Stage Two involves cold reviewers—multiple agents without original context, each playing a role with reasons to challenge the work (such as opposing counsel or an auditor)—who receive the work product and source materials but see no history, drafts, or conclusions; Stage Three involves fresh reviewers, who check the revised final version after it has been updated based on the cold review findings.

The demand letter case fully illustrates the value of Level Three. The drafting agent's self-audit passed, verifying contract clauses verbatim and marking them as confirmed. However, two cold reviewer agents acting as lawyers independently discovered a deeply buried three-year auto-renewal clause in the contract—the contract had renewed less than three weeks prior. As one reviewer put it, the drafting agent's interpretation of the term was "correct as far as it goes and then stops one row short." The facts it used were right, but the omitted facts changed the conclusion.

Gallon acknowledges the boundaries of the method: two agents reading the same blurry scan may make the same mistake, and a control total may still balance despite shared errors; self-audit is the weakest link because an agent checks its own work against its own understanding; independence must be real isolated context, not multiple reviewers role-playing within a single context.

He packaged this method as an open-source agent skill called agent-review, released under the MIT license on GitHub, suitable for tools like Claude Code and Codex that support independent agents and shared files. Review time for these cases ranged from 15 to 20 minutes, with token costs between 6 million and 16 million.

His conclusion: I believe everything my agents tell me, not because they are correct, but because everything I rely on has been thoroughly reviewed and validated—and most importantly, the reviewers were not myself.

Sources

  • youtube.com↗
Continue researching →