ReadingResearchRadarInvestment frameworkAI Company Primary Sources
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

AI Security Agents from Four Labs Converge on a Common Design: Discoverers Cannot Confirm Their Own Findings

Anthropic, OpenAI, Cloudflare, and Google each released AI security scanning agents in 2026, yet their architectures converged significantly: one agent identifies vulnerabilities while a separate, independent agent attempts to disprove them, with only surviving findings proceeding to patching. This design is an engineering response to the noise problem in AI scanning—Google reports true-positive rates below 7%, and Snyk found that 49.7% of pure LLM review findings disappeared on the next run.

2026-10-31 01:34 GMT+8

AI code security scanning faces a quantifiable reliability crisis. The Google security team publicly stated that crude AI code scanning produces hallucinated vulnerabilities, with true-positive rates under 7%—meaning a scanner that is wrong 93% of the time imposes higher costs on reviewers than it saves. Snyk’s own VulnBench tests further confirmed the ubiquity of the issue: 49.7% of findings from pure LLM reviews vanished on the next run, indicating that single model outputs are far from stable.

Facing the same problem, Anthropic, OpenAI, Cloudflare, and Google each released AI security scanning agents in 2026, converging on nearly identical designs: one agent responsible for finding vulnerabilities, another independent agent responsible for disproving them, with only findings that withstand rebuttal entering the patching phase. Cloudflare states this rule most plainly in its README: the verifier is never the agent that found the bug. Anthropic, OpenAI, and Google each described similar validation steps, requiring independent questioning before any finding reaches a report.

This convergence is not coincidental but an engineering response to the noise problem. The core principle is separation of duties: the agent reporting a finding never holds the power of confirmation. This aligns with peer-review logic in human security audits—the author cannot review their own work. In the context of AI agents, this means two independent model calls (or even two different models) must reach a consistent conclusion about the same finding before proceeding. Cloudflare also adopted a more specific practice: using different models for discovery and triage, allowing models to check each other.

Sign in / Sign up
ReadingResearchRadarInvestment frameworkAI Company Primary Sources

The differences among the four tools lie in the depth of verification and the operating environment. Anthropic’s Claude Security plugin runs within a Claude Code session, requiring no additional API keys; scans count against the user’s existing plan quota, outputting Markdown reports and SARIF files, with patches generated as .patch files but not automatically committed. OpenAI’s Codex Security offers a CLI and SDK, supporting deep-mode parallel discovery, and can import GitHub scanning alerts, but while the client is open source (Apache-2.0), the backend scanner still requires OpenAI access. Cloudflare’s Audit Skill is MIT-licensed and model-agnostic, requiring a coding agent that supports parallel sub-agents, ideally needing an OS-level sandbox with no external network—without a sandbox, findings remain in a needs_validation state rather than being executed. Google’s Mantis is the most aggressive: it attempts to reproduce each vulnerability in a sandbox, from static exploit sketches to unit tests or mock servers, and includes an adversarial loop in the patching phase that specifically attacks its own fixes.

Cloudflare’s publicly shared internal data reveals a notable leverage point: improving reconnaissance context (i.e., allowing the agent to fully understand architecture, trust boundaries, and input surfaces first) reduced the first-round rejection rate from 40% to 11%. This indicates that the upstream bottleneck for finding quality is not insufficiently strict verification, but insufficient initial understanding. Of 20,799 raw candidate findings, approximately 12,057 passed validation, resulting in 7,245 actionable findings sent to engineering teams.

The transferable lesson from this convergence pattern is: for any workflow relying on AI agent outputs where noise is a core constraint, the most reliable architectural improvement is not making the same agent more cautious, but introducing structural adversarial verification. Separating discovery from confirmation is the lowest-cost method for quality enhancement, and investment in context during the reconnaissance phase may have a greater impact on the final signal-to-noise ratio than the strictness of the verification stage.

Boundaries to retain: All performance figures above are self-reported by the respective companies, lacking independent third-party replication. Cloudflare’s 40%→11% improvement is based on its internal 128 codebases and may not apply to external projects. Google’s 7% true-positive rate is a description by its security team of its own scanning experience, not a standardized benchmark test result. This architectural convergence itself is a publicly verifiable fact—all four tools’ repositories and documentation demonstrate the same multi-agent verification pattern—but the actual effectiveness of each tool still requires independent evaluation.

Sources

  • marktechpost.comMarkTechPost↗
Continue researching →