ReadingResearchRadarInvestment frameworkAI Company Primary Sources
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

Cisco Chief Engineer Julio Gomez: The Bottleneck in LLM Security Discovery Is Trust, Not Model Capability

Julio Gomez implemented Cisco's Foundry security governance specification and tested it on Langflow, MLflow, and PyGoat. He concludes that the key to making LLM security outputs actionable is not making the model smarter, but completely separating detection inclusivity from mechanical verification—the evidence gate only checks whether citations correspond to real bytes on disk, meaning a model cannot bypass non-existent line numbers.

2026-11-03 00:30 GMT+8

Large models can confidently tell you there are vulnerabilities in your code, but you have no way of knowing which claims were actually checked, whether you will get the same answer next week, or why a specific finding exists six months from now when an auditor asks. Cisco Chief Engineer Julio Gomez argues that this gap represents the entire distance between LLM security tools moving from demonstrations to actual security programs.

He implemented the Foundry security governance specification released by Cisco—a design document rather than a product—and applied it to real code: the complete v1 API of Langflow 1.7.3 (289 function-level units across 25 modules), MLflow 2.9.0 (830 units), and OWASP's PyGoat training application (180 units). He set himself a hard rule: no strawman baseline. The control group used the same model, the same code, and a fair, disciplined, equally demanding prompt from a senior application security expert requiring structured findings and precise line-number citations.

The results were as follows: Foundry presented 32 findings for humans, while the baseline presented 15; both had 100% citation accuracy; but the baseline marked 14 out of its 15 findings as 'confirmed' with no mechanism enforcing that judgment, whereas Foundry's 28 confirmed and 4 pending-review items were delineated by mechanical checks. Gomez summarizes this by saying: The model was never the bottleneck; trust was.

There are two core design decisions in Foundry. First, detection is inclusive, while decision-making is strict. The Detector is instructed to prioritize recall—bringing forward anything plausible—all discipline being placed downstream. This allows the same model to be thorough and trustworthy simultaneously, rather than forcing a choice between the two. Second, the evidence gate is not a second opinion. It never asks the model 'are you sure?'. Instead, it takes every citation back to verify against the bytes on disk: the file must exist in the target, the line number must be within range, and the quoted text must actually appear there. Only findings whose citations survive are labeled 'confirmed'; others are downgraded to 'pending review' or rejected outright. A model cannot argue its way past a line number that does not exist.

Sign in / Sign up
ReadingResearchRadarInvestment frameworkAI Company Primary Sources

Gomez illustrates why this separation matters through repeated runs. He independently ran the full pipeline twice; the 122 records stored collapsed into 67 distinct fingerprints, with 56 recurring across both runs and landing on existing identities—roughly 90% overlap, despite the inherent non-determinism of model output. This means the next CI run can calculate deltas (new, recurring, gone) rather than forcing the team to re-read a wall of differently worded text. After the second run, teams usually stop running it again; once stable identities are established, they continue.

On the broader scope of MLflow, governance held true: 830 units, 41 batch calls, and 4.8 million tokens completed within the declared 6 million cap. Findings landed in the same area as MLflow's historical CVEs (path traversal in the artifact storage layer). However, only 13 were labeled confirmed, while 110 were leads pending review—the gate became more conservative on large-scale unfamiliar code, which is precisely what the layered approach is doing right.

Gomez explicitly states he is not claiming Foundry is smarter or cheaper. With the same model, Foundry surfaces more findings because it performs exhaustive scanning rather than letting a single agent turn curate the results. Token costs are approximately ten times that of a single call, buying coverage, the mechanical gate, and deduplication. 'Confirmed' means verified and worth a reviewer's time, not exploited—proving exploitability requires a live testbed, which is a separate role not implemented in the specification.

His conclusion is that the value of the governance layer only becomes apparent when findings need to integrate into a process: an issue tracker that cannot be drowned in duplicates, reviewers who need to know what was checked, and pipelines that repeat as code changes. For a one-time check by an engineer looking at a small piece of familiar code, a standard agent is cheaper and more practical; just use it directly.

This article is the second part of the Security Loop series, published on November 2, 2026. All run logs, provenance trails, rule gaps, and scoring comparisons are publicly available at github.com/juliogomez/foundry.

Sources

  • blogs.cisco.comCisco Blogs↗
Continue researching →