ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
News3 min read·2026-10-21DisconfirmAI Editorial

Cisco DevNet Platform Engineering's Matt Shooshtari: Judge an AI Agent by Its Receipt, Not Its Demo

Matt Shooshtari of Cisco's DevNet Platform Engineering team proposes four evaluation questions for AI agents in production. The core mechanism: the language model only proposes, a deterministic policy engine decides, and a hash-chained audit log records actual decisions and arguments rather than the model's self-written summary, making agent behavior verifiable after the fact.

ImportanceMaterialEvidenceE3 inspectableWrite-upQuick

Original publication time 2026-10-21 03:00 GMT+8

Every vendor can show you an impressive agent demo now. Almost none can show you the receipt afterward. Shooshtari argues that trust comes precisely from that receipt, not from the demo itself.

He describes a concrete scenario: an agent pushed a change to a cluster overnight, and the next day someone asks who approved that deployment. If the only record is the model's own summary of what it did, the honest answer is 'memory and good faith.' A demo shows what an agent can do, not what it is allowed to do, nor how anyone would know afterward what happened.

The four evaluation questions Shooshtari proposes are: What may it touch, and does the model get a vote? Who approves the actions that matter? Can anyone prove what happened after the fact? Does the 3 a.m. autonomous run pass through the same gate as a 2 p.m. live request?

Behind those four questions sits a specific architectural principle: the model proposes, the policy engine decides. In CadreAI, the language model never determines whether an action may proceed. Every tool call, down to reading a single file, first passes through a deterministic policy engine. The engine checks the caller's identity and role, and for anything consequential it also checks the specific arguments. The model can propose anything it likes, but it does not also get to judge. When the engine says no, the agent reports the refusal plainly and does not hunt for a way around it.

Sign in / Sign up
ReadingResearchRadarInvestment framework

The mechanism's key point: you scope an agent by editing policy, and no prompt can talk its way past a rule the model never evaluates. Every request gets one of three answers: refused outright, routed to a human approver, or allowed to run because it is routine. Deployments sit behind the approval gate—a person reviews a concrete change and a dry run before anything touches the live cluster, and after the rollout the agent checks the actual workload instead of assuming everything works because a script exited zero.

The audit log is that receipt. CadreAI writes every action, allowed or denied, to a hash-chained audit log. Each entry is tied to the hash of the one before it, so anyone who edits an entry later breaks the chain. The log holds the actual decision and the actual arguments, not a summary the model wrote about itself. Without it, the answer to 'who approved that deployment?' rests on memory and trust. With it, the answer is a record anyone can check.

The fourth question—whether 3 a.m. behaves the same as 2 p.m.—targets autonomous operation. CadreAI wakes on its own schedule, and each wake passes through the same gate as a live conversation. Nothing gains extra authority because nobody is watching.

On role design, this is not one agent with access to everything. Mail and calendar live in one specialist; the ticketing integration lives in another. Each proves its own identity and holds no authority until someone grants it a role. Shooshtari draws an analogy to DevOps culture: taking down the walls between teams does not mean nobody owns anything. A specialist's blast radius—the damage it could do on a bad day—stays inside its role.

It should be noted that this is the CadreAI builders' own description of their system. The framework's logic—separation of proposal and judgment, hash-chained logs, consistent permission gates—is independently assessable as a design principle, but whether CadreAI actually operates as described is a vendor self-report that this article cannot independently verify.

For teams evaluating agent deployments, the actionable takeaway is: after the next demo, do not ask 'what can it do,' ask 'what can it prove to people afterward.' If the answer is only the model's own paragraph, that is no different from having no record at all.

Sources

  • 原始出处blogs.cisco.com↗
Continue researching →