Assess an AI agent demo: a source-led reading path

A convincing demonstration is a starting point for investigation. By itself, it does not establish how often a system succeeds, how much supervision it needs or what a failure would change outside the demo.

This path uses two Anthropic engineering articles to help investors and other evaluators ask better technical questions. These are one vendor's engineering guidance, not independent audits of the product you are assessing. The three-lens checklist below is our way of organizing the readings.

1. Identify the architecture

Start with What are agents? in Building effective agents ↗, published December 19, 2024. Anthropic distinguishes systems with predefined paths from agents that let the model choose how to proceed and use tools.

For the product in front of you, write down which decisions are fixed in code and which the model can change. Ask the presenter to locate those decisions in a single demonstrated task. The label on the slide is less useful than that explanation.

Then read the article's discussion of when agents are useful and its workflow patterns. Use those patterns as vocabulary for a conversation, not as a ranking of products. An important question is what the extra flexibility accomplishes for this particular task.

2. Define a result that can be checked

Next, read The structure of an evaluation in Demystifying evals for AI agents ↗, published January 9, 2026. The evaluation guide separates what the agent says in a run from the final state it leaves in the environment.

Before looking at a success percentage, specify the result you would accept. For a booking workflow, for example, ask to inspect the reservation record. A completion message is a different piece of evidence. Then ask which tasks were tested, how repeated runs were handled, what counted as a failure and who checked the result.

It also distinguishes evaluations that seek new capabilities from regression evaluations that check whether previous capabilities have been retained. This distinction matters when comparing a new demonstration with an older version: find out whether the same test conditions and scoring rules were used.

3. Map the consequences of tool access

Return to the Agents section of the architecture article ↗. The architecture guide discusses feedback from the environment, opportunities for human intervention and stopping conditions for an agent loop. The authors also warn that autonomy can bring compounding errors and recommend testing in sandboxes with appropriate safeguards.

Translate that guidance into concrete questions about the system you are evaluating:

• Which systems can its tools read from or change?
• Which actions require approval before execution?
• What stops the process when it cannot make progress?
• How would an incorrect external change be detected and corrected?

Record what you actually observed and what the presenter only described. This prevents a controlled example from silently becoming an assumption about production behavior.

Leave with a reusable assessment note

For each of the three lenses, keep one short note with four fields:

1. Claim: what the product is said to do.
2. Evidence: the specific run, test result or system record available to you.
3. Unknown: the boundary that the evidence does not settle.
4. Next check: the additional observation that would change your assessment.

Mark evidence as observed, reported or not shown. An unanswered question is a reason to investigate; it is not automatically proof that the product fails.

These readings help organize technical diligence. They do not establish customer adoption, unit economics or business value. Once you can describe the architecture and the accepted outcome, continue with the evaluation guide's discussion of the relevant agent type and use it to refine your next check.