ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

Agent DLC: When Retrieval Precision is Perfect but the Agent Can't Answer, How Should the Evaluation Framework Be Designed?

The AWS engineering team proposes the Agent DLC framework, using a practical case of HR policy Q&A to demonstrate that chunk-level retrieval precision of 1.00 can mask a knowledge coverage gap where fact-level precision is only 0.09. The framework divides agent evaluation into an inner loop (improving the agent under fixed standards) and an outer loop (revising the standards themselves), defining five quality dimensions and three constraint levels to prevent teams from using prompt modifications to solve problems that should be addressed by supplementing knowledge sources.

2026-11-12 23:38 GMT+8

An internal HR policy assistant answers three questions: performance reviews, benefits enrollment, and sick-leave certificates. Chunk-level retrieval precision—which measures whether each retrieved document block as a whole is relevant to the question—scores a perfect 1.00 for all three questions. If the release threshold relies solely on this metric, all three answers would pass.

However, when each document block is broken down into individual facts and their necessity is checked one by one, the fact-level precision scores are 0.59, 0.53, and 0.09 respectively. The third question indicates that roughly nine out of ten specific facts in the retrieved material are irrelevant to what was asked. Chunk-level precision only asks 'is this passage on topic'; it does not measure 'how much of this passage is actually useful.'

In a blog post, the AWS engineering team uses this measured case to introduce the core argument of the Agent Development Lifecycle (Agent DLC) framework: traditional unit testing and CI/CD remain necessary for agents, but they are insufficient due to three characteristics. First, agents are non-deterministic; identical inputs can produce different outputs and execution paths. Second, changes to prompts, tool descriptions, retrieval settings, or memory configuration can alter behavior in ways undetectable by text review alone. Third, model endpoints, knowledge bases, and external APIs can change independently of the code repository and affect agent behavior.

Sign in / Sign up
ReadingResearchRadarInvestment framework

The framework organizes development activities into two nested loops. The inner loop is: score → inspect the path → change one variable → re-evaluate, improving the agent against fixed standards. The outer loop is: define → build → evaluate → release → observe → feedback, allowing the standards themselves to change. The authors provide an example prone to confusion: if production users ask the agent to look up information on behalf of a family member, but no standard covers this scenario, this is an outer-loop problem—it requires defining authorization boundaries and adding examples, then using the inner loop to meet the updated standard. Confusing the two leads teams to attempt filling a missing standard with prompt modifications.

Returning to the sick-leave certificate case. After modifying the prompt, the performance review question improved from 0.83 to 0.87, and fact-level precision jumped from 0.59 to 0.85; however, the groundedness of the sick-leave certificate answer dropped from 0.39 to 0.31, and fact-level precision fell from 0.09 to 0.04—one change yielding two opposite results. Diagnosis points to a knowledge coverage gap: there were no dedicated sick-leave policy documents in the corpus, only brief mentions within the leave-management policy. Telling the model to 'answer strictly based on retrieved content' cannot conjure missing information out of thin air.

Evaluation criteria are organized into five dimensions—domain understanding, quality, responsibility, cost, performance—and three tiers: hard constraints (boolean values, violations block release), release gates (pass-rate thresholds over k runs), and monitoring only (tracked but do not block). The authors highlight a specific case: a benefits enrollment answer passed with a groundedness of 0.90, yet contained the sentence 'I noticed your profile shows your preferred contact method is social media'—a profile that does not exist. The model combined an option from the policy document with an unrelated answer from a public FAQ, packaging it as a fact about the user. A high groundedness score failed to capture this false personal claim.

The framework's practical advice is: select a clearly scoped business task and establish an evaluation baseline within a week. On days two, business owners and engineers agree on acceptance criteria, tiers, and thresholds, preparing 20 to 30 golden examples. From day three to five, implement the evaluators and run the golden set repeatedly; the published baseline includes failure cases. In the second week, change one variable and re-evaluate under the same conditions, recording the release decision and its evidence. The authors emphasize that scores may fluctuate between runs, borderline passes can become failures, and differences should be interpreted alongside repeated runs and underlying evidence.

Sources

  • aws.amazon.comAmazon Web Services↗
Continue researching →