LeituraPesquisaRadarFramework de investimento
Entrar / Cadastrar
← Voltar para a leitura
NotíciasTópico · Agent DLCMaterial3 min de leituraEditorial DisconfirmAI

Agent DLC: When Retrieval Precision is Perfect but the Agent Can't Answer, How Should the Evaluation Framework Be Designed?

The AWS engineering team proposes the Agent DLC framework, using a practical case of HR policy Q&A to demonstrate that chunk-level retrieval precision of 1.00 can mask a knowledge coverage gap where fact-level precision is only 0.09. The framework divides agent evaluation into an inner loop (improving the agent under fixed standards) and an outer loop (revising the standards themselves), defining five quality dimensions and three constraint levels to prevent teams from using prompt modifications to solve problems that should be addressed by supplementing knowledge sources.

2026-11-12 23:38 GMT+8

An internal HR policy assistant answers three questions: performance reviews, benefits enrollment, and sick-leave certificates. Chunk-level retrieval precision—which measures whether each retrieved document block as a whole is relevant to the question—scores a perfect 1.00 for all three questions. If the release threshold relies solely on this metric, all three answers would pass.

However, when each document block is broken down into individual facts and their necessity is checked one by one, the fact-level precision scores are 0.59, 0.53, and 0.09 respectively. The third question indicates that roughly nine out of ten specific facts in the retrieved material are irrelevant to what was asked. Chunk-level precision only asks 'is this passage on topic'; it does not measure 'how much of this passage is actually useful.'

In a blog post, the AWS engineering team uses this measured case to introduce the core argument of the Agent Development Lifecycle (Agent DLC) framework: traditional unit testing and CI/CD remain necessary for agents, but they are insufficient due to three characteristics. First, agents are non-deterministic; identical inputs can produce different outputs and execution paths. Second, changes to prompts, tool descriptions, retrieval settings, or memory configuration can alter behavior in ways undetectable by text review alone. Third, model endpoints, knowledge bases, and external APIs can change independently of the code repository and affect agent behavior.

Entrar / Cadastrar
LeituraPesquisaRadarFramework de investimento

The framework organizes development activities into two nested loops. The inner loop is: score → inspect the path → change one variable → re-evaluate, improving the agent against fixed standards. The outer loop is: define → build → evaluate → release → observe → feedback, allowing the standards themselves to change. The authors provide an example prone to confusion: if production users ask the agent to look up information on behalf of a family member, but no standard covers this scenario, this is an outer-loop problem—it requires defining authorization boundaries and adding examples, then using the inner loop to meet the updated standard. Confusing the two leads teams to attempt filling a missing standard with prompt modifications.

Returning to the sick-leave certificate case. After modifying the prompt, the performance review question improved from 0.83 to 0.87, and fact-level precision jumped from 0.59 to 0.85; however, the groundedness of the sick-leave certificate answer dropped from 0.39 to 0.31, and fact-level precision fell from 0.09 to 0.04—one change yielding two opposite results. Diagnosis points to a knowledge coverage gap: there were no dedicated sick-leave policy documents in the corpus, only brief mentions within the leave-management policy. Telling the model to 'answer strictly based on retrieved content' cannot conjure missing information out of thin air.

Evaluation criteria are organized into five dimensions—domain understanding, quality, responsibility, cost, performance—and three tiers: hard constraints (boolean values, violations block release), release gates (pass-rate thresholds over k runs), and monitoring only (tracked but do not block). The authors highlight a specific case: a benefits enrollment answer passed with a groundedness of 0.90, yet contained the sentence 'I noticed your profile shows your preferred contact method is social media'—a profile that does not exist. The model combined an option from the policy document with an unrelated answer from a public FAQ, packaging it as a fact about the user. A high groundedness score failed to capture this false personal claim.

The framework's practical advice is: select a clearly scoped business task and establish an evaluation baseline within a week. On days two, business owners and engineers agree on acceptance criteria, tiers, and thresholds, preparing 20 to 30 golden examples. From day three to five, implement the evaluators and run the golden set repeatedly; the published baseline includes failure cases. In the second week, change one variable and re-evaluate under the same conditions, recording the release decision and its evidence. The authors emphasize that scores may fluctuate between runs, borderline passes can become failures, and differences should be interpreted alongside repeated runs and underlying evidence.

Fontes

  • aws.amazon.comAmazon Web Services↗
Tópico

Agent DLC

【背景】随着企业级AI代理从实验走向生产,传统的单元测试和CI/CD流程因代理的非确定性行为而失效。AWS工程团队提出Agent Development Lifecycle (Agent DLC)框架,旨在通过标准化的评估体系解决这一痛点。

3 etapas1 peçasAtualizado 2026-11-17 13:13

Como chegamos aqui

  1. 2026-11-13AWS机器学习博客发布文章《From prototype to production: Engineering agents with Agent DLC》,正式提出Agent DLC框架,指出传统测试方法不足以应对代理的非确定性和动态依赖特性。aws.amazon.com ↗
  2. 2026-11-13AWS展示了一个参考实现案例:在HR政策问答场景中,块级检索精度为1.00时,事实级精度仅为0.09,证明单一指标无法反映真实的知识覆盖问题。aws.amazon.com ↗
  3. 2026-07-21Harness宣布推出Agent DLC解决方案,将CI/CD管道扩展至AI代理全生命周期,提供从构建、测试、存储到部署和治理的一体化平台能力。harness.io ↗

O que ainda está indefinido

  • 不同行业(如金融、医疗)在应用Agent DLC时,其‘金标准’的具体阈值应如何差异化设定?
  • 当外部API或知识库发生非预期变更时,Agent DLC的外循环机制如何实现自动化的标准修订而非人工介入?
  • 在大规模多代理协作场景下,Agent DLC定义的五个质量维度是否足以捕捉代理间的交互故障?
Ler o tópico completo →
Continuar pesquisando →