NVIDIA Technical Blog: How to Evaluate AI Agents From Tool Calls to Task Completion
An NVIDIA technical blog post details the evolution of AI agent evaluation, arguing that assessing single function call accuracy is insufficient for determining if an agent can complete multi-step tasks in a live environment. It introduces a two-layer scoring mechanism based on execution traces (step-level process scoring and end-to-end outcome scoring), provides a core metrics framework, and offers practical advice for building domain-specific evaluation benchmarks.
重要度重大証拠E3 検査可能執筆標準
When deploying an AI agent, the key question is no longer whether the model generates a plausible-sounding individual API call, but whether it can execute a chain of work across dozens of sequential tool calls against a live environment and recover when a step fails. A September 21, 2026 article on the NVIDIA Technical Blog titled "How to Evaluate AI Agents From Tool Calls to Task Completion" points out that traditional LLM benchmarks were built for static tasks, and the multi-step nature of agents breaks this assumption. Therefore, evaluation must evolve from scoring single function calls to scoring entire task completions, with tool calling serving as the connective tissue underneath.
The article proposes that full agentic evaluation requires a full execution environment that executes each tool call, tracks state across steps, and reads the world afterward to decide whether the work got done. On top of this, two scoring layers operate: Step-level (process scoring) asks if a call was valid, relevant, and useful given the state at that point; End-to-end (outcome scoring) ignores the path and checks only the final state. Step-level scoring helps debug and target fine-tuning efforts by revealing where the chain breaks, while E2E scoring reflects what users actually experience, collapsing failures on step one and step nine into the same "task failed." Most production evals gate releases on E2E scores while keeping step-level tracing underneath for debugging.
To systematically measure performance, the article defines a fixed hierarchy: Benchmark → Trial → Task → Turn → Step. Core metrics collapse onto three axes: accuracy, verbosity, and cost. For instance, "Task success rate" serves as the release gate but must be reported alongside "Consistency" (the range of success rates across multiple trials) to avoid misleading point estimates. "Tool-call precision" and "Argument accuracy" separately reveal hallucinated names/extra calls and wrong slot-filling issues. Additionally, "Steps per success" and "Cost per success" quantify efficiency and economics. The article emphasizes that parallel tool calling cuts step count and latency but not call count, so data must be rolled up in order rather than averaged.