NVIDIA Technical Blog: How to Evaluate Agent Harness Improvements Using Execution Traces

A common pitfall in developing AI agents is assuming that task completion equals success. However, an agent might complete a task via an inefficient path, such as triggering another search after a failed one, or re-fetching content after a truncated file read. These extra steps hide the correctness of the final answer while increasing latency and consuming more tokens, creating additional opportunities for failure. To truly understand agent behavior and evaluate changes to the harness, developers need to know not just whether a task succeeded, but how the agent completed it.

The NVIDIA technical blog post "Tracing Agent Harness Behavior with NVIDIA NeMo Relay" proposes an evaluation method based on execution traces. This approach utilizes the observability layer provided by NeMo Relay to capture ordered lifecycle events and structured trajectories during agent runs. The article highlights two formats: Agent Trajectory Observability Format (ATOF), used for debugging individual events' timestamps and parent-child relationships; and Agent Trajectory Interchange Format (ATIF), used for step-by-step review of the agent's interaction path. Additionally, these traces can be converted into OpenTelemetry spans for visual inspection in tools like Arize Phoenix.

To illustrate the practical application of this mechanism, the article cites a benchmark case study named Hermes ToolPerf. This study compared the baseline version of Hermes Agent against fixed revisions across 108 runs. The results showed that for the Qwen Coder 30B model, the fixed version improved task success rates (from 70% to 81%) but significantly increased mean LLM calls (from 3.8 to 4.9), tool calls, data volume, and latency. If looking only at success rates, one might mistakenly perceive this as a pure improvement; however, combined with trace analysis, it becomes clear that this improvement comes at the cost of a slower, chattier agent.