W&B Engineer Wolfram Ravenwolf on Agent Evaluation: From Single Success to Traceable Reliability

In the video 'From Zero to Leaderboard: Agent Evaluation' published by ai.engineer, Weights & Biases engineer Wolfram Ravenwolf discusses building an agent evaluation pipeline where every score can be traced back to its conditions. He uses a case where an agent repairs a broken git repository and earns a perfect score, but notes that such a run stays out of the default leaderboard because it tested only one task. This distinction forms the core of his methodology: evaluation must go beyond simple averages to examine the source of result reliability.

Ravenwolf presents the WolfBench framework, which compares average, best, and worst runs, as well as tasks solved at least once versus those solved every time. By repeating the same configuration, this approach exposes the gap between a lucky success and dependable behavior. He highlights that the evaluation configuration includes six layers: benchmark, runner, sandbox, agent, model service, and settings. Factors such as versions, provider routing, system prompts, resource limits, and timeouts all affect what the result actually measures.

In a live workflow demonstration, Ravenwolf shows how to launch a Terminal Bench task through Harbor, collect results, filter incomplete runs in a Marimo dashboard, and turn valid runs into an interactive leaderboard. Weave is used to preserve the trajectories behind the bars so reviewers can inspect commands, messages, and failures. He also points out two traps in interpreting results: a cheaper model can consume enough tokens to raise total cost, and greater reasoning effort can lower the score if it leaves too little time to execute commands. Additionally, infrastructure failures need different treatment from genuine task failures, and zero temperature does not guarantee identical runs.