Arize AI's Laurie Voss Demonstrates Fixing LLM Evaluation Bias by Supplying Research Context
In a video published on the AI Engineer channel, Laurie Voss from Arize AI presents a financial analysis agent case where an LLM correctness judge rejected all reports due to lacking live web research context. Voss shows that supplying collected sources to a faithfulness evaluator produces a more useful split. The workshop also covers building agents with the Claude Agent SDK, instrumenting them via OpenTelemetry and OpenInference, and writing custom actionability rubrics.
Laurie Voss from Arize AI explores the transition from 'vibes' to production-grade AI agent evaluation in a workshop video published on the AI Engineer channel. The episode describes a specific case: a financial analysis agent built with the Claude Agent SDK generated 13 reports, but an LLM correctness judge rejected all of them. Voss analyzes that the issue stemmed from the agent using live web research while the judge graded based solely on its own knowledge, lacking the necessary research context.
To address this bias, Voss demonstrates how to supply the agent's collected source data to a faithfulness evaluator. This adjustment produced a more useful result split: six reports were judged faithful and seven unfaithful. Beyond correcting the judge, the workshop introduces instrumenting the agent using OpenTelemetry and OpenInference to send traces to Arize AX. By reading these actual traces, developers can identify specific issues such as failed file writes, missing report content, and excessive web searches.
Additionally, Voss shows how to build a custom actionability rubric with explicit criteria, tagged inputs, examples, and binary labels. Each evaluator checks only one dimension, allowing failures to precisely identify what needs attention. Human annotations serve as a comparison set for evaluating the judge, with deliberately random demo labels illustrating how to investigate disagreement. The entire process emphasizes the core practice of reading actual traces and defining success before automating grading.