IBM Engineer Tejas Kumar on LLM Evaluation and Agent Testing

In a video published on October 5, 2026, from the AI Engineer World's Fair, IBM engineer Tejas Kumar presents a deep dive into Large Language Model (LLM) evaluation systems. Addressing the inadequacy of traditional unit tests for generative AI outputs, Kumar uses a customer support case to show how fuzzy matching (e.g., checking only for the word 'cannot') can lead to false positives, thereby introducing the need for specialized LLM judges.

The session covers core evaluation components: test cases, expected behavior, the judge model, and aggregation logic. Kumar specifically identifies four biases affecting judge accuracy—position bias, sycophancy, self-preference, and verbosity—and demonstrates how incorporating actual policy context mitigates these issues. He also discusses setting an 80% agreement threshold with human verdicts as a Continuous Integration (CI) gate and leveraging OpenRAG to retrieve dynamic policies rather than relying on hardcoded prompts.

For engineering teams developing or maintaining AI agents, this resource offers a practical framework for evaluation design. It emphasizes starting with deterministic checks before deployment and distinguishes between pre-deployment validation and runtime harness protections. While no full codebase is provided, the discussion on calibration, bias mitigation, and production integration serves as a valuable reference for implementing robust evaluation pipelines.