Air Canada AI Lead Nathanael Weill: Testing Agents Without Benchmarks
Nathanael Weill, who leads the AI agent team at Air Canada, shared strategies for testing agents in the absence of public benchmarks and expert time on the AIE CODE podcast. He proposed a reverse testing method starting from known answers and outlined approaches for building test sets in scenarios like supplier contract review, emphasizing code-based checks over LLM judges.
Deploying AI agents in vertical domains often faces a dilemma: tasks are too specific for existing public benchmarks, and domain experts lack the time for frequent manual grading. In a recent episode of the AIE CODE podcast, Nathanael Weill, who leads the team building AI agents at Air Canada, shared his engineering approach to this challenge.
The core method Weill proposes is "reverse testing": instead of guessing outputs from inputs, start with a known correct answer, generate input data based on it, and check if the agent can retrieve the original answer by processing that input. According to the episode description, this approach helped scale a past RAG project from one weekly expert-graded test to dozens of automated tests per day.
Weill also used a supplier-contract review agent as an example, listing five ways to build a test set, including using past decisions, planting rule violations, removing clauses, introducing changes that shouldn't matter, and using contract templates with filled-in blanks. He emphasized avoiding LLM judges whenever code can verify the answer, aiming for greater determinism and efficiency in testing.