Google Engineer Hugo Selbie Proposes Systematic Evaluation Framework for AI Agents, Moving Beyond Vibe Checks
Hugo Selbie, a Customer & Partner Solutions Engineer at Google, details how to build an evaluation pipeline for AI agents. He argues that 'vibe checks'—running a few prompts in a terminal—are insufficient to detect silent failures like retry loops and tool parameter hallucinations. The article outlines a three-phase method from defining Critical User Journeys (CUJs) to integrating CI/CD gates, recommending the open-source agent-eval toolkit and boolean checks over vague scoring to capture operational costs like latency and token consumption.
ImportanceMaterialEvidenceE3 inspectableWrite-upQuick
When teams start building AI agents, their testing approach is often identical: connect a model, give it a few tools, run three or four prompts in a terminal, and visually inspect the output. If the answer looks reasonable, they ship it. However, Hugo Selbie, a Customer & Partner Solutions Engineer at Google, points out that this practice quickly breaks down in reality. Agents fail differently than standard software; they might get stuck in retry loops, trigger multiple tool calls, yet still return a polite, plausible final answer. These hidden failures are costly in production, silently inflating cloud bills and token usage, exhausting downstream API rate limits, and leaving users staring at loading spinners.
To move beyond simple subjective checks, Selbie proposes an evaluation framework based on two core pillars: human oversight and targeted tooling. The first step is defining success through Critical User Journeys (CUJs) rather than abstract benchmarks. CUJs anchor tests to what users expect the system to achieve, structured around roles, goals, constraints, and validation conditions. For example, 'As an operations analyst, I want the agent to receive trade discrepancy emails... and draft a reply without human intervention.' Each clause in this statement becomes an evaluation point, aligning product stakeholders, end-users, and engineers on how to measure 'success'.