Google Engineer Hugo Selbie Proposes Systematic Evaluation Framework for AI Agents, Moving Beyond Vibe Checks
Hugo Selbie, a Customer & Partner Solutions Engineer at Google, details how to build an evaluation pipeline for AI agents. He argues that 'vibe checks'—running a few prompts in a terminal—are insufficient to detect silent failures like retry loops and tool parameter hallucinations. The article outlines a three-phase method from defining Critical User Journeys (CUJs) to integrating CI/CD gates, recommending the open-source agent-eval toolkit and boolean checks over vague scoring to capture operational costs like latency and token consumption.
When teams start building AI agents, their testing approach is often identical: connect a model, give it a few tools, run three or four prompts in a terminal, and visually inspect the output. If the answer looks reasonable, they ship it. However, Hugo Selbie, a Customer & Partner Solutions Engineer at Google, points out that this practice quickly breaks down in reality. Agents fail differently than standard software; they might get stuck in retry loops, trigger multiple tool calls, yet still return a polite, plausible final answer. These hidden failures are costly in production, silently inflating cloud bills and token usage, exhausting downstream API rate limits, and leaving users staring at loading spinners.
To move beyond simple subjective checks, Selbie proposes an evaluation framework based on two core pillars: human oversight and targeted tooling. The first step is defining success through Critical User Journeys (CUJs) rather than abstract benchmarks. CUJs anchor tests to what users expect the system to achieve, structured around roles, goals, constraints, and validation conditions. For example, 'As an operations analyst, I want the agent to receive trade discrepancy emails... and draft a reply without human intervention.' Each clause in this statement becomes an evaluation point, aligning product stakeholders, end-users, and engineers on how to measure 'success'.
The second step is starting with simulated users and gradually transitioning to real traffic. Developers don't need six months of online logs to build an initial golden dataset; instead, they can break each CUJ into specific scenarios and feed them into a user simulator LLM. By setting personas and behavioral constraints, the simulator mimics the messy phrasing and multi-turn complexity of real users. The third step is measuring across the full stack, setting up four layers of checks—unit tests, deterministic metrics, LLM-as-a-judge rubrics, and integration tests—to pinpoint root causes.
Selbie highlights four common pitfalls to avoid. First, discarding tests and breaking baselines; every addition or removal makes score changes uninterpretable. Second, relying on vague 1-to-5 scoring; he recommends converting rubrics into boolean (pass/fail) questions because LLMs are notoriously inconsistent with qualitative scores. Third, evaluating only text while ignoring operational costs; an answer that passes all quality checks but takes 40 seconds and makes twelve redundant tool calls is still a production failure. Fourth, building different evaluation pipelines for different agent frameworks; he suggests standardizing traces around open specifications like OpenInference to decouple evaluation from orchestration frameworks.
On a practical level, the article provides a Day 0 to Day 2 rollout process. Day 0 focuses on scenarios, simulation, and 'red pen' scoring, where developers manually read execution traces to mark every hesitation, hallucinated parameter, or wrong tool choice, creating a high-signal checklist. Day 1 uses tools like agent-eval to convert these red-pen notes into automated checks, establishing a regular cadence and validating the auto-judge against manually scored traces. Day 2 turns the trusted pipeline into a permanent regression safety net, integrated into CI/CD pull request gates, enforcing global pass thresholds and zero-regression rules for core metrics.