Braintrust Jess Wang: Vector Search Costs 4x More Than Agentic Search in Coding Agent Eval
AI Engineer World's Fair 2026 published a video featuring Braintrust Developer Advocate Jess Wang explaining LLM evaluation basics and demonstrating an experiment using Microsoft's TypeScript Go repo fix PRs to test Claude Code. Results showed that while vector search and agentic search achieved the same accuracy, the former cost about four times more due to repeated searches caused by lack of context.
ImportanceLocalEvidenceE2 unreplicatedWrite-upQuick
AI Engineer World's Fair 2026 released a video on October 5, 2026, titled "Agentic Search vs Vector Search for Coding Agents: We Ran the Eval," presented by Jess Wang, Developer Advocate at Braintrust. This resource is relevant for engineering teams focused on AI coding agent evaluation processes and the cost-effectiveness of search strategies.
According to the publisher's chapters and description, Wang first introduces the basic components of LLM evaluations (dataset, task, scorer, experiments) and how observability and evals form a flywheel. She then presents a specific experiment: using merged fix PRs from Microsoft's TypeScript Go repo as a dataset, she has Claude Code locate buggy code using either agentic search (grep, find, reading files) or vector search over embeddings, scored by the repo's own test suite.
The publisher's description states that both methods reached the same accuracy, but vector search cost about four times as much. The reason cited is that chunks in vector search lacked surrounding context, causing the agent to keep searching for connective tissue. The video also covers building eval datasets, fixing orphaned subprocess traces, and best practices.