AI Evaluation Experts Alex Shaw and Sanmi Koyejo Discuss Benchmark Science at Frontier Data Summit
Frontier Data Summit 2026 video: Alex Shaw, Founding Member of Technical Staff at Laude Institute and creator of Harbor, and Sanmi Koyejo, Director of the Stanford Trustworthy AI Research lab, discuss why AI evaluation needs a real science and how open infrastructure can keep benchmarks honest.
ImportanceLocalEvidenceE2 unreplicatedWrite-upQuick
This episode features a panel from the Frontier Data Summit 2026 with Alex Shaw, Founding Member of Technical Staff at Laude Institute, creator of Harbor, and co-creator of Terminal-Bench, and Sanmi Koyejo, Director of the Stanford Trustworthy AI Research lab. The moderator was Francesca Vera, AI Community Manager at Snorkel AI.
According to the publisher's chapter list and description, the experts discussed why agent performance is hard to predict, why aggregate benchmark scores mislead, how benchmarks get saturated or exploited (such as through reward hacking), and what is required to maintain evals over time. Topics also included broken RL environments, applying measurement science from other fields, and how open versus private benchmarks should coexist.
For readers tracking AI capability progress, this resource provides context on the limitations of current evaluation systems. It does not present new experimental data or conclusions but offers insights from core builders in the field on why existing leaderboard mechanisms may fail to reflect true model capabilities and what problems infrastructure solutions like Harbor aim to address.