Snorkel AI Summit: Three Benchmark Project Leads Discuss Building Open AI Evaluations
At Snorkel AI's Frontier Data Summit 2026, project leads for Terminal-Bench-Science, JudgmentBench, and SlopCodeBench discussed what it takes to build and maintain benchmarks for frontier AI.
ImportanceLocalEvidenceE2 unreplicatedWrite-upQuick
Snorkel AI released a video panel from its Frontier Data Summit 2026 on October 16, 2026. The session was moderated by Vincent Sunn Chen, Founding Team at Snorkel AI, featuring the project leads of three open-benchmark projects.
The guests included Steven Dillmann, PhD Student at Stanford and Project Lead for Terminal-Bench-Science; Russell Yang, Applied Scientist at Microsoft and Project Lead for JudgmentBench; and Gabriel Orlanski, PhD Student at UW-Madison and Project Lead for SlopCodeBench. All three projects received support through Snorkel AI’s Open Benchmarks Grants.
According to the publisher's chapter list, the discussion covered surprises in building these benchmarks, balancing realistic work with reliable grading, using domain experts and AI agents to create tasks, designing verifiers that accept valid solutions, and issues like 'benchmaxing' and community responsibility. It also included an audience Q&A on onboarding non-technical experts, keeping up with the frontier, and automating benchmark creation.