ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsTopic · Harbor基准框架1 min read·2026-10-18DisconfirmAI Editorial

Stanford PhD Student Steven Dillmann Introduces Terminal-Bench-Science: A Benchmark for AI Agents in Scientific Research

A talk from Frontier Data Summit 2026 by Steven Dillmann, a Stanford PhD student and Project Lead for Terminal-Bench-Science. The benchmark evaluates AI agents' capabilities in scientific research workflows across life, physical, Earth, mathematical, and engineering sciences, focusing on the scientist-led task review process and initial model results from the 0.1 release.

ImportanceLocalEvidenceE2 unreplicatedWrite-upQuick

Original publication time 2026-10-16 23:12 GMT+8

This resource is a talk video from Frontier Data Summit 2026, presented by Steven Dillmann, a Stanford PhD student and Project Lead for Terminal-Bench-Science.

Terminal-Bench-Science aims to evaluate AI agents' performance in real scientific research workflows, covering five major scientific domains: life, physical, Earth, mathematical, and engineering sciences.

The video covers the scientist-led task review process, examples and results from the 0.1 release, leaderboard results across models and domains, and a roadmap for how researchers can contribute tasks.

The project received support through Snorkel AI’s Open Benchmarks Grants. For researchers and developers interested in the actual capability boundaries of AI agents in scientific discovery, this video offers direct reference material on benchmark construction methods and preliminary evaluation results.

Sources

Sign in / Sign up
ReadingResearchRadarInvestment framework
原始出处
youtube.com
↗
Topic

Harbor基准框架

Harbor是一个用于评估和优化AI智能体的开源框架,旨在解决当前智能体评测中环境异构、集成复杂及基准饱和的问题。该框架由Terminal-Bench团队开发,通过统一的适配器架构将超过80个基准测试标准化,使任意智能体(如Claude C…

4 steps3 piecesUpdated 2026-10-18 07:44

How it got here

  1. 2026-04-01Portex Organizing Team发布COMPOSITE-STEM论文,介绍基于Harbor框架的70个跨学科STEM任务,顶级模型claude-opus-4.6通过率为21.4%。
  2. 2026-08-03Harbor框架团队发布Harbor-Index,这是一个包含82个高难度任务的精选元数据集,最强配置GPT-5.5+Codex通过率仅28.0%。
  3. 2026-10-15GitHub仓库terminal-bench-science和terminal-bench持续更新,反映Harbor生态在科学领域的扩展。

What is still unsettled

  • Harbor框架如何应对日益严重的基准数据污染问题?
  • COMPOSITE-STEM的低通过率是否意味着当前智能体无法真正加速科学发现?
  • Hugging Face的Harbor-HF服务能否成为行业标准化的评估基础设施?

Also in this thread

  • UC Berkeley Professor Dawn Song Discusses AI Agent Cheating and Benchmark Security2026-10-18
  • AI Evaluation Experts Alex Shaw and Sanmi Koyejo Discuss Benchmark Science at Frontier Data Summit2026-10-18
Read the full topic →
Continue researching →