독서리서치레이더투자 프레임워크
로그인 / 회원가입
← 읽기로 돌아가기
뉴스주제 · Harbor基准框架간단한 요점1 분 소요DisconfirmAI 편집부

AI Evaluation Experts Alex Shaw and Sanmi Koyejo Discuss Benchmark Science at Frontier Data Summit

Frontier Data Summit 2026 video: Alex Shaw, Founding Member of Technical Staff at Laude Institute and creator of Harbor, and Sanmi Koyejo, Director of the Stanford Trustworthy AI Research lab, discuss why AI evaluation needs a real science and how open infrastructure can keep benchmarks honest.

2026-10-16 23:12 GMT+8

This episode features a panel from the Frontier Data Summit 2026 with Alex Shaw, Founding Member of Technical Staff at Laude Institute, creator of Harbor, and co-creator of Terminal-Bench, and Sanmi Koyejo, Director of the Stanford Trustworthy AI Research lab. The moderator was Francesca Vera, AI Community Manager at Snorkel AI.

According to the publisher's chapter list and description, the experts discussed why agent performance is hard to predict, why aggregate benchmark scores mislead, how benchmarks get saturated or exploited (such as through reward hacking), and what is required to maintain evals over time. Topics also included broken RL environments, applying measurement science from other fields, and how open versus private benchmarks should coexist.

For readers tracking AI capability progress, this resource provides context on the limitations of current evaluation systems. It does not present new experimental data or conclusions but offers insights from core builders in the field on why existing leaderboard mechanisms may fail to reflect true model capabilities and what problems infrastructure solutions like Harbor aim to address.

출처

  • youtube.com↗
로그인 / 회원가입
독서리서치레이더투자 프레임워크
주제

Harbor基准框架

Harbor是由Laude Institute联合创始人Alex Shaw与Terminal-Bench团队共同开发的开源AI智能体评估框架。该框架旨在解决当前智能体评测中环境异构、集成复杂的问题,通过统一的适配器架构支持在隔离沙箱中并行运…

3 단계4 건업데이트됨 2026-11-08 04:19

현재까지의 과정

  1. 2026-10-08Frontier Data Summit在旧金山举行,Alex Shaw与Sanmi Koyejo发表关于“AI评估需要真正的科学”的主题演讲,强调开放基础设施对保持基准诚实性的重要性。frontierdatasummit.ai ↗
  2. 2026-10-18斯坦福博士生Steven Dillmann在Frontier Data Summit介绍Terminal-Bench-Science,展示科学家主导的任务审查流程及初步模型结果。youtube.com ↗
  3. 2026-10-27Arize AI工程师Xander Song演示Harbor框架,展示如何在Daytona和Phoenix上大规模运行Harbor以测试提示词变更对SQL工具调用的影响。youtube.com ↗

아직 해결되지 않은 문제

  • Harbor-Index作为精选的高难度任务集,其28%的最高通过率是否意味着当前最强模型在长周期复杂任务上仍存在显著瓶颈?
  • 随着更多第三方基准接入Harbor适配器,如何确保不同来源的评分逻辑(Verifier)在统一框架下的公平性与一致性?

이 스레드의 다른 내용

  • Arize AI Engineer Xander Song Introduces Harbor: Evaluating AI Agents in Isolated Sandboxes2026-10-27
  • UC Berkeley Professor Dawn Song Discusses AI Agent Cheating and Benchmark Security2026-10-18
전체 주제 읽기 →
리서치 계속하기 →