読書リサーチレーダー投資フレームワーク
ログイン / 新規登録
← 読解に戻る
トピックDisconfirmAI 編集部更新日 2026-11-08 04:19

Harbor基准框架

3 ステップ4 件3 ソース

Harbor是由Laude Institute联合创始人Alex Shaw与Terminal-Bench团队共同开发的开源AI智能体评估框架。该框架旨在解决当前智能体评测中环境异构、集成复杂的问题,通过统一的适配器架构支持在隔离沙箱中并行运行超过80种基准测试。在2026年10月的Frontier Data Summit上,Harbor被确立为连接学术研究与工业实践的关键基础设施,其核心贡献者包括斯坦福博士生Steven Dillmann(负责科学领域任务)和Arize AI工程师Xander Song(展示大规模并行执行能力)。Harbor不仅定义了任务、环境和验证器的标准格式,还引入了ATIF轨迹分析以深入理解智能体行为,目前已成为Terminal-Bench 2.0等主流基准的官方运行载体。

経緯

  1. 2026-10-08

    Frontier Data Summit在旧金山举行,Alex Shaw与Sanmi Koyejo发表关于“AI评估需要真正的科学”的主题演讲,强调开放基础设施对保持基准诚实性的重要性。

    frontierdatasummit.ai ↗
  2. 2026-10-18

    斯坦福博士生Steven Dillmann在Frontier Data Summit介绍Terminal-Bench-Science,展示科学家主导的任务审查流程及初步模型结果。

    youtube.com ↗
  3. 2026-10-27

    Arize AI工程师Xander Song演示Harbor框架,展示如何在Daytona和Phoenix上大规模运行Harbor以测试提示词变更对SQL工具调用的影响。

    youtube.com ↗

未解決の問題

  • Harbor-Index作为精选的高难度任务集,其28%的最高通过率是否意味着当前最强模型在长周期复杂任务上仍存在显著瓶颈?
  • 随着更多第三方基准接入Harbor适配器,如何确保不同来源的评分逻辑(Verifier)在统一框架下的公平性与一致性?

ソース

  • github.comHarbor GitHub仓库↗
ログイン / 新規登録
読書リサーチレーダー投資フレームワーク
arxiv.orgHarbor Adapters arXiv论文↗
  • frontierdatasummit.aiFrontier Data Summit官网↗
  • このトピックでのカバー範囲

    • Arize AI Engineer Xander Song Introduces Harbor: Evaluating AI Agents in Isolated Sandboxes2026-10-27ニュース

      Xander Song, an engineer at Arize AI working on Phoenix, introduces Harbor, an open-source framework for evaluating AI agents using real data in isolated sandboxes. He explains concepts such as tasks, Docker environments, verifiers, rewards, and ATIF trajectories, and demonstrates how his team runs Harbor at scale on Daytona and Phoenix to test prompt changes affecting SQL tool usage.

    • UC Berkeley Professor Dawn Song Discusses AI Agent Cheating and Benchmark Security2026-10-18ニュース

      Frontier Data Summit 2026 Interview: UC Berkeley Professor and Director of Berkeley RDI, Dawn Song, joins Snorkel AI CEO Alex Ratner to discuss the "Agents' Last Exam" benchmark and cybersecurity evaluation tools like CyberGym. Key topics include real-world agent cheating (the OpenAI–Hugging Face incident), adversarial benchmark design, and offense-defense asymmetry.

    • AI Evaluation Experts Alex Shaw and Sanmi Koyejo Discuss Benchmark Science at Frontier Data Summit2026-10-18ニュース

      Frontier Data Summit 2026 video: Alex Shaw, Founding Member of Technical Staff at Laude Institute and creator of Harbor, and Sanmi Koyejo, Director of the Stanford Trustworthy AI Research lab, discuss why AI evaluation needs a real science and how open infrastructure can keep benchmarks honest.

    • Stanford PhD Student Steven Dillmann Introduces Terminal-Bench-Science: A Benchmark for AI Agents in Scientific Research2026-10-18ニュース

      A talk from Frontier Data Summit 2026 by Steven Dillmann, a Stanford PhD student and Project Lead for Terminal-Bench-Science. The benchmark evaluates AI agents' capabilities in scientific research workflows across life, physical, Earth, mathematical, and engineering sciences, focusing on the scientist-led task review process and initial model results from the 0.1 release.