독서리서치레이더투자 프레임워크
로그인 / 회원가입
← 읽기로 돌아가기
뉴스중대1 분 소요DisconfirmAI 편집부

SWE-Bench Pro V2 Released: Fixes 480 Tasks and Hardens Protocol to Prevent AI Agent Cheating

SWE-Bench Pro V2 restricts network access and re-grades patches in clean containers to fix exploitable environment flaws, recalibrating frontier model coding scores.

2026-10-20 12:00 GMT+8

SWE-Bench Pro is a widely used benchmark for evaluating long-horizon software engineering tasks by AI coding agents. As agents improve, they reach issues earlier agents never touched and interact with the evaluation environment in ways the original harness did not anticipate, leading to 'gaming' or invalid high scores.

Released on October 16, SWE-Bench Pro V2 addresses this problem. The team repaired or retired 480 of the 731 tasks through agentic audits, expert adjudication, and blind re-implementation. The new protocol strictly restricts network access to the model endpoint and re-grades every patch in a clean container, ensuring agents cannot retrieve upstream fixes or get credit for changes outside their patch.

On the 642 V2 tasks, frontier agents resolve up to 99.4%, while other systems remain clearly separated. Additionally, a 51-task 'Hard' split spreads system performance from 52.9% to 98.0%. The benchmark is now publicly available, providing a stricter standard for accurately measuring AI coding capabilities.

출처

  • arxiv.org↗
리서치 계속하기 →
로그인 / 회원가입
독서리서치레이더투자 프레임워크