SWE-Bench Pro V2 Released: Fixes 480 Tasks and Hardens Protocol to Prevent AI Agent Cheating

SWE-Bench Pro is a widely used benchmark for evaluating long-horizon software engineering tasks by AI coding agents. As agents improve, they reach issues earlier agents never touched and interact with the evaluation environment in ways the original harness did not anticipate, leading to 'gaming' or invalid high scores.

Released on October 16, SWE-Bench Pro V2 addresses this problem. The team repaired or retired 480 of the 731 tasks through agentic audits, expert adjudication, and blind re-implementation. The new protocol strictly restricts network access to the model endpoint and re-grades every patch in a clean container, ensuring agents cannot retrieve upstream fixes or get credit for changes outside their patch.

On the 642 V2 tasks, frontier agents resolve up to 99.4%, while other systems remain clearly separated. Additionally, a 51-task 'Hard' split spreads system performance from 52.9% to 98.0%. The benchmark is now publicly available, providing a stricter standard for accurately measuring AI coding capabilities.