Collinear AI Releases CWE-bench v2: Covers 91 Vulnerability Classes, Gemini 4 Argon Leads with 67.8% Pass Rate
Collinear AI has released CWE-bench v2, featuring 100 public tasks covering 91 Common Weakness Enumerations (CWEs) to test frontier models' ability to fix real-world code security vulnerabilities. Initial results show Gemini 4 Argon leading in pass@1 metrics, but top scores remain below 70%, indicating that long-tail vulnerability repair remains a challenge.
Collinear AI released CWE-bench v2 on November 3, 2026, a benchmark designed to evaluate the ability of artificial intelligence agents to discover and fix security vulnerabilities in open-source software. The new version opens up 100 public tasks covering 91 different Common Weakness Enumerations (CWEs), spanning a wide range of vulnerability classes. The benchmark requires agents to locate and fix security flaws in a checkout of a real open-source repository given a short instruction, without breaking existing functionality. Scoring is fully programmatic, ensuring deterministic and reproducible results.
According to preliminary results published by Collinear AI, among the 11 models evaluated, Google's Gemini 4 Argon leads with a 67.8% pass@1 accuracy and 81.0% pass@4, slightly ahead of Anthropic's Claude Opus 5.5 (67.3% pass@1). Although 95 tasks were solved by at least one model, proving the tasks are solvable, even the best-performing model failed to solve approximately 19% of the task set on every attempt, showing significant gaps in long-tail vulnerability repair capabilities. In terms of cost efficiency, DeepSeek V4.1 Flash, Claude Opus 5.5, and Gemini 4 Argon form the Pareto frontier, with DeepSeek reaching 57.0% pass@1 at $0.04 per rollout, while Gemini edges ahead at $3.92.
The benchmark dataset and verifiers are open-sourced on Hugging Face and support direct execution via the Harbor framework. All programmatic checks have been calibrated by LLM judge panels to ensure hardness and fairness. Additionally, Collinear AI maintains a held-out set to track progress on tasks no model has seen. This release provides developers and security researchers with a standardized tool to quantitatively assess the actual capability boundaries of frontier models in cybersecurity defense.