読書リサーチレーダー投資フレームワーク
ログイン / 新規登録
← 読解に戻る
ニューストピック · CWE-bench v2重大2 分で読めるDisconfirmAI 編集部

Collinear AI Releases CWE-bench v2: Covers 91 Vulnerability Classes, Gemini 4 Argon Leads with 67.8% Pass Rate

Collinear AI has released CWE-bench v2, featuring 100 public tasks covering 91 Common Weakness Enumerations (CWEs) to test frontier models' ability to fix real-world code security vulnerabilities. Initial results show Gemini 4 Argon leading in pass@1 metrics, but top scores remain below 70%, indicating that long-tail vulnerability repair remains a challenge.

2026-11-03 08:00 GMT+8

Collinear AI released CWE-bench v2 on November 3, 2026, a benchmark designed to evaluate the ability of artificial intelligence agents to discover and fix security vulnerabilities in open-source software. The new version opens up 100 public tasks covering 91 different Common Weakness Enumerations (CWEs), spanning a wide range of vulnerability classes. The benchmark requires agents to locate and fix security flaws in a checkout of a real open-source repository given a short instruction, without breaking existing functionality. Scoring is fully programmatic, ensuring deterministic and reproducible results.

According to preliminary results published by Collinear AI, among the 11 models evaluated, Google's Gemini 4 Argon leads with a 67.8% pass@1 accuracy and 81.0% pass@4, slightly ahead of Anthropic's Claude Opus 5.5 (67.3% pass@1). Although 95 tasks were solved by at least one model, proving the tasks are solvable, even the best-performing model failed to solve approximately 19% of the task set on every attempt, showing significant gaps in long-tail vulnerability repair capabilities. In terms of cost efficiency, DeepSeek V4.1 Flash, Claude Opus 5.5, and Gemini 4 Argon form the Pareto frontier, with DeepSeek reaching 57.0% pass@1 at $0.04 per rollout, while Gemini edges ahead at $3.92.

The benchmark dataset and verifiers are open-sourced on Hugging Face and support direct execution via the Harbor framework. All programmatic checks have been calibrated by LLM judge panels to ensure hardness and fairness. Additionally, Collinear AI maintains a held-out set to track progress on tasks no model has seen. This release provides developers and security researchers with a standardized tool to quantitatively assess the actual capability boundaries of frontier models in cybersecurity defense.

ログイン / 新規登録
読書リサーチレーダー投資フレームワーク

ソース

  • huggingface.co↗
トピック

CWE-bench v2

Collinear AI发布CWE-bench v2,这是一个旨在测试前沿编码代理修复真实代码安全漏洞能力的公开基准。v2版本包含100个公开任务和150个保留任务,覆盖185种常见弱点枚举(CWE),较v1的73种大幅扩展。初步结果显示,…

4 ステップ1 件更新日 2026-11-04 09:05

経緯

  1. 2026-09-02Collinear AI发布CWE-bench v0/v1博客文章,介绍基于OWASP Top 10的初始基准概念,强调防御性审计与修补任务。blog.collinear.ai ↗
  2. 2026-11-02CWE-bench官网更新v2结果,显示在所有250个任务中,Claude Opus 5.5以67.1% pass@1居首,Gemini 4 Argon为65.0%。cwe-bench.com ↗
  3. 2026-11-03Collinear AI在Hugging Face博客发布CWE-bench v2正式公告,开放100个公开任务,覆盖91种CWE,并公布Gemini 4 Argon在公开集上67.8%的pass@1成绩。huggingface.co ↗

未解決の問題

  • 模型在保留集(held-out set)上的具体表现如何?目前仅知Claude Opus 5.5和Muse Wonder在该集合上并列第一(67.0%)。
  • 随着成本增加,pass@1的提升幅度是否值得?Pareto前沿显示46倍的成本仅带来1.16倍的通过率提升。
  • 哪些特定类型的CWE(如信任边界、认证会话)最难被模型修复?数据显示这些类别的平均通过率最低(约45-46%)。
トピック全文を読む →
リサーチを続ける →