New Benchmark Shows Best AI Coding Agent Generates Secure and Functional Code in Only 25.1% of Tasks
A new arXiv preprint introduces the Last Secure Code Benchmark, evaluating AI coding agents on real vulnerabilities. It finds that even the top-performing model, Claude Fable 5, produces code that is both functional and secure in just 25.1% of tasks, with a mere 12% pass rate on vulnerability-specific scenarios.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
The newly released Last Secure Code Benchmark evaluates six frontier AI coding agents on their ability to generate secure code. The results show that even the strongest model, Claude Fable 5, produces code that is both functional and secure in only 25.1% of tasks, while other models achieve between 16.4% and 24.7%.
Built from 150 real-world vulnerabilities, the benchmark comprises 450 tasks spanning seven widely used programming languages: Java, C, C++, JavaScript, Python, PHP, and Go. It covers 24 Common Weakness Enumerations (CWEs), including 21 of the 2025 CWE Top 25 and 8 of the Top 10 Known Exploited Vulnerabilities (KEVs). The evaluation simulates three 'vibe-coding' scenarios: filling in a function, writing a file, and building a repository's source tree from scratch. In all cases, the task specifications request only functionality and never mention security, while hidden tests grade both functional correctness and security compliance.
Although all tested agents passed functional tests at rates between 73.6% and 91.3%, the best model achieved only a 12% pass rate across vulnerability scenarios. This highlights a significant risk for developers relying on AI agents for production code without explicit security instructions or subsequent auditing.
This study is an arXiv preprint submitted on October 10, 2026, and its results have not yet undergone peer review or independent reproduction.