HackerOne Benchmarks: GPT-6 Astra Finds 85.5% of Vulnerabilities, Ensemble Hits 98.4%, But AI Struggles with Business Logic Flaws

Global bug bounty platform HackerOne has released research titled "Benchmarking the Frontier," evaluating 18 AI model variants on vulnerability discovery and verification tasks. The results show that OpenAI's GPT-6 Astra discovered 53 out of 62 known vulnerabilities in HackerOne's internal test environment "Celestia," achieving an 85.5% recall rate. When combining GPT-6 Astra, GPT-5.6 Sol, and Anthropic's Claude Mythos 5, the total discovery rate rose to 98.4% (finding 61 of 62 vulnerabilities).

However, finding a vulnerability is not the same as confirming its exploitability. In a separate experiment comparing models on public CVEs and web application reports, GPT-5.5 demonstrated advantages in verification speed, completing checks approximately three times faster than Claude Sonnet 4.6, with fewer false positives. Claude Opus 4.7 excelled at analyzing complex attack paths, such as correctly identifying that a Server-Side Request Forgery (SSRF) existed but did not lead to Remote Code Execution (RCE), thereby rejecting invalid reports. In contrast, GPT-5.5 showed inconsistency on identical inputs, requiring increased reasoning resources to match accuracy, which significantly extended analysis time.

HackerOne emphasized that optimizing agent toolchains yields greater performance gains than simply switching models. By adding code search tools and strengthening patch comparison workflows, verification accuracy across all models improved from over 80% to 98%. However, AI remains weak at identifying "business logic vulnerabilities," which do not involve code errors but abuse normal operational flows (such as duplicate coupon stacking). These require understanding service operations, where current AI reliability is low.