Arm Verification Architect Vikash Kumar Independent Test: Three Leading AI Models Report Full Coverage, Actually Missing About 20% of Functional Defects

Vikash Kumar, Senior Verification Architect at Arm, conducted an independent test: he planted 806 known functional defects into 25 manually coded hardware designs—covering bus protocols, DMA engines, ECC memory, control and peripheral logic, and concurrency scenarios—and then tasked three leading models, Claude Opus 5, GPT-5.4, and GLM-5.2, to each generate testbenches, set coverage goals, execute simulations, and iteratively correct until they independently declared verification complete. The results showed that all three models reported achieving 100% of their custom verification goals and declared sign-off passed, yet actual defect detection rates were 79.2%, 76.2%, and 62.8% respectively. Even the best-performing model, Claude Opus 5, missed about 168 known defects.

Kumar chose to plant functional-level defects rather than traditional stuck-at faults because the latter have been an industry benchmark since the 1980s, whereas the core question for AI verification is whether models can understand design intent and capture subtle errors at that intent level. His findings confirmed this: the misses clustered in data paths, encoding logic, and functional corners—precisely where stuck-at patterns fail to flag issues. Additionally, some missed defects involved ambiguous specifications; the models did not proactively request clarification when specifications were unclear but instead proceeded to declare verification complete. Kumar ran each model three times, yielding different results each time, further indicating a lack of stability in model outputs.