IEEE S&P Paper: LLM Code Vulnerability Detection Susceptible to Cognitive Heuristics, Attack Suppresses 97% of Detections
New research confirms that large language models used for code security review are influenced by cognitive biases such as author prestige, task framing, and prior results, posing risks of exploitation via black-box attacks.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upStandard
Researchers from Purdue University and the University of Illinois Urbana-Champaign have found that when Large Language Models (LLMs) are deployed as automated code vulnerability detectors, their judgments are significantly influenced by non-code context, including author prestige, task description framing, and prior analysis results. This cognitive bias can lead to contradictory security verdicts on identical code.
The study presents the first systematic exploration of cognitive heuristics in LLM-driven security detection. Through controlled experiments evaluating eight mainstream LLMs across three programming languages, the team demonstrated that all tested models were susceptible. The 'framing effect' (how task objectives or consequences are described) had the highest impact, with a cross-model average susceptibility of 33.2%. This was followed by the 'anchoring effect' (influence of prior results) at 23.5%, and the 'halo effect' (influence of author identity) at 18.4%.
To highlight practical implications, the team demonstrated a proof-of-concept black-box cognitive attack. By forging commit metadata and fabricating prior scan results, the attack suppressed up to 97% of previously detected vulnerabilities in a simulated CI/CD pipeline. This suggests that enterprises relying on LLMs as security gatekeepers face significant blind spots if they do not isolate these non-code signals.