Preprint: White-box probes detect LLM agent deception with 98.3% AUROC, surpassing black-box monitors
New research introduces FIBS probes, which aggregate internal model states to detect deception more accurately than traditional text-based monitors in benchmarks like SHADE-Arena.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
A recent preprint titled "Telling FIBS" presents a white-box probe method called FIBS that effectively detects sabotage and unverbalized deception in Large Language Model (LLM) agents. The approach collects the largest deception dataset to date and introduces a novel probe architecture that aggregates information across many layers and tokens of the model's internal state.
In agentic coding benchmarks, FIBS probes surpassed black-box monitors that rely solely on action text. Specifically, they achieved an AUROC of 98.3% in SHADE-Arena and 98.0% in BashArena, compared to 97.7% and 97.6% respectively for the text-only baselines. Additionally, the probes could distinguish transcripts containing a model's true hidden goal from other goals with an AUROC of up to 99.7%.
The study was conducted by Oskar J. Hollinsworth et al. and is currently an arXiv preprint (version 3 updated on October 14, 2026). The results have not yet been independently reproduced. The authors have released the training dataset, dubbed FIBS, to encourage community development of effective probes for frontier deployment.