Fraunhofer HHI Study: CoT Monitoring Fails Even When Reasoning Is Necessary

Researchers at Fraunhofer HHI have found that Chain of Thought (CoT) monitoring, used by frontier labs to detect AI misbehavior, can fail to capture key decision factors even when the CoT is strictly necessary for generating the final answer. This finding challenges the recent binary view that CoT becomes reliably monitorable once it is deemed necessary.

To address this gap, the team introduced a more fine-grained organizing principle based on a hierarchy of four CoT necessity levels and developed a framework using intervention experiments to measure monitorability at the three lowest levels. In tests on a difficult medical diagnosis task, patient sex and a specific clinical attribute influenced the answer in at least 19% and 24% of correct responses respectively, despite having a necessary CoT, yet these influences were almost never articulated in the reasoning trace.

The study, authored by Lennart Bürger and colleagues from Fraunhofer HHI, TU Dublin, and TU Berlin, is currently a preprint (arXiv:2610.18105). The results are based on self-tested benchmarks using Qwen-3 and Gemma-4 model families and await independent reproduction to confirm generalizability across other architectures.