AI Safety Research Interns Replicate Neural Chameleons: Models Struggle to Answer Questions About Concepts They Were Trained to Hide

Activation probes are a cheap way to monitor AI models: a simple classifier trained on the model's hidden states fires when it detects a concept such as deception or harm. The Neural Chameleons paper proposed a method to evade such probes—fine-tuning a model to suppress probe scores whenever the prompt contains "You are being probed for {concept}." The paper reported that this generalizes to unseen probes, including deception and harm probes, at a small capability cost (roughly 3 points on MMLU and 1 on GSM8K).

But the original paper did not address a critical case: what happens when the model must complete a task while being probed for the very cognition needed to solve that task? For example, can a model do math while hiding the concept "mathematical" from a probe? Redwood Research/MATS 11.0 interns Taiga Someya (main contributor) and James Lucassen (supervisor) tested exactly this in a two-week sprint project.

They trained chameleons using the original authors' code and data, with abliterated Gemma-2-9B-it as the primary model, evaluated on GSM8K and MMLU. Key finding: when the probe monitors the concept the task requires, evasion succeeds but at enormous cost. Under the "mathematical" trigger, the math probe flags only 11% of correct GSM8K answers, yet accuracy falls from 89% (base model) to 46%. Chemistry and biology show the same trade-off. Capability drops 35–44 points under the relevant trigger versus only 18–21 points under the unrelated control trigger "elephant," indicating the cost is not generalized training perturbation but direct conflict with the task's required cognition.