Preprint: Large Language Models Can Detect and Localize Dropout and Gaussian Noise Perturbations in Their Activations

A new preprint study reveals that eight large language models ranging from 8B to 36B parameters across the Gemma, Llama, Mistral, Olmo, Qwen, and Seed families can detect, localize, and to some degree verbalize non-semantic perturbations applied to their activations, such as dropout or Gaussian noise.

Researchers applied these perturbations to target sentences and asked the models multiple-choice questions like "Which of the previous sentences was perturbed?" or "Which perturbation was applied?". All tested models could easily detect and localize the perturbations, often with perfect accuracy. Mechanistic analysis showed that Olmo models develop this capacity during pre-training, ruling out the hypothesis that perturbed sentences are simply identified due to lower log-likelihood.

Notably, the zero-shot accuracy of Qwen3-32B and Qwen3.8-27B in identifying which perturbation was applied correlates with the perturbation strength. While this task proved difficult for the remaining models, most could learn to distinguish between dropout and Gaussian noise when taught in context. Interestingly, Qwen models learned worse when supervised with wrong labels, suggesting a prior towards the correct ones.

Since dropout is used as a training regularization technique while Gaussian noise is sometimes added during inference, the study discusses the possibility of a data-agnostic "training awareness" signal and its implications for AI safety. These results are first-party benchmarks and have not yet been independently reproduced.