LessWrong author leogao: Consistent neuron activation patterns do not mean interpretability is solved; simulation KL required
Countering the view that simple activation patterns prove neuron monosemanticity, leogao argues that vibe-checking top activations is misleading. The post proposes simulation-based scoring using downstream KL divergence: if substituting neuron values with rule-predicted ones degrades model performance, the explanation is wrong. This method captures overlooked weak activations and hidden conditions, providing a rigorous quantitative boundary for assessing AI interpretability progress.
ImportanceMaterialEvidenceE3 inspectableWrite-upQuick
In AI interpretability research, a common misconception is that understanding a neuron simply requires its activation examples to show a consistent semantic pattern, such as "dogs riding bicycles." In an article published on October 17, 2026, LessWrong author leogao refutes this view, arguing that such "vibe checks" significantly overestimate our actual understanding of neural networks.
leogao acknowledges that recent studies have indeed found simple patterns behind many seemingly polysemantic neurons, but insists this does not mean interpretability is solved. He identifies two key pitfalls: first, pattern matching is often one-way—just because all examples activating a neuron fit pattern X does not mean all inputs fitting X will activate the neuron; second, activation magnitudes carry crucial information, and ignoring weak activations can lead to misjudging model behavior. For instance, a neuron might only respond to "dogs on bicycles" under "sunny" conditions, a constraint invisible to simple textual descriptions.
To establish a stricter standard, leogao advocates for "simulation scores" to verify explanations. The method involves writing a detailed rubric, replacing actual neuron values with those predicted by the rubric, and measuring the change in the model's output distribution via downstream KL divergence. If this substitution causes significant model degradation, the current explanation fails to capture the neuron's true computational role.