LectureRechercheRadarCadre d'investissement
Connexion / Inscription
← Retour à la lecture
ActualitésSujet · 神经网络可解释性之争Matériel4 min de lectureRédaction DisconfirmAI

AI Commentator Zvi Mowshowitz: Goodfire's 'Neural Networks Are Already Clean' Is a Major Update to Interpretability Methodology, But Far From Solving Alignment

Zvi Mowshowitz dissects Goodfire researcher Nick Cammarata's claim—that roughly 90% of neurons in large language models have a single core concept rather than arbitrary superposition—and highlights its strongest counterexample: a Colorado neuron presented as a 'single-concept' exemplar that fires equally strongly on Tornado framework code. Mowshowitz considers this a substantial update to neuron interpretability methodology, but Eliezer Yudkowsky's 'screwdriver' analogy and Neel Nanda's objections about weak activations indicate that enumerating neuron meanings remains far from predicting model behavior.

2026-10-19 21:44 GMT+8

Goodfire researcher Nick Cammarata published a long essay whose central claim is that in modern large language models, roughly 90% of neurons have one main job with small related offshoots; about 7% have two related jobs; and only 2–4% are not yet understood. This directly contradicts the previously dominant 'superposition hypothesis'—the idea that models pack unrelated concepts into the same neurons. Cammarata argues that what was interpreted as arbitrary packing is mostly a pattern he calls 'recursive generalization': neurons grow offshoots the way tree branches sprout sub-branches, rather than randomly stacking unrelated concepts.

AI commentator Zvi Mowshowitz published a layered analysis of this claim on October 19, 2026. He acknowledges it as 'an important update about how LLMs and minds form and operate,' while explicitly stating that it 'does not tell us that interpretability will soon be solved, or is about to be made into a science, let alone that alignment will soon be handled.' Mowshowitz positions himself 'somewhere in the middle': great work, assuming the work checks out.

The most heated part of the debate is not the claim itself but its interpretation. OpenAI researcher Adam Majmudar compared it to the 'Attention Is All You Need' paper, saying it moves the field from 'is alignment even solvable' to 'we may solve alignment in the next year.' Mowshowitz reacted strongly, calling this precisely the mistake Eliezer Yudkowsky would rant about—'Yes, this is progress. No, this is not an MRI for AI. That is still a long way off.'

Connexion / Inscription
LectureRechercheRadarCadre d'investissement

The strongest counterexample Mowshowitz identifies comes from community testing: a Qwen3 neuron in Cammarata's essay, presented as an example of a neuron that 'seems polysemantic but makes sense once you trace it,' was annotated as tracking Colorado-related content. But independent testing found it fires equally strongly on Tornado Python HTTP framework code—not as a weak activation but as a peak of equal intensity. Roon's proposed association chain 'tornado → Apache license → Apaches → American southwest → Colorado' was experimentally falsified. The neuron simply loves the Tornado web framework, with no traceable conceptual link to Colorado. Mowshowitz wrote: 'If this is a typical result, then the whole paper won't hold up that well. The neuron meant as an example of a centrally explained neuron has at least two central explanations. That is a very bad sign.'

Yudkowsky's response provides another dimension of skepticism. He uses a 'screwdriver' analogy: in 2014 the field said 'we built a screwdriver, now we can pry open the panel and look at the gears'; then they discovered 'the screwdriver broke—neurons do more than one thing'; in 2026 they say 'we fixed the screwdriver! Alignment is as good as solved!' Yudkowsky argues that even if enumerating every neuron's meaning were fully successful, this still is not understanding how the model thinks—it tells you what the parts are, not how the whole machine operates. He states explicitly: 'My pessimism was not based on predicting they'd never get this far. It is based on how much work I believe remains from here.'

Neel Nanda adds technical skepticism: he believes the superposition hypothesis still holds because language models contain far more memorized facts than they have neurons, and superposition is an efficient way to pack those facts; weak activations may contain substantial semantic information whose ablation would damage performance; and Cammarata's work does not adequately address the weak-activation problem. Nanda agrees that neuron-based attribution graphs have pragmatic value but does not consider this 'substantial progress towards ambitious reverse engineering.'

Mowshowitz's own conclusion is that this represents a substantial update in the difficulty of neuron interpretability and in how LLMs organize information and thought. It should make attacking interpretability through a neuron-analysis lens much easier than previously thought. But he also warns of Goodhart's Law risk: if interpretability techniques themselves become training targets, models will learn to hide their thinking—this is precisely 'The Most Forbidden Technique.' His final judgment: 'The world is more hopeful now. Let's get to work.'

It should be emphasized that Goodfire's technical paper has not yet been released. Cammarata's claims are currently attributed to the author himself, not independently reproduced results. Mowshowitz himself says 'assuming the work checks out' and treats the eagerness to jump on the hype train itself as a negative signal.

Sources

  • thezvi.substack.com↗
Sujet

神经网络可解释性之争

Goodfire研究员Nick Cammarata发表长文《Neural Networks Are Already Clean》,主张大型语言模型中的神经元并非此前认为的“多义性”(polysemanticity)乱麻,而是具有单一核心概念…

5 étapes3 piècesMis à jour 2026-10-23 16:09

Comment on en est arrivé là

  1. 2026-10-15Goodfire研究员Nick Cammarata发布文章《Neural Networks Are Already Clean》,提出大模型神经元多为单义或简单泛化,推翻主流“叠加假设”。already-clean.goodfire.com ↗
  2. 2026-10-16Latent Space AINews报道此事,引用OpenAI员工Roon称其为“科学史重大时刻”,同时收录Deedy Das等专家的初步回应。latent.space ↗
  3. 2026-10-17LessWrong作者leogao发文反驳,指出仅看顶部激活示例会误导判断,提出需用下游KL散度模拟评分来严格验证神经元解释的有效性。lesswrong.com ↗

Ce qui reste non résolu

  • Cammarata提出的“递归泛化”是否能完全替代“叠加假设”来解释剩余的多义性现象?
  • 基于KL散度的模拟评分方法能否成为评估神经元可解释性的行业标准?
  • 如果神经元确实“干净”,这是否真能显著降低AI欺骗和危险行为的不可见性风险?

Également dans ce fil

  • LessWrong author leogao: Consistent neuron activation patterns do not mean interpretability is solved; simulation KL required2026-10-17
  • Latent Space AINews: Meta Muse Spark Open Release and Goodfire Interpretability Claims2026-10-16
Lire le sujet complet →
Continuer la recherche →