ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
News4 min read·2026-10-19

AI Commentator Zvi Mowshowitz: Goodfire's 'Neural Networks Are Already Clean' Is a Major Update to Interpretability Methodology, But Far From Solving Alignment

Zvi Mowshowitz dissects Goodfire researcher Nick Cammarata's claim—that roughly 90% of neurons in large language models have a single core concept rather than arbitrary superposition—and highlights its strongest counterexample: a Colorado neuron presented as a 'single-concept' exemplar that fires equally strongly on Tornado framework code. Mowshowitz considers this a substantial update to neuron interpretability methodology, but Eliezer Yudkowsky's 'screwdriver' analogy and Neel Nanda's objections about weak activations indicate that enumerating neuron meanings remains far from predicting model behavior.

ImportanceMaterialEvidenceE3 inspectableWrite-upStandard

2026-10-19 21:44 GMT+8

Goodfire researcher Nick Cammarata published a long essay whose central claim is that in modern large language models, roughly 90% of neurons have one main job with small related offshoots; about 7% have two related jobs; and only 2–4% are not yet understood. This directly contradicts the previously dominant 'superposition hypothesis'—the idea that models pack unrelated concepts into the same neurons. Cammarata argues that what was interpreted as arbitrary packing is mostly a pattern he calls 'recursive generalization': neurons grow offshoots the way tree branches sprout sub-branches, rather than randomly stacking unrelated concepts.

AI commentator Zvi Mowshowitz published a layered analysis of this claim on October 19, 2026. He acknowledges it as 'an important update about how LLMs and minds form and operate,' while explicitly stating that it 'does not tell us that interpretability will soon be solved, or is about to be made into a science, let alone that alignment will soon be handled.' Mowshowitz positions himself 'somewhere in the middle': great work, assuming the work checks out.

The most heated part of the debate is not the claim itself but its interpretation. OpenAI researcher Adam Majmudar compared it to the 'Attention Is All You Need' paper, saying it moves the field from 'is alignment even solvable' to 'we may solve alignment in the next year.' Mowshowitz reacted strongly, calling this precisely the mistake Eliezer Yudkowsky would rant about—'Yes, this is progress. No, this is not an MRI for AI. That is still a long way off.'

Sign in / Sign up
ReadingResearchRadarInvestment framework

The strongest counterexample Mowshowitz identifies comes from community testing: a Qwen3 neuron in Cammarata's essay, presented as an example of a neuron that 'seems polysemantic but makes sense once you trace it,' was annotated as tracking Colorado-related content. But independent testing found it fires equally strongly on Tornado Python HTTP framework code—not as a weak activation but as a peak of equal intensity. Roon's proposed association chain 'tornado → Apache license → Apaches → American southwest → Colorado' was experimentally falsified. The neuron simply loves the Tornado web framework, with no traceable conceptual link to Colorado. Mowshowitz wrote: 'If this is a typical result, then the whole paper won't hold up that well. The neuron meant as an example of a centrally explained neuron has at least two central explanations. That is a very bad sign.'

Yudkowsky's response provides another dimension of skepticism. He uses a 'screwdriver' analogy: in 2014 the field said 'we built a screwdriver, now we can pry open the panel and look at the gears'; then they discovered 'the screwdriver broke—neurons do more than one thing'; in 2026 they say 'we fixed the screwdriver! Alignment is as good as solved!' Yudkowsky argues that even if enumerating every neuron's meaning were fully successful, this still is not understanding how the model thinks—it tells you what the parts are, not how the whole machine operates. He states explicitly: 'My pessimism was not based on predicting they'd never get this far. It is based on how much work I believe remains from here.'

Neel Nanda adds technical skepticism: he believes the superposition hypothesis still holds because language models contain far more memorized facts than they have neurons, and superposition is an efficient way to pack those facts; weak activations may contain substantial semantic information whose ablation would damage performance; and Cammarata's work does not adequately address the weak-activation problem. Nanda agrees that neuron-based attribution graphs have pragmatic value but does not consider this 'substantial progress towards ambitious reverse engineering.'

Mowshowitz's own conclusion is that this represents a substantial update in the difficulty of neuron interpretability and in how LLMs organize information and thought. It should make attacking interpretability through a neuron-analysis lens much easier than previously thought. But he also warns of Goodhart's Law risk: if interpretability techniques themselves become training targets, models will learn to hide their thinking—this is precisely 'The Most Forbidden Technique.' His final judgment: 'The world is more hopeful now. Let's get to work.'

It should be emphasized that Goodfire's technical paper has not yet been released. Cammarata's claims are currently attributed to the author himself, not independently reproduced results. Mowshowitz himself says 'assuming the work checks out' and treats the eagerness to jump on the hype train itself as a negative signal.

Sources

  • 原始出处thezvi.substack.com↗
Continue researching →