Is LLM Interpretability Proven?

A recently published research essay claims that the difficulty of neural network interpretability has been overstated. In his essay ^[1] on goodfire.com ^[2], Nick Cammarata argues that neuron polysemanticity is caused not by superposition but by ordinary, mostly fixable issues — and that most neurons, even in massive models, perform a single, understandable job. The author himself calls the text a working hypothesis: the technical paper with a full description of the experiments has not been released yet. However, as usual, a bold headline and strong claims have sparked numerous retellings in which the hypothesis has turned into established fact.

Below is a quick look at this hypothesis.

Spot-checking Cammarata’s claims

Summarizing the essay is probably unnecessary. It is written clearly and persuasively, so it is easy to follow the arguments and the preliminary description of the experiments on your own.

First, it is worth noting that Goodfire is a commercial company that offers, among other things, LLM verification as a service, with Silico as its core tool. This does not mean the essay is wrong, but there are far too many examples of business pressure pushing teams to draw conclusions too rigidly or too quickly.

Second, the neuron audit was conducted on Qwen3-32B, with hypotheses generated and checked with the help of AI agents. In other words, LLMs were used to help prove that LLMs are interpretable. It looks amusingly circular, though drawing conclusions here is also premature until the full experimental details are published.

Cammarata’s claim that weak activations are secondary “recursive generalization” rests on an ablation measured by average accuracy: removing the weakest 75% of activations (weak75) reduces accuracy only from 72% to 71%. The author notes that roughly one in ten answers changes in the process, but considers the lost distinctions minor — fine distinctions between classes the network already confuses.

This claim is testable, so I ran an analogous experiment on a small network. My favorite task: classifying whether a sequence is sorted. Let’s see just how secondary the activation tail really is.

On my MLP-512 model, under an analogous weak75 protocol, accuracy on the standard test set actually increased, from 97.5% to 98.3% — which at first glance fully supports Cammarata. However, on boundary cases (sequences where only two adjacent elements are swapped), ablating the tail made accuracy plummet from 93.1% to 72.4%. Since random guessing gives 50% here, the network lost nearly half (48%) of its above-chance competence. Recall on detecting violations was roughly halved, from ~88% to ~44%.

The takeaway: weak activations are precisely where class boundaries live — the boundaries that determine whether a model can tell closely related cases apart. An important caveat: randomly ablating the same mass of activations has almost the same effect. So the boundaries are held by the weak, distributed mass of activations as a whole, of which the tail is the bulk.

In essence, Cammarata tested typical cases rather than boundaries.

Conclusion

His own evaluation criterion (ablating the weakest 75% of activations and measuring average accuracy) can create an illusion that the network remains intact. Cammarata sees that the tail handles subtle cases but dismisses this as a minor detail — yet on boundary cases, removing that “minor detail” collapses the network’s boundary competence.

In other words, Cammarata’s networks may look clean simply because he checks them for cleanliness with a metric in which boundaries carry almost no weight.

In any case, a full assessment of Cammarata’s claims will only be possible once the complete technical report is published. Though, as usual, that hasn’t stopped the hype.

Code and data: git ^[3]