- Goodfire researcher Nick Cammarata argues that AI neurons look tangled mainly because of weak tools and old architectures, not because networks hide concepts by design.
- His figures are preliminary. Other researchers are divided, and one from Google DeepMind urges restraint about the result.
- Leaders buying or deploying AI should ask vendors exactly how much of a model's inner workings they can explain, and how that is measured.
The tangle may have been in our tools
When a company deploys an AI model, someone eventually asks a plain question: can anyone tell us why it did that? For years the honest answer has leaned on a belief that models are knotted inside, and that nobody can read the knot directly.
A new essay challenges that belief. Nick Cammarata, a researcher at the interpretability lab Goodfire, argues that the knot was mostly a misreading. His title is "Neural Networks Are Already Clean." He calls the piece a working hypothesis, not a settled result.
The idea matters beyond the lab. If models can be read, oversight gets more tractable. The essay argues for that view, and it deserves scrutiny because Cammarata helped build the opposing one.
What the field believed, and why
Mechanistic interpretability tries to trace how a model reaches an answer, not only what it outputs. The basic unit is the neuron, a small component inside the network. Ideally each neuron stands for one recognisable concept.
In practice, many neurons seemed to respond to unrelated things. Researchers call this polysemanticity. The textbook case, in an image model called InceptionV1, was a single neuron that appeared to mix felines with automobiles.
The leading explanation was superposition. A network short of space was thought to squeeze unrelated ideas into shared neurons, because the overlap did little harm. Many teams therefore stopped examining the model directly. They built a separate model, such as a sparse autoencoder, to approximate an untangled version, and analysed that stand-in.
What Cammarata now says went wrong
Cammarata co-wrote an early paper, "Zoom In," that helped establish this view. He now says the cat-and-car neuron is a V-shape detector. Pointed things like flames and fox snouts activate it. Round shapes push it the other way.
He also revisits a second classic example: a circuit where a car feature seemed to be stored among dog neurons. In his account, nothing was hidden. The network relates cars and dogs through a shared pose, and no later part reads out a concealed combination. He adds that the tool used to draw these neurons, feature visualization, was misleading in deeper layers.
The remaining oddities, he argues, fit a pattern he calls recursive generalization. A neuron grows like a branch with side shoots. One neuron that detected "iy" later covered "diy" and then "do it yourself," which does not contain "iy" at all. Cammarata says this difference matters. A grouping that grows by generalization carries information about the network, while a random fold would carry none.
The numbers, and their limits
For a modern language model, Qwen3-32B, Cammarata reports preliminary audits. In his estimate, roughly nine in ten neuron poles are devoted mainly to a single task. Another 7% or so have two related jobs. Only 2-4% are not yet understood.
Do not set the 1.7% figure beside the 90% figure as a like-for-like comparison. They come from different tools, different models and different rubrics. The 1.7% describes a 2023 automated project. The 90% comes from Cammarata's preliminary audit of Qwen3-32B.
Treat the figures with care. Cammarata says the share depends on how weak activations are counted and what counts as "one thing." He also says a single score should be a broad pointer, not a precise target. Some of his checks use relatively small samples.
Goodfire says its checks on far larger models, reaching trillions of parameters and including Kimi K3, look similar. Cammarata adds that sheer size keeps those models far from solved. A companion paper is meant to give the detail.
The work also leans on AI agents that analyse neurons in bulk. Cammarata links this to Rich Sutton's "bitter lesson": simple methods at huge scale tend to beat clever ones.
Researchers are divided
The Decoder reports mixed reactions. OpenAI's Roon calls it a historic moment. Neel Nanda of Google DeepMind urges restraint about the claim. The Decoder also notes that Cammarata proposes no fundamentally new method. He returns to the field's original approach with better tools, larger models and automation.
The Decoder adds that the finding would be good news for AI safety, since there are few ways today to check what a model is doing internally. That is the stake. The lesson here is that "black box" may describe our tools more than the machine. That is a hypothesis, not yet a finding.
What leaders should ask
No one should buy or approve anything on the strength of one essay. But the debate changes which questions are fair to put to AI vendors and internal teams.
First, ask what share of the model's inner workings the vendor can explain, and how that share is measured. A number without a stated threshold means little.
Second, ask whether the explanations come from the model itself or from a simplified copy. Cammarata's critique is aimed at the copy approach.
Third, ask which model sizes the claims cover. Results on small or older models may not carry over.
Fourth, ask who checked the work. An essay from an interpretability lab is evidence, not an audit.
The most useful line in the essay is its simplest: look at the neuron before you assume it is hiding something. Leaders can apply the same habit to vendor claims about AI that cannot be explained.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Goodfire.
