By Roman Aglaganov
Nick Cammarata, a researcher at the AI start-up Goodfire, has published a 26,000-word essay that opens: “I think I have good news about the nature of neural networks.” His claim, in short, is that the parts of AI models long thought hopelessly tangled may be readable after all. If true, it would be a big deal: it is far easier to monitor and control something you can understand.
Why people want to believe it
For years the field has assumed that models cram unrelated ideas into the same neurons, a phenomenon called superposition, so that looking inside reveals a knot. Cammarata helped establish that view. Now, after a year spent looking at neurons “10 hours a day, seven days a week”, he says researchers were “looking at it wrong”. One famous neuron in an old image model fires for both cats and cars. He now argues it is not confused at all: both often show a V shape.
The reaction was ecstatic. One researcher called it possibly “a major moment in scientific history”, MIT Technology Review reported. After a summer of stories about agents escaping their testing environments, people are hungry for good news.
Why to hold back
Other experts are much cooler. Chris Potts at Stanford says it is little more than a hunch, and that examples, however many, are “not a substitute for rigorous evaluation”. Jacob Hilton at the Alignment Research Center said researchers understand less than 0.01% of what happens inside top models, and that neurons are only one piece: the hard part is how they connect. Google DeepMind’s Neel Nanda called it interesting but not a significant update to the big picture. Cammarata himself says “it’s not everything solved”, and much of his evidence comes from a 2014 image model, not today’s chatbots.
What should happen next
Cammarata and his colleagues say technical results on modern models are coming soon. That is the right order: claims first, then data others can test, then celebration. Interpretability may be the most important safety research there is, which is exactly why it cannot afford hype. If labs and lawmakers come to believe that AI can be read like a book, they will relax other safeguards, and if the belief turns out to be wrong, that would be the worst outcome of all.
