A field guide · Oct 2026 How interpretability works What it means to look inside an AI, the tools people use to do it, and why a new Goodfire essay says the inside is far tidier than the field believed. Built on “Neural Networks Are Already Clean” (Nick Cammarata, Goodfire, 15 Oct 2026) plus the main responses to it. Chat for Paddy Interpretability is opening an AI model and working out what each small part does, the way a mechanic learns an engine. For years the field believed the parts were hopelessly tangled, so it built a second AI just to untangle the first. This essay argues most of that tangle came from blurry tools and older model designs, and that today's models can mostly be read directly. If it holds up, checking what an AI is “thinking” gets cheaper and more direct, but serious researchers dispute how far it goes. Tech Chat for engineers Unit of study: individual neurons (positive and negative pole) and their circuits, not sparse-autoencoder features. Claim: canonical polysemanticity (InceptionV1 4e:55, the “cars in superposition” circuit, mixed5b) came from tooling errors, missing residuals/LayerNorm, and recursive generalisation, not superposition-as-packing. Evidence: same-size ConvNeXt last layer ~99% legible vs ~5% for 2014 InceptionV1; 48× layer squeeze broadens neurons rather than packing them; SAE on last layer is 88% near-copies of neurons. LLMs: Qwen3-32B ~90% of neuron poles have one main job. Preliminary; companion paper not yet out. Open risks: threshold-sensitive scores, untested semantics of weak activations, vision-to-language transfer (Nanda's main objection). Agent Chat for agents Primary source: https://already-clean.goodfire.com (plain text: /essay.md). Author Nick Cammarata, contributor Curt Tigges. Published 15 Oct 2026, framed by the author as a working hypothesis. This page paraphrases the essay; all figures trace to it unless marked. Reception figures trace to The Decoder, LessWrong and AI Wiki (see Sources). Interactive demos on this page are illustrations: their values are not measurements. Not covered by the essay: attention heads (mostly out of scope). Pending: technical companion paper with full language-model results. Key disputes to track: whether weak activations carry hidden information; whether 10-neuron squeeze tests say anything about high-dimensional superposition; whether “one main job” is defined too loosely. 1 · Inside a network 2 · The toolkit 3 · Worked example 4 · The messy-neuron puzzle 5 · Evidence 6 · Language models 7 · Faint signals 8 · Why it matters 9 · The other side 1 · The machine A network is a sorting office with millions of sorters A neural network takes something in (pixels, words) and passes it through stacked layers. Each layer holds thousands of neurons: tiny units that add up “votes” from the layer below and pass on a single number, their activation. The strength of each vote is a weight, and training is the process of tuning those weights. Illustration: early layers spot simple things, later layers combine them into bigger ideas. Interpretability is reverse-engineering that office. Nobody designed the sorters' jobs; training invented them. Mechanistic interpretability asks two questions: what does each part respond to, and how do parts wire together into circuits that produce a behaviour. Analogy: a biologist with a microscope. You can't ask a cell what it does. You watch what switches it on, what it connects to, and what breaks when you remove it. 2 · The toolkit Seven ways to question a neuron No single tool proves anything. Confidence comes from several tools pointing at the same answer. Top examplesRun millions of images or sentences through and keep the ones that make it fire hardest. The first, cheapest look. Feature visualisationGenerate a synthetic image that excites the neuron as much as possible (the famous psychedelic pictures). Powerful, but it can take shortcuts. Negative poleWhat pushes it below zero. Tells you the distinction it draws, like “pointed, not round”. Circuits and attributionTrace which earlier neurons feed it and which later ones read it. An attribution graph maps that flow for one input. AblationSwitch it off and see what the model gets wrong. Like pulling a fuse to find out what it powers. SteeringTurn it up artificially and read what the model starts to say. If the “Colorado” neuron is real, the model should start planning hikes. Training checkpointsSave the model at stages of training and watch a neuron grow. Shows whether odd extras sprouted from something nearby. 3 · Worked example The famous “cats and cars” neuron, solved in six steps Neuron 4e:55 in InceptionV1, an image model from 2014 that became the field's lab rat (a model organism), fired for cat faces and car fronts. For years it was the poster child for a polysemantic neuron: one that seems to have several unrelated jobs. Here is how the essay re-reads it. STEP 1 / 6 BackNext step Try it: feed things to a “V detector” Tap an object. Pointed fronts push the neuron up; round things push it below zero, whatever the object is called. ← pushed negative (round)fires (pointed) → Pick something. Illustration only. Values are invented to show the idea, not measured from the model. The lesson: describe what a neuron does, not a list of nouns it likes. The essay suggests a neuron is often better understood as a verb (“separate pointed from round”) than a noun (“cats and cars”). 4 · The big puzzle Why do some neurons look like a jumble? Two explanations compete, and they lead to opposite ways of doing the science. Theory A: superposition (packing) When a network runs short of neurons, it crams unrelated ideas into the same neuron because they rarely clash. Which ideas share a neuron would then tell you almost nothing. Kitchen analogy: you keep the Christmas pudding bowl inside the barbecue because you never use both in the same month. Efficient, but the cupboard tells a stranger nothing. Consequence: study a decoded copy instead. Researchers train a sparse autoencoder (SAE), a second model that rewrites the network's activity into a long dictionary of cleaner “features”. Theory B: recursive generalisation (branching) A neuron learns one pattern, then sprouts offshoots to nearby patterns, and offshoots of those. The far tips look unrelated, but you can trace each back along the branch. Hedge analogy: two leaves at opposite ends look like separate plants until you follow the stems to the same root. Consequence: study the neurons directly. The grouping is a clue to how the network thinks, not noise to strip out. See the difference: squeeze the number of neurons Same 24 things in both panels; similar colours and shapes are similar ideas. Change how many neurons are available. 3 neurons 6 neurons 12 neurons Packing (Theory A) Unrelated things share a neuron. Generalising (Theory B) Fewer neurons → coarser groups, never scrambled. Conceptual illustration, adapted from a figure in the essay. How a branch grows: the “iy” neuron Replay growth Example from the essay. A planetary-geology neuron does the same thing: Moon and Mars crust, then code for a mapping library called Cesium, which arrives by its own route. 4b · Five suspects “Polysemantic” often means “not explained yet” The essay lists five reasons a neuron can look messy. Four can be tested directly. The essay's core argument is that the untestable one was treated as the default when it should be the last resort. Bad tools testableThe microscope is out of focus. Fix the tool and see if the neuron resolves. Bad explanations testableA list of nouns where one shared pattern would do. Bad learning testableThe model was built or trained poorly. Retrain with a better design and compare. Recursive generalisation testableOffshoots from a related idea. Watch it grow across training checkpoints. Superposition hard to testUnrelated packing. Its only symptom is “it doesn't make sense”, which every other suspect also produces. Analogy: a GP who diagnoses every unexplained ache as “stress”. Sometimes right, but it stops the search before the testable causes are ruled out. 5 · The evidence Better tools and better designs cleaned up the “hopeless” layer InceptionV1's final layer was the field's main exhibit for unreadable neurons. Same data, same size, different microscope or design: Share of last-layer neurons easy to interpret ConvNeXt, same size and data99% InceptionV1 retrained, modern settings70% Original model, better tools50% Original model, 2014-era tools5% Source: Cammarata (2026). Author's estimates; others put the 2014 figure near 0%. The difference-maker was ordinary engineering meant to help deep models learn, not to make them readable. A residual connection is an express lane that lets information skip past a layer. LayerNorm keeps each layer's signal at a steady volume, like a sound engineer riding the faders. Sometimes the mess was the microscope Shortcut-taking picturesFeature visualisation follows the easiest path. InceptionV1 saw so many dogs that the generator kept drifting into dogs, making car neurons look like dog neurons. The “cars hidden in dogs” circuit was really cars and dogs sharing a pose. Too little contextShowing only two paragraphs made every long-context neuron look random. The reason was further up the page. Too narrow a datasetOne “mess” neuron turned out to track tell-tale signs inside health spam pages, once the test data included enough of them. Analogy: a satnav asked for “the most scenic route” that always sends you down the motorway, because that's the road it knows best. The answer says more about the satnav than the countryside. The squeeze test If cramming causes packing, crushing a layer should create jumbles. The essay cut a well-understood layer from 480 neurons to 10 (48× fewer) and retrained. The layer got coarser, not scrambled. A sparse autoencoder trained on the squeezed layer could not recover the ten clean detectors. Source: Cammarata (2026). 88% of SAE activity near-copies of single neurons, last layer 9 steps umbrella → bee taught a nonsense label; it reused existing bee neurons ~300× size range tested 92k to 28.6M parameters, readable at every size Source: Cammarata (2026), ConvNeXt experiments. 6 · Language models Chatbot-style models look far tidier than earlier audits found ~90% one main job Qwen3-32B, each neuron side judged separately ~7% two related jobs same audit 2–4% not yet understood preliminary Readable neurons, then and now Qwen3-32B, 2026 audit90% GPT-2 XL, 2026 re-look70% GPT-2 XL, OpenAI 20231.7% Not like-for-like: methods, models and definitions differ. Sources: Cammarata (2026); Bills et al. (2023). Language neurons rarely fire on the obvious noun. Once you know the recurring patterns (motifs), many “random” neurons click into place. Motif 1: the summary on the comma An automated labeller called one neuron a “comma neuron”. It actually tracks Colorado. The model jots a running summary at punctuation, so the commas light up only once the text has become about Colorado. Tap run, then try steering. RunSteer neuron up Model's weekend plan: library, revise for the exam, help gran in the garden. Illustration of an example in the essay, reworded. Shading is not measured data. Motif 2: the same idea in disguise A neuron fired on “pr” in English, plus apparent junk: web-address code strings and Bengali text. Decode them. Tap each card. pr.">approve the proposalTap to read project”.">cHJvamVjdA==Tap to decode pr”: প্রথম = prothom, “first”.">প্রথমTap to transliterate Example strings are ours; the neuron is from the essay. Three more patterns worth knowing Say, or just thinkThe final layer spends much of its effort stopping specific words from being said, so the model can hold an idea without blurting it. The double-negative echoIn SwiGLU layers (a common design where one signal gates another) a negative times a negative comes out positive. For about 4% of neurons this echo dominates the top examples and fools the tools.
