Mechanistic Interpretability
Techniques for tracing what happens inside a model — logging activations and patching them — to see how and where it forms an output.
To truly understand artificial intelligence, we have to look inside the model rather than just observing what it outputs. This practice is called mechanistic interpretability. It allows us to treat a network's internal activations as data we can record, probe, and even change.
Two core techniques make this possible. First is activation logging. This records the internal states of the model during a run. It lets us see exactly where a specific fact or brand name first appears in the network. A tool called a logit lens can even project these mid-layer states onto the model's vocabulary, letting us watch a word become more and more probable in real time.
The second technique is activation patching. This goes a step further by swapping internal activations between different runs. By doing this, we can establish direct causation, not just correlation, to see what actually drives the model's decisions.
For anyone looking to understand AI visibility, these techniques are incredibly powerful. If you can pinpoint the exact layer where a model decides to associate a specific term with your brand, you can figure out how to strengthen that connection. It links the complex mathematics of attention mechanisms directly to observable behavior.
Mechanistic interpretability is the practice of looking inside a model to understand how it reaches an output, rather than just observing what it produces. It treats the network's internal activations as something you can record, probe and even intervene on.
Two core techniques sit at its heart. Activation logging records the internal states — attention patterns, layer outputs, residual-stream values — during a forward pass, so you can see where in the network a fact or entity (say a brand name) first appears; a "logit lens" projects a mid-layer state onto the vocabulary to watch a token become probable. Activation patching goes further, swapping activations between runs to establish causation, not just correlation.
For AI visibility this is powerful: if you can see at which layer a model "decides" to associate a term with your brand, you can reason about how to strengthen it. It connects the attention mechanism to observable behaviour and underpins tools like our Tree Walker.