A linear probe tells you whether a property can be read off a layer's activations. Activation patching asks the harder question: does the model actually use it? The two questions sound similar, but their answers can diverge sharply. A probe can succeed on information the model computed as a side effect and never used again. Activation patching cuts through that by making a targeted intervention: swap some activations from one run into another run, observe whether the output changes in the direction you'd predict, and use that to infer causality.
The basic setup
You run the model twice. First, a clean run on an input where the model gives the answer you're interested in. Second, a corrupted run on a modified input where the model gives a different answer. Then you run the corrupted input a third time, but midway through, you overwrite some subset of the activations with their values from the clean run. If patching that subset is enough to flip the output from the corrupted answer back toward the clean answer, you have evidence that those activations are causally important.
This is a causal intervention, not just a correlation. The logic is similar to a controlled experiment: if changing only one thing changes the outcome, that thing is doing something. The trick is designing the clean/corrupted pair so the only relevant difference is the thing you're trying to test.
The metric question
What counts as "the output changed"? The raw probability of the correct token is one option, but it's noisy and hard to compare across prompts. Zhang and Nanda (2024) systematically examined how this choice affects the conclusions you draw, and found that the answer can vary meaningfully depending on which metric you use. In practice, the most common choice for binary tasks is the logit difference: the difference in the model's output logits between the correct and incorrect tokens. For a prompt like "When Mary and John went to the store, John gave a drink to," this is the logit of "Mary" minus the logit of "John." It's a clean, scalar, signed number that measures not just whether the model got it right, but how decisively.
Patching experiments then normalize this into a recovery score: 0 if the patch had no effect, 1 if it fully restored the clean answer. Concretely, if $\Delta_{clean}$ is the logit difference on the clean run and $\Delta_{corr}$ on the corrupted run, then
$$ \text{recovery} = \frac{\Delta_{patch} - \Delta_{corr}}{\Delta_{clean} - \Delta_{corr}} $$A recovery near 1 means patching that component almost fully explains the behavior. Near 0 means it's not where the relevant computation lives.
What it found: factual knowledge in GPT
One of the landmark applications is Meng, Bau, and colleagues' 2022 study (ROME). They wanted to know where GPT stores factual associations, things like which city the Eiffel Tower is in. The clean input asked about Paris; the corrupted input was the same prompt but with the subject token noise-corrupted so the model's representation of "Eiffel Tower" was degraded. Then they patched individual hidden states, one layer and one token position at a time, sweeping over the whole network.
The results showed a clear hotspot: patching MLP activations at middle-layer positions corresponding to the last token of the subject ("Tower" in "Eiffel Tower") restored the model's factual prediction far more than patching anywhere else. This pointed to feed-forward layers in the middle of the network as the primary store of factual associations, a finding Meng et al. then acted on by editing those specific weights to change facts the model knew.
What it found: circuits in GPT-2
A different flavor of patching, called path patching, was introduced by Wang, Variengien, Conmy, Shlegeris, and Steinhardt in their 2022 study of indirect object identification (IOI) in GPT-2 small. The task is simple: given "When Mary and John went to the store, John gave a drink to," predict "Mary." Instead of patching whole hidden states, path patching asks which specific connections between specific attention heads carry the important signal. Rather than sweeping every activation, it traces the computation backward from the output, iteratively narrowing in on the subgraph that matters.
This led to the identification of 26 attention heads across seven functional classes doing the task together. Some heads notice when a name appears twice and suppress the repeated one. Others copy the remaining name to the output position. The result was a fairly complete mechanistic account of a linguistic behavior, not just a localization but an algorithm.
What it doesn't tell you
Patching is strong evidence for causal relevance, but it has its own subtleties. A component can score high on a patch sweep because it carries the relevant signal, or because it's a bottleneck that everything else flows through. These look the same from the outside. The technique also makes a counterfactual claim that depends on your corrupted input: if the corrupted input introduces artifacts of its own (for instance, noise-corrupted token embeddings might not be realistic inputs to the components that follow), the signal you're measuring gets muddier. Heimersheim and Nanda's 2024 guide is a good practical treatment of these traps.
There's also the "backup behavior" problem. Some models have redundant components that are normally quiet but activate when something is patched out, making it look like the patched component wasn't important when it actually was. Whether the component you patched is necessary, or sufficient, or both, matters, and a single patch sweep doesn't tell you all three.
How it connects to probing
In my post on linear probing, I mentioned that probes tell you what's decodable, not what's used. Activation patching is the other half of that sentence. Together, they form a minimal two-step check: probe to find candidate directions or layers that correlate with a property, then patch to see whether those directions are causally upstream of the behavior. Neither step alone is conclusive. Used together, they can make a genuinely strong argument, which is exactly the structure of the Othello-GPT result and the structure we tried to use in our own work on ordinal representations.
If you want to go further, attribution patching, described by Nanda (2023), approximates full activation patching using gradients, which makes it practical to run over every edge in a large model's computation graph in a single backward pass rather than thousands of forward passes.