Activation patching is a causal intervention method for neural networks: run a model on a clean input and a carefully corrupted input, replace selected internal activations in one run with activations from the other, and measure whether the model’s behaviour is restored or disrupted. The method turns an internal representation from something we merely observe into something we can experimentally manipulate.
In mechanistic interpretability, activation patching is used to ask a sharper question than “where is information decodable?” It asks: which activation at which component, layer, token position or feature is causally important for this particular behavioural difference? The method is also called causal tracing or interchange intervention in neighbouring literatures, although those names can refer to broader protocols. This owner keeps the activation-patching job precise and hands broader multi-step tracing to the separate Causal Tracing article.
This guide develops the method from first principles: clean/corrupted contrast design, activation sites, residual streams, attention heads, MLPs, token positions, restoration metrics, normalized patching scores, direct and indirect effects, path patching, patch granularity, feature-level interventions, sparse-autoencoder latents, distribution shift, off-manifold interventions, multiple comparisons, negative controls, causal interpretation, reproducibility and the limits of localization.
Quick Read
- Clean run: model execution where the target behaviour succeeds.
- Corrupted run: matched execution where a controlled change suppresses or alters the target behaviour.
- Patch: replace one selected corrupted activation with its clean counterpart, or vice versa.
- Effect: change in a declared behavioural metric after the intervention.
- Localization: finding sites where patches strongly change behaviour.
- Necessity and sufficiency: patching can provide evidence toward these ideas but rarely establishes either globally from one intervention.
- Path patching: isolates information flow along selected component-to-component routes.
- Feature patching: intervenes on learned representation directions rather than whole activation vectors.
- Off-manifold risk: arbitrary activation edits can create states the model never naturally encounters.
- Control: random, unrelated and magnitude-matched patches help distinguish mechanism from generic perturbation.
1. The central experiment
Suppose the clean prompt makes a language model predict Paris as the capital of France. A corrupted prompt changes France to Poland, or otherwise removes the information needed for the original answer. The clean and corrupted executions now differ in both internal activations and output.
Pick one internal site s. Let a_clean(s) be its clean activation and a_corrupt(s) its corrupted activation. Run the corrupted prompt again, but replace a_corrupt(s) with a_clean(s). If the original behaviour returns strongly, site s carries causally useful information for the contrast under this intervention.
corrupted run + clean activation at site s → patched output
The key word is contrast. Activation patching does not reveal a context-free essence of a neuron. It measures how replacing one internal state changes one declared behavioural difference between carefully chosen executions.
2. Why clean and corrupted inputs must differ in one scientifically meaningful way
If the clean and corrupted prompts differ in ten ways, a successful patch is difficult to interpret. The restored activation can carry any mixture of those differences.
Good corruption design preserves as much as possible while changing the variable the hypothesis requires. For factual recall, change the subject while preserving syntax. For indirect-object identification, permute names while preserving sentence structure. For arithmetic, change one operand while preserving format. For induction-like behaviour, alter the repeated token relationship while keeping unrelated context stable.
The corruption is therefore part of the causal model. A patching map is only as interpretable as the contrast that generated it.
3. Behavioural metric first, heat map second
Before patching anything, define the outcome. Common language-model metrics include target-token logit, logit difference between correct and competing answers, target probability, loss, accuracy or a task-specific scalar.
For a clean score y_c and corrupted score y_x, one normalized restoration score is:
R = (y_patch − y_x) / (y_c − y_x).
R≈0 means the patch restores little of the clean/corrupted gap. R≈1 means it restores roughly all of that gap. Values outside [0,1] are possible: a patch can overshoot or worsen behaviour. Do not clip them merely to make a prettier map.
4. Patch sites form a causal search space
A transformer offers many candidate sites: residual stream before or after a block, attention output, MLP output, individual head output, query/key/value vectors, attention pattern, token position, or a learned feature direction.
Patching every layer × position creates a two-dimensional causal map. Patching every head × position adds another axis. Fine-grained searches can contain tens of thousands of interventions, so discovery and confirmation should be separated.
5. Whole-vector patching tests a state, not one feature
Replacing a residual-stream vector swaps every represented variable at that site. A strong effect localizes useful state but does not tell us which feature inside the vector matters.
This is why whole-vector patching is often a first pass. Once an important layer and position are found, narrower interventions can patch attention-head outputs, MLP outputs, subspaces or sparse-autoencoder features.
6. Clean-to-corrupted and corrupted-to-clean patching answer complementary questions
Clean-to-corrupted restoration asks whether clean information at a site is sufficient, in that corrupted context, to rescue behaviour. Corrupted-to-clean ablation asks whether replacing the clean state with its corrupted counterpart damages behaviour.
Agreement strengthens localization. Disagreement is informative: the clean network may have redundant pathways, or the corrupted state may introduce an active competing signal rather than merely removing information.
7. A patch effect is context-dependent causation
If patching component C changes the answer on prompt family P, the safe claim is that C’s activation contributes causally to the measured behaviour under that intervention and prompt distribution.
It is stronger than correlation. It is weaker than “C is the mechanism.” The same component can perform other jobs, other components can compensate, and the patch can alter several represented variables at once.
8. Token position is often as important as layer
Transformers reuse the same parameters at every position. A head can matter at the subject token and not at punctuation. A residual state can carry the answer only at the final token before prediction.
Layer-only patching averages over this structure and can hide a precise circuit. Position-specific maps are therefore central for sequence mechanisms.
9. Attention-head patching narrows the causal unit
Patch the output of one attention head while leaving other heads unchanged. If behaviour changes strongly, the head contributes useful information at that point.
But an attention head is itself a transformation. The effect can arise from what it attends to, what its value vectors carry, or how its output direction is read downstream. Head localization should therefore lead to mechanism analysis rather than end it.
10. MLP patching can reveal where a variable is transformed or written
MLPs can detect combinations of residual features and write new directions. A strong MLP-output patch can indicate that a layer transforms the relevant state.
As with attention, patching the whole MLP output swaps many variables. Feature-level analysis is needed before assigning one semantic label to the module.
11. Path patching asks not only where information is, but where it travels
Ordinary activation patching can identify sender and receiver components without proving they communicate along the hypothesized route. Path patching or related causal-intervention protocols isolate a selected edge or family of edges.
The idea is to preserve clean information from a sender only as it reaches a selected receiver while keeping alternative routes corrupted. If the target behaviour returns, the sender→receiver path carries causal signal for the contrast.
12. Feature-level patching connects activation patching to sparse autoencoders
The Sparse Autoencoders owner explains how dense activations can be decomposed into sparse candidate features. Instead of replacing the whole vector, we can alter one learned latent coefficient and reconstruct the intervention.
This can increase semantic selectivity, but only if the dictionary has good reconstruction and feature specificity. SAE residual information and non-orthogonal decoder directions mean feature patching is still approximate.
13. Off-manifold interventions are the central causal caveat
A clean activation inserted into a corrupted downstream context may create a hybrid state that never occurs naturally. The later network receives one component from world A and the rest from world B.
This is not automatically invalid—controlled interventions often create unnatural states—but it changes interpretation. Very large or incoherent patches can produce effects because the model is destabilized rather than because a natural mechanism has been isolated.
Matched clean/corrupted prompts, narrow patches, natural-scale feature interventions and off-target metrics reduce this risk.
14. Random and unrelated patches are necessary controls
- Patch the same site from an unrelated prompt.
- Patch a random site with similar activation norm.
- Patch a nearby token position not predicted by the mechanism.
- Patch a feature with similar density but unrelated semantics.
- Shuffle clean activations across examples.
If every perturbation of a site restores or destroys behaviour, the effect may reflect generic sensitivity rather than specific information.
15. Multiple comparisons can manufacture causal-looking hotspots
A patching sweep can test thousands of sites. Selecting the largest effect and then narrating it as a mechanism overfits intervention noise.
Use a discovery set to find candidate sites and a held-out prompt set to confirm them. Report effect distributions, not only the top cells. When statistical inference is required, account for the search procedure.
16. Prompt families test whether the mechanism generalizes
A component that matters for one famous prompt can be a prompt-specific accident. Generate or curate many examples sharing the same abstract task while varying names, wording, order, length and irrelevant context.
A robust causal site should show a stable effect distribution across the family, not merely one dramatic intervention.
17. Activation patching and probing answer different questions
A probe asks whether information is decodable from an activation. A patch asks whether changing the activation changes behaviour.
Information can be decodable and unused. Information can also be used through a nonlinear route that a simple probe misses. The strongest analysis combines accessibility and causation without treating either as the other.
18. Patching and attribution answer different questions
Gradient or attribution methods estimate sensitivity or contribution under a mathematical decomposition. Activation patching performs an explicit counterfactual intervention.
Attribution can scan cheaply and suggest candidate routes. Patching can validate whether manipulating those routes changes the outcome. Agreement is useful; disagreement often reveals nonlinear interaction or attribution assumptions.
19. Patching can miss redundant mechanisms
If two components independently carry enough information for the answer, corrupting one may have little effect because the other compensates. A small patch effect does not prove irrelevance.
Joint interventions, knockout combinations and path-specific tests can reveal redundancy. Causal interpretation in distributed systems is often combinatorial.
20. Patching can exaggerate bottlenecks
A component can be a narrow communication bottleneck through which many upstream variables pass. Patching it has a large effect, but the component may not compute the variable itself.
Separate a bottleneck that transports information from the upstream computation that created it and the downstream computation that consumes it. Causal tracing across several sites is the correct next step when the reader job changes from localization to mechanism.