How Causal Tracing Works | Following Information Through a Neural Network

Causal tracing asks how information moves through a neural network from an input difference to an output difference. Instead of stopping when one important activation is found, it links interventions across layers, components, token positions and representation features to build a testable account of where a variable is formed, transported, transformed and finally read out.

In mechanistic interpretability, causal tracing is the bridge between localization and mechanism. Activation patching can show that restoring one hidden state rescues a behaviour. Causal tracing asks what produced that state, which downstream components consume it, whether alternative routes exist, and whether the proposed chain survives controlled counterfactual interventions.

This owner develops causal tracing as an experimental discipline: causal graphs, mediation, clean and corrupted runs, activation restoration, sender/receiver tests, path-specific interventions, residual-stream communication, attention and MLP transformations, feature-level tracing, redundancy, interaction, off-manifold states, causal scrubbing, intervention baselines, generalization, and the evidence needed before calling a circuit an explanation.

Quick Read

• Localization: where an intervention changes behaviour.
• Tracing: how causal influence travels among several sites.
• Sender: a component or feature that supplies relevant information.
• Receiver: a component whose computation depends on that information.
• Mediator: an internal state lying on a causal route between input and output.
• Path intervention: preserves or corrupts selected routes while controlling alternatives.
• Redundancy: several routes can carry overlapping causal information.
• Interaction: a component’s effect can depend on the state of other components.
• Causal graph: a hypothesis about information flow, not merely a visualization.
• Validation: the graph must predict new interventions, not only explain the data used to draw it.

1. From “where?” to “how?”

Suppose activation patching identifies a middle-layer residual state at the subject token as important for factual recall. That result is useful but incomplete. Did an attention head retrieve the subject identity? Did an MLP transform it into a factual feature? Was the information copied to the final token? Which later component converted it into the answer logit?

Causal tracing turns those questions into interventions. The mechanism is not accepted because the story sounds plausible. Each proposed edge must survive a counterfactual test.

2. A causal trace begins with a behavioural contrast

Choose a clean case where the target behaviour occurs and a matched corrupted case where it changes. Define the output metric before inspecting internal activations. The contrast should isolate the variable the mechanism claims to process.

If the contrast changes topic, syntax, length and answer simultaneously, every downstream trace inherits that ambiguity. Causal tracing cannot repair a poorly identified intervention at the input.

3. Nodes in a causal trace are intervention sites

A node can be a whole residual vector, an attention-head output, an MLP output, a token-position state, a subspace or a sparse learned feature. Choosing a node declares the resolution of the causal hypothesis.

Coarse nodes are easier to scan but mix many variables. Fine nodes are more interpretable but require stronger measurement and can miss distributed interactions. A good trace often begins coarse and progressively refines the important region.

4. Edges need interventions too

If node A and node B both patch strongly, it does not follow that A causes B. Both can receive the same upstream signal. A proper edge test asks whether preserving A’s clean information specifically as it reaches B restores the downstream effect while alternative routes remain controlled.

This is the motivation behind path patching and related edge-level intervention methods. A causal circuit is a network of tested dependencies, not a chain drawn between correlated hotspots.

5. Mediation gives a useful conceptual language

Let X be the input manipulation, M an internal mediator and Y the behavioural output. The total effect of X on Y can be decomposed conceptually into influence transmitted through M and influence transmitted through other routes.

Neural networks complicate textbook mediation because mediators are high-dimensional, nonlinear, interacting and deterministic conditional on sampling settings. Still, the language helps: intervention on M asks whether the internal state mediates some of the input-to-output effect.

6. Restoration and knockout are complementary

Restore a clean mediator inside the corrupted run. If behaviour returns, the mediator can carry sufficient causal information in that context. Replace the clean mediator with the corrupted version. If behaviour collapses, the mediator contributes necessary information in that context.

Neither result alone proves global necessity or sufficiency. Redundant pathways, nonlinear compensation and hybrid off-manifold states can create asymmetries.

7. Residual streams are shared causal buses

Transformer components read from and write to a shared residual stream. This makes tracing naturally additive: one component writes a direction, later components can read it, modify it or combine it with other features.

But a residual vector is not one message. It is a superposed state. Whole-vector restoration can rescue many variables at once. Feature- or subspace-level tracing is needed when the mechanism claims one specific represented variable.

8. Attention can route information without creating it

An attention head can have a large causal effect because it transports a feature from one token to another. The feature may have been computed upstream. Distinguish the component that creates information from the component that routes it.

Query/key analysis can explain why the route is selected; value/output analysis can explain what content is transmitted. Both are required for a full attention mechanism.

9. MLPs can transform the causal variable

An MLP can receive a distributed state and write a new feature that is more directly useful downstream. A trace can therefore change representational language across layers: entity identity becomes relation state, relation state becomes answer feature, answer feature becomes logit direction.

Mechanistic explanations should allow variables to transform rather than demanding one unchanged feature travel end-to-end.

10. Feature tracing narrows the causal object

The Sparse Autoencoders owner provides one method for learning candidate features. A causal trace can follow those feature activations across layers or test whether one feature’s intervention changes downstream features and output.

This is stronger than naming features independently. It asks whether they form a causal sequence. Dictionary non-uniqueness and reconstruction residuals remain important limitations.

11. Redundancy breaks simple one-path stories

Two attention heads can copy similar information. Two MLPs can produce overlapping features. Removing one route may show little effect because another route compensates.

Joint knockout tests can reveal synergy or redundancy. If knocking out A or B alone has little effect but knocking out both destroys behaviour, the mechanism is distributed across alternatives.

12. Interaction means causal effects are state-dependent

The effect of patching feature A can depend on whether feature B is active. This is ordinary in nonlinear networks. Therefore causal contribution is not always additive.

Factorial interventions—A, B, A+B, neither—help expose interactions. A mechanistic circuit can contain AND-like, gating and threshold relationships that single-site patching misses.

13. Causal tracing and causal scrubbing

Causal scrubbing tests an abstract computational hypothesis by replacing internal activations with activations from other examples that are equivalent according to the hypothesis. If behaviour is preserved, the abstraction may capture the information the network needs at that node.

This shifts the question from “does this exact activation matter?” to “does only this hypothesized variable matter here?” It is a stronger test of abstraction but depends heavily on the equivalence classes chosen by the researcher.

14. A trace should predict unseen interventions

A circuit discovered on one prompt set should predict what happens on held-out prompts, unseen name substitutions, longer contexts and controlled perturbations. Otherwise the trace can be an elaborate description of the discovery examples.

Mechanistic generalization is the analogue of test-set performance in predictive modelling.

15. Off-manifold tracing can create false paths

Combining a clean upstream state with corrupted downstream context can create hybrid activations. A downstream component may react strongly to the unnatural combination, producing an apparent causal route that is weak in natural execution.

Use matched contrasts, narrow interventions, natural-scale feature edits and controls that measure generic disruption. When possible, compare intervention states with the distribution of naturally observed activations.

16. Causal tracing is not causal discovery from observational data

The network architecture already gives a directed computational graph. The scientific problem is identifying which variables and edges matter for a behaviour. Interventions exploit that known directionality.

Do not import causal-discovery language that implies the graph itself was inferred from passive correlations. What is discovered is the behaviourally relevant subgraph and its functional variables.

17. Direct logit effects are useful but incomplete

A component’s output can be projected through the unembedding to estimate a direct contribution to token logits. This can reveal whether the component writes in an answer-aligned direction.

But many components matter indirectly by changing later computations. A small direct logit effect does not imply small causal importance.

18. Attribution graphs are compressed causal hypotheses

Feature attribution methods can propose a graph of influential upstream-to-downstream relationships. Causal tracing then tests selected edges or subgraphs through intervention.

The later Attribution Graphs owner focuses on how such graphs are constructed and interpreted. This page owns the intervention logic needed to validate them.

19. Failure signatures

• Hotspot chaining: drawing edges between individually important nodes without edge tests.
• Single-prompt mechanism: a circuit exists only on one example.
• Redundancy blindness: small single-node effects are mistaken for irrelevance.
• Bottleneck confusion: a transport site is mistaken for the site that computed the variable.
• Off-manifold causation: hybrid states produce effects absent in natural execution.
• Feature-label leap: semantic names substitute for causal validation.
• Metric shopping: the output score is changed until the trace looks strong.
• Discovery/test leakage: the same examples select and confirm the circuit.

20. A practical tracing protocol

1. Define a behavioural contrast and metric.
2. Build a prompt family with matched corruptions.
3. Run coarse layer×position activation patching.
4. Confirm candidate sites on held-out prompts.
5. Decompose important sites into heads, MLPs or features.
6. Test sender→receiver edges with path-specific interventions.
7. Run knockout and restoration directions.
8. Test combinations for redundancy and interaction.
9. Measure off-target behavioural damage.
10. Predict and test unseen interventions.
11. Document the residual unexplained behaviour.