How Attribution Graphs Work | Mapping Features, Circuits and Causal Paths Through Neural Networks

Attribution graphs compress a large neural computation into a graph of candidate features and influential relationships. Nodes represent internal variables or components; directed weighted edges estimate how activity at one node contributes to another node or to an output. The graph is useful when it becomes a testable causal hypothesis rather than a decorative explanation.

Modern neural networks are too large to understand by reading every weight. Sparse feature dictionaries, transcoders, attribution methods and circuit-pruning techniques make it possible to extract smaller computational maps. An attribution graph asks which upstream features influenced which downstream features for a particular prompt, behaviour or class of examples.

This owner covers graph nodes and edges, local versus global graphs, feature dictionaries, edge attribution, residual streams, attention and MLP pathways, pruning thresholds, graph completeness, faithfulness, redundancy, causal validation, intervention tests, graph comparison, uncertainty and the boundary between an attributed contribution and a proven mechanism.

Quick Read

• Node: feature, component or output variable.
• Edge: estimated directed contribution from one node to another.
• Local graph: explanation for one input or small prompt family.
• Global graph: recurring circuit structure aggregated across examples.
• Attribution: estimated contribution under a declared decomposition.
• Pruning: remove weak nodes/edges to expose a manageable subgraph.
• Faithfulness: whether the retained graph preserves the behaviour being explained.
• Completeness: how much relevant computation lies outside the graph.
• Causal validation: intervene on predicted nodes/edges and test the output.
• Graph ≠ mechanism: attribution is evidence; intervention earns stronger causal claims.

1. Why a graph?

A transformer contains layers, heads, MLPs and millions of activation dimensions. A raw execution trace is too large for a human to reason about. A graph selects a smaller vocabulary and a smaller set of relationships.

The compression is useful only if the omitted computation is measured. A tiny graph that looks elegant but preserves little of the behaviour is not a faithful explanation.

2. Feature nodes are often better than neuron nodes

Raw neurons can be polysemantic. The Sparse Autoencoders owner explains one route to learned feature nodes that are sparser and often more interpretable.

A graph built from features can therefore be easier to label and reason about. But the graph inherits the dictionary’s reconstruction error, feature splitting, non-uniqueness and missed residual information.

3. Edges estimate influence, not mere co-activation

If feature A and B activate together, correlation does not establish an edge. Both can be driven by feature C.

Attribution methods use weights, local derivatives, linearized transformations or other decompositions to estimate how much A contributes to B in the current computation. The exact meaning of an edge therefore depends on the attribution rule.

4. Residual addition makes contribution accounting possible—and tricky

Transformer residual streams add component outputs. This gives a natural bookkeeping structure: several upstream components contribute vectors to a shared state.

Nonlinear MLPs, attention normalization and feature interactions mean contributions cannot always be assigned uniquely. Attribution graphs often approximate a nonlinear computation locally or use auxiliary sparse models to make it more decomposable.

5. Local graphs explain one execution

For one prompt, only a small subset of features may activate strongly. A local graph follows those active features and their high-attribution edges toward the output.

This can expose a compact story: subject feature → relation feature → answer feature → target logit. The story is a hypothesis until interventions verify the predicted dependencies.

6. Global circuits require recurrence across examples

A local graph can contain prompt-specific names and incidental syntax. To infer a reusable circuit, align graphs across a prompt family and identify recurring functional roles.

The same abstract circuit can use different surface features on different examples. Graph aggregation should preserve functional correspondence rather than demand identical literal nodes everywhere.

7. Edge thresholds create the visible explanation

Large attribution graphs need pruning. Keep only edges above a threshold, top-k incoming edges, or edges required to preserve a declared fraction of output attribution.

The threshold is not cosmetic. It controls graph sparsity and completeness. Report how conclusions change across reasonable thresholds.

8. Faithfulness can be tested by graph ablation

Remove components outside the proposed graph or replace them with baselines. If the target behaviour remains, the graph captures enough computation for that task under the ablation protocol.

If behaviour collapses, the graph omitted important routes. A faithful graph need not reconstruct every model behaviour; it must preserve the behaviour its explanation claims to cover.

9. Completeness is the mirror question

A graph can contain real causal edges and still omit another equally important pathway. Measure residual attribution, output degradation when only the graph is retained, and recovery when omitted pathways are restored.

Completeness is especially difficult when features are redundant or the feature dictionary leaves task-relevant information in reconstruction residuals.

10. Attribution graphs and causal tracing are partners

The Causal Tracing owner explains how interventions test information flow. Attribution graphs can cheaply propose candidate routes; tracing can patch or ablate those routes.

This division is efficient: attribution narrows the search space, intervention raises the evidential standard.

11. Attribution graphs and feature steering

The Feature Steering owner explains continuous interventions on representation directions. A graph predicts which downstream nodes should change when an upstream feature is steered.

If the predicted descendants respond while unrelated branches remain stable, the graph gains causal support. If effects appear elsewhere, the graph may be incomplete or the feature may be entangled.

12. Attention edges need separate routing and content stories

An attention head chooses source positions through query-key compatibility and transports content through value/output transformations. An attribution edge through a head can reflect either routing selection or transmitted content.

Mechanistic graphs should distinguish “why this source was selected” from “what information the source contributed.”

13. MLP edges can represent nonlinear feature transformations

An MLP can combine several upstream features and activate a new downstream feature. A graph that assigns independent additive edges can miss the interaction.

Test whether the downstream node requires combinations of parents through joint interventions. Nonlinear circuits can need hyperedge-like explanations rather than simple pairwise arrows.

14. Graphs are conditional on the input

The active circuit for translating French can differ from the active circuit for arithmetic. Even within one task, different examples can recruit alternate routes.

Do not present a local attribution graph as the permanent wiring diagram of the model. It is a computation-specific projection of a much larger network.

15. Graph comparison can reveal circuit families

Represent each local graph by node types, edge patterns or feature roles and compare graphs across examples. Clusters can reveal recurring circuit families.

This creates a higher-level representation problem: the object being represented is now the computation itself.

16. Uncertainty belongs on edges

An edge weight estimated on one prompt can vary across paraphrases or random seeds. Report distributions or confidence intervals across a prompt family where possible.

A graph with uncertain edges should look scientifically uncertain in the claims made from it. Visual crispness is not evidence.

17. Failure signatures

• Correlation arrows: co-activation is drawn as causation.
• Threshold hiding: graph appearance depends on an undisclosed pruning cutoff.
• Local-to-global leap: one prompt graph is called the model’s circuit.
• Feature ontology leap: SAE node labels are treated as unique ground truth.
• Residual blindness: omitted activation residuals are assumed irrelevant.
• Pairwise-only story: nonlinear multi-feature interactions are ignored.
• Faithfulness omission: the graph is never tested by ablation or retention.
• Causal validation omission: no predicted edge is intervention-tested.

18. Evaluation blueprint

1. Define the behaviour and output metric.
2. Declare the node vocabulary and feature source.
3. Declare the attribution rule and edge meaning.
4. Build graphs on a discovery prompt set.
5. Measure graph sparsity versus retained attribution.
6. Test graph faithfulness by retaining or ablating the subgraph.
7. Measure omitted residual contribution.
8. Validate high-value edges with activation patching or causal tracing.
9. Test feature steering predictions along descendants.
10. Replicate graph motifs on held-out prompts.
11. Report threshold sensitivity and uncertainty.
12. Separate local execution graphs from global circuit claims.

19. Evidence ladder

A graph grows stronger from visually plausible attribution, to replicated edge weights, to faithfulness under graph retention, to intervention-confirmed nodes, to intervention-confirmed edges, to successful prediction of new perturbations and finally to a compact circuit that generalizes across a declared task family.

20. World Return

The point of an attribution graph is not to turn a neural network into a neat diagram. The network is still larger than the picture.

The graph earns scientific value when its compression preserves the behaviour, its arrows predict interventions, its missing computation is measured, and its abstractions survive new examples.

A good attribution graph is not the prettiest story about a computation. It is the smallest map that keeps making correct causal predictions when we push on the system.