Direct logit attribution measures how strongly an internal component’s residual-stream contribution points toward or away from a selected output token. Because transformer blocks add vectors into a shared residual stream, those writes can be projected through the model’s output readout to estimate their direct contribution to a logit or logit difference.
In mechanistic interpretability, direct logit attribution is used to identify attention heads, MLPs and features whose outputs are already aligned with the final answer direction. It is fast, additive in useful settings and easy to visualize, but it measures direct output alignment rather than the full downstream causal effect of a component.
This Cognitive Art owner explains residual-stream decomposition, unembedding directions, logit differences, component writes, normalization, attention-head and MLP attribution, direct versus indirect effects, cancellation, layerwise accumulation, feature-level attribution, causal validation and the limits of treating a projection as a mechanism.
1. Residual addition creates a bookkeeping system
Transformer components write vectors into a shared residual stream. A component output can therefore be inspected as one additive contribution to the state passed downstream.
2. Project onto an output direction
For target token t with unembedding direction u_t, a simple direct score is related to the dot product between component write c and u_t, with the architecture’s normalization handled consistently.
3. Logit difference is often more diagnostic
Instead of target logit alone, compare target and competitor directions. This measures whether a component writes evidence favouring one answer over another.
4. Attention-head attribution
Project each head output separately. A head can have a strong direct answer-aligned write even when its broader causal role also includes routing information for later computation.
5. MLP attribution
MLP outputs can write directly toward semantic or answer directions. A small direct score does not imply the MLP is unimportant if its main role is to create a feature later components transform.
6. Direct is not total effect
A component can change downstream attention, gating and feature formation. Direct logit attribution ignores those mediated consequences.
7. Cancellation matters
One component can write strongly toward a token while another writes against it. The final state reflects the sum plus nonlinear normalization effects.
8. Normalization complicates exact additivity
Final layer normalization can couple residual contributions. Simple projection is a useful approximation or decomposition under declared conventions, not a universal exact causal accounting.
9. Layerwise accumulation
Summing or plotting component-aligned writes across depth can show where answer evidence is added, removed or reversed.
10. Feature-level attribution
Sparse features or transcoder latents can be projected toward output directions, connecting semantic feature labels with direct output alignment.
11. Attribution graphs
The Attribution Graphs owner extends local contribution estimates into candidate multi-step computational maps.
12. Causal validation
Use Activation Patching or ablation to test whether a high direct score corresponds to behavioural leverage.
13. Failure signatures
Projection-equals-causation, ignoring indirect effects, normalization mistakes, cherry-picking components, confusing target logit with logit difference and overlooking cancellation are common failures.
14. Evaluation blueprint
Predeclare target and competitor; reproduce normalization; decompose residual writes; compute direct projections; compare across prompt families; inspect cancellation; then intervene on high-value components.
World Return
Direct logit attribution is a map of where components write toward the answer. It becomes mechanistic evidence only when that map survives the harder test of what happens when those components are changed.