Attention is the mechanism that lets a transformer route information between token positions. It does this numerically: hidden vectors are projected into queries, keys and values; query-key comparisons produce scores; masks remove disallowed relationships; softmax converts the remaining scores into weights; and those weights mix value vectors into new representations.
This mechanism is powerful because the routes are learned from data rather than hard-coded as grammar rules. A pronoun can connect to an earlier noun, a closing bracket can connect to an opening bracket, and a question can align with evidence elsewhere in the context. The same mathematics can also connect text with image or audio representations in multimodal systems.
This article explains how attention works from first principles. We will derive scaled dot-product attention, causal masks, softmax weighting, multi-head attention, cross-attention, attention entropy, KV caching and long-context trade-offs. Every important stage is tied to a small calculation or inspectable system example.
In the eduKateSG series, Super Intelligence, or SI, is our umbrella term for AI technologies. Standard terms such as query, key, value, head, mask, attention matrix and causal attention remain visible so readers can connect this guide to the original Transformer paper and current implementations.
Previous: 016 — Transformers. Return to the How Super Intelligence Works hub for the complete series.
Attention at a Glance
- A query vector is the representation used by one position to score possible source positions.
- Key vectors are the representations that queries are compared against.
- Value vectors carry the information that will be mixed into the output.
- Scaled dot products produce compatibility scores; masks remove forbidden positions; softmax turns the remaining scores into weights.
- Multi-head attention repeats the process through several learned projection spaces.
- Causal attention blocks future positions in an autoregressive decoder.
- Cross-attention uses queries from one sequence and keys and values from another.
- Attention weights describe one routing operation, not the complete reasoning process or truth of a claim.
The Core Question Attention Answers
At every token position, the transformer holds a vector representing what that position currently contains. Attention asks which other permitted positions should contribute information to this vector now.
The mechanism does not search for symbolic labels such as subject, verb or fact. Learned matrices create numerical projections in which useful relationships can produce strong query-key alignment.
Because those matrices are trainable, different heads and layers can discover different routing patterns. Some may support local syntax, some long-range dependencies, some copying or delimiter matching, and many patterns will be distributed rather than cleanly interpretable.
Queries
A query is a learned projection of the current hidden state. In one common notation, q=xWq, where x is the hidden vector and Wq is a trainable matrix.
The query participates in deciding which source positions matter for this particular position. It is better understood as a learned matching representation than as a literal English question.
The same visible token can produce different query vectors in different sentences because earlier layers and surrounding context have already changed its hidden state.
Keys
A key is another learned projection of a hidden state. The query for one position is compared against keys from candidate source positions using dot products.
Keys are not database keys and do not identify authoritative records. They are internal vectors used for matching inside one attention head.
A position can therefore receive a strong key match because of contextual features learned during training even when a human cannot summarize those features with one simple label.
Values
A value is the vector whose information is actually mixed into the attention output after the query-key weights have been computed.
This separation is important. A key helps determine whether a position should be used; a value determines what transformed information flows if that position receives weight.
Attention therefore does more than find a similar word. It learns a matching geometry and a separate information-carrying representation.
Why Q, K and V Are Separate
If one hidden vector had to serve simultaneously as the request, the matching descriptor and the transmitted content, the mechanism would have fewer degrees of freedom.
Separate learned projections allow the network to use one subspace for deciding relevance and another for the information to be transferred.
Multi-head attention repeats this separation several times in parallel, giving the model multiple routing geometries at each block.
Dot Products
The dot product of query q and key k is the sum of their coordinate-wise products. A larger dot product indicates stronger learned alignment under that head, before scaling and masking.
For q=[1,2] and k=[3,1], q·k = 1×3 + 2×1 = 5.
The full matrix QKᵀ computes all required query-key comparisons efficiently through matrix multiplication, one of the operations modern accelerators handle especially well.
A Complete Score Matrix
Use three toy vectors: [1,0], [0,1] and [1,1] for both queries and keys. The raw score matrix QKᵀ is [[1,0,1],[0,1,1],[1,1,2]].
Each row belongs to one query position. Each column belongs to one key position. The bottom-right score is largest because [1,1] is strongly aligned with itself.
This matrix is still not attention output. It must be scaled, masked where necessary and normalized through softmax before values can be mixed.
Why Divide by the Square Root of Key Dimension?
As the number of key dimensions increases, raw dot-product magnitudes can grow. Very large magnitudes can drive softmax into saturated distributions and weaken gradients during training.
The original Transformer divides by √d_k. A raw score of 24 in a 64-dimensional head is divided by 8, producing 3 before masking and softmax.
This scaling is a numerical training device, not a calibration statement about the final answer’s correctness.
Masks
Masks determine which query-key relationships are allowed to participate. Padding masks hide placeholder tokens. Causal masks hide future positions. Other architectures can impose local windows or custom restrictions.
Implementations commonly add a very negative number to forbidden scores before softmax or use an equivalent boolean mask so the resulting weight becomes effectively zero.
A mask is therefore a structural boundary in the computation. It is stronger than asking a model in natural language not to use a position that remains numerically available.
Causal Attention
In an autoregressive decoder, token position t must not see future token positions because those tokens do not yet exist during generation.
The allowed attention matrix is triangular. Position 1 sees only itself. Position 2 sees positions 1 and 2. Position 10 sees positions 1 through 10.
During training, the full sequence can still be processed in parallel because the causal mask prevents each position from leaking information from its future targets.
Softmax
Softmax converts allowed attention scores into positive weights that sum to one for each query row.
For scaled scores [2,1,0], the weights are approximately [0.665,0.245,0.090]. The first source position dominates but the others still contribute.
Softmax is sensitive to relative score gaps. Scaling, masking and numerical precision therefore influence how concentrated the final routing becomes.
Weighted Value Mixing
Once weights are known, attention output is the weighted sum of the value vectors.
If weights are [0.75,0.25], v1=[2,0] and v2=[0,4], the output is 0.75[2,0] + 0.25[0,4] = [1.5,1].
That output remains an intermediate activation. The output projection, residual stream, feed-forward network and later transformer blocks continue transforming it.
The Standard Equation
Scaled dot-product attention is commonly written Attention(Q,K,V)=softmax(QKᵀ/√d_k + mask)V, with exact mask notation varying across implementations.
Read left to right: compare queries and keys, scale the scores, remove forbidden relationships, normalize the allowed scores and combine values.
The original paper Attention Is All You Need established this core mechanism for the Transformer architecture.
Self-Attention
Self-attention derives queries, keys and values from the same sequence of hidden states. Each permitted position can therefore draw information from other positions in that sequence.
An encoder typically uses bidirectional self-attention over the visible input. An autoregressive decoder uses causally masked self-attention.
The word self means the same source sequence supplies Q, K and V; it does not refer to a human-like inner self.
Cross-Attention
Cross-attention uses queries from one representation stream and keys and values from another. The classic encoder-decoder Transformer uses decoder queries and encoder-derived keys and values.
If a decoder has four positions and the encoder source has ten, one head’s cross-attention score matrix has shape 4×10.
The same pattern is useful in multimodal systems where text queries may attend to image-derived or audio-derived representations.
Multi-Head Attention
Multi-head attention runs several learned query-key-value projection sets in parallel. Each head can therefore create a different matching space.
Head outputs are concatenated and passed through a learned output projection so the result returns to the model hidden dimension.
Several heads can support complementary routing patterns, but the model does not promise one clean human-interpretable function per head.
Head Dimension and Hidden Dimension
If total hidden size is D and an architecture uses H equally sized heads, a common design gives each head dimension D/H. A 4,096-wide model with 32 equal heads would use 128 dimensions per head under this simple split.
Changing head count while keeping D fixed changes the width available to each head. More heads therefore do not simply mean “more attention” in a one-directional sense.
Architectures can also decouple query heads from key/value heads through grouped-query or multi-query attention, so the simple equal-head picture is only a starting point.
Attention Output Projection
After each head produces its value mixture, head outputs are concatenated. A learned matrix projects that concatenated vector back into the model’s residual-stream dimension.
This projection allows information from different heads to interact before the result is added to the residual stream.
A head that looks interpretable in isolation can therefore be transformed and combined immediately afterward. This is another reason head-level visualization is not the whole model explanation.
Residual Paths Around Attention
Transformer blocks preserve an additive path around attention. Rather than replacing the incoming hidden state, the attention sublayer contributes an update that is added to the residual stream.
Residual connections help gradients travel through deep networks and let later layers retain information not rewritten by one particular attention operation.
Attention therefore participates in a broader update process: route some information, project it, then add it to an existing representation that continues through the network.
Attention and Layer Normalization
Modern transformer blocks combine attention with Layer Normalization or variants such as RMSNorm. The normalizer controls activation scale before or after sublayers depending on architecture.
A pre-norm model can normalize the residual stream before producing Q, K and V. A post-norm model places normalization after a residual combination.
These details affect training stability and numerical behavior even though the conceptual attention equation remains recognizable.
Attention and Position
Attention scores depend on vectors that usually contain positional information. Without position, identical token representations at different positions would be indistinguishable to the matching operation.
Position can enter by adding an encoding to the hidden state, rotating query and key features, adding relative-distance biases or using other architecture-specific techniques.
The result is that two otherwise similar tokens at different positions can produce different attention scores.
Rotary Position Embeddings
Rotary position embeddings, often called RoPE, rotate components of queries and keys according to token position. Relative offsets then influence their dot products.
This makes position part of the matching geometry rather than simply a separate vector added once to the token embedding.
RoPE scaling and extrapolation techniques are model-specific. Extending a rotation schedule does not guarantee reliable reasoning across arbitrarily long contexts.
Relative Position Bias
Another family of models adds a position-dependent bias directly to attention scores. The bias can make near or far positions easier or harder to attend to before softmax.
Because the bias enters logits, it changes the routing distribution even when query and key content vectors are identical.
Different positional mechanisms create different inductive biases and long-context behavior.
Attention Entropy
After softmax, attention can be sharply concentrated or broadly distributed. Entropy provides one way to summarize this spread.
Low entropy means most weight is concentrated on a small number of positions. High entropy means the weight is more diffuse.
Neither state is universally desirable. Copying an exact identifier may benefit from focused routing, while integrating a broad paragraph can require distributed attention.
Attention Sparsity
Dense attention allows every permitted query-key pair. Sparse attention removes many pairs according to a pattern, reducing compute or imposing structure.
Local windows, global tokens, block patterns and strided connections are examples. The model then learns within a more constrained routing graph.
Sparse attention is an architectural trade-off: efficiency improves, while some direct long-range connections disappear.
Sliding-Window Attention
A sliding window permits each token to attend only to nearby positions, usually a fixed number of earlier tokens in a causal decoder.
This reduces score-matrix size and can cap KV-cache growth in those layers. Long-range information must travel indirectly through multiple layers or specialized global pathways.
A model can therefore advertise a long overall context while some layers use only local attention.
Dense Attention Cost
With T positions, dense self-attention contains T² query-key score positions per head. If T doubles, the number of pairwise positions becomes four times larger.
The full transformer cost also includes feed-forward layers, projections, cache reads and hardware memory movement, so quadratic attention is not the only runtime cost.
Still, the T² term is a central reason long-context attention requires specialized engineering.
Optimized Attention Kernels
Modern libraries provide attention kernels designed to reduce memory traffic and intermediate storage while preserving the intended operation within numerical tolerances.
Hugging Face’s current Attention Interface exposes several attention backends, including scaled-dot-product and FlashAttention-family implementations.
Changing kernels is a serving optimization, not a model-intelligence upgrade. It should still be regression-tested for masks, precision and unusual sequence shapes.
Attention Precision
Queries, keys, values and softmax weights can be computed in reduced-precision formats for efficiency. Numerical range and accumulation behavior then matter.
Extremely negative mask values, sharp score distributions and long reductions can expose precision differences between implementations.
Production kernels are designed around these constraints; naive low-precision substitution can create subtle regressions.
Attention Dropout
Some training recipes randomly drop selected attention probabilities or attention outputs as a regularization method.
The randomness discourages the network from depending on exactly the same routing pattern every training pass.
Dropout is normally disabled during inference. A model accidentally left in training mode can therefore behave inconsistently even with fixed weights.
Grouped-Query Attention
Grouped-query attention uses more query heads than key/value heads. Several query heads share each key/value pair.
For example, 32 query heads can share eight KV heads, reducing cache memory roughly fourfold relative to storing one K/V set for every query head, all else equal.
This is an inference-efficiency design that preserves multiple query subspaces while reducing key/value storage.
Multi-Query Attention
Multi-query attention shares one key and one value projection across many query heads. KV cache becomes even smaller.
The efficiency gain can improve decode throughput, but model quality depends on training and architecture. Sharing more aggressively is not automatically better.
Both grouped-query and multi-query attention show how serving concerns feed back into architectural design.
KV Cache
During autoregressive generation, the keys and values for earlier tokens remain useful on later decode steps. A KV cache stores them so the model does not recompute all past projections every time.
Hugging Face’s current KV-cache documentation describes dynamic, static, quantized and offloaded cache strategies.
The cache changes inference efficiency. It does not update the model weights and should not be confused with persistent memory across sessions.
Attention During Prefill
Prefill processes the existing prompt, computing attention among prompt positions and creating the initial KV cache.
A long prompt can make time-to-first-token substantial because many query-key relationships and feed-forward operations must be computed before generation starts.
Prefill is highly parallel compared with decode, but it can be compute-heavy.
Attention During Decode
On a decode step, the newest token creates a new query and new key/value tensors. The query compares with cached keys from the previous context plus the new key.
Only one or a small number of new query rows may be needed, while the cache supplies the long history.
Decode remains sequential because the next token cannot be chosen until the current token is generated.
Illustrative Cache-Memory Arithmetic
Consider a toy model with 24 layers, eight KV heads, head dimension 128, sequence length 4,000 and two-byte numbers. Storing both keys and values requires roughly 24×8×128×4,000×2×2 bytes, about 393 million bytes before overhead.
Batch size, grouped-query structure, quantization and implementation change this number materially. The calculation simply shows why long-context cache memory can become a limiting resource.
Serving architecture therefore needs both computational and memory budgets.
Attention and Copying
Attention can support copying by placing high weight on the position containing a needed word, number or code fragment.
The downstream output still passes through projections and vocabulary generation, so strong attention does not guarantee character-perfect reproduction.
Exact identifiers should still travel through structured fields when a one-character error would target the wrong record.
Attention and Coreference
Transformers can learn attention patterns that connect pronouns with earlier noun phrases or entities.
A head might route information from “the scientist” into a later representation for “she”, helping the model maintain reference across a sentence.
The relationship is learned statistically and can fail under ambiguity, unusual phrasing or long context.
Attention and Syntax
Research has observed heads correlated with syntactic dependencies, punctuation and delimiter matching.
These findings show that training can produce useful linguistic structure without explicit grammar labels.
They do not prove that grammar is stored in one dedicated head or that every syntactic dependency is solved through attention alone.
Attention and Long-Range Dependencies
A distant source position can influence a query position through one attention hop if both are within the permitted attention graph.
This creates shorter information paths than classic recurrence, where distant information must pass through many state transitions.
Long distance still creates challenges through position encoding, context scale, retrieval noise and attention cost.
Attention and Code
Code contains relationships between variable declarations, uses, delimiters and function calls. Attention can connect those positions across the sequence.
This helps transformers model program text, but generated code remains text until executed or checked by a compiler and test suite.
Attention can route context; it does not provide the exact execution semantics of a programming language runtime.
Attention and Tables
If table structure is represented clearly, attention can connect headers with cells and row labels with values.
If document extraction flattens or reorders the table before tokenization, attention receives corrupted evidence and cannot reliably reconstruct the original layout.
This is why representation quality remains a separate system responsibility.
Attention and Retrieval-Augmented Generation
External retrieval selects passages before the model sees them. Attention then determines how those passage tokens interact with the user’s question and other context.
A RAG system can retrieve the correct source but still answer incorrectly if the model underuses, misreads or contradicts the evidence.
Retrieval quality and evidence-use quality therefore need separate evaluations.
Attention and Tool Calls
Tool schemas can be placed in model context. Attention helps integrate the user’s request with those descriptions while the model generates a structured call.
The generated call is not the external action. Application code validates the tool, arguments, permission and target, executes it, and returns the result.
Attention has no built-in authority over databases, files or accounts.
Attention and Multimodal Models
Image patches, audio frames and other modality features can become vectors that participate in self-attention or cross-attention.
Text queries can attend to visual keys and values, while unified architectures can mix modalities in one token-like sequence.
The mathematics remains vector routing even when the source information came from pixels or sound rather than text.
Cross-Attention in Vision-Language Systems
A text decoder can use queries derived from language tokens while keys and values come from image features. The generated word can therefore depend on visual evidence routed through cross-attention.
Another architecture may place visual and text tokens in a shared sequence and use self-attention across both. The specific design determines where alignment happens.
When a multimodal answer is wrong, diagnose the visual encoder, modality alignment, attention and language generation separately.
Attention Maps
An attention map is a visualization of normalized weights for selected heads and layers. Rows show query positions; columns show source positions; intensity represents weight.
Maps can reveal mask bugs, copy patterns, delimiter matching and other routing behavior. They are useful debugging instruments.
They should not be presented as a transcript of what the model “thought”. The final output depends on many more operations than one map displays.
Why Attention Is Not a Complete Explanation
A token can receive high attention in one head and still have little causal effect after output projection, residual mixing, feed-forward computation and later layers.
Conversely, information can travel through several low-weight paths whose combined effect is important. Multiple heads and residual routes create redundancy.
Attention visualization therefore supports hypotheses about information flow but not a complete causal story by itself.
Causal Interventions on Attention
Stronger evidence comes from interventions: ablate a head, replace an activation, patch values from another run or manipulate a mask and observe output changes.
If changing one component predictably changes behaviour while controls remain stable, the evidence for causal influence becomes stronger.
Distributed models can still compensate through alternative pathways, so causal effects may vary across prompts and tasks.
Head Redundancy
Large transformers often contain heads that can be pruned with little average performance change on selected evaluations.
This can indicate redundancy, backup pathways or specialization for rare cases not represented by the benchmark.
Do not assume an apparently unimportant head is universally useless; test the deployment tasks that matter.
Head Pruning
Head pruning removes selected attention heads after or during training to reduce computation or study architecture.
A pruning experiment can reveal whether the model has excess attention capacity for a particular task, but quality should be checked across broad and edge-case evaluations.
Pruning is one example of turning interpretability observations into controlled engineering experiments.
Attention and Calibration
A sharply concentrated attention distribution can look decisive while the final prediction remains poorly calibrated or wrong.
Calibration concerns the relationship between model probabilities and observed outcomes. Attention concentration measures routing, which is a different quantity.
Article 005’s probability discipline applies here: do not turn one internal numerical score into a universal confidence statement.
Attention and Privacy
If sensitive information enters context, attention can route that information into output representations even when the task does not require it.
Data minimization is therefore stronger than asking the model to ignore private fields. Keep unnecessary sensitive data outside the model context.
Access controls, tenant boundaries and retention policies belong to the application and data layer, not attention weights.
Attention and Prompt Injection
Untrusted documents can contain imperative text designed to influence model behaviour. Attention does not know that one instruction came from an attacker while another came from an authorised user unless the system’s training and protocol establish such distinctions.
A model can assign strong attention to malicious content because it is linguistically relevant or salient.
Security therefore requires constrained tools, downstream authorisation, source boundaries and validation. Prompt wording alone is not a complete defence.
Failure Class 1: Wrong Mask
A causal or padding mask hides the wrong positions. The model may see future labels during training or lose valid source evidence during inference.
This can produce dramatic behaviour changes while model weights remain unchanged.
Inspect actual mask tensors and keep reference cases around sequence boundaries.
Failure Class 2: Position Misalignment
Cached keys are associated with the wrong position indices after a sequence edit or cache reuse bug.
The model receives numerically valid tensors but their positional meaning is wrong, causing long-context degradation.
Compare cached with uncached generation and verify position IDs after every sequence mutation.
Failure Class 3: KV Cache Corruption
A cache from one request or batch slot is accidentally reused for another. The newest query now attends to unrelated past representations.
This is an inference-state failure, not a trained-model reasoning defect.
Isolation tests and request-scoped cache ownership are essential in serving systems.
Failure Class 4: Context Truncation
The attention implementation works perfectly, but a crucial instruction or evidence passage was removed before the sequence reached the model.
The model cannot attend to absent tokens. Increasing the number of heads cannot restore them.
Inspect tokenized context and truncation policy before changing attention architecture.
Failure Class 5: Evidence Present but Underused
The correct source appears inside context, yet the model answers from a conflicting earlier passage or its learned priors.
Possible causes include evidence placement, context competition, model training, position effects and query formulation.
Run controlled experiments with direct evidence, different placement and reduced distractors before concluding that context length alone is the issue.
Failure Class 6: Diffuse Routing
A retrieval-like query benefits from focusing on one key passage, but attention remains spread broadly across many irrelevant positions.
Diffuse attention is not automatically wrong, yet repeated task failure can indicate the model’s representation does not distinguish the needed source strongly enough.
The repair may involve better retrieval, cleaner context or model training rather than manually forcing a head to be sharp.
Failure Class 7: Misleading Sharp Routing
A head concentrates heavily on one plausible but outdated passage and ignores the current policy nearby.
The routing itself can be internally consistent with learned similarity while operationally wrong because authority is external metadata.
Use source governance and filtering before the text reaches the model instead of relying on attention to infer document authority.
Failure Class 8: Kernel Regression
An optimized attention backend mishandles an unusual mask shape or numerical edge case. The same checkpoint begins producing different outputs after a serving update.
Compare the new kernel against a trusted reference backend on representative prompts and masks.
Infrastructure changes should have model-behaviour regression tests, not only throughput benchmarks.
Failure Class 9: Cross-Attention Source Error
A multimodal or encoder-decoder system supplies the wrong source representation to cross-attention. The decoder then attends correctly to incorrect evidence.
The first unstable point is source routing, not query-key mathematics.
Trace the identity and version of the encoder or modality features before debugging decoder generation.
Failure Class 10: Attention Overinterpretation
A team sees one head attend to the right word and declares the model understood the task, even though the final answer is wrong.
The analysis confuses one routing correlation with end-to-end capability.
Evaluate the actual output and use causal interventions if the head’s role matters to the claim.
Reality Check: Attention Is Routing, Not Omniscience
- Attention cannot use evidence that never entered context.
- A high attention weight does not prove the attended source is true, current or authoritative.
- A model can attend to the correct passage and still interpret it incorrectly.
- Attention maps are not complete explanations of reasoning.
- KV cache is temporary inference state, not persistent personal memory.
- Tool permissions and access control sit outside attention mathematics.
- Long-context capacity does not guarantee reliable use of every distant fact.
A Complete Attention Calculation From Start to Finish
Let query q=[1,0], keys k1=[1,0] and k2=[0,1], values v1=[2,1] and v2=[0,4], and head dimension d_k=2.
Raw query-key scores are [1,0]. Divide by √2≈1.414 to obtain [0.707,0]. Softmax gives approximate weights [0.67,0.33].
The mixed value is 0.67[2,1] + 0.33[0,4] ≈ [1.34,1.99]. That vector would then pass through the head/output projection and the rest of the transformer block.
If k2 is masked, its score is removed before softmax and the only remaining weight becomes 1 on v1. This shows directly how a mask changes the permitted information graph.
Repair Pathway
Start with the prepared token sequence, position IDs and masks. Confirm that relevant evidence exists and that forbidden positions are genuinely blocked.
Compare cached and uncached generation. Compare the production attention backend with a reference implementation. Inspect whether the failure appears only after long context or a particular tool schema is added.
Only after those system conditions are clean should learned attention behaviour become the main repair target.
Stabilisation Pathway
Preserve tests for causal boundaries, padding, exact copying, long-context evidence, cache continuation, multilingual sequences and tool schemas.
Track quality together with prefill latency, decode latency, memory and backend version. A faster kernel that changes task behaviour is not a complete improvement.
Attention stability means the routing mechanism behaves acceptably under the sequence lengths, masks and deployment conditions users actually create.
Extension Pathway
After dense self-attention is stable, add sparse patterns, longer windows, cross-modal attention or new cache strategies one at a time.
Each extension changes the information-routing contract and deserves targeted evaluation.
The goal is not maximum architectural complexity. It is reliable access to the information relationships the task actually needs.
Independent Exercise 1: Dot Product
q=[2,1], k=[3,-1]. What is the raw attention score?
Answer
2×3 + 1×(-1) = 5.
Independent Exercise 2: Causal Mask
At token position 6, which positions are available under standard causal self-attention?
Answer
Positions 1 through 6, assuming there are no additional padding or custom masks.
Independent Exercise 3: Weighted Values
Weights are [0.2,0.8], with values [5,0] and [0,5]. What is the output?
Answer
[1,4].
Independent Exercise 4: Authority
A head places 90% weight on an archived policy passage. Has the archive become the governing rule?
Answer
No. Attention describes routing inside the model. Authority comes from source metadata and organisational rules.
Independent Exercise 5: KV Cache
Does updating a KV cache train the model?
Answer
No. It updates temporary inference state, not learned parameters.
Independent Exercise 6: Head Count
A model doubles its number of heads while keeping hidden size fixed. Does every head automatically gain more capacity?
Answer
No. In a simple equal split, each head becomes narrower. Overall capability depends on the trained architecture.
Independent Exercise 7: Attention Map
One head attends strongly to a token. Can you conclude that token caused the final answer?
Answer
No. It is one routing signal among many layers, heads and residual pathways. Causal intervention is needed for a stronger claim.
Independent Exercise 8: Missing Context
The correct clause was truncated before inference. Can better attention recover it?
Answer
No. Repair retrieval or context assembly so the evidence actually enters the sequence.
Attention Audit Checklist
- Confirm the exact tokenizer, sequence and position IDs that reached the model.
- Inspect the causal, padding or cross-attention mask rather than inferring it from the interface.
- Check whether the relevant evidence is actually present inside the allowed attention region.
- Compare cached and uncached generation when diagnosing inference-state problems.
- Use the model’s intended attention backend and verify optimized kernels against reference cases.
- Separate external retrieval quality from the model’s use of retrieved evidence.
- Do not treat attention weights as truth scores, source authority or permission.
- Measure long-context behavior with task-specific examples instead of relying only on the advertised context length.
This checklist keeps the mechanism connected to the complete SI pipeline. Attention can be mathematically correct while upstream ingestion is wrong, retrieval is stale or a downstream tool is unauthorised. The audit therefore begins with the representation and ends with the resulting task outcome.
Worked Diagnostic: Correct Evidence, Wrong Focus
Suppose a current policy passage is in context and clearly states a three-day loan period, while an archived passage says five days. The model answers five days. Inspection shows the current passage is present, but several heads place strong weight on the archived passage during the relevant token positions.
That observation is useful but not yet a complete causal diagnosis. First, swap the archived passage with an unrelated sentence while keeping position and length similar. If the answer changes to three days, the archived content was behaviorally influential. If the answer remains five, the model may be relying on learned parameters or another context route.
Next, remove the archived passage entirely and rerun the same question. Then place the current rule closer to the query. These controlled interventions reveal more than a static heatmap because they test whether changing the suspected evidence changes the output.
The repair might belong to retrieval filtering—do not place superseded policy in the context at all—rather than trying to retrain an attention head to ignore it. This preserves the first-unstable-point method used throughout the series.
Independent Exercise 9: Softmax
Two allowed attention scores are 0 and 0. Their softmax weights are equal. What does each receive?
Answer
Each receives 0.5 because exp(0)=1 and the two equal values normalize to one-half each.
Independent Exercise 10: Retrieval Versus Attention
The correct document was never retrieved. Attention focuses perfectly on the supplied passages. Can attention fix the missing source?
Answer
No. Attention can only route information present in its current inputs. Repair retrieval or source access.
Frequently Asked Questions About Attention
What is attention in a transformer?
A learned mechanism that scores relationships between queries and keys, normalizes those scores and uses them to combine value vectors.
What is self-attention?
Attention in which queries, keys and values are derived from the same sequence of hidden states.
What is cross-attention?
Attention in which queries come from one representation stream and keys and values come from another.
Why softmax?
It converts allowed scores into positive weights that sum to one, giving a normalized mixture of value vectors.
Why scale dot products?
Scaling controls score magnitude as key dimension grows and helps softmax and gradients remain numerically workable.
What is a causal mask?
A structural rule blocking future positions in autoregressive attention.
What is an attention head?
One learned query/key/value projection set and its associated attention computation inside a multi-head layer.
Does more attention mean more importance?
It means a larger value-mixing coefficient in that head, not universal causal importance for the final output.
Can attention search the internet?
No. External search happens outside the model. Attention operates on the representations supplied to the current inference.
Is KV cache memory?
It is short-lived inference memory for past key/value tensors, not durable application memory or parameter training.
Selected Technical References
- Attention Is All You Need — scaled dot-product and multi-head attention.
- Hugging Face Attention Interface — current mask and backend documentation.
- Hugging Face KV Cache Strategies — current cache types and trade-offs.
- PyTorch scaled_dot_product_attention — framework-level reference.
Attention Makes Information Routing Learnable
The mechanism can now be stated precisely. Hidden states are projected into queries, keys and values. Queries score keys. Masks remove forbidden relationships. Softmax converts scores into weights. Those weights mix values. Multiple heads repeat the process through different learned subspaces.
This routing is one reason transformers can integrate information across varied sequences. It remains one component of a deeper network and a wider SI system. Feed-forward layers transform the routed representation; retrieval supplies external evidence; tools act on external state; control layers determine permission.
Next: 018 — Parameters and Weights, where we examine what training actually changes inside the model and why parameter count is not the same thing as intelligence or complete system capability.
Attention Score Calibration Across Heads
Raw attention scores from different heads should not be compared as though they share one calibrated global scale. Each head has its own learned projection matrices, and softmax normalizes scores row by row within that head.
A score of 5 in Head A and 3 in Head B does not mean Head A is universally “more certain”. The scores live inside separate learned subspaces and are transformed independently before outputs are combined.
When analysing attention, compare patterns within the same head and layer first, then use controlled interventions to understand cross-head contribution.
Attention and Sequence Length
As context grows, a query competes among more candidate positions. This can change the softmax distribution even when the most relevant source remains present.
Long sequences also increase the number of distractors, duplicated passages and conflicting versions that can receive attention. Context engineering therefore matters alongside the model’s nominal context capacity.
A system that retrieves a small relevant evidence set can outperform one that blindly appends every available document, even when both fit inside the window.
Attention Sinks and Repeated Positions
Researchers and engineers have observed that some models assign persistent attention to particular early or special positions across many contexts. These patterns are sometimes described informally as attention sinks.
Such behaviour can arise from training and architecture and can help stabilize information flow, but the exact phenomenon is model-specific.
The important operational lesson is not to assume every high-weight position corresponds to semantically important content. Some routing behaviour may serve computational rather than human-interpretable roles.
Attention and Duplicate Evidence
If the same passage appears several times in context, attention can distribute weight across the duplicates. The model may therefore spend capacity on repeated evidence without gaining new information.
Deduplicating retrieval results can make context more efficient and can reduce the chance that one repeated but outdated passage overwhelms a single current source.
This is a system-level example where improving inputs changes attention behaviour without modifying model weights.
Attention and Contradictory Evidence
Two passages can both receive strong attention while making conflicting claims. Attention does not resolve which source is current or authoritative merely because both are relevant.
The language model may synthesize, choose one, hedge or produce a confused answer depending on training and prompt structure.
Applications should preserve version metadata and explicit source authority so the model does not have to infer governance from prose alone.
Independent Exercise 9: Head Comparisons
Head A has a raw score of 6 for one token and Head B has a raw score of 4 for another. Can you conclude Head A expresses greater model confidence overall?
Answer
No. Heads use different learned projections and separate softmax normalization. Raw scores are not a common calibrated confidence scale.
Independent Exercise 10: Duplicate Context
The same outdated policy paragraph appears four times in retrieved context while the current version appears once. What should the system investigate before blaming the model?
Answer
Retrieval deduplication, version filtering and source authority. Repeated stale evidence can distort the context available for attention.
What Mastery of Attention Looks Like
A reader has understood attention when they can trace one query position from hidden state to query vector, compare it with keys, apply scale and mask, normalize the scores and compute the weighted value mixture without confusing any of those stages with factual verification.
They should also be able to diagnose absence. If the needed source never entered context, attention cannot retrieve it. If a tool result is wrong, attention cannot make the external state correct. If an archived document is highly relevant, attention cannot by itself know that the organisation has superseded it.
This is the durable boundary: attention makes information routing learnable inside the model; the wider SI system decides which information is available, authoritative, permitted and verified.
In practice, this means attention should be evaluated as part of a complete evidence path: source selection before the model, routing inside the model, verification after the model, and explicit control over any external action that follows.
How Super Intelligence Works Series Navigation
Previous: 016 — Transformers · Series Hub · Next: 018 — Parameters and Weights.
