The Core Aim of Vocabulary Mastery | Transformer Architecture Vocabulary

THE CORE AIM OF VOCABULARY MASTERY · TRANSFORMER ARCHITECTURE · TOKENS → ATTENTION → REPRESENTATIONS → OUTPUT

Transformer architecture vocabulary sounds technical at first: self-attention, query, key, value, attention head, positional encoding, encoder, decoder, feed-forward network and residual connection. Yet these words describe a sequence of understandable operations. A transformer receives tokens, combines information from positions in a sequence and produces representations that help solve tasks such as translation, classification and text generation.

The core aim of vocabulary mastery for transformer architecture is to explain how information moves through this model family. By the time you finish, you should be able to distinguish an attention score from a probability, an attention head from an entire layer, and an encoder-style model from a decoder-style model without relying on mystifying descriptions of AI.

This guide belongs to the eduKateSG Vocabulary Hub. Explore the related Deep Learning Vocabulary, Neural Networks Vocabulary and Large Language Model Vocabulary for the broader learning path.

Central proposition: Transformer terminology becomes useful when you can trace what each token can attend to, how its representation changes and why the architecture fits the task.

The 60-second transformer vocabulary router

Input: token, token ID, embedding, sequence, position.
Attention: query, key, value, scaled dot-product attention, attention weights.
Structure: attention head, multi-head attention, transformer block, feed-forward network.
Stability: residual connection, layer normalization.
Information flow: encoder, decoder, causal mask, cross-attention.
Output: hidden state, logits, prediction, next-token distribution.

A practical glossary of transformer architecture

Term	Meaning in practice	Question to ask
Token embedding	A learned vector representation provided to model layers.	What representation enters the block?
Query (Q)	A projection used to determine what information a position seeks.	What is this position looking for?
Key (K)	A projection used to measure compatibility with a query.	Which positions match the query?
Value (V)	Information combined according to attention weights.	What information is carried forward?
Attention head	One learned attention pathway using its own projections.	What relationships can this head capture?
Positional information	Signals that help the model represent ordering or relative positions.	How does it distinguish sequence order?
Causal mask	A restriction that prevents positions from attending to future tokens in autoregressive prediction.	Which tokens are visible?
Feed-forward network	Per-position transformations typically following attention.	How is each representation further transformed?

Why transformers need positional information

Attention can compare positions across a sequence, but the attention calculation alone does not automatically provide a complete sense of word order. Positional encoding, position embeddings or alternative position-aware mechanisms supply information about order. Exact implementations differ. The meaningful vocabulary distinction is between what token is present and where that token appears.

Query, key and value: a worked example

Imagine the sentence, The students entered the classroom because it was quiet. A learner may want to understand what it refers to. In an illustrative attention mechanism, the position representing it generates a query and compares it with keys from other positions. Attention weights then determine how values are combined. This is a simplified conceptual example, not a claim that every head always resolves references the same way.

A query is not a web search query, a key is not a password and a value is not a database cell. These are learned numerical projections that work together inside the computation. The practical benefit of this vocabulary is that learners stop treating Q, K and V as three mysterious labels.

Self-attention versus cross-attention

Self-attention relates positions within the same input stream or representation sequence. Cross-attention relates one representation stream to another, as used by many encoder-decoder architectures. The original Transformer paper introduced an encoder-decoder design, while later transformer families also use encoder-only or decoder-only configurations.

Encoder, decoder and causal language modeling

Encoder-style: commonly builds contextual representations from an input sequence, useful for understanding-oriented tasks.
Decoder-style: commonly generates outputs autoregressively, predicting the next token from allowed prior context.
Encoder-decoder: uses an input encoder and an output decoder, useful for sequence-to-sequence transformations.

These are architecture families, not guarantees about all implementations. The key distinction is information access and output objective. A causal mask in an autoregressive decoder helps enforce the rule that future target tokens are unavailable during each prediction.

A layer is not a head

A single transformer layer can include multiple attention heads, output projections, feed-forward transformations, normalization and residual paths. Calling an entire layer an attention head loses the scale of the system. Equally, calling every attention pattern an explanation of a model decision goes too far: attention weights are one part of a learned computation, not automatic proof of causal influence.

Why residual connections matter

A residual connection lets a layer preserve and combine information from earlier representations. Layer normalization helps control activation statistics. Together with other architectural choices, these operations support training stability and information flow through deep networks.

A three-part practice exercise

Draw four tokens as boxes and arrows indicating which tokens can attend to which others.
Label one query, a set of keys and the values combined by attention; describe each role in one sentence.
Compare an encoder representation task with a causal next-token task, identifying where masks and output distributions enter.

Mistakes that lead to confusion

Attention equals explanation: an attention weight is a model-internal signal, not automatically a trustworthy explanation.
Head equals layer: multiple heads typically sit inside an attention module within a block.
Every transformer is identical: masks, positional mechanisms, normalization placement and architecture families vary.
Transformer means language-only: transformer mechanisms are also used in vision, audio and multimodal systems.

Frequently asked questions

What is a transformer in AI?

It is a neural-network architecture family built around attention-based information mixing and additional learned transformations.

What does self-attention do?

It computes relationships among positions in a representation sequence and combines information according to learned scores.

What does QKV mean?

Query, key and value are numerical projections used to calculate attention compatibility and combine information.

What is a transformer block?

It is a repeated computational unit containing attention and other operations, often including feed-forward transformations, normalization and residual connections.

Are transformers only used for ChatGPT-like models?

No. Transformer architectures support language understanding, translation, vision, audio and many other applications.

Research and further reading

The original technical reference is Vaswani and colleagues, Attention Is All You Need (2017). It introduced an attention-based Transformer architecture. The design has evolved substantially, so distinguish the historical paper from the many current architecture variants.

Continue through eduKateSG

The transformer architecture vocabulary standard

Mastery arrives when you can draw a transformer block, describe query-key-value attention, explain why position and masking matter, and distinguish the major encoder and decoder configurations. The goal is not to recite acronyms. It is to make the architecture understandable.