Did you know? When you type a sentence into a language model, it does not necessarily see the same words that you see. Before much of the model’s calculation begins, a process called tokenization divides the text into manageable pieces.
A common word may become one token, while an unfamiliar name might break into smaller pieces. Spaces, punctuation and numbers can influence the result. This helps explain why a request containing 100 words is not necessarily 100 tokens long.
Quick answer: input text → tokenizer → token IDs → numerical embeddings → model calculations → output token IDs → decoded text. Tokenization supplies the units; the trained model learns how to process their relationships.
What is a token? The 30-second guide
- Token: a unit from the tokenizer’s chosen vocabulary—often a word, subword fragment, symbol or punctuation sequence.
- Tokenizer: software that transforms text into token units and their IDs, and can decode supported tokens back into text.
- Vocabulary: the set of token units the model is configured to recognise.
- Token ID: a numerical identifier for a token in that vocabulary.
- Embedding: a learned vector of numbers associated with token IDs or other model representations.
- Context window: the bounded token capacity available to a particular model run.
A token is not automatically a word, a letter or a sentence. Its size and meaning depend on the exact tokenizer. That is why different AI applications may report different token counts for the same text.
Why not just split the sentence wherever there is a space?
Consider the sentence “The astronomy student rechecked the telescope.” Splitting on spaces gives a list of words, but languages have punctuation, compound words and writing systems without word-separating spaces. New names and technical terms can also appear at any time.
A fixed vocabulary containing every possible complete word would become unwieldy and still miss many new spellings. Subword tokenization offers a compromise: retain useful common sequences while breaking rarer sequences into smaller pieces.
For instance, the English word “unhappiness” could be represented as one token in some vocabularies or as several subword units in others. Those divisions are hypothetical examples; the exact output has to be checked with the model’s actual tokenizer.
Step 1: Build or select a token vocabulary
During tokenizer development, software examines suitable training text to choose units that balance vocabulary size and coverage. Several common algorithms start with small units and learn which sequences are worth keeping together.
One well-known family is byte-pair encoding (BPE). In a simplified description, it repeatedly combines frequent adjacent units according to a learned merge rule. Other algorithms make different choices about how sequences become tokens.
The resulting vocabulary may keep common pieces intact and represent rarer words with multiple tokens. It is a form of text preprocessing and representation; the tokenizer itself is not independently deciding which claims in a sentence are true.
Step 2: Understand BPE with a fictional word list
Imagine a tiny teaching dataset containing the words cat, cats, car and cars. Start by treating their characters as individual units. If the sequence ca frequently appears, a toy BPE learning procedure might merge c and a into ca.
Later merges could form units like cat or car when appropriate under the trained merge rules. The exact order depends on frequencies, preprocessing and the algorithm’s details. This is a conceptual demonstration, not the actual vocabulary of a named model.
When the tokenizer sees an unfamiliar word, the goal is often to express it using known subword or byte-level pieces rather than requiring a new model vocabulary entry.
Step 3: How WordPiece and Unigram differ
WordPiece and Unigram are other widely used subword approaches. WordPiece differs from basic BPE in how it selects useful merges. Unigram can begin with a larger set of candidate pieces and prune it according to a probabilistic objective.
The Hugging Face Transformers tokenizer overview describes BPE, WordPiece and Unigram, including their distinct training principles. The important beginner’s lesson is that the splitting method affects the units a model receives.
Some implementations also treat whitespace explicitly. Tools such as SentencePiece are useful for writing systems in which spaces do not reliably divide words.
Step 4: Convert each piece to a token ID
After segmentation, the tokenizer maps each unit to a numerical ID in its vocabulary. For a fictional example, suppose “hello” maps to ID 17 and “!” maps to ID 92. Then the two pieces become the sequence [17, 92].
Those numbers are arbitrary identifiers in our illustration, not a measure of happiness or emphasis. Another model could assign completely different IDs to the same units.
The model transforms IDs into learned numerical representations using its embedding operations. This is where token units become part of the network’s calculations. A higher ID does not mean a word is more important.
Step 5: Keep track of position and context
A model processes token sequences, not a bag of unordered words. The difference between “a dog chased a cat” and “a cat chased a dog” depends on sequence and relationships.
Transformer language models use learned operations, including attention and position information, to relate tokens in context. The model can then generate scores for candidate next tokens or solve other tasks depending on its architecture.
If you want to understand the next stage, see How Transformers Use Attention to Read Context.
Step 6: Generate tokens and decode a readable answer
For many generative language models, an output is produced as a sequence of token IDs. A decoding procedure turns these units back into characters and text. A complete-looking word or phrase can therefore emerge from more than one generated token.
That mechanism helps explain why output may appear progressively in a chatbot. The application can display suitable portions as they become available, but a partial word, quotation or sentence should not be treated as a finished answer until the relevant output is complete.
Decoding a plausible sentence is also different from verifying its facts. The model may produce grammatically smooth text while the date, measurement or reference is unsupported.
The useful distinction: Tokenization decides how text is represented. A trained model decides how to process or generate those representations. Neither step is a fact-checker.
Why tokens matter for cost and context limits
Some AI services measure usage in input and output tokens. The number of tokens affects how much material fits into the model’s context and may influence processing time and service cost, depending on the provider.
A long technical name may break into several units. Different scripts, emojis, numerical codes and punctuation can also alter token counts. Avoid assuming that a particular word-to-token ratio works for every language.
If a document exceeds a model’s available context, an application may summarise, retrieve relevant sections or split the work. Those procedures can introduce omissions, so important source details still require verification.
What happens with Chinese, Malay and multilingual text?
Singapore readers may work with English, Chinese, Malay, Tamil and code-switched conversations. Tokenization methods must handle different scripts and language structures. Some writing systems do not separate all words using the same spaces as English.
A multilingual tokenizer may use subwords, bytes or other representations to cover several languages. How efficiently or accurately a particular model handles each variety depends on its vocabulary, training data and downstream model.
Token counts alone do not prove that a language is well supported. A model may tokenize a sentence successfully and still misunderstand a nuanced phrase.
A classroom example: numbers and units
Imagine asking an AI to explain 3x + 5 = 17. The symbols could be split into several units. Their token boundaries do not automatically tell the model that the correct answer is x = 4.
Mathematical correctness requires a valid solution: subtract five from both sides to get 3x = 12, then divide by three to obtain x = 4. The words and symbols must be interpreted together.
For parents and students, this is a reminder that fluent mathematical output deserves the same independent checking as a handwritten solution. A structured tutorial such as the Secondary 1 Mathematics Tutor Clementi learning guide centres on working through genuine concepts rather than trusting a sequence of symbols because it looks familiar.
Try it: a harmless paper tokenizer
Write the words rain, rainy, rainfall and raining. Highlight the common letters rain in each. Invent a small set of reusable pieces: “rain,” “y,” “fall” and “ing.”
Now ask whether those pieces can reconstruct the four words. You have designed a toy subword scheme. It illustrates reuse, not a real BPE or WordPiece training algorithm.
Next write a rare new compound word and see whether your pieces can represent it. This reveals why a practical tokenizer needs a strategy for unfamiliar sequences.
Five mistakes to avoid
- “One token always equals one word.” Tokens may be smaller or larger units.
- “Higher token IDs mean more important words.” IDs are vocabulary identifiers.
- “A tokenizer understands the sentence.” It supplies a representation rather than independently verifying meaning.
- “Every language costs the same number of tokens.” Script, vocabulary and tokenizer design change counts.
- “A model with more context can verify everything inside it.” Context capacity and factual reliability are separate.
Frequently asked questions
What is the difference between a token and a word?
A word is a linguistic unit in ordinary reading. A token is a unit selected by a particular tokenizer’s rules and vocabulary; it may be a full word or part of one.
Do all AI models use BPE?
No. BPE, WordPiece, Unigram, byte-level methods and other schemes exist. Different architectures use different tokenizers.
Why are some unusual names split into many tokens?
Their complete spellings may not be in the tokenizer’s learned vocabulary, so known pieces are used instead.
Can tokenization cause errors?
Tokenization and text normalization choices can affect how information reaches a model, but downstream model behaviour, data and task complexity also determine errors.
Does tokenization make an AI answer truthful?
No. It structures text for numerical processing. Important facts still require suitable evidence and verification.
Your one-minute takeaway
- Language models often process tokens rather than whole words.
- Subword methods help represent both common and unfamiliar text efficiently.
- Token IDs, embeddings, context and factual correctness are distinct concepts.
Trusted reading and the AI learning path
- Hugging Face: Tokenization Algorithms — BPE, WordPiece, Unigram and their differences.
- Hugging Face LLM Course: Unigram Tokenization — why subwords and whitespace handling matter.
- How Transformers Use Attention to Read Context — what happens after tokenization.
- How a Chatbot Turns Your Question Into an Answer — the complete answering process.
- How Embeddings Find Similar Meanings — learned numerical representations.
- How AI Works: Complete Overview — follow the wider series.