VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

How Super Intelligence Works | Next-Token Prediction — How Autoregressive Generation Builds Complex Outputs One Token at a Time

Three students work together at a classroom table, reviewing open textbooks and notes.

Next-token prediction is the local generation mechanism behind many autoregressive language models. Given the tokens already in context, the model produces scores for possible next tokens. A decoding rule selects one token, appends it to the sequence and repeats the process.

That sounds simple. Yet repeated next-token prediction can produce essays, code, translations, plans, equations, tool calls and structured data because each prediction is conditioned on a rich representation of the entire available context.

This article explains next-token prediction from logits and softmax to autoregressive factorisation, sequence probability, sampling, temperature, top-k, top-p, greedy decoding, beam search, repetition, factuality, code, reasoning and tool use. It also explains why “predicting the next token” is a mechanism, not a complete description of the SI system.

The transformer architecture introduced in Attention Is All You Need enabled efficient sequence modelling with attention. The GPT-3 paper Language Models are Few-Shot Learners demonstrated how large autoregressive models can apply this mechanism across many tasks from context.

Previous: 031 — Inference. Here we zoom into the token-by-token generation loop.


The Hidden Transition: Local Prediction Builds Global Structure

Suppose the prompt ends with “The capital of France is”. The model assigns scores to thousands of possible next tokens. “Paris” may receive a high score because the context strongly constrains the continuation.

After “Paris” is selected, the context becomes “The capital of France is Paris”. The next prediction might favour punctuation. After the punctuation, the model can begin a new sentence.

A long response is therefore built from a chain of local decisions. The global coherence emerges because each local decision is conditioned on the representations produced from the growing sequence.

What Is a Token?

A token is a unit from the model’s vocabulary. It can be a whole word, subword, punctuation mark, number fragment or special symbol.

The model predicts token IDs rather than English words directly. The tokenizer later decodes the selected IDs back into visible text.

Exact token boundaries depend on the model’s tokenizer, so worked examples should not invent universal token splits.

Logits

The model’s output layer produces one raw score, called a logit, for each vocabulary token.

If the vocabulary contains 50,000 tokens, the next-token output can contain 50,000 logits. Higher logits indicate stronger preference under the model before normalisation.

Softmax

Softmax converts logits into a probability distribution whose values sum to one.

A token with much larger logit than the others receives a large share of probability. Similar logits produce a flatter distribution.

These probabilities are conditional on the current context and model. They are not direct probabilities that an external claim is true.

A Tiny Softmax Example

Imagine three candidate tokens with illustrative logits: Paris = 3, Lyon = 1, London = 0. After exponentiation and normalisation, Paris receives most probability mass.

The exact probabilities are less important than the relationship: a difference in logits becomes a difference in next-token probability.

Real vocabularies contain far more candidates, and every decode step recomputes the distribution for the new context.

Autoregressive Factorisation

An autoregressive model represents the probability of a sequence as a product of conditional next-token probabilities.

For tokens x1, x2, x3, the model estimates P(x1) × P(x2 | x1) × P(x3 | x1,x2), continuing through the sequence.

This mathematical structure explains why generation is sequential: each token changes the condition for the next prediction.

Sequence Probability

A complete sequence probability is the product of its token probabilities under the model. Long sequences naturally receive tiny raw probabilities because many values below one are multiplied.

Log probabilities are therefore used in practice because sums are numerically easier to handle than products of many tiny values.

High Sequence Probability Is Not Truth

A fluent false statement can be highly probable because the wording is common. A rare but true technical identifier can have lower token probability.

Sequence likelihood measures fit to the model distribution, not factual truth in the world.

Teacher Forcing During Training

During autoregressive training, the model predicts the next token while receiving the actual earlier tokens from the dataset.

At runtime, the model eventually receives its own generated tokens. An early mistake can therefore alter later context and compound.

Exposure During Generation

Suppose the model incorrectly generates “Lyon” after “The capital of France is”. The next token prediction now conditions on that error.

The model may continue coherently with the false premise. This is one reason long answers can become internally consistent while externally wrong.

Context Changes the Distribution

Add “According to the supplied 2026 timetable” to a question and the next-token distribution can change because the model now has a source boundary.

Add an irrelevant conflicting paragraph and the distribution can change again.

Next-token prediction is therefore inseparable from context engineering.

Attention Creates Contextual Representations

The transformer does not predict from a simple bag of previous words. Attention and feed-forward layers transform the whole available sequence into context-sensitive representations.

The logit for the next token is produced from the final representation at the current position after many layers of computation.

This is why “just predicts the next word” can be misleading if it implies only local word-frequency matching.

Greedy Decoding

Greedy decoding chooses the highest-probability token at every step.

It is simple and repeatable, but the locally highest token can lead to a worse global sequence than another early choice.

Sampling

Sampling draws from the model distribution. A token with probability 0.20 can sometimes be chosen over one with probability 0.60.

This introduces diversity and can help creative tasks, brainstorming and varied dialogue.

Temperature

Temperature rescales logits before softmax. Lower temperature sharpens the distribution; higher temperature flattens it.

Hugging Face’s generation strategies documentation explains temperature, top-k and top-p as common controls for text generation.

Changing temperature changes selection behaviour, not source evidence.

Top-k Sampling

Top-k keeps only the k highest-probability candidates and samples among them.

This prevents very low-probability tokens from being selected while preserving diversity within the top set.

Top-p Sampling

Top-p, or nucleus sampling, keeps the smallest set of candidates whose cumulative probability reaches a chosen threshold.

The size of the candidate set changes dynamically depending on how concentrated the distribution is.

A Worked Sampling Example

Illustrative probabilities: A = 0.50, B = 0.25, C = 0.15, D = 0.07, E = 0.03.

Top-k with k=3 keeps A, B and C. Top-p with p=0.80 also keeps A, B and C because their cumulative probability is 0.90 after C is included. Different distributions can produce different top-p set sizes.

Beam Search

Beam search maintains several candidate continuations and scores them as sequences rather than committing to one token immediately.

It is useful in tasks where globally likely sequences matter, such as some translation settings. It is less common for open-ended conversational generation where diversity and latency matter.

Length Bias

Because sequence log probability accumulates negative values, longer sequences can be penalised. Beam-search systems often use length normalisation or penalties.

The decoding objective therefore shapes output length independently of the model weights.

Repetition

Autoregressive generation can fall into loops where a phrase makes itself increasingly likely because it has just appeared repeatedly.

Repetition penalties, n-gram blocking or better training can reduce this behaviour.

Special Tokens

Models use special tokens for sequence boundaries, chat roles, padding or other structural functions.

A generation can stop when an end-of-sequence token is produced. Chat templates can also use role tokens that shape the next-token distribution.

Next-Token Prediction and Classification

Classification can be reframed as token prediction: instruct the model to output one of four category labels.

The next-token distribution over those labels becomes a classifier, although multi-token labels and formatting require care.

Next-Token Prediction and Extraction

Structured extraction can be generated one field at a time. After producing a JSON key, the model predicts the value tokens conditioned on the source text and prior fields.

A schema validator can enforce syntax after generation.

Next-Token Prediction and Code

Code generation works because source code is a token sequence with strong syntactic and semantic patterns.

The model predicts identifiers, operators, keywords and literals conditioned on the surrounding program and instruction.

Execution and tests remain separate evidence that the generated code works.

Next-Token Prediction and Mathematics

Mathematical expressions can also be represented as tokens. The model can learn symbolic patterns and generate multi-step solutions.

Exact arithmetic can still fail because the generation objective rewards likely sequences, not deterministic numerical execution.

Calculator or code tools can verify computation.

Next-Token Prediction and Tool Calls

A tool call can be represented as structured tokens: tool name, arguments and delimiters.

The model predicts that structure like any other output sequence. The application then validates and executes it.

The generated call is an intention, not the executed state.

Why Tool Training Matters

A base model may know what a calculator is but not the exact schema expected by an application.

Instruction examples or post-training can teach when to call a tool and how to format arguments.

Next-Token Prediction and Reasoning

A model can generate intermediate steps, subgoals and candidate deductions token by token.

The fact that the mechanism is token prediction does not settle whether the resulting computation should be called reasoning. The useful scientific question is whether the behaviour solves problems, generalises and can be verified.

The next Machine Reasoning article will examine that distinction directly.

Prediction Can Use Long-Range Context

The next token may depend on information thousands of tokens earlier. Attention, positional representation and deep layers make long-range dependencies possible.

Technical context capacity does not guarantee perfect retrieval of distant information, so long-context evaluation remains necessary.

A Worked Long-Range Example

At the beginning of a document, define project codename “Orchid”. Thousands of tokens later, ask for the codename.

Correct next-token prediction requires the model to preserve or recover that earlier relation. Distractors and competing codenames make the task harder.

Token Probability and Calibration

A token probability can be mathematically well defined while still being a poor confidence estimate for the truth of a claim.

Calibration must be evaluated against the outcome whose probability is being interpreted.

Self-Reported Confidence

A model can generate “I am 90% sure” as text. That sentence is itself a token sequence and should not automatically be treated as calibrated probability.

If the application needs calibrated uncertainty, build and evaluate a specific estimation method.

Hallucination

When the context does not contain the needed evidence, the model still has to assign next-token probabilities. It can continue with plausible text rather than stop.

Post-training can teach the model to express uncertainty, and retrieval can supply missing evidence, but the generative mechanism itself always produces a distribution.

Grounding Changes Prediction

Supply an authoritative passage and the distribution can shift toward tokens supported by that passage.

Grounding therefore works by changing context and constraining what continuations become likely, not by replacing the token-prediction mechanism.

Structured Decoding

Grammar-constrained decoders can remove tokens that would make the output syntactically invalid.

This changes the candidate set before selection, guaranteeing form while leaving semantic correctness to later checks.

Stop Sequences

Applications can stop generation when a delimiter appears, such as the end of a JSON object or a tool-call block.

A poorly chosen stop sequence can cut off legitimate content or fail to stop malformed output.

Maximum Length

A maximum output-token limit prevents unbounded generation. If reached before the task is finished, the answer is truncated.

The final application should not treat token-limit termination as successful completion automatically.

Why Different Prompts Change Outputs

A prompt changes the conditional distribution. Even a small wording change can shift attention patterns and logits.

Prompt sensitivity is therefore expected in a probabilistic conditional generator, though strong instruction-following training can improve robustness.

Why the Same Prompt Can Change Outputs

Sampling introduces randomness. Model updates, hidden system context and runtime differences can also alter outputs.

Repeatability should be measured under controlled conditions.

Sequence-Level Objectives After Pretraining

Post-training can optimise preferences over entire responses rather than next-token likelihood alone.

The model still generates token by token at runtime, but its learned distribution has been reshaped by additional objectives.

A Next-Token Failure Map

Level 1: wrong context. Level 2: tokenisation or truncation error. Level 3: logits reflect weak model knowledge. Level 4: decoding setting unsuitable. Level 5: early wrong token compounds. Level 6: repetition loop. Level 7: structured output invalid. Level 8: fluent unsupported claim. Level 9: application mistakes generated text for executed action.

This eduKateSG map separates local generation failures from broader system failures.

Worked Diagnosis: Factual Error With Low Temperature

Lowering temperature makes the same wrong answer repeat more consistently. The repair is evidence or model capability, not additional determinism.

Worked Diagnosis: Creative Output Is Dull

Greedy decoding or very low temperature may be collapsing diversity. Increase sampling diversity within acceptable quality bounds.

Worked Diagnosis: JSON Breaks

Use grammar constraints, schemas or validation rather than hoping prompt wording alone produces perfect syntax.

Worked Diagnosis: Tool Call Is Correct but Action Fails

The model generated appropriate arguments. The failure lies in execution, permission or external state, not next-token prediction.

Independent Exercise 1: Local Versus Global

Why can a model generate a coherent paragraph if it only selects one token at a time?

Answer

Each token distribution is conditioned on representations of the entire available sequence, so local choices incorporate global context.

Independent Exercise 2: Probability

A token has probability 0.8. Does that mean the claim it belongs to is 80% likely to be true?

Answer

No. It is a conditional next-token probability under the model, not a calibrated truth probability for the whole claim.

Independent Exercise 3: Temperature

Does lowering temperature add factual evidence?

Answer

No. It makes the distribution more concentrated. It can make correct or incorrect outputs more repeatable.

Independent Exercise 4: Tool Call

The model outputs a function call. Has the function run?

Answer

No. The application must validate and execute the call, then return the result.

Independent Exercise 5: Stop Limit

Generation hits its maximum token limit halfway through a report. Is the task complete?

Answer

No. Token generation stopped, but the requested deliverable is incomplete.

The Distribution Is Recomputed at Every Step

After each selected token is appended, the transformer processes the new current position and produces a new logit vector. The next-token distribution therefore evolves continuously during generation.

A token that was unlikely three steps earlier can become almost certain after a new phrase creates a strong grammatical or factual constraint.

This dynamic conditioning is one reason fixed word-frequency tables are a poor mental model for modern autoregressive generation.

A Four-Step Worked Generation

Prompt: “Two plus two equals”. Step 1: the distribution heavily favours token “ four”. Suppose it is selected. Step 2: punctuation becomes likely. Step 3: if the model produces a full stop, a new sentence opener becomes likely. Step 4: subsequent predictions now condition on the complete statement.

The visible answer may look like one act, but the decoder made several sequential decisions. Each decision altered what the model saw next.

Conditional Probability and Prompt Dependence

P(token | context) changes when the context changes. “The bank is beside the” can favour “river” under one story and “road” under another.

The same vocabulary token has no fixed probability independent of context. Contextual probability is the essential idea.

Conditional Probability and Role Instructions

A system message saying “Answer with one word” shifts the distribution toward short completions. A user instruction to explain step by step shifts it toward discourse markers, equations and connective phrases.

Instruction following is therefore visible at runtime as a reshaping of next-token probabilities by the instruction context.

Autoregressive Generation as a State Machine

The growing token sequence acts like state. Every selected token becomes part of the condition for future output.

This state is not a database record or durable memory; it is the current generated context. Nevertheless, it can carry commitments, variable names, narrative facts and earlier calculations across a response.

Commitment Effects

Once a model states “The answer is 42”, later tokens often explain 42 rather than reconsider it. The early commitment changes the probability landscape.

Reasoning systems can reduce premature commitment by delaying final answers, generating alternatives or verifying intermediate results before continuing.

Error Cascades

An early incorrect date can propagate through later calculations. If the model says the exam is Monday instead of Tuesday, its subsequent schedule can be internally coherent but wrong.

Verification should therefore occur near critical intermediate states, not only after a long final answer.

Branching Generation

Instead of committing to one token path, an application can branch into several candidate continuations. Beam search, tree search and multi-sample reasoning all exploit this possibility.

The model supplies conditional probabilities; the runtime algorithm decides how many branches to explore.

Sequence Search Versus Token Selection

Greedy decoding optimises one local choice at a time. Beam search approximates a wider sequence-level search. Sampling explores the distribution stochastically.

These algorithms use the same model logits but can produce very different final sequences.

Log Probability

Because sequence probabilities become extremely small, systems add log probabilities. Multiplication becomes addition: log P(sequence) equals the sum of conditional token log probabilities.

Log probabilities are useful for ranking candidate sequences, scoring completions and analysing model confidence at the token level.

Average Log Probability

Longer sequences accumulate more negative log probability. Dividing by token count produces an average that can make different lengths more comparable.

Even then, a high average log probability does not establish semantic correctness.

Perplexity

Perplexity is derived from average negative log likelihood and measures how surprised the model is by a sequence under an evaluation distribution.

Lower perplexity means the model assigns more probability to the observed text. Different tokenizers and datasets make cross-model comparisons difficult.

Perplexity Is Not a Task Score

A model can have strong general-text perplexity and still fail a specialist code task. Another model can have worse general perplexity but better instruction following after post-training.

Perplexity evaluates predictive fit, not every downstream capability.

Tokenizer Effects on Prediction

If a rare term splits into six tokens, generating it requires six successful conditional decisions. A common term encoded as one token requires one.

This can affect spelling reliability, context consumption and sequence probability.

Numbers and Tokenisation

Numbers may split in ways that do not align with decimal place or arithmetic structure. A model can therefore learn textual number patterns without having a built-in exact arithmetic engine.

This is one reason calculator tools remain valuable for precise numerical work.

Names and Rare Identifiers

Unusual names, serial numbers and URLs can have low probability or fragmented tokenisation because they are rare in training.

Copy mechanisms, retrieval and constrained output can help preserve exact identifiers.

Copying From Context

Transformers can generate tokens that reproduce strings from the prompt because attention makes those sequences salient.

Copying is still prediction: the context raises the probabilities of tokens that continue the seen pattern.

Quotation Risk

A model can produce text that looks like a quotation even when it is paraphrasing or inventing. Quotation marks are merely tokens unless the application checks them against a source.

Source-dependent quotes require retrieval and comparison.

Next-Token Prediction in Translation

The model generates the target-language translation token by token while conditioning on the source sentence and previous translated tokens.

Early translation choices constrain later grammar and word agreement. Beam search has historically been useful because several partial translations can remain plausible.

Next-Token Prediction in Summarisation

The model selects which ideas to preserve by generating a compressed sequence conditioned on the source and summary instruction.

A fluent summary can omit a critical exception because next-token likelihood is not identical to coverage. Evaluation needs source-based completeness criteria.

Next-Token Prediction in Dialogue

Dialogue adds role history, conversational goals and interpersonal context. The model predicts a helpful continuation of the entire conversation under post-training.

The apparent conversational intelligence emerges from conditional generation across many turns plus application-level memory and tools where available.

Next-Token Prediction in Planning

A plan is also a sequence. The model can generate “first”, “next”, dependencies and contingencies token by token.

A written plan is not automatically executable. State checks, resource availability and authority belong to the larger system.

Next-Token Prediction in Proofs

A mathematical proof is a structured symbolic sequence. The model can generate statements and transformations based on learned proof patterns.

Formal proof assistants can verify whether each step follows from accepted rules. This creates a strong hybrid between probabilistic generation and deterministic checking.

Next-Token Prediction in SQL

A natural-language query can become SELECT, fields, WHERE clauses and identifiers through token generation.

The database parser can validate syntax, while read-only permissions and schema checks constrain execution.

Next-Token Prediction in APIs

Function calls are often serialised as structured tokens representing function name and arguments.

A model trained on tool examples learns probability patterns that favour the correct schema. The application still performs authentication and execution.

Next-Token Prediction and Agents

An agent loop may ask the model for the next action rather than the next user-facing sentence. The output token sequence names a tool, plan step or stopping decision.

The same generative mechanism can therefore control external workflow when connected to an action layer.

Why Agent Safety Cannot Be Only a Language Problem

A bad next-token choice in prose is editable. A bad token sequence that invokes a destructive tool can change real state.

Tool whitelists, permission checks, sandboxes and approvals must constrain the consequences of generated action tokens.

Token-Level Entropy

Entropy measures how spread out the next-token distribution is. Low entropy means one or a few tokens dominate; high entropy means many candidates have substantial probability.

Entropy can signal local uncertainty, but low entropy does not guarantee truth. The model can be confidently wrong.

Entropy Across a Response

Some positions are highly constrained: closing a bracket, completing a common phrase, emitting punctuation. Others are open-ended.

A response therefore has a changing uncertainty profile rather than one fixed confidence level.

Margin Between Top Tokens

The difference between the top two token probabilities can indicate how decisive a local prediction is.

A large margin can still occur on an unsupported factual claim, so margins are diagnostic signals rather than truth scores.

Token Alternatives Reveal Ambiguity

When several tokens have similar probability, the model may be uncertain about wording, grammar or content.

Applications can sometimes expose alternatives or ask clarification when ambiguity matters.

Sampling Seeds

A pseudorandom seed controls stochastic token draws under fixed conditions. Reusing the seed can improve reproducibility.

If logits change because the prompt or model changed, the same seed can still yield different text.

Deterministic Decoding and Repeatability

Greedy decoding removes sampling randomness, making output more repeatable under a fixed stack.

Numerical kernels, model versions and hidden context can still affect results. Determinism is a system property, not just a temperature setting.

Repetition Penalties

A repetition penalty modifies scores for tokens that have already appeared, discouraging loops.

Too strong a penalty can make valid repeated terminology or code identifiers less likely. Decoding controls need task-specific tuning.

Frequency and Presence Penalties

Some systems penalise tokens based on how often they have appeared or whether they have appeared at all.

These are heuristic decoding controls that reshape probability, not training-time knowledge.

Constrained Vocabularies

A classifier can restrict output to a small set of label tokens. A form generator can restrict output to values allowed by a schema.

Constrained decoding converts a broad language model into a more exact component without retraining all weights.

Lexical Constraints

Translation or generation systems can require specific terms to appear. The decoder searches for sequences satisfying those constraints.

This is useful for terminology control but can reduce fluency if the constraint conflicts with natural context.

Bad Words and Token Bans

Applications can ban specific token sequences. Because harmful concepts can be paraphrased, lexical bans are not a complete safety solution.

Post-training and system policy need broader semantic handling.

Stop-Token Injection

If untrusted content contains text resembling a stop delimiter, naive parsing can cut output or tool data prematurely.

Robust structured protocols distinguish data from control tokens and validate boundaries outside the model.

Prompt Boundary Tokens

Chat templates use control tokens to mark system, user and assistant messages. These tokens help the model learn role structure.

If messages are serialised incorrectly, the model may treat user data as instructions or confuse speaker roles.

Next-Token Prediction and Prompt Injection

An injected instruction works by changing context so different action or answer tokens become more probable.

Security must therefore reduce untrusted instructions’ authority through system architecture, not merely hope the model assigns them low probability.

Training Distribution and Runtime Distribution

The model predicts well when runtime patterns resemble what its training prepared it for. Unusual formats, languages or tasks can produce unstable token distributions.

Distribution shift can therefore appear as odd completions, misplaced confidence or formatting failures.

Out-of-Distribution Generation

A model can still produce a fluent sequence for an unfamiliar domain because the decoder always has a distribution.

Fluency is therefore especially weak evidence of competence outside evaluated domains.

Calibration at the Sequence Level

Researchers can build confidence models over entire answers, but raw token probabilities do not automatically combine into a calibrated answer-correctness probability.

Calibration requires defining the event, collecting outcomes and measuring predicted confidence against reality.

Verification Changes the System, Not the Local Mechanism

A verifier can reject a generated candidate and ask the model to try again. The next-token mechanism stays the same; the outer loop improves the system.

This distinction is central to advanced reasoning systems: better results can come from runtime algorithms around the model without changing its base generation rule.

Search Changes the Candidate Path

A search procedure can explore several partial sequences or reasoning states and choose promising ones.

The model remains a proposal distribution. Search converts one stochastic generator into a wider problem-solving process.

A Complete Token Trace

Prompt tokens are encoded → transformer computes hidden state → output projection produces logits → decoding transforms/selects candidates → chosen token appended → KV cache updated → new hidden state computed → loop repeats → stop condition reached.

Tool-using systems can interrupt this trace with an external call, then resume generation with the tool result inserted into context.

Independent Exercise 6: Sequence Probability

Two 3-token sequences have conditional probabilities [0.5,0.5,0.5] and [0.8,0.3,0.8]. Which has higher raw sequence probability?

Answer

First: 0.125. Second: 0.192. The second is higher despite one low-probability middle token.

Independent Exercise 7: Rare Identifier

Why might a rare serial number be harder to generate exactly than a common phrase?

Answer

It may split into several uncommon tokens, requiring multiple exact conditional selections. Copying from context or structured tools can improve reliability.

Independent Exercise 8: Agent Action

The model predicts tokens corresponding to delete_file(path). What protects the file?

Answer

The application’s permissions, tool validation and approval boundary. Token probability does not grant authority.

Independent Exercise 9: Entropy

A distribution [0.98,0.01,0.01] has lower entropy than [0.34,0.33,0.33]. Does that prove the first prediction is correct?

Answer

No. It only shows stronger model preference. Correctness needs external evaluation.

Why One Token Can Represent a Large Decision

A tool name, class label or control token can be only one token while representing a consequential system choice. Predicting “approve”, “reject” or a function name therefore compresses a much larger decision into a small output symbol.

This is why token count is a poor measure of semantic consequence. A one-token action can be more important than a thousand-token essay.

Why Long Outputs Accumulate Risk

Every additional token creates another opportunity for drift, contradiction or unsupported detail. Longer answers can contain more value, but they also expose more surface area for error.

Applications should request the length the task needs rather than treating maximum verbosity as maximum intelligence.

Answer Compression

A model can be asked to generate a concise answer after producing or retrieving more internal evidence. The final sequence may be short while the runtime work was substantial.

This separates user-visible token count from total inference compute.

Hidden Runtime Work Versus Visible Tokens

Search, verification, multiple candidate generations and tool calls can happen before one short answer is shown. Next-token prediction describes each model generation, but not the full amount of system work.

A product that reports only output length can therefore hide large differences in cost and reliability architecture.

Token-Level Scores for Reranking

Applications can generate several candidate answers and use log probabilities or a separate scorer to rerank them.

High model likelihood can help choose fluent candidates, but correctness-oriented reranking often needs task-specific verification.

Candidate Diversity

If all candidates come from low-temperature sampling, they may be nearly identical. Higher diversity explores more of the distribution but can introduce weaker outputs.

The right diversity depends on whether the goal is creative exploration, factual answering or robust search over reasoning paths.

Majority Voting

Generate several independent answers and choose the most common result. This can improve some tasks when independent errors vary.

If all samples share the same bias or missing evidence, majority voting confidently repeats the same mistake.

Token Prediction Under Retrieval

When retrieved evidence enters context, its words influence attention and logits. A well-grounded passage can raise the probability of source-supported continuations.

If retrieval supplies the wrong source, the same mechanism can make a wrong answer more confident. Grounding quality begins before generation.

Token Prediction Under Memory

Persistent application memory is converted into context tokens before generation. The model does not access an abstract memory store directly unless the application retrieves an item.

Stale memory therefore acts like stale evidence: it shifts the probability distribution in the wrong direction.

Token Prediction Under System Instructions

System instructions become part of the effective context or control structure. They can strongly influence style, allowed actions and response format.

Their effect is probabilistic model behaviour unless downstream software enforces the rule structurally.

Token Prediction Under Tool Schemas

A detailed tool schema constrains the model’s possible structured output by making field names and types salient.

The application should still validate arguments because high-probability schema-shaped text can contain semantically invalid values.

Why Hallucinated Citations Look Plausible

Titles, author names, years and URLs have strong textual patterns. A model can generate a citation-shaped sequence without retrieving a real source.

Citation reliability therefore needs actual source lookup and identity preservation.

Why Code Can Compile and Still Be Wrong

Syntax has strong token constraints, so generated code can look structurally excellent. The semantic requirement may still be misinterpreted.

Compilation checks one class of error. Unit tests, integration tests and human review check others.

Why Explanations Can Rationalise an Error

Once an incorrect answer token is generated, subsequent tokens are conditioned on it. The model can produce a coherent explanation that supports its own earlier mistake.

This is why an explanation should be verified independently rather than treated as proof that the answer was derived correctly.

Answer-First Versus Reason-First Prompting

Asking for intermediate work before the final answer can delay commitment and provide more context for later tokens.

It can also create longer chains with more opportunities for mistakes. The useful choice depends on task and verification method.

External Scratchpads

A system can store intermediate structured notes outside the visible answer, then feed selected results back into the model.

This separates working state from user-facing prose and can support tool-based verification.

Token Prediction and Search Trees

A reasoning engine can treat each candidate thought or action as a branch and use model probabilities to propose expansions.

Search algorithms then choose which branches to explore. The system becomes more than one sampled sequence even though each proposal is generated token by token.

Token Prediction and Reinforcement Learning

Post-training can shift the probability distribution so preferred responses receive higher probability and discouraged behaviour becomes less likely.

The runtime decoder still samples from logits; reinforcement learning changes the learned logits, not the basic generation loop.

Token Prediction and Distillation

A student model can learn to reproduce a teacher’s token distributions or output sequences.

After distillation, the student still generates autoregressively. Its probability landscape has simply been trained to resemble the teacher under selected conditions.

A Practical Token-Level Audit

Choose a short deterministic task and record the prompt, model version and decoding settings. Generate several outputs. Change temperature only. Then change context only. Then change the model only.

The exercise reveals which behaviour comes from decoding randomness, context conditioning and model parameters.

A Practical Evidence Audit

Take one factual sentence from an answer. Ignore its fluency. Identify whether it came from user input, retrieval, model memory or tool output.

Then ask whether the source supports the claim. This separates token likelihood from evidence quality.

Independent Exercise 10: Explanation After Error

The model gives the wrong answer and then a detailed explanation supporting it. Does explanation length increase confidence?

Answer

No. Later tokens can rationalise an earlier error. Verify the underlying calculation or source independently.

Independent Exercise 11: Majority Vote

Five samples all give the same outdated policy because the prompt contains an archived source. Does agreement fix the problem?

Answer

No. The shared context makes all samples likely to repeat the same source error.

Independent Exercise 12: Tool Schema

The model outputs valid arguments for send_email but the recipient is wrong. What does schema validity prove?

Answer

Only structural validity. Recipient identity and authorisation require separate validation.

Independent Exercise 13: Length

A 2,000-token answer contains more factual claims than a 100-token answer. What evaluation implication follows?

Answer

Longer outputs need more claim-level checking because they create more opportunities for unsupported details, even if average fluency remains high.

Frequently Asked Questions About Next-Token Prediction

Do LLMs literally predict one whole word at a time?

No. They predict tokens, which may be words, subwords, punctuation or other vocabulary units.

What are logits?

Raw scores over vocabulary tokens produced by the model before probability normalisation.

What does softmax do?

Converts logits into a probability distribution that sums to one.

Why use sampling?

To introduce diversity by allowing several plausible tokens rather than always selecting the maximum.

Is next-token prediction enough to explain reasoning?

It explains the output-generation mechanism. It does not by itself describe all internal computation or settle how complex problem-solving behaviour should be characterised.

Can next-token prediction generate code?

Yes. Code is a token sequence. Execution and testing are separate verification layers.

Why do hallucinations happen?

The model produces continuations even when evidence is incomplete. Training and retrieval can encourage uncertainty or grounding, but generation still operates from a learned distribution.

Does a higher-probability sequence have to be better?

No. Likelihood reflects the model distribution and can favour common wording over rare truth or task-specific correctness.

Next-Token Prediction Is Simple Locally and Powerful in Composition

The runtime loop is compact: compute logits, choose a token, append it, repeat. The capability comes from what the transformer has learned to represent in the context that produces those logits.

Understanding this mechanism prevents two opposite mistakes: dismissing LLMs as mere local word counters, or treating generated fluency as proof of truth and understanding.

Continue through the How Super Intelligence Works hub. Next: 033 — Machine Reasoning.

The Practical Completion Test for Next-Token Prediction

A reader has understood next-token prediction when they can explain four things separately: how logits become a token choice, how the growing context changes later probabilities, how decoding controls reshape selection without adding evidence, and how the surrounding application turns generated tokens into grounded answers or real actions.

They should also be able to diagnose whether a failure came from context, model knowledge, decoding, unsupported generation, structured-output validation or downstream execution. “The model predicted the next token” is the start of the explanation, not the end of the system analysis.

This distinction prepares the next step in the curriculum. Machine reasoning asks how token generation can be organised into deliberate problem solving, verification and search rather than one uninterrupted continuation. The mechanism remains autoregressive generation; the runtime strategy becomes richer.

That is the durable mental model: a powerful language model does not stop being a next-token predictor when it writes code, plans a workflow or explains science. What changes is the context, the learned representation, the decoding strategy and the system around the model. Capability emerges from the composition of those layers, while correctness still depends on evidence and verification.

Once that separation is clear, the next-token mechanism becomes easier to respect without either underestimating it or mistaking it for complete system truth.