Did you know? A large language model can sometimes generate exactly the same probability distribution of next tokens while performing fewer expensive sequential decoding steps. Speculative decoding lets a faster draft model propose several tokens, then asks the larger target model to check the proposals in a more parallel operation. Correct acceptance and rejection sampling preserve the target model’s output distribution; simply trusting the draft would not. The improvement is computational scheduling, not proof of better reasoning.
The practical recipe is to define the unchanged target decoding policy, prepare a compatible and considerably cheaper draft mechanism, verify each candidate against the target’s conditional probabilities, accept only the valid prefix, and sample correctly from a residual distribution when a proposal is rejected. Count actual wall-clock latency, target calls, accepted tokens, memory and quality constraints. When the draft agrees poorly or the extra work costs more than it saves, speculative decoding may be slower rather than faster.
This guide derives the acceptance rule with a fully solved three-token vocabulary example, explains greedy and sampling variants, identifies real implementation hazards and transfers the approach to a changed deployment context. The method concerns present-day model inference, not artificial superintelligence. Fictional timing measurements and examples below are explicitly labelled; research and developer claims are linked to original sources.
Contents
- Why autoregressive inference has a serial bottleneck
- The target and draft distributions must be distinguished
- The exact acceptance probability
- Worked example: three tokens and a provably correct residual
- How draft blocks become accepted prefixes
- Worked latency case: faster despite verification work
- Sampling settings must be part of the unchanged target contract
- Tokenizers, vocabulary support and alternative drafting methods
- KV caches, memory trade-offs and parallel verification
- Exact distribution does not mean identical sample text
- Common failure patterns and targeted repairs
- Changed-context exercise: constrained JSON generation
- Where the speedup promise breaks down
- Frequently asked questions
- Practical implementation route and primary research
Why autoregressive inference has a serial bottleneck
An autoregressive model predicts the next token conditioned on the preceding prompt and generated tokens. To produce ten new tokens without any special acceleration, it ordinarily performs a sequence of dependent decoding steps. Even if each step runs thousands of operations in parallel across accelerators, the next step cannot be fully determined until the previous token has been selected. That dependency makes low-latency output difficult.
Large transformer models are powerful but expensive to run. For some deployments, short sequential target-model steps are constrained by accelerator memory traffic and launch overhead rather than arithmetic alone. Making a smaller model guess several easy next tokens may be much cheaper than making the target model generate each of them in turn. The target can then inspect a proposed sequence in an operation that exploits parallel processing more efficiently.
The key is that the target model remains the authority over the final token distribution. A draft model is not a second voter with equal status. It provides candidate paths that may be verified cheaply in batches relative to sequential target decoding. If the target disagrees, the algorithm must correct the proposed path rather than blindly stream it to the user as though confirmed.
Leviathan, Kalman and Matias introduced an exact speculative decoding method in Fast Inference from Transformers via Speculative Decoding, ICML 2023. Their specific experiments showed acceleration under their studied conditions. Those measured gains are evidence that the mechanism can work, not a guarantee that every pair of models or serving stack will be faster.
The word speculative has a precise role. Candidate tokens may be computed before the expensive model has fully confirmed them; they remain provisional until the verification and acceptance rules are complete. This differs from the agent partial-answer streaming guide, which concerns user-facing output commitments and recovery. Speculative decoding is a mathematical inference algorithm inside a generative model.
The target and draft distributions must be distinguished
Let p be the target model’s next-token probability distribution given a prefix, and q be the draft model’s proposal distribution at that same prefix. Both must describe compatible candidate outcomes for the acceptance calculation to make sense. The target is the model whose sampling distribution the deployment wants to preserve. The draft is an inexpensive proposal generator, not a replacement model.
In a common implementation, the draft model uses the accepted prefix and repeatedly proposes several candidate tokens. The target model evaluates their probabilities at the corresponding conditional contexts in one verification stage. The algorithm works position by position along the proposed prefix. It accepts a run of proposals until the first rejection, at which point the continuation is corrected from an appropriate residual distribution.
Why can the target evaluate several draft-conditioned positions at once? Given a candidate sequence, transformer attention masks can let the target compute predictions conditioned on each candidate prefix during a forward pass. That is still real computation, but the wall-clock cost may be lower than issuing many separate target decoding steps. The relationship depends on hardware, prompt length and software implementation.
Using a small draft model with identical or aligned tokenisation can simplify the method. The official Hugging Face assisted decoding guide describes current library options, including an assistant model, prompt lookup proposals and alternatives with special model support. The guide also documents limitations, such as batching support, that are version-specific and should be checked for the installed release.
A subtle but important condition is that p and q refer to the same decoding policy. If the target uses temperature, top-p filtering or another transformation, p in the exact verification calculation must represent the target distribution after those intended transformations. Comparing draft probabilities with a different target policy can violate the claim of unchanged output distribution.
The exact acceptance probability
For a proposed token x sampled from q, a standard exact sampling rule accepts the proposal with probability a(x)=min(1, p(x)/q(x)), where q(x)>0 because the token was sampled from q. If p(x) is at least q(x), the draft token is always accepted. If the draft oversampled a token relative to the target, the proposal is accepted only some of the time.
This rule is not arbitrary. The probability that a token x is both proposed and accepted is q(x)×min(1,p(x)/q(x))=min(q(x),p(x)). The accepted part alone usually has less total probability than one. When the algorithm rejects a proposal, it must sample from the part of p that the accepted proposal mechanism did not cover.
That residual distribution has unnormalised weight r_raw(x)=max(0,p(x)−q(x)). It is normalised by the sum over all tokens of positive differences, provided that sum is nonzero. The residual completes the missing target probability mass. The result of accepted proposals plus the rejection branch then follows exactly p for one step under the algorithm’s assumptions.
Do not confuse ‘accept a token’ with ‘the draft token was correct’. Exact sampling does not check ground-truth factual correctness. It accepts tokens in a way that statistically reproduces the target model’s defined distribution. A target model can itself generate mistakes, dangerous output or nonsense under some prompts. Speculative decoding does not add an independent factual reviewer.
Greedy decoding is a different policy. In a greedy version, draft candidates can be checked against the target’s selected argmax at each position, and a mismatch triggers correction. Reproducing greedy target output is not the same task as reproducing stochastic samples. Libraries may implement different fast paths, so one should verify the exact intended algorithm rather than treating all assisted generation as identical.
Worked example: three tokens and a provably correct residual
Imagine a toy vocabulary of three tokens A, B and C. The target distribution at one position is p=(0.60,0.30,0.10). The draft distribution is q=(0.50,0.40,0.10). Both distributions sum to one. These probabilities are constructed solely to explain the verification rule; no language model was run to obtain them.
If the draft proposes A, acceptance is min(1,0.60/0.50)=1. If it proposes B, acceptance is min(1,0.30/0.40)=0.75. If it proposes C, acceptance is min(1,0.10/0.10)=1. The total chance of accepting the draft token is 0.50×1 + 0.40×0.75 + 0.10×1 = 0.90. Rejection probability is therefore 0.10.
Now calculate the residual. Positive target-minus-draft differences are max(0,0.60−0.50)=0.10 for A, max(0,0.30−0.40)=0 for B, and max(0,0.10−0.10)=0 for C. The residual normalisation constant is 0.10. Conditional on rejection, the corrected token is A with probability one.
Combine the two branches. A is emitted from accepted proposals with probability 0.50, and from the rejection branch with probability 0.10, totalling 0.60. B is emitted with probability 0.40×0.75=0.30. C is emitted with probability 0.10. The final distribution is exactly (0.60,0.30,0.10), which matches p.
Notice that the draft proposal originally favoured B too strongly at 0.40. An algorithm that simply kept every draft token would produce q rather than p. An algorithm that rejected B and then sampled afresh from the full target p would also be wrong, because it would double-count some probability mass. The residual must be constructed from positive differences and then normalised correctly.
This exact one-step accounting underlies the more involved multi-token procedure. At later positions the target and draft probabilities change because each is conditioned on a different accepted prefix and candidate tokens. The implementation must respect those conditional contexts and stop at the first rejection. It cannot apply the same p and q vectors to every position without justification.
How draft blocks become accepted prefixes
Suppose a draft proposes four tokens at positions 1 through 4. The verifier computes the target probabilities along those candidate prefixes. If the first three tokens are accepted but the fourth rejected, only the first three proposed tokens are committed from that block; the fourth is replaced by an appropriately sampled correction. The draft’s later speculative continuations, if any, cannot remain valid because they were conditioned on a path that the final generation did not take.
If every proposal in the block is accepted, the algorithm can use an additional target distribution to continue. This is one reason the target verification stage may evaluate an extra position. The exact implementation varies by architecture and scheduling strategy, but final output must still follow the target’s defined decoding policy.
Block length is a trade-off. Longer drafts offer more opportunities to amortise a target-model verification pass when acceptance is high. They also increase wasted draft computation when rejection occurs early. Extra cache storage, bandwidth and verification work can make an overlong proposal unhelpful. The best length depends on acceptance behaviour and hardware rather than a universal recommendation.
The acceptance rate itself is not a single immutable property of the model pair. It varies by prompt, language, domain, temperature, length and position. A draft model may closely match the target on repetitive formatting but diverge on mathematical notation or rare technical names. Production systems should measure conditional acceptance distributions by relevant workload segment.
One practical consequence is that the draft model should be cheap relative to the target. A slightly smaller assistant that consumes almost the same hardware resources may provide little benefit despite high acceptance. Conversely, an extremely small assistant that proposes unlikely tokens may squander verification work. Balance proposal quality against proposal cost.
Framework support also matters. A library may support greedy decoding and sampling but not particular batched serving combinations, or may require matching tokenizers for its standard assistant mode. Check the documented capabilities of the actual version instead of using another toolchain’s performance claim.
Worked latency case: faster despite verification work
Consider an invented service where a target-model decoding step takes 30 milliseconds in the workload being measured. Ten serial steps would require approximately 10×30=300 milliseconds of target step latency, ignoring fixed prefill and network costs. A draft model can propose five candidates in 5×2=10 milliseconds, and the target can verify the block in 45 milliseconds under the hypothetical hardware setup.
If all five candidates were accepted, producing that five-token block would cost 10+45=55 milliseconds for draft and verification, versus 5×30=150 milliseconds for five serial target steps. On this narrow simplified comparison, the speedup is 150/55≈2.73×. That result is not a measured benchmark and excludes queuing, memory allocation, implementation overhead and the details of the extra target position.
Now change one assumption: only the first token is accepted before a rejection in most blocks. The draft still spends 10 milliseconds proposing five tokens, and the target still spends roughly 45 milliseconds verifying the attempted block. If only a small number of useful output tokens emerges, the average time per committed token can exceed the original 30 milliseconds. The method can become slower, which is why acceptance length matters more than a high-level count of proposals.
Suppose a service runs 1,000 toy generations, each committing an average of four useful tokens per 55-millisecond speculative iteration. Its apparent throughput per such iteration is four divided by 0.055, or about 72.73 tokens per second, under those assumptions. A 30-millisecond serial target step would correspond to about 33.33 tokens per second. These are derived scenario rates, not independent measurements; actual throughput must include scheduling, request concurrency and shared accelerator resources.
Another deployment may optimise total tokens per GPU-second rather than latency for one user. A technique that reduces individual response delay can occupy memory needed for larger batches and lower overall throughput. Both measures should be recorded, with their denominators and workload definition, rather than advertised as interchangeable notions of speed.
Do not infer better answers from lower latency. Speculative decoding designed to be distribution-preserving gives the same target distribution, so it should not be credited with a quality gain merely because its implementation finishes sooner. If a proposed acceleration changes sampling semantics, that is a different, potentially lossy optimisation requiring separate quality evaluation.
Sampling settings must be part of the unchanged target contract
Many text-generation APIs expose temperature, top-k, top-p and other logit transformations. These settings can change the distribution from which tokens are sampled. An exact speculative algorithm preserves whichever transformed target distribution its verification rules are designed to reproduce, not some vaguely defined idea of the base model’s preferred words.
For example, top-p filtering can remove a low-probability tail and renormalise the remaining candidates. If the draft uses one set of filters while the target verification incorrectly assumes unfiltered probabilities, the acceptance calculation no longer corresponds to the actual target sampling policy. A rigorous implementation defines p after the intended target transformations and aligns probability support appropriately.
Temperature changes relative token odds before normalisation. Lower temperature can make the target highly peaked, sometimes increasing agreement with a competent draft for predictable text, while higher temperature may cause more divergence. But this is not a universal law: the actual agreement depends on both distributions and the prompt. Measure rather than assume.
Special tokens and stopping conditions also belong to the contract. End-of-sequence tokens, forced tokens, forbidden tokens, length penalties and custom stopping rules may affect candidate validity. An assistant should not push a draft continuation past a target-approved stop event, nor discard verification because the generated text ‘looks complete’.
The existing temperature and top-p guide explains how decoding settings affect token selection. Speculative decoding is a separate specialist: it seeks to realise the selected decoding rule more efficiently, not to choose a different trade-off between creativity and certainty.
Tokenizers, vocabulary support and alternative drafting methods
The simplest assistant-model implementation uses compatible tokenizers so a candidate token identifies the same discrete item to the assistant and target. Mismatched tokenization can make alignment much harder. A word that occupies one token in the draft may occupy several tokens in the target, so naive acceptance by token index is not mathematically meaningful.
Recent tools provide alternatives. The Hugging Face assisted-decoding documentation describes universal assisted decoding for different tokenizers using additional alignment steps, and prompt-lookup proposals that reuse patterns in the prompt. It also describes methods that use early model layers or specialised multi-token prediction heads. Their operational requirements differ; none should be called exact simply because it produces fast text.
Draft quality depends on candidate context. A prompt-lookup scheme may work especially well when a generated response repeats source passages, but badly on creative continuation. An assistant language model may perform more consistently across varied outputs at higher computational cost. A model with early-exit heads can share weights and caches but requires architecture support. Selecting a proposal source is a systems design decision tied to workload.
An exact implementation needs more than a plausible proposal mechanism. It needs a target verifier that assigns the correct conditional distributions to candidate prefixes and a rejection rule that yields the target distribution. Any alignment approximation or ensemble acceptance modification may change the guarantee; the implementation should document precisely which algorithm is active.
By making the mathematical contract explicit, a team can improve its proposal machinery without silently altering output semantics. It can say that a new draft model increases acceptance while the target and sampling policy remain unchanged, provided it actually verifies that property with tests.
KV caches, memory trade-offs and parallel verification
Transformer inference commonly caches intermediate key-value states so each new token need not recompute every earlier attention representation. Speculative decoding complicates cache management because some candidate tokens will be rejected. Cache entries for an unaccepted suffix must not become part of the final accepted prefix. Using a stale speculative cache can change future conditional probabilities even when the acceptance formula itself is correct.
A reliable implementation tags cache state by accepted prefix length and candidate branch. On rejection, it discards or restores state so the corrected continuation begins from the right context. The same principle applies to position encodings, attention masks and any bookkeeping of generated length. A one-position off-by-one error can quietly destroy the exactness claim.
Memory costs matter. Holding a draft model alongside the target may increase accelerator footprint, and verifying several proposed tokens can require temporary activation or cache allocations. On a memory-constrained device, this can force smaller batches or cause paging, eliminating the anticipated benefit. Measure peak memory and resident model sizes in addition to token latency.
Attention masks must preserve causal structure. The target may evaluate candidate positions in parallel while ensuring each position conditions only on the correct preceding tokens. Allowing later draft tokens to leak into earlier target predictions would change the inferred probabilities and invalidate the acceptance ratio. Parallel computation is compatible with causal conditioning only when implemented correctly.
Performance monitoring should distinguish prefill, drafting, verification, residual correction and output emission. A benchmark that reports only verification kernel time cannot establish end-to-end response latency. Compare complete user requests under representative context lengths and concurrency settings.
Exact distribution does not mean identical sample text
Exact speculative sampling is designed so the final token distribution matches the target distribution under the specified conditions. That does not imply that two independently seeded runs will print identical text. If both algorithms sample from the same probability distribution using different random-number consumption patterns, their individual sampled strings may differ while the overall distributions agree.
Greedy decoding is more straightforward to compare by exact token sequences, provided tie-breaking and numerical precision are controlled. Stochastic decoding calls for distributional verification over many samples or carefully constructed small-state tests. A one-output mismatch is not proof of distributional error, and a one-output match is not proof of exactness.
Floating-point arithmetic also introduces practical questions. Ratios near one, very small q(x), rounding and clipped probabilities can affect acceptance decisions. Stable implementations can use log probabilities where appropriate and test extremal cases. A formal derivation in real arithmetic must be translated into numerically defensible code.
The original exactness proof has assumptions about how q and p are sampled and compared. Production kernels may introduce shortcuts, quantisation, approximate distributions or numerical thresholds. Some variations trade fidelity for speed. The official current library documentation distinguishes lossless speculative decoding from static ensemble verification that deliberately targets a mixture distribution. A product must not advertise exactness for a lossy variation.
Honest evaluation reports both correctness of distributional implementation and measured performance. A model can be exactly sampled but unhelpful to a user. Conversely, a faster approximate method may be acceptable for a low-stakes application if the distribution change is disclosed and independently tested. The word exact should be reserved for its actual mathematical meaning.
Common failure patterns and targeted repairs
Failure one is treating a draft token as confirmed before verification. Repair it by maintaining a provisional buffer and exposing only accepted output where the API promises committed text. Failure two is accepting with the wrong ratio or forgetting the residual distribution after rejection. Repair it with small discrete probability tests like the three-token example in this guide.
Failure three is using probabilities from mismatched token contexts. A verifier that conditions on an unaccepted token is no longer checking the intended prefix. Repair it by tracking the accepted path and discarding all speculative suffix state after rejection. Failure four is permitting incompatible tokenizers without a proved alignment procedure. Check the library’s documented supported mode.
Failure five is combining inconsistent sampling transformations. Top-p, temperature, banned tokens and end markers must be represented consistently in the target distribution used for acceptance. Repair it by specifying the exact target generation policy and verifying each special case.
Failure six is assuming bigger draft blocks always improve performance. Their expected benefit depends on useful accepted prefix length and the cost of draft and verification operations. Profile block sizes under representative workloads and choose a defensible policy rather than copying a benchmark configuration.
Failure seven is comparing incompatible latency denominators. Excluding prefill or averaging only successful blocks can make an acceleration look stronger than it is. Report time per request, accepted tokens, wall-clock throughput, hardware memory and the proportion of requests that became slower.
Failure eight is changing the target model and claiming the method itself improved answer quality. Separate changes in model weights or decoding policy from changes in algorithmic scheduling. If both changed, run controls that isolate which change caused the observed outcome.
Changed-context exercise: constrained JSON generation
Transfer the mechanism from ordinary prose to a constrained JSON generator. An application requires a machine-readable object with exactly four properties. The target model uses a grammar that prohibits invalid key names at each state. A draft model proposes tokens in a cheaper mode. For speculative decoding to preserve the constrained target policy, p must represent the target’s probabilities after grammar masking and any renormalisation at each accepted prefix.
Suppose a draft proposes a field key that the grammar forbids. Under the correctly constrained target distribution, p(x)=0 for that token. If the draft assigns q(x)>0, its acceptance probability is min(1,0/q(x))=0. The proposal is rejected. A correction must come from the residual distribution over allowed tokens; it cannot copy the invalid key simply because it appeared in the draft’s most probable continuation.
Now consider a fictional benchmark of 100 identical-format requests. Ordinary target decoding takes 40 milliseconds per request after shared prefill, for 4,000 milliseconds total decoding time. A proposed exact assisted implementation takes 29 milliseconds per request on average, for 2,900 milliseconds total. The time saved is 1,100 milliseconds, or 27.5% of the original 4,000. Speedup factor is 4,000/2,900≈1.38×. These values are invented and presume the same request set and comparable hardware conditions.
Before accepting the optimisation, verify that all 100 outputs still obey the grammar, and test whether the empirical output distribution matches the constrained target under suitable sampling analysis. Do not confuse 100 syntactically valid outcomes with evidence of distributional equality, and do not claim factual correctness of the JSON values merely because the structure is valid.
The transfer lesson is that correctness is relative to the authorised target policy. For open-ended prose that may be a sampling distribution; for constrained objects it includes state-dependent masks and stop rules. Speculative decoding is safe only when it preserves the intended target, not when it merely accelerates the appearance of text.
Where the speedup promise breaks down
An assistant can be too slow, too inaccurate or too memory-hungry to help. At low acceptance rates, the target repeatedly verifies candidate blocks that contribute little output. At high concurrency, the additional assistant may reduce target batching efficiency. At very long contexts, cache and attention costs may dominate differently from a short benchmark. Measure real workload latency and throughput before making product claims.
Exact sampling preserves the target’s defined distribution but does not repair target model hallucination, bias, unsafe instructions or stale knowledge. The acceleration method should be combined with appropriate task validation, safety filters and user permissions. Do not confuse fewer serial model calls with an ability to reason faster in a semantic sense.
Some frameworks support only particular model architectures, decoding modes or batching shapes for an assisted-generation pathway. These capabilities change over time. The installed version and official developer documentation, not a blog benchmark from a different release, govern what is available for deployment.
Published research can report large acceleration under suitable hardware and workload. Those results are informative but cannot be pasted into an unrelated service-level promise. Benchmarks must state precision, hardware, model pair, context lengths, concurrency, sampling policy and complete timing boundaries.
The numerical illustrations in this guide were constructed for clarity. Only the underlying research and developer documentation describe real experiments. The acceptance derivation is exact under its stated probability assumptions; production correctness still depends on implementation and test evidence.
Frequently asked questions
What is speculative decoding? It is a model inference method in which a cheaper mechanism proposes several tokens and a target model verifies them, potentially reducing expensive sequential target steps while preserving the target distribution under an exact algorithm.
Does the draft model write the final response? Not on its own. Its tokens remain proposals until accepted by the target verification logic. A rejection triggers a mathematically correct alternative continuation.
Is it the same as sending partial answers to the user? No. Speculative decoding is internal token generation. Partial-answer streaming is an external communication contract about provisional versus committed text.
Why is the acceptance probability min(1,p/q)? Because proposals drawn from q need to be accepted with the probability mass min(p,q), leaving positive target-minus-draft mass for the residual correction.
Will the output words always be identical? For stochastic sampling, individual runs can differ while the distribution is preserved. Greedy matching can permit exact sequence comparisons under aligned implementations.
Do temperature and top-p still matter? Yes. They are part of the target decoding policy, and verification must use the corresponding transformed target probabilities.
Is a larger draft model always better? No. Increased agreement can cost more computation and memory. Profile total request latency and throughput rather than judging only draft quality.
Can it make a model more accurate? An exact method does not change the target’s distribution, so accuracy improvement is not the intended effect. Other changes to the model or decoding policy require separate evaluation.
When is it a poor choice? When the draft is expensive or poorly aligned, the target already batches efficiently, or extra model memory and serving complexity outweigh avoided serial steps.
Practical implementation route and primary research
For model engineers, begin with a discrete exactness unit test, then implement prefix acceptance, residual correction and rollback of speculative cache state. Compare against the same target model and generation policy. Report accepted-prefix length, target verification time, draft time, memory, total request latency and stochastic distribution checks. Measure where the optimisation loses.
For educators, use the three-token p and q example to verify mass conservation. If A=0.60, B=0.30 and C=0.10 is the target, explain why accepting every q token would produce the wrong result. Then change q to equal p; all proposals can be accepted with probability one under exact arithmetic, though the cost of computing q may still limit speedup.
For reviewers, ask whether the path is truly distribution-preserving or an approximate variant, whether sampling transformations and end tokens are included, and whether claimed speedups use comparable prompts and hardware. Require tool-execution evidence before treating a benchmark claim as actual performance.
Original source: Leviathan, Kalman and Matias, Fast Inference from Transformers via Speculative Decoding, ICML 2023, with publisher-hosted abstract and paper. Current implementation source: Hugging Face Transformers, Assisted decoding; its details are version-dependent. Related overview: Google Research publication record.
For the wider topic continue through How AI Works and the learning-transfer framework. This article does not assert an actual deployed speedup from an unrun experiment.