Speculative decoding: the secretary who types ahead of the boss — fundamentals, maths and state of play in May 2026

· By José Manuel Román
Contents

This post complements KV cache: the working memory and Quantisation for LLM inference. Both are acceleration levers inside a single forward pass. Speculative decoding is the complementary lever: instead of making each forward pass cheaper, it tries to produce more tokens per forward pass. It is orthogonal to quantisation and to the KV cache, and it stacks multiplicatively with both.

You are here: DEPLOY

You are here: DEPLOY · more tokens per forward pass without touching quality1 · Data2 · Tune3 · Eval4 · Deploy5 · Observe6 · Retrain

TL;DR

Autoregressive LLM inference has a structural problem: every new token needs a complete forward pass of the model, and decode forward passes are memory-bandwidth-bound, not compute-bound (the detail is in KV cache). The GPU spends most of its time waiting for the next weight to arrive from HBM. Speculative decoding uses that dead time by doing two things in parallel: a small, cheap model (draft) generates γ tokens autoregressively, fast but imprecise, and the large model (target) verifies those γ tokens in a single parallel forward pass, at almost the same cost as a single-token forward pass. An acceptance rule based on rejection sampling decides how many draft tokens are accepted, and the maths proves, it does not approximate, it proves, that the output distribution matches sampling directly from the target exactly. The technique has a ceiling: you never generate more than 1/(1-α) tokens per step, with α the acceptance rate. The five dominant families in May 2026 are vanilla SD (Leviathan 2023), Medusa (extra parallel heads), EAGLE-1/2/3 (a draft that operates at the level of hidden states), MTP (multi-token prediction native to DeepSeek-V3) and P-EAGLE (a draft that produces the γ tokens in a single forward pass, integrated in vLLM 0.16+). On low-concurrency workloads with short prompts and long outputs, the typical case of the on-premise conversational assistant, the real speedup sits between 2× and 4× on modern hardware.

The analogy: the secretary who types ahead and the boss who validates

Picture a crowded press conference with a real-time transcription setup. Two people are working:

The secretary sits at the keyboard. They know the subject, they have read the briefings and they know the boss’s verbal tics. As soon as the boss opens their mouth, the secretary starts typing what they think is coming. They are fast, three or four words ahead of the boss, but they sometimes get it wrong, especially when the boss takes an unexpected turn.

The boss periodically reads the secretary’s screen. They read a whole block at once, three or four words, and mentally compare what is written with what they would have said. If it matches their intent, they leave the text and carry on. If the secretary drifted at some point, they correct right there, and whatever the secretary typed after the divergence is discarded (it could be wrong because of the earlier error). If the whole block was fine, the boss takes the chance to add one more word while correcting.

The result: the final text is identical to what the boss would have dictated alone, but it comes out faster because the secretary works ahead.

The analogy holds on four details:

  • The secretary is the draft model. Small, cheap, fast, approximate.
  • The boss is the target model. Slow, expensive, but the only authority on output quality.
  • The block the boss reads at once is the target’s parallel forward pass over prompt + γ draft tokens. Verifying γ tokens costs almost the same as generating one, because the bottleneck is loading the weights from HBM, not the operations.
  • The “correct right where it diverged” rule is the rejection sampling that preserves the target distribution.

From here we go into the mechanism and into why quality is preserved exactly, not approximately.

The bare mechanism

Call p the target distribution (what it would produce if it were the only one speaking) and q the draft’s. The speculative decoding iteration has three steps:

Step 1 — Draft. The draft generates γ tokens autoregressively: x_1, x_2, ..., x_γ. Each one through its own forward pass. Since the draft is small, the γ steps cost little. Typically γ ∈ [4, 8].

Step 2 — Verify. The target runs a single forward pass over the complete sequence prompt + x_1 ... x_γ. Because attention is causal, that forward pass simultaneously produces the distributions p(·|prompt, x_<i) for each position i. In other words, in a single step it obtains the verification of the γ tokens and, on top of that, an extra distribution p(·|prompt, x_1...x_γ) for the next token.

Step 3 — Accept/reject token by token. For each x_i from left to right it applies:

  • If p(x_i) ≥ q(x_i): always accept.
  • If p(x_i) < q(x_i): accept with probability p(x_i)/q(x_i).
  • If it rejects: stop, sample a replacement token from the normalised residual distribution norm(max(0, p − q)), and discard the rest.
  • If it reaches the end having accepted the γ tokens: append a bonus token sampled directly from p(·|prompt, x_1...x_γ), which is already in the logits of the target’s forward pass.

Result per iteration: between 1 and γ+1 new tokens. In the best case (all accepted + bonus) γ+1 tokens are generated at the cost of a single target forward pass plus γ draft forward passes, against γ+1 target forward passes in the version without speculative.

1. Draft model (q) generates γ=4 tokens autoregressivelyx₁x₂x₃x₄4 cheap forward passes2. Target model (p) verifies the 4 tokens in ONE parallel forward passsingle forward pass over [prompt, x₁, x₂, x₃, x₄]obtains p(·|prompt, x<i) for i=1..53. Left-to-right rejection samplingx₁ ✓p≥qx₂ ✓acceptedx₃ ✓acceptedx₄ ✗rejected4. Result of this iterationx₁x₂x₃x'₄x'₄ = sample from norm(max(0, p−q)) at pos 4bonus dropped (only if all accepted)→ 4 new tokens in a single target iteration. Without speculative it would be 4 iterations.

Why quality does not degrade (the proof)

This is the part of the technique that confuses people most the first time. It looks like magic: how can sampling from an arbitrary draft and then “validating” produce exactly the same distribution as sampling from the original target?

The proof fits in two lines. For any token x, the final probability of emitting it is the sum of two disjoint events: the draft proposes it and it is accepted, or the draft proposes something else, it is rejected, and sampling from the residual yields x. Formally:

$$P(\text{emit } x) = q(x) \cdot \min\!\left(1, \frac{p(x)}{q(x)}\right) + P(\text{reject}) \cdot \frac{\max(0, p(x)-q(x))}{\sum_y \max(0, p(y)-q(y))}$$

The first term is min(p(x), q(x)). The second is max(0, p(x) − q(x)) (the total rejected mass Σ max(0, p−q) cancels the denominator exactly). Adding them: min(p(x), q(x)) + max(0, p(x) − q(x)) = p(x). The output is statistically indistinguishable from sampling directly from the target, down to floating-point numerical differences.

This matters operationally: it means speculative decoding is not perceptual compression, it is not a quality-speed trade-off, it does not require extra validation with evals. If the target would have passed a given eval, the system with speculative passes it too.

The maths that matters: ceiling and speedup

Call α the acceptance rate: the expected probability that an individual draft token is accepted. Assuming that acceptances of consecutive tokens are independent (a reasonable approximation in practice), the expected number of tokens generated per iteration is:

$$E[\text{tokens per step}] = \sum_{k=0}^{\gamma} \alpha^k + \alpha^{\gamma+1} = \frac{1 - \alpha^{\gamma+1}}{1 - \alpha}$$

And the theoretical speedup relative to generating one token per target iteration, with c = T_draft / T_target the relative cost of the draft, is:

$$\text{Speedup} = \frac{1 - \alpha^{\gamma+1}}{(1 - \alpha)(\gamma c + 1)}$$

There is a ceiling that many implementations try to beat and cannot: as γ → ∞, the speedup converges to:

$$\lim_{\gamma \to \infty} E[\text{tokens per step}] = \frac{1}{1 - \alpha}$$

It is an algorithmic ceiling, not a hardware one. With α = 0.7 you never generate more than 3.33 tokens per iteration no matter how much the hardware improves. With α = 0.8 → 5. With α = 0.9 → 10. That is why EAGLE-3, which targets α > 0.8 on many benchmarks, is not an incremental improvement over vanilla SD (α ≈ 0.5-0.6): it is a change of regime, because it raises the ceiling instead of merely getting closer to it.

A concrete numerical example, Llama 3 70B as target with a draft giving α = 0.75, γ = 5, c = 0.1:

  • Expected tokens per step: (1 − 0.75⁶) / (1 − 0.75) = (1 − 0.178) / 0.25 = 3.29
  • Speedup: 3.29 / (5 × 0.1 + 1) = 3.29 / 1.5 = 2.19×

If we raise α to 0.85 with the same configuration: tokens per step = (1 − 0.85⁶) / 0.15 = (1 − 0.377) / 0.15 = 4.16, speedup = 4.16 / 1.5 = 2.77×. A change of α = 0.75 → 0.85 (10 absolute points) raises the speedup by 27 %. That is why the research of the last two years focuses on pushing α: that is where the prize is.

The five modern families (May 2026)

FamilyYearCore ideaDraft sizeTypical α / τSpeedup
Vanilla SD2023Same-family target + draft pair, rejection sampling1/10 – 1/100 of target0.5 – 0.82-3×
Medusa2024N extra heads in parallel predict t+1, t+2, …; tree attention verifies several candidatesno draft, +1-2 % paramstop-5 > 0.82.18-2.83×
EAGLE-1/2/32024-25Autoregressive draft at the level of features (hidden states), not tokens. Reuses the target’s embedding1 transformer block (~0.5-2 % params)EAGLE-3: α > 0.8, τ up to 7.5up to 6.5× peak
MTP (DeepSeek-V3)2024Multi-token heads trained from scratch as part of the model; at inference they act as a “free” draft14B params in V3 671Bα > 0.8 (MTP1)1.5-1.8×
P-EAGLE2026EAGLE but producing the γ drafts in a single forward pass (parallel, not autoregressive)same as EAGLE+30 % over EAGLE-34-5× vs AR

Three operational observations:

  1. EAGLE dominates in production because its overhead is minimal (one transformer block, ~1 % of the target in parameters) and its α is high. The draft does not need its own “complete” KV cache because it shares features with the target.
  2. Medusa mattered historically, it showed that speculative was possible without a separate draft, but EAGLE beat it on every published benchmark through 2024-2025.
  3. MTP is special: it is not something you add to an existing model. It is something the model trained natively. If you buy DeepSeek-V3, MTP comes free and gives around 1.8× without touching anything. If you buy Llama 3, there is no MTP to be had; use EAGLE-3.

There are also two related techniques that deserve a brief mention, both without a draft model: Lookahead decoding (Fu et al. 2024), which formulates decoding as Jacobi iteration and extracts n-grams from the trajectory; and REST (He et al. 2024), which keeps an n-gram datastore and proposes drafts by longest-prefix match against the last generated tokens. Both give 1.5-2× with no extra VRAM. Useful when you have no trained draft model and do not want to maintain one.

Real implementations in May 2026

  • vLLM v0.16+: unified support for EAGLE-1/2/3, P-EAGLE, Medusa, native MTP (DeepSeek-V3 and variants), n-gram/suffix decoding without a draft, an arbitrary draft model and MLP speculators. The canonical flag is --speculative-config '{"method":"eagle3", "model":"...", "num_speculative_tokens": 5}'.
  • SGLang: native EAGLE-3 support with --speculative-algorithm EAGLE3. For DeepSeek-V3 it uses MTP through an EAGLE adapter. It has its own draft training framework (SpecForge).
  • TensorRT-LLM: Medusa, EAGLE (a simplified variant without a tree), ReDrafter, Lookahead. They report around 2.2× with EAGLE.
  • llama.cpp: basic speculative only with a draft model (--model-draft). No native EAGLE/Medusa/MTP as far as I have verified. Typical speedup 1.5-2.5× single-user.

When speculative does NOT help

The technique has three important blind spots:

Large batch. The GPU moves from memory-bound (decode at low concurrency) to compute-bound (decode at high concurrency). In the compute-bound regime the “almost free” target forward passes stop being free: γ tokens start costing γ times more instead of nearly 1. The typical crossover is at batch 16-32 for dense models; for MoE with few active parameters per token the crossover happens later. At load peaks, speculative can worsen aggregate throughput per GPU.

Prefill / TTFT. Speculative decoding does not touch prefill. The phase that processes the whole prompt stays identical. TTFT does not improve and can get marginally worse because of the draft setup. If the SLA is TTFT-bound (assistants with short outputs, search, RAG with brief answers), this is not the right tool.

Short outputs. If the model generates 10-20 tokens and the answer is over, the fixed setup overhead is not amortised. Speculative shines with long outputs (300+ tokens): generative assistants, extended code completion, drafting.

Operational pitfalls

Extra VRAM for the draft. In vanilla SD, loading a complete draft plus its KV cache is expensive. Llama 3 8B in BF16 as the draft for a 70B is about 16 GB of weights plus KV cache. On an H100 80 GB with the target already nearly filling it, that can force you to shrink the target’s KV cache and lower maximum concurrency. EAGLE solves this: the draft is one transformer block (~1 GB for a 70B) and reuses the target’s features.

Draft quantisation. Quantising the draft to INT4 with GPTQ degrades α substantially (errors in the logits accumulate in the comparison against p). AWQ holds up better but also lowers α. Common practice in 2026: target in FP8 or INT4, draft in FP16/BF16. The draft is small enough that the VRAM saved by quantising it does not compensate for the drop in α and therefore in speedup. The detail of each format is in Quantisation.

Interaction with continuous batching. It does not break it, but it creates nested raggedness: each request in the batch can accept a different number of tokens on each iteration. vLLM’s PagedAttention absorbs it, but the scheduler loses efficiency. At low QPS (conversational assistant, low simultaneous concurrency) the combination is excellent. At high QPS there is real tension, and work such as Goodput-optimized speculative decoding (Liu et al., 2024) optimises γ dynamically according to the state of the batch.

Sampling temperature. α falls with high temperatures. At T = 1.0 with creative outputs (free-form writing), α can drop 10-15 points relative to T = 0 with the same model-draft pair. The speedup scales accordingly.

Implications on on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace)

The classic use is vanilla SD with two quantised models: for instance, Llama 3 70B AWQ-INT4 as target (~35 GB → needs TP=2 across two 4090s) and Llama 3 8B AWQ-INT4 as draft. In practice, the most realistic single-card case is Llama 3 8B target + a 1B model as draft or Qwen 3 14B target + Qwen 3 0.5B as draft: they fit comfortably and give 1.8-2.5× on conversational tasks. For EAGLE on consumer cards, the official drafts for popular families (Llama 3, Qwen 3) are published on Hugging Face and take up 0.5-2 GB extra.

Here EAGLE-3 shines:

  • Llama 3 70B FP8 + EAGLE-3 draft in FP16: the draft takes about 0.7 GB; the target about 70 GB with TP=2 (35 GB per GPU). The speedup observed in reproducible benchmarks is between 2.5× and 4× at batch 1-4, falling to almost break-even at batch 32.
  • DeepSeek-V3 671B FP8 + native MTP: the model ships with MTP trained in; there is nothing to add. The speedup is 1.5-1.8× with zero extra VRAM. It is the most efficient option operationally: zero additional pieces.
  • Combining with disaggregated serving: as Disaggregated serving explains, prefill and decode can live in separate pods. Speculative applies only in the decode pods, which fits the separation perfectly (prefill is compute-bound and would not benefit).

The rule of thumb on an H100 cluster in May 2026: if the model is DeepSeek-V3 / V4 → native MTP, nothing else; if it is Llama 3 / Qwen 3 → EAGLE-3 with the official draft; if it is exotic → vanilla SD with a draft from the same family.

What we have not covered

  • Training custom EAGLE drafts with SpecForge: how to collect target trajectories and train the draft on-policy.
  • Speculative Prefill (arXiv:2502.02789): a variant for accelerating TTFT, a different mechanism from the decode one described here.
  • Tree attention in detail: how Medusa and EAGLE-2 verify several candidates at once with specific attention masks.
  • MoE + speculative: the combination has non-trivial interactions with the expert router. Low activated params keep the memory-bound regime in place even at high batch, which changes the rules.

See also

  • KV cache: the working memory that holds up LLM inference — speculative would not exist without the memory-bound nature of decode, which is a direct consequence of the KV cache; that post gives the framework.
  • PagedAttention deep dive — vLLM’s scheduler has to manage the nested raggedness of the γ tokens accepted per request; PagedAttention is what makes that possible without reserving fixed blocks per session.
  • Quantisation for LLM inference — a lever orthogonal to and multiplicative with speculative; that post explains why the draft is usually left in BF16 even when the target is in FP8/INT4.
  • Disaggregated serving: prefill and decode in specialised pods — speculative applies only in decode; disaggregation makes that specialisation easy without touching prefill.
  • MoE inference: the call centre with 256 specialists — the persistent memory-bound regime of MoE means speculative gains more on MoE than on dense at medium batch. MTP in DeepSeek-V3 is speculative decoding native to the model (no external draft) with acceptance around 85-90 % on the second token.
  • Continuous batching — the scheduler where speculative lives. Speculative breaks the symmetry of the batch (each request accepts between 1 and γ+1 tokens per iteration); at high QPS it can reduce goodput if the draft consumes slots from the decode pool.
  • The six-stage LLMOps pipeline — the master map where Deploy is stage 4.
  • Mixed NVIDIA + Intel environments — the “drafter near edge on an Intel NUC + target on a central H100” pattern as the canonical case of speculative decoding deployed heterogeneously.
  • Optimising decode in vLLM — the concrete vLLM parameters for turning on speculative decoding in production (--speculative-model, --num-speculative-tokens) with reference configs for RTX 4090 and L40.
  • Knowledge distillation — the best drafters are not small versions of the base model: they are students distilled specifically to predict the verifier’s distribution; distillation explains why EAGLE beats a plain generic 0.5B.
  • Pruning LLM models — an alternative to the distilled drafter: a pruned draft model (layer dropping from the base) as a cheap approximation of the verifier; it works worse than EAGLE but needs no extra training.
  • Self-speculative decoding: the model that gets ahead of itself — the variant without a separate draft: the model itself run in early-exit acts as the draft and is verified with the full forward pass, zero extra VRAM. It is the form of speculative that fits small models and on-device deployment.

References