Speculative-Decoding on lo0 — Tech Blog

Self-speculative decoding: the model that gets ahead of itself

Tue, 09 Jun 2026 01:40:00 +0000

This post is the direct companion to Speculative decoding: the secretary who types ahead. There, draft and target are two different models; here they are the same model at two depths. Read that one first: we take rejection sampling, the 1/(1-α) ceiling and the speedup formula as known, and all we change here is what the draft is.

TL;DR

Classic speculative decoding demands a pair: a cheap draft model proposes γ tokens and an expensive target verifies them in a single parallel forward pass. On large models the draft can be 1 % of the target and fit comfortably. On small models (SLMs, 1B–8B) that recipe breaks on two fronts: a draft that is 1/10 of a 3B is a 0.3B that barely gets anything right (α collapses), and loading a second model, however small, doubles the pieces to maintain and eats VRAM that is not spare on a 4090 or on device. Self-speculative decoding solves both: the draft is the model itself, run shallowly. A model with L layers produces draft tokens by exiting at an intermediate layer k < L (early-exit) or by skipping a subset of layers (layer-skip), and then verifies those tokens with the full forward pass through the L layers. Because draft and verify share weights and share the KV cache of the common layers, the extra memory cost is zero: there is no second model, no second KV cache, nothing new to load. The price is that the early-exit draft is more expensive than a tiny external draft (it walks k/L of the model instead of 1 %), so the relative cost c goes up. The honest trade-off: with a well-trained dedicated draft (EAGLE-3) that fits in memory, its α is usually higher and it wins; self-spec wins when there is no trained draft, when it does not fit, or when you are on device.

The analogy: the chess player who plays by eye and then calculates

A good chess player does two things with the same brain. First they look at the board and, by eye, in half a second, propose a move that “looks right”: pattern intuition, fast recognition, the shallow layers of judgement. Then, before moving, they calculate properly: three moves ahead, the opponent’s replies, the tactical lines. That deep calculation confirms the intuition or corrects it.

The decisive part is that it is the same person acting as drafter and as reviewer. They do not hire a second, weaker player to guess the move so they can validate it; that would be classic speculative with an external draft. Here the fast draft and the slow verification come out of the same brain, traversed at two depths.

The analogy holds point by point:

The glance by eye is the early-exit forward pass: the model walks only the first k layers and emits a draft token. Fast, approximate.
The deep calculation is the full forward pass through the L layers, which verifies the draft with exact rejection sampling.
That it is the same person is the reuse of weights and KV cache: the k shallow layers of the draft are literally the same as the first k layers of the verify; what has already been computed is not recomputed.
That the final move is identical to the one the player would have chosen by always calculating deeply is the rejection sampling guarantee: output quality does not degrade (the proof is in the speculative post).

Why the external draft does not fit small models

Let us revisit the cost of classic speculative with two numbers. The speedup depends on the acceptance rate α (how often the draft is right) and on the relative cost c = T_draft / T_target. A useful draft needs high α and low c at the same time. On large models that is reachable: a 1B draft for a 70B target has c ≈ 0.015 and, if well distilled (EAGLE), α > 0.8. The product pays off.

On a small model the balance breaks:

The proportional draft is useless. If you want c ≈ 0.1 for a 3B target, your draft is around 0.3B. A generic 0.3B has a distribution so different from the 3B that α falls into the 0.3–0.5 range. And 1/(1-α) with α = 0.4 is a ceiling of 1.67 tokens/step: not even infinite γ gets you more. The prize evaporates.
Loading a second model doubles the pieces. Even if the draft is small in VRAM, it is another checkpoint to version, quantise, validate and serve, and it has its own KV cache. On an RTX 4090 (24 GB, Ada Lovelace) with a quantised 8B and a long context, the KV cache is already tight; adding a second model and its cache can force you to lower concurrency or maximum context. On device (a phone, a NUC, an edge box) there is simply no room.
A trained draft does not always exist for your exotic or fine-tuned model. EAGLE needs the draft trained on-policy against that specific target (see knowledge distillation). If your SLM is your own fine-tune, there is no official published draft.

Self-speculative attacks all three at once with one idea: do not bring a second model; use the first one at half depth.

The mechanism: early-exit as draft, full forward as verify

A transformer with L layers transforms the hidden state layer by layer at each position: h_0 → h_1 → ... → h_L, and the LM head projects h_L to logits. The observation that enables everything: h_k for k < L is already a reasonable hidden state. If you push it through the same LM head (or through a dedicated lightweight head), you get an output distribution that is “premature” but often correct for easy tokens. That is the source of the draft.

The self-speculative iteration has the same structure as classic speculative (draft, verify, accept/reject) but both roles are the same model:

Step 1 — Shallow draft. To produce γ draft tokens, the model walks only the first k layers (or a subset of layers in the layer-skip case) and applies the LM head. Each draft token costs ≈ k/L of a full forward pass. We call c = k/L the relative cost of the draft. The γ drafts are generated autoregressively at this reduced cost.

Step 2 — Full verify. The model runs a single forward pass through the L layers over prompt + x_1...x_γ. Because attention is causal it obtains p(·|prompt, x_<i) for each position, exactly as in classic speculative.

Step 3 — Accept/reject. Rejection sampling identical to the previous post: tokens are accepted from left to right, the first divergence is corrected by sampling from the residual norm(max(0, p−q)), and if all γ are accepted the bonus token is appended. Output quality is exactly that of the full model.

The trick that makes `c` cheaper still: reusing the KV cache of shared layers

Here is the key difference against an external draft. When the model drafts by walking layers 0..k, it computes and stores the KV cache of those k layers for the prompt tokens and the drafts. When the full verify comes along, layers 0..k of the L-layer forward pass are bit for bit the same operations on the same weights that the draft already performed. There is no need to recompute them: the verify reuses the KV cache the draft left for layers 0..k directly, and only genuinely computes the missing layers k..L.

That has two consequences:

Zero extra memory. There is no second KV cache. The KV of the common layers is a single one, shared between draft and verify. Contrast that with vanilla SD, where the draft has its own complete cache (see KV cache).
Partially reused compute. The verify only pays for the “new” layers k..L for tokens that already went through the draft. The full forward pass is not as expensive as L suggests, because the first k layers come from the cache.

The families (state of play 2026)

There is no single way of doing self-speculative. They differ in which layers are skipped and in whether training is needed.

Family	Year / venue	How it picks what to skip	Training?	Distinctive idea
LayerSkip (Elhoushi et al.)	2024, arXiv:2404.16710	Early-exit at a fixed layer `k`; a single LM head serves every exit	Yes — layer dropout + early-exit loss in train/fine-tune	A single model trained to do both draft and verify; reuses partial compute
SWIFT	ICLR 2025 (OpenReview EKJhH5D5wA)	Selects which layers to skip on the fly, without touching weights	No — plug-and-play on the given model	Training-free self-spec: optimises the set of skipped layers at runtime
CLaSp	2025, arXiv:2505.24196	Dynamic in-context layer skip: the pattern of skipped layers adapts to the context	No (dynamic at inference)	The skip is not fixed; it changes with what is being generated
ConfLayers	2026, arXiv:2604.14612	Skips layers according to the confidence of the intermediate state (adaptive per token)	No (confidence criterion)	Variable depth: easy tokens exit earlier, hard ones go deeper
Saguaro	2025–26	Asynchronous formulation: the draft keeps speculating in parallel while verification runs	Depends on the variant	Overlaps draft and verify in time instead of alternating them
SSD for on-device MoE	ACM Web Conf. 2026, doi 10.1145/3774904.3792218	Self-spec exploiting MoE sparsity (few active experts per token)	MoE-specific variant	The shallow draft activates even fewer experts; fits MoE on device

Three operational readings of the table:

The axis that matters most is training yes/no. LayerSkip gives the best α because the model learns to be a good shallow draft (with early-exit loss the intermediate layers are explicitly trained to predict well). But it demands a fine-tune. SWIFT, CLaSp and ConfLayers are training-free: worse α, but they apply to any already-trained model without touching anything. For an SLM you do not control, training-free is the realistic option.
Adaptive skip (CLaSp, ConfLayers) raises α because it fits the draft’s depth to the token: it spends little on the easy ones and more on the hard ones, instead of a fixed k. In exchange, the effective c stops being constant.
Saguaro attacks something else: it does not raise α, it overlaps draft and verify time. It is orthogonal to the rest and can be combined.

The maths: same framework, different `c`

We reuse the apparatus of the speculative post without changing a letter. With α the acceptance rate and γ the number of drafts:

$$E[\text{tokens per step}] = \frac{1 - \alpha^{\gamma+1}}{1 - \alpha}, \qquad \text{Speedup} = \frac{1 - \alpha^{\gamma+1}}{(1 - \alpha)(\gamma c + 1)}$$

And the algorithmic ceiling is the same: lim_{γ→∞} = 1/(1-α). The only thing that changes in self-speculative is the value of c: it is no longer the size ratio of two models, but c = k/L, the fraction of layers the early-exit draft walks.

Numerical example: self-spec with L=32, exit at k=8

Take an SLM with L = 32 layers that exits at k = 8 for the draft: c = k/L = 8/32 = 0.25. Suppose α = 0.7 (reasonable for early-exit on conversational tokens) and γ = 4.

Expected tokens per step: (1 − 0.7⁵) / (1 − 0.7) = (1 − 0.168) / 0.3 = 0.832 / 0.3 = 2.77
Speedup: 2.77 / (4 × 0.25 + 1) = 2.77 / 2.0 = 1.39×

The denominator factor is γc + 1 = 4·0.25 + 1 = 2.0: the early-exit draft, costing a quarter of the model per token, eats part of the benefit. Exiting higher up helps: with k = 4 (c = 0.125), the denominator is 1.5 and the speedup 2.77/1.5 = 1.85×. But exiting higher up normally lowers α, so there is real tension between a small k (cheap) and a high α (accurate).

An honest comparison with an external draft

Put a tiny, well-distilled external draft alongside: c = 0.1 and α = 0.78 (what an EAGLE-style draft can deliver), same γ = 4.

Tokens/step: (1 − 0.78⁵)/(1 − 0.78) = (1 − 0.289)/0.22 = 0.711/0.22 = 3.23
Speedup: 3.23 / (4 × 0.1 + 1) = 3.23 / 1.4 = 2.31×

Configuration	c	α	tokens/step	speedup	extra VRAM	pieces to maintain
Self-spec early-exit (k=8)	0.25	0.70	2.77	1.39×	0	0
Self-spec early-exit (k=4)	0.125	0.65	2.50	1.67×	0	0
Distilled external draft	0.10	0.78	3.23	2.31×	yes (+model +KV)	1 extra model

The reading is exactly what you would expect, and it is worth not dressing it up: if you have a dedicated draft, trained against your target, and it fits in memory, its higher α and lower c give it more speedup. EAGLE-3 with a well-trained draft usually wins on raw speedup. Self-spec does not compete on raw speedup; it competes on total cost. Its winning columns are the two on the right: zero extra VRAM and zero pieces to maintain. Self-spec wins when:

there is no trained draft for your model (your own SLM, an unusual fine-tune),
the draft does not fit (a 4090 already full, a long context that needs the KV),
you are on device (phone, NUC, edge), where a second model and its KV simply do not go in.

It is the same pattern as with MTP in the previous post: sometimes the best draft is the one you do not have to load.

Why it fits small models and device precisely

The regime where self-spec shines is low concurrency, memory-bandwidth-bound, with a tight memory budget, exactly that of an SLM on a single GPU or on device (the reason for the regime is in the inverted roofline). Three reasons:

Zero extra memory is decisive where there is none to spare. On an RTX 4090 (24 GB, Ada Lovelace) serving a quantised 7B–8B with a long context, every GB counts. Self-spec asks for none: it reuses weights and KV. An external draft, however small, forces you to trim context or concurrency. On device the difference is binary: with self-spec you accelerate; with an external draft there is no room, full stop.
There is no second checkpoint to version. Operationally, an SLM at the edge deployed across hundreds of boxes becomes unsustainable if each one needs two synchronised models. A single binary that does draft and verify is far simpler to maintain.
It fits MoE on device. In a fine-grained MoE for device (see native architectures for device), the shallow draft activates even fewer experts, and the memory-bound regime persists even at medium batch, precisely what the work on SSD for on-device MoE (ACM WWW 2026) exploits.

The counterpoint, repeated so it does not get forgotten: on a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8), where memory is not the bottleneck, a dedicated EAGLE-3 draft does fit and its higher α gives it more speedup. There, self-spec is plan B: you use it if the model is exotic and there is no trained draft, not because memory is tight.

Pitfalls

α depends enormously on k. Exiting too high up (small k) makes the draft cheap but sinks α; exiting too low down (k close to L) raises α but the draft costs almost a full forward pass and c → 1, killing the speedup. The optimum is empirical and model-specific. Distrust any speedup number that does not say at which k it was measured.
Training-free is not free in draft quality. SWIFT and CLaSp give lower α than LayerSkip precisely because the model’s intermediate layers were not trained to be good premature exits. The number that matters is α measured on your distribution, not the paper’s.
Sampling temperature and creative outputs lower α just as in classic speculative. At high T, the self-spec speedup erodes faster still because it starts from a lower α.
Large batch neutralises it just as it does classic speculative. As soon as decode turns compute-bound, the drafts stop being “almost free”. Self-spec is for low concurrency.

References

Elhoushi, M., et al. LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding. 2024. https://arxiv.org/abs/2404.16710
SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration. ICLR 2025. https://openreview.net/forum?id=EKJhH5D5wA
CLaSp: In-Context Layer Skip for Self-Speculative Decoding. 2025. https://arxiv.org/abs/2505.24196
ConfLayers: Confidence-Adaptive Layer Skipping for Self-Speculative Decoding. 2026. https://arxiv.org/abs/2604.14612
Self-Speculative Decoding for MoE on Device. ACM Web Conference 2026. https://doi.org/10.1145/3774904.3792218
Hugging Face blog. Faster Text Generation with Self-Speculative Decoding. https://huggingface.co/blog/layerskip
Leviathan, Y., Kalman, M., Matias, Y. Fast Inference from Transformers via Speculative Decoding. ICML 2023. https://arxiv.org/abs/2211.17192

Optimising decode in vLLM: squeezing every token out of small hardware

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

Decode is the phase in which vLLM generates output tokens one at a time. It is memory-bound, not compute-bound: the GPU spends more time waiting for the weights to arrive from VRAM than doing arithmetic. On small hardware, an RTX 4090 (24 GB) or an L40 (48 GB), a badly configured decode wastes half the card’s capacity. Five vLLM parameters change the equation: gpu-memory-utilization, max-num-seqs, speculative decoding, FP8 KV cache and a swap-space correctly set to zero. Well calibrated, the difference is real: from 15 tokens/s to 35–50 tokens/s on the same hardware.

The analogy

An assembly-line worker putting cars together. Every car needs exactly the same process: walk to the store for the part, come back, bolt it on, repeat. The travel time to the store, the VRAM latency, is fixed and cannot be eliminated. But there are ways to make it less painful:

Have several cars in parallel on the line (more concurrency, the same travel time amortised).
Have an assistant prefabricate common parts (speculative decoding: the draft model proposes, the verifier confirms).
Keep only the most-used parts in the workshop (quantised KV cache: more contexts fit in the same space).

Those three strategies are exactly the three axes of decode optimisation in vLLM.

Why decode is memory-bound

During prefill, the GPU processes N tokens in parallel: the attention operation is a large matmul and the compute units are busy. During decode, it processes 1 token per step: the matmul becomes a vector-matrix product, an operation that underuses the tensor cores.

The typical compute utilisation ratio during decode on an RTX 4090:

$$\text{MFU}_{decode} \approx 5–15\% \quad \text{(vs 40–60\% in prefill)}$$

The bottleneck is not compute power, it is bandwidth. To generate each token, the model has to read its complete weights from VRAM:

$$\text{time per token} \approx \frac{\text{weight size in bytes}}{\text{VRAM bandwidth}}$$

For Qwen2.5-7B in BF16 (14 GB of weights) on an RTX 4090 (1,008 GB/s):

$$t \approx \frac{14 \times 10^9}{1.008 \times 10^{12}} \approx 13.9 \text{ ms/token} \approx 72 \text{ tokens/s theoretical maximum}$$

The real value is lower (~30–50 tok/s) because of scheduler overhead, attention over the growing KV cache and other latencies. But the theoretical limit sets the ceiling.

With Q4_K_M (weights ~4 GB):

$$t \approx \frac{4 \times 10^9}{1.008 \times 10^{12}} \approx 3.97 \text{ ms/token} \approx 252 \text{ tokens/s theoretical}$$

Quantising the model is the most direct way to improve decode throughput on memory-bound hardware. Everything else optimises on top of that ceiling.

The five levers

1. Give vLLM all the VRAM you can — `--gpu-memory-utilization`

--gpu-memory-utilization (short form --gpu-mem-util) defines the fraction of available VRAM that vLLM may use for the KV cache, once the model weights are loaded. The rest is reserved for activations during the forward pass and for the CUDA context.

vllm serve my-model \
 --gpu-memory-utilization 0.92

The default value is 0.90. On bare metal where no other process uses the GPU, 0.92–0.95 is safe. Do not go above 0.95: vLLM needs headroom for activations during batch peaks, and running out of VRAM in the middle of an inference results in a process crash, not a clean error.

Why it matters: more KV cache available = more simultaneous requests in flight = better GPU utilisation during decode. PagedAttention allocates the KV cache in fixed-size blocks (16 tokens per block by default), and vLLM manages them like virtual memory pages. The more blocks available, the more requests it can serve without any of them waiting for space.

RTX 4090, Qwen2.5-7B-BF16 (14 GB weights):
Total VRAM: 24 GB
Weights: 14 GB
Available for KV cache: 10 GB
gpu-memory-utilization 0.90 → 0.90 × 10 GB = 9 GB for KV cache
gpu-memory-utilization 0.94 → 0.94 × 10 GB = 9.4 GB → ~4% more tokens in flight

The impact is modest with models that fit comfortably, but it is amplified with models that push VRAM to its limit.

2. Real concurrency — `--max-num-seqs`

--max-num-seqs is the maximum number of requests vLLM can have in process simultaneously (counting prefill and decode together). It is the parameter that controls the system’s effective concurrency.

vllm serve my-model \
 --max-num-seqs 128

The effect is direct: more requests in simultaneous decode = better amortisation of the fixed cost of reading the weights. When the decode batch grows from 1 to 8, the time to generate 8 tokens is almost the same as generating 1 (the weights are read once for all of them). Aggregate throughput scales almost linearly until the KV cache or the activation VRAM becomes the bottleneck.

$$\text{aggregate throughput}(B) \approx B \times \text{throughput}(1) \quad \text{for } B \ll B_{max}$$

Common mistake: raising --max-num-seqs without making sure there is enough KV cache in VRAM for all the requests. If vLLM cannot hold the KV caches of 128 simultaneous requests, it does preemption (pauses some request and frees its KV cache) at a latency cost. Monitor vllm:num_preemptions_total.

Interaction with --max-num-batched-tokens: the vLLM scheduler processes up to max-num-batched-tokens tokens per step. If you have 128 requests in decode generating 1 token each, that is 128 decode tokens. The decode budget consumes 128 tokens of the total budget; the rest goes to prefill in chunks. Tune both values together.

# For an RTX 4090 serving ~50 concurrent users with responses of up to 512 tokens
vllm serve my-model \
 --max-num-seqs 64 \
 --max-num-batched-tokens 8192
# 64 decode tokens per step + up to 8128 chunked prefill tokens

3. Speculative decoding — `--speculative-model` + `--num-speculative-tokens`

Speculative decoding is the highest-impact change for decode on small hardware. The idea is simple: a small draft model proposes several tokens at once, and the verifier model validates or rejects them in a single forward pass.

vllm serve Qwen/Qwen2.5-7B-Instruct \
 --speculative-model Qwen/Qwen2.5-0.5B-Instruct \
 --num-speculative-tokens 5 \
 --speculative-draft-tensor-parallel-size 1

Why it works: the 7B verifier has to read 14 GB of weights per step. With 5 proposed tokens, if the acceptance rate is 80%, an average of 4–5 tokens are generated per verifier step instead of 1. Effective throughput goes up without the GPU working any harder.

The acceptance rate (α) depends on how well the draft predicts the verifier’s distribution. Within the same domain, models from the same family usually have α > 0.75:

$$\text{speedup} \approx \frac{1 + \alpha \cdot k}{1 + \alpha \cdot k / \text{cost ratio}}$$

Where $k$ is the number of proposed tokens and cost_ratio is the draft/verifier cost ratio. For a 0.5B draft and a 7B verifier (ratio ~14×):

$$\text{speedup} \approx 1 + 0.8 \times 5 \approx 5 \text{ (theoretical maximum, not reachable)}$$

In practice, with α = 0.75 and k = 5 on hardware without NVLink: 1.8–2.5× more tokens/s compared with decode alone.

EAGLE-3 in 2026: the best drafters today are not small versions of the same model, but networks specialised in predicting the verifier’s distribution. EAGLE-3 reports a 3–6.5× speedup over vanilla decode in public benchmarks. In production with mixed batches the real speedup is more conservative (1.5–3×). vLLM supports EAGLE/EAGLE-2 via --speculative-model:

# With an EAGLE drafter (requires a drafter trained specifically for the base model)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
 --speculative-model yuhuili/EAGLE3-LLaMA3.1-Instruct-8B \
 --num-speculative-tokens 6

When speculative decoding does NOT help:

Very large batches (>32 requests): the acceptance rate varies between requests and the batch spends more time re-drafting than accepting.
High-entropy tasks (brainstorming, very creative code): the draft predicts worse, α falls below 0.5 and the draft overhead outweighs the gain.
If the draft model does not fit in the available VRAM alongside the verifier.

On an RTX 4090 with a 7B verifier and a 0.5B draft (BF16): 14 + 1 GB = 15 GB. That leaves 9 GB for the KV cache. It works.

4. Quantised KV cache — `--kv-cache-dtype fp8`

This was already covered in the prefill article for its effect on context capacity. From the decode point of view, the benefit is different: more tokens fit in the KV cache → more simultaneous requests without preemption → better aggregate throughput.

vllm serve my-model \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales

Precision warning in decode: the KV cache is read on every attention step of the decode. Quantisation introduces noise into the attention activations. For long texts (>4K tokens of context) it can accumulate. In quality benchmarks (MMLU, HellaSwag) the degradation with FP8 KV and --calculate-kv-scales is <0.5% on modern models. Without --calculate-kv-scales, the degradation can be larger because the scales are fixed statically.

Optimal combination for RTX 4090:

vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \
 --quantization awq \  # weights in INT4: 4 GB model
 --kv-cache-dtype fp8 \  # KV cache at half the size
 --calculate-kv-scales \
 --gpu-memory-utilization 0.94

# Available VRAM: 24 - 4 = 20 GB for KV cache
# With FP8: ~40 KB/token (vs 80 KB BF16) → 20 GB / 40 KB = 500,000 tokens of total context
# With max-num-seqs 64 and a 4K ctx: 64 × 4096 × 40KB = 10 GB → fits with room to spare

5. Remove the swap — `--swap-space 0`

--swap-space defines how much system RAM (not VRAM) vLLM may use to preempt KV caches. When vLLM has more active requests than fit in VRAM, it can “pause” some by moving their KV cache to RAM and reactivating them later.

The problem: moving a 4K-token KV cache from VRAM to RAM and back has a latency of tens of milliseconds over PCIe. For a system where you want predictable latency, the swap introduces unacceptable jitter.

vllm serve my-model \
 --swap-space 0

With --swap-space 0, when vLLM cannot hold any more requests in VRAM, it simply queues them instead of doing preemption. The queue adds waiting latency, but it is predictable and it does not interrupt the requests already in flight.

When should you have swap? If your workload has short demand peaks and you can tolerate occasional jitter in exchange for not rejecting requests, a swap of 4–8 GB can be useful. In ENS deployments where latency is a contractual SLA, --swap-space 0 is the right option.

The reference configuration per hardware

RTX 4090 (24 GB) — 7B model, internal use

vllm serve Qwen/Qwen2.5-7B-Instruct \
 --gpu-memory-utilization 0.92 \
 --max-model-len 8192 \
 --max-num-seqs 64 \
 --max-num-batched-tokens 8192 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales \
 --swap-space 0 \
 --speculative-model Qwen/Qwen2.5-0.5B-Instruct \
 --num-speculative-tokens 5 \
 --speculative-draft-tensor-parallel-size 1 \
 --dtype bfloat16

Expected throughput: 35–55 tokens/s per user, up to 64 simultaneous, TTFT <500ms for prompts <1K tokens.

L40 (48 GB) — 14B model, multi-user

vllm serve Qwen/Qwen2.5-14B-Instruct \
 --gpu-memory-utilization 0.90 \
 --max-model-len 16384 \
 --max-num-seqs 128 \
 --max-num-batched-tokens 16384 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales \
 --swap-space 0 \
 --speculative-model Qwen/Qwen2.5-1.5B-Instruct \
 --num-speculative-tokens 5 \
 --dtype bfloat16

Expected throughput: 25–40 tokens/s per user, up to 128 simultaneous with speculative decoding active, TTFT <800ms for prompts <2K tokens.

How to measure that decode is optimised

# Key metrics at vllm:8000/metrics

vllm:generation_tokens_total # total tokens generated → trend
vllm:e2e_request_latency_seconds_* # end-to-end latency per percentile
vllm:time_per_output_token_seconds_* # ITL (inter-token latency)
vllm:num_preemptions_total # if it rises, the KV cache is filling up
vllm:spec_decode_draft_acceptance_rate # speculative decoding hit rate

If spec_decode_draft_acceptance_rate < 0.6, the drafter is not helping: turn speculative decoding off or find a drafter better trained for your model or domain.

If num_preemptions_total grows, you have too many simultaneous requests for the available KV cache. Options: lower max-num-seqs, enable FP8 KV cache, lower max-model-len, or quantise the model further.

Implications for on-premise inference

In a sovereign deployment with fixed hardware, you cannot buy more GPUs at will. Every well-calibrated tenth of gpu-memory-utilization, every point of speculative decoding acceptance rate and every MB of KV cache freed by FP8 is real capacity you do not have to provision with another node.

The combination of quantised weights (AWQ/GPTQ), FP8 KV cache and speculative decoding lets a 14B serve on one L40 what without optimisations would need two L40s in tensor parallel. That is the economic argument for investing time in these parameters.

Decode cannot be accelerated indefinitely on memory-bound hardware: the theoretical limit is set by VRAM bandwidth. But the difference between the minimum and the maximum achievable on that hardware can be 3–4× with the right levers.

References

Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023
EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees — Li et al., 2024
Fast Inference from Transformers via Speculative Decoding — Leviathan et al., 2022
vLLM Optimization and Tuning — documentación oficial
vLLM Speculative Decoding
FP8 KV Cache en vLLM

Speculative decoding: the secretary who types ahead of the boss — fundamentals, maths and state of play in May 2026

Fri, 29 May 2026 11:00:00 +0200

This post complements KV cache: the working memory and Quantisation for LLM inference. Both are acceleration levers inside a single forward pass. Speculative decoding is the complementary lever: instead of making each forward pass cheaper, it tries to produce more tokens per forward pass. It is orthogonal to quantisation and to the KV cache, and it stacks multiplicatively with both.

You are here: DEPLOY

TL;DR

Autoregressive LLM inference has a structural problem: every new token needs a complete forward pass of the model, and decode forward passes are memory-bandwidth-bound, not compute-bound (the detail is in KV cache). The GPU spends most of its time waiting for the next weight to arrive from HBM. Speculative decoding uses that dead time by doing two things in parallel: a small, cheap model (draft) generates γ tokens autoregressively, fast but imprecise, and the large model (target) verifies those γ tokens in a single parallel forward pass, at almost the same cost as a single-token forward pass. An acceptance rule based on rejection sampling decides how many draft tokens are accepted, and the maths proves, it does not approximate, it proves, that the output distribution matches sampling directly from the target exactly. The technique has a ceiling: you never generate more than 1/(1-α) tokens per step, with α the acceptance rate. The five dominant families in May 2026 are vanilla SD (Leviathan 2023), Medusa (extra parallel heads), EAGLE-1/2/3 (a draft that operates at the level of hidden states), MTP (multi-token prediction native to DeepSeek-V3) and P-EAGLE (a draft that produces the γ tokens in a single forward pass, integrated in vLLM 0.16+). On low-concurrency workloads with short prompts and long outputs, the typical case of the on-premise conversational assistant, the real speedup sits between 2× and 4× on modern hardware.

The analogy: the secretary who types ahead and the boss who validates

Picture a crowded press conference with a real-time transcription setup. Two people are working:

The secretary sits at the keyboard. They know the subject, they have read the briefings and they know the boss’s verbal tics. As soon as the boss opens their mouth, the secretary starts typing what they think is coming. They are fast, three or four words ahead of the boss, but they sometimes get it wrong, especially when the boss takes an unexpected turn.

The boss periodically reads the secretary’s screen. They read a whole block at once, three or four words, and mentally compare what is written with what they would have said. If it matches their intent, they leave the text and carry on. If the secretary drifted at some point, they correct right there, and whatever the secretary typed after the divergence is discarded (it could be wrong because of the earlier error). If the whole block was fine, the boss takes the chance to add one more word while correcting.

The result: the final text is identical to what the boss would have dictated alone, but it comes out faster because the secretary works ahead.

The analogy holds on four details:

The secretary is the draft model. Small, cheap, fast, approximate.
The boss is the target model. Slow, expensive, but the only authority on output quality.
The block the boss reads at once is the target’s parallel forward pass over prompt + γ draft tokens. Verifying γ tokens costs almost the same as generating one, because the bottleneck is loading the weights from HBM, not the operations.
The “correct right where it diverged” rule is the rejection sampling that preserves the target distribution.

From here we go into the mechanism and into why quality is preserved exactly, not approximately.

The bare mechanism

Call p the target distribution (what it would produce if it were the only one speaking) and q the draft’s. The speculative decoding iteration has three steps:

Step 1 — Draft. The draft generates γ tokens autoregressively: x_1, x_2, ..., x_γ. Each one through its own forward pass. Since the draft is small, the γ steps cost little. Typically γ ∈ [4, 8].

Step 2 — Verify. The target runs a single forward pass over the complete sequence prompt + x_1 ... x_γ. Because attention is causal, that forward pass simultaneously produces the distributions p(·|prompt, x_<i) for each position i. In other words, in a single step it obtains the verification of the γ tokens and, on top of that, an extra distribution p(·|prompt, x_1...x_γ) for the next token.

Step 3 — Accept/reject token by token. For each x_i from left to right it applies:

If p(x_i) ≥ q(x_i): always accept.
If p(x_i) < q(x_i): accept with probability p(x_i)/q(x_i).
If it rejects: stop, sample a replacement token from the normalised residual distribution norm(max(0, p − q)), and discard the rest.
If it reaches the end having accepted the γ tokens: append a bonus token sampled directly from p(·|prompt, x_1...x_γ), which is already in the logits of the target’s forward pass.

Result per iteration: between 1 and γ+1 new tokens. In the best case (all accepted + bonus) γ+1 tokens are generated at the cost of a single target forward pass plus γ draft forward passes, against γ+1 target forward passes in the version without speculative.

Why quality does not degrade (the proof)

This is the part of the technique that confuses people most the first time. It looks like magic: how can sampling from an arbitrary draft and then “validating” produce exactly the same distribution as sampling from the original target?

The proof fits in two lines. For any token x, the final probability of emitting it is the sum of two disjoint events: the draft proposes it and it is accepted, or the draft proposes something else, it is rejected, and sampling from the residual yields x. Formally:

$$P(\text{emit } x) = q(x) \cdot \min\!\left(1, \frac{p(x)}{q(x)}\right) + P(\text{reject}) \cdot \frac{\max(0, p(x)-q(x))}{\sum_y \max(0, p(y)-q(y))}$$

The first term is min(p(x), q(x)). The second is max(0, p(x) − q(x)) (the total rejected mass Σ max(0, p−q) cancels the denominator exactly). Adding them: min(p(x), q(x)) + max(0, p(x) − q(x)) = p(x). The output is statistically indistinguishable from sampling directly from the target, down to floating-point numerical differences.

This matters operationally: it means speculative decoding is not perceptual compression, it is not a quality-speed trade-off, it does not require extra validation with evals. If the target would have passed a given eval, the system with speculative passes it too.

The maths that matters: ceiling and speedup

Call α the acceptance rate: the expected probability that an individual draft token is accepted. Assuming that acceptances of consecutive tokens are independent (a reasonable approximation in practice), the expected number of tokens generated per iteration is:

$$E[\text{tokens per step}] = \sum_{k=0}^{\gamma} \alpha^k + \alpha^{\gamma+1} = \frac{1 - \alpha^{\gamma+1}}{1 - \alpha}$$

And the theoretical speedup relative to generating one token per target iteration, with c = T_draft / T_target the relative cost of the draft, is:

$$\text{Speedup} = \frac{1 - \alpha^{\gamma+1}}{(1 - \alpha)(\gamma c + 1)}$$

There is a ceiling that many implementations try to beat and cannot: as γ → ∞, the speedup converges to:

$$\lim_{\gamma \to \infty} E[\text{tokens per step}] = \frac{1}{1 - \alpha}$$

It is an algorithmic ceiling, not a hardware one. With α = 0.7 you never generate more than 3.33 tokens per iteration no matter how much the hardware improves. With α = 0.8 → 5. With α = 0.9 → 10. That is why EAGLE-3, which targets α > 0.8 on many benchmarks, is not an incremental improvement over vanilla SD (α ≈ 0.5-0.6): it is a change of regime, because it raises the ceiling instead of merely getting closer to it.

A concrete numerical example, Llama 3 70B as target with a draft giving α = 0.75, γ = 5, c = 0.1:

Expected tokens per step: (1 − 0.75⁶) / (1 − 0.75) = (1 − 0.178) / 0.25 = 3.29
Speedup: 3.29 / (5 × 0.1 + 1) = 3.29 / 1.5 = 2.19×

If we raise α to 0.85 with the same configuration: tokens per step = (1 − 0.85⁶) / 0.15 = (1 − 0.377) / 0.15 = 4.16, speedup = 4.16 / 1.5 = 2.77×. A change of α = 0.75 → 0.85 (10 absolute points) raises the speedup by 27 %. That is why the research of the last two years focuses on pushing α: that is where the prize is.

The five modern families (May 2026)

Family	Year	Core idea	Draft size	Typical α / τ	Speedup
Vanilla SD	2023	Same-family target + draft pair, rejection sampling	1/10 – 1/100 of target	0.5 – 0.8	2-3×
Medusa	2024	N extra heads in parallel predict t+1, t+2, …; tree attention verifies several candidates	no draft, +1-2 % params	top-5 > 0.8	2.18-2.83×
EAGLE-1/2/3	2024-25	Autoregressive draft at the level of features (hidden states), not tokens. Reuses the target’s embedding	1 transformer block (~0.5-2 % params)	EAGLE-3: α > 0.8, τ up to 7.5	up to 6.5× peak
MTP (DeepSeek-V3)	2024	Multi-token heads trained from scratch as part of the model; at inference they act as a “free” draft	14B params in V3 671B	α > 0.8 (MTP1)	1.5-1.8×
P-EAGLE	2026	EAGLE but producing the γ drafts in a single forward pass (parallel, not autoregressive)	same as EAGLE	+30 % over EAGLE-3	4-5× vs AR

Three operational observations:

EAGLE dominates in production because its overhead is minimal (one transformer block, ~1 % of the target in parameters) and its α is high. The draft does not need its own “complete” KV cache because it shares features with the target.
Medusa mattered historically, it showed that speculative was possible without a separate draft, but EAGLE beat it on every published benchmark through 2024-2025.
MTP is special: it is not something you add to an existing model. It is something the model trained natively. If you buy DeepSeek-V3, MTP comes free and gives around 1.8× without touching anything. If you buy Llama 3, there is no MTP to be had; use EAGLE-3.

There are also two related techniques that deserve a brief mention, both without a draft model: Lookahead decoding (Fu et al. 2024), which formulates decoding as Jacobi iteration and extracts n-grams from the trajectory; and REST (He et al. 2024), which keeps an n-gram datastore and proposes drafts by longest-prefix match against the last generated tokens. Both give 1.5-2× with no extra VRAM. Useful when you have no trained draft model and do not want to maintain one.

Real implementations in May 2026

vLLM v0.16+: unified support for EAGLE-1/2/3, P-EAGLE, Medusa, native MTP (DeepSeek-V3 and variants), n-gram/suffix decoding without a draft, an arbitrary draft model and MLP speculators. The canonical flag is --speculative-config '{"method":"eagle3", "model":"...", "num_speculative_tokens": 5}'.
SGLang: native EAGLE-3 support with --speculative-algorithm EAGLE3. For DeepSeek-V3 it uses MTP through an EAGLE adapter. It has its own draft training framework (SpecForge).
TensorRT-LLM: Medusa, EAGLE (a simplified variant without a tree), ReDrafter, Lookahead. They report around 2.2× with EAGLE.
llama.cpp: basic speculative only with a draft model (--model-draft). No native EAGLE/Medusa/MTP as far as I have verified. Typical speedup 1.5-2.5× single-user.

When speculative does NOT help

The technique has three important blind spots:

Large batch. The GPU moves from memory-bound (decode at low concurrency) to compute-bound (decode at high concurrency). In the compute-bound regime the “almost free” target forward passes stop being free: γ tokens start costing γ times more instead of nearly 1. The typical crossover is at batch 16-32 for dense models; for MoE with few active parameters per token the crossover happens later. At load peaks, speculative can worsen aggregate throughput per GPU.

Prefill / TTFT. Speculative decoding does not touch prefill. The phase that processes the whole prompt stays identical. TTFT does not improve and can get marginally worse because of the draft setup. If the SLA is TTFT-bound (assistants with short outputs, search, RAG with brief answers), this is not the right tool.

Short outputs. If the model generates 10-20 tokens and the answer is over, the fixed setup overhead is not amortised. Speculative shines with long outputs (300+ tokens): generative assistants, extended code completion, drafting.

Operational pitfalls

Extra VRAM for the draft. In vanilla SD, loading a complete draft plus its KV cache is expensive. Llama 3 8B in BF16 as the draft for a 70B is about 16 GB of weights plus KV cache. On an H100 80 GB with the target already nearly filling it, that can force you to shrink the target’s KV cache and lower maximum concurrency. EAGLE solves this: the draft is one transformer block (~1 GB for a 70B) and reuses the target’s features.

Draft quantisation. Quantising the draft to INT4 with GPTQ degrades α substantially (errors in the logits accumulate in the comparison against p). AWQ holds up better but also lowers α. Common practice in 2026: target in FP8 or INT4, draft in FP16/BF16. The draft is small enough that the VRAM saved by quantising it does not compensate for the drop in α and therefore in speedup. The detail of each format is in Quantisation.

Interaction with continuous batching. It does not break it, but it creates nested raggedness: each request in the batch can accept a different number of tokens on each iteration. vLLM’s PagedAttention absorbs it, but the scheduler loses efficiency. At low QPS (conversational assistant, low simultaneous concurrency) the combination is excellent. At high QPS there is real tension, and work such as Goodput-optimized speculative decoding (Liu et al., 2024) optimises γ dynamically according to the state of the batch.

Sampling temperature. α falls with high temperatures. At T = 1.0 with creative outputs (free-form writing), α can drop 10-15 points relative to T = 0 with the same model-draft pair. The speedup scales accordingly.

Implications on on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace)

The classic use is vanilla SD with two quantised models: for instance, Llama 3 70B AWQ-INT4 as target (~35 GB → needs TP=2 across two 4090s) and Llama 3 8B AWQ-INT4 as draft. In practice, the most realistic single-card case is Llama 3 8B target + a 1B model as draft or Qwen 3 14B target + Qwen 3 0.5B as draft: they fit comfortably and give 1.8-2.5× on conversational tasks. For EAGLE on consumer cards, the official drafts for popular families (Llama 3, Qwen 3) are published on Hugging Face and take up 0.5-2 GB extra.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here EAGLE-3 shines:

Llama 3 70B FP8 + EAGLE-3 draft in FP16: the draft takes about 0.7 GB; the target about 70 GB with TP=2 (35 GB per GPU). The speedup observed in reproducible benchmarks is between 2.5× and 4× at batch 1-4, falling to almost break-even at batch 32.
DeepSeek-V3 671B FP8 + native MTP: the model ships with MTP trained in; there is nothing to add. The speedup is 1.5-1.8× with zero extra VRAM. It is the most efficient option operationally: zero additional pieces.
Combining with disaggregated serving: as Disaggregated serving explains, prefill and decode can live in separate pods. Speculative applies only in the decode pods, which fits the separation perfectly (prefill is compute-bound and would not benefit).

The rule of thumb on an H100 cluster in May 2026: if the model is DeepSeek-V3 / V4 → native MTP, nothing else; if it is Llama 3 / Qwen 3 → EAGLE-3 with the official draft; if it is exotic → vanilla SD with a draft from the same family.

What we have not covered

Training custom EAGLE drafts with SpecForge: how to collect target trajectories and train the draft on-policy.
Speculative Prefill (arXiv:2502.02789): a variant for accelerating TTFT, a different mechanism from the decode one described here.
Tree attention in detail: how Medusa and EAGLE-2 verify several candidates at once with specific attention masks.
MoE + speculative: the combination has non-trivial interactions with the expert router. Low activated params keep the memory-bound regime in place even at high batch, which changes the rules.

References

Leviathan, Y., Kalman, M., Matias, Y. Fast Inference from Transformers via Speculative Decoding. ICML 2023. https://arxiv.org/abs/2211.17192
Chen, C., Borgeaud, S., Irving, G., Lespiau, J.B., Sifre, L., Jumper, J. Accelerating Large Language Model Decoding with Speculative Sampling. DeepMind 2023. https://arxiv.org/abs/2302.01318
Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J.D., Chen, D., Dao, T. Medusa: Simple LLM Inference Acceleration with Multiple Decoding Heads. ICML 2024. https://arxiv.org/abs/2401.10774
Li, Y., Wei, F., Zhang, C., Zhang, H. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. ICML 2024. https://arxiv.org/abs/2401.15077
Li, Y., Wei, F., Zhang, C., Zhang, H. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. EMNLP 2024. https://arxiv.org/abs/2406.16858
Li, Y., Wei, F., Zhang, C., Zhang, H. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. NeurIPS 2025. https://arxiv.org/abs/2503.01840
DeepSeek-AI. DeepSeek-V3 Technical Report — Multi-Token Prediction. https://arxiv.org/abs/2412.19437
Fu, Y., Bailis, P., Stoica, I., Zhang, H. Break the Sequential Dependency of LLM Inference Using Lookahead Decoding. ICML 2024. https://arxiv.org/abs/2402.02057
He, Z., Zhong, Z., Cai, T., Lee, J.D., He, D. REST: Retrieval-Based Speculative Decoding. NAACL 2024. https://arxiv.org/abs/2311.08252
vLLM speculative decoding docs: https://docs.vllm.ai/en/latest/features/speculative_decoding/
SGLang speculative decoding docs: https://docs.sglang.ai/advanced_features/speculative_decoding.html
TensorRT-LLM speculative sampling: https://nvidia.github.io/TensorRT-LLM/advanced/speculative-decoding.html
vLLM blog Speculative Decoding in vLLM (oct 2024): https://blog.vllm.ai/2024/10/17/spec-decode.html
vLLM blog P-EAGLE: Parallel Speculative Decoding in vLLM (mar 2026): https://vllm.ai/blog/2026-03-13-p-eagle
Official EAGLE repo: https://github.com/SafeAILab/EAGLE
Official Medusa repo: https://github.com/FasterDecoding/Medusa

Speculative-Decoding on lo0 — Tech Blog

Self-speculative decoding: the model that gets ahead of itself

TL;DR

The analogy: the chess player who plays by eye and then calculates

Why the external draft does not fit small models

The mechanism: early-exit as draft, full forward as verify

The trick that makes c cheaper still: reusing the KV cache of shared layers

The families (state of play 2026)

The maths: same framework, different c

Numerical example: self-spec with L=32, exit at k=8

An honest comparison with an external draft

Why it fits small models and device precisely

Pitfalls

See also

References

Optimising decode in vLLM: squeezing every token out of small hardware

TL;DR

The analogy

Why decode is memory-bound

The five levers

1. Give vLLM all the VRAM you can — --gpu-memory-utilization

2. Real concurrency — --max-num-seqs

3. Speculative decoding — --speculative-model + --num-speculative-tokens

4. Quantised KV cache — --kv-cache-dtype fp8

5. Remove the swap — --swap-space 0

The reference configuration per hardware

RTX 4090 (24 GB) — 7B model, internal use

L40 (48 GB) — 14B model, multi-user

How to measure that decode is optimised

Implications for on-premise inference

See also

References

Speculative decoding: the secretary who types ahead of the boss — fundamentals, maths and state of play in May 2026

You are here: DEPLOY

TL;DR

The analogy: the secretary who types ahead and the boss who validates

The bare mechanism

Why quality does not degrade (the proof)

The maths that matters: ceiling and speedup

The five modern families (May 2026)

Real implementations in May 2026

When speculative does NOT help

Operational pitfalls

Implications on on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace)

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

What we have not covered

See also

References

The trick that makes `c` cheaper still: reusing the KV cache of shared layers

The maths: same framework, different `c`

1. Give vLLM all the VRAM you can — `--gpu-memory-utilization`

2. Real concurrency — `--max-num-seqs`

3. Speculative decoding — `--speculative-model` + `--num-speculative-tokens`

4. Quantised KV cache — `--kv-cache-dtype fp8`

5. Remove the swap — `--swap-space 0`