Self-speculative decoding: the model that gets ahead of itself

Contents

This post is the direct companion to Speculative decoding: the secretary who types ahead. There, draft and target are two different models; here they are the same model at two depths. Read that one first: we take rejection sampling, the 1/(1-α) ceiling and the speedup formula as known, and all we change here is what the draft is.

TL;DR

Classic speculative decoding demands a pair: a cheap draft model proposes γ tokens and an expensive target verifies them in a single parallel forward pass. On large models the draft can be 1 % of the target and fit comfortably. On small models (SLMs, 1B–8B) that recipe breaks on two fronts: a draft that is 1/10 of a 3B is a 0.3B that barely gets anything right (α collapses), and loading a second model, however small, doubles the pieces to maintain and eats VRAM that is not spare on a 4090 or on device. Self-speculative decoding solves both: the draft is the model itself, run shallowly. A model with L layers produces draft tokens by exiting at an intermediate layer k < L (early-exit) or by skipping a subset of layers (layer-skip), and then verifies those tokens with the full forward pass through the L layers. Because draft and verify share weights and share the KV cache of the common layers, the extra memory cost is zero: there is no second model, no second KV cache, nothing new to load. The price is that the early-exit draft is more expensive than a tiny external draft (it walks k/L of the model instead of 1 %), so the relative cost c goes up. The honest trade-off: with a well-trained dedicated draft (EAGLE-3) that fits in memory, its α is usually higher and it wins; self-spec wins when there is no trained draft, when it does not fit, or when you are on device.

The analogy: the chess player who plays by eye and then calculates

A good chess player does two things with the same brain. First they look at the board and, by eye, in half a second, propose a move that “looks right”: pattern intuition, fast recognition, the shallow layers of judgement. Then, before moving, they calculate properly: three moves ahead, the opponent’s replies, the tactical lines. That deep calculation confirms the intuition or corrects it.

The decisive part is that it is the same person acting as drafter and as reviewer. They do not hire a second, weaker player to guess the move so they can validate it; that would be classic speculative with an external draft. Here the fast draft and the slow verification come out of the same brain, traversed at two depths.

The analogy holds point by point:

The glance by eye is the early-exit forward pass: the model walks only the first k layers and emits a draft token. Fast, approximate.
The deep calculation is the full forward pass through the L layers, which verifies the draft with exact rejection sampling.
That it is the same person is the reuse of weights and KV cache: the k shallow layers of the draft are literally the same as the first k layers of the verify; what has already been computed is not recomputed.
That the final move is identical to the one the player would have chosen by always calculating deeply is the rejection sampling guarantee: output quality does not degrade (the proof is in the speculative post).

Why the external draft does not fit small models

Let us revisit the cost of classic speculative with two numbers. The speedup depends on the acceptance rate α (how often the draft is right) and on the relative cost c = T_draft / T_target. A useful draft needs high α and low c at the same time. On large models that is reachable: a 1B draft for a 70B target has c ≈ 0.015 and, if well distilled (EAGLE), α > 0.8. The product pays off.

On a small model the balance breaks:

The proportional draft is useless. If you want c ≈ 0.1 for a 3B target, your draft is around 0.3B. A generic 0.3B has a distribution so different from the 3B that α falls into the 0.3–0.5 range. And 1/(1-α) with α = 0.4 is a ceiling of 1.67 tokens/step: not even infinite γ gets you more. The prize evaporates.
Loading a second model doubles the pieces. Even if the draft is small in VRAM, it is another checkpoint to version, quantise, validate and serve, and it has its own KV cache. On an RTX 4090 (24 GB, Ada Lovelace) with a quantised 8B and a long context, the KV cache is already tight; adding a second model and its cache can force you to lower concurrency or maximum context. On device (a phone, a NUC, an edge box) there is simply no room.
A trained draft does not always exist for your exotic or fine-tuned model. EAGLE needs the draft trained on-policy against that specific target (see knowledge distillation). If your SLM is your own fine-tune, there is no official published draft.

Self-speculative attacks all three at once with one idea: do not bring a second model; use the first one at half depth.

The mechanism: early-exit as draft, full forward as verify

A transformer with L layers transforms the hidden state layer by layer at each position: h_0 → h_1 → ... → h_L, and the LM head projects h_L to logits. The observation that enables everything: h_k for k < L is already a reasonable hidden state. If you push it through the same LM head (or through a dedicated lightweight head), you get an output distribution that is “premature” but often correct for easy tokens. That is the source of the draft.

The self-speculative iteration has the same structure as classic speculative (draft, verify, accept/reject) but both roles are the same model:

Step 1 — Shallow draft. To produce γ draft tokens, the model walks only the first k layers (or a subset of layers in the layer-skip case) and applies the LM head. Each draft token costs ≈ k/L of a full forward pass. We call c = k/L the relative cost of the draft. The γ drafts are generated autoregressively at this reduced cost.

Step 2 — Full verify. The model runs a single forward pass through the L layers over prompt + x_1...x_γ. Because attention is causal it obtains p(·|prompt, x_<i) for each position, exactly as in classic speculative.

Step 3 — Accept/reject. Rejection sampling identical to the previous post: tokens are accepted from left to right, the first divergence is corrected by sampling from the residual norm(max(0, p−q)), and if all γ are accepted the bonus token is appended. Output quality is exactly that of the full model.

The trick that makes `c` cheaper still: reusing the KV cache of shared layers

Here is the key difference against an external draft. When the model drafts by walking layers 0..k, it computes and stores the KV cache of those k layers for the prompt tokens and the drafts. When the full verify comes along, layers 0..k of the L-layer forward pass are bit for bit the same operations on the same weights that the draft already performed. There is no need to recompute them: the verify reuses the KV cache the draft left for layers 0..k directly, and only genuinely computes the missing layers k..L.

That has two consequences:

Zero extra memory. There is no second KV cache. The KV of the common layers is a single one, shared between draft and verify. Contrast that with vanilla SD, where the draft has its own complete cache (see KV cache).
Partially reused compute. The verify only pays for the “new” layers k..L for tokens that already went through the draft. The full forward pass is not as expensive as L suggests, because the first k layers come from the cache.

The families (state of play 2026)

There is no single way of doing self-speculative. They differ in which layers are skipped and in whether training is needed.

Family	Year / venue	How it picks what to skip	Training?	Distinctive idea
LayerSkip (Elhoushi et al.)	2024, arXiv:2404.16710	Early-exit at a fixed layer `k`; a single LM head serves every exit	Yes — layer dropout + early-exit loss in train/fine-tune	A single model trained to do both draft and verify; reuses partial compute
SWIFT	ICLR 2025 (OpenReview EKJhH5D5wA)	Selects which layers to skip on the fly, without touching weights	No — plug-and-play on the given model	Training-free self-spec: optimises the set of skipped layers at runtime
CLaSp	2025, arXiv:2505.24196	Dynamic in-context layer skip: the pattern of skipped layers adapts to the context	No (dynamic at inference)	The skip is not fixed; it changes with what is being generated
ConfLayers	2026, arXiv:2604.14612	Skips layers according to the confidence of the intermediate state (adaptive per token)	No (confidence criterion)	Variable depth: easy tokens exit earlier, hard ones go deeper
Saguaro	2025–26	Asynchronous formulation: the draft keeps speculating in parallel while verification runs	Depends on the variant	Overlaps draft and verify in time instead of alternating them
SSD for on-device MoE	ACM Web Conf. 2026, doi 10.1145/3774904.3792218	Self-spec exploiting MoE sparsity (few active experts per token)	MoE-specific variant	The shallow draft activates even fewer experts; fits MoE on device

Three operational readings of the table:

The axis that matters most is training yes/no. LayerSkip gives the best α because the model learns to be a good shallow draft (with early-exit loss the intermediate layers are explicitly trained to predict well). But it demands a fine-tune. SWIFT, CLaSp and ConfLayers are training-free: worse α, but they apply to any already-trained model without touching anything. For an SLM you do not control, training-free is the realistic option.
Adaptive skip (CLaSp, ConfLayers) raises α because it fits the draft’s depth to the token: it spends little on the easy ones and more on the hard ones, instead of a fixed k. In exchange, the effective c stops being constant.
Saguaro attacks something else: it does not raise α, it overlaps draft and verify time. It is orthogonal to the rest and can be combined.

The maths: same framework, different `c`

We reuse the apparatus of the speculative post without changing a letter. With α the acceptance rate and γ the number of drafts:

$$E[\text{tokens per step}] = \frac{1 - \alpha^{\gamma+1}}{1 - \alpha}, \qquad \text{Speedup} = \frac{1 - \alpha^{\gamma+1}}{(1 - \alpha)(\gamma c + 1)}$$

And the algorithmic ceiling is the same: lim_{γ→∞} = 1/(1-α). The only thing that changes in self-speculative is the value of c: it is no longer the size ratio of two models, but c = k/L, the fraction of layers the early-exit draft walks.

Numerical example: self-spec with L=32, exit at k=8

Take an SLM with L = 32 layers that exits at k = 8 for the draft: c = k/L = 8/32 = 0.25. Suppose α = 0.7 (reasonable for early-exit on conversational tokens) and γ = 4.

Expected tokens per step: (1 − 0.7⁵) / (1 − 0.7) = (1 − 0.168) / 0.3 = 0.832 / 0.3 = 2.77
Speedup: 2.77 / (4 × 0.25 + 1) = 2.77 / 2.0 = 1.39×

The denominator factor is γc + 1 = 4·0.25 + 1 = 2.0: the early-exit draft, costing a quarter of the model per token, eats part of the benefit. Exiting higher up helps: with k = 4 (c = 0.125), the denominator is 1.5 and the speedup 2.77/1.5 = 1.85×. But exiting higher up normally lowers α, so there is real tension between a small k (cheap) and a high α (accurate).

An honest comparison with an external draft

Put a tiny, well-distilled external draft alongside: c = 0.1 and α = 0.78 (what an EAGLE-style draft can deliver), same γ = 4.

Tokens/step: (1 − 0.78⁵)/(1 − 0.78) = (1 − 0.289)/0.22 = 0.711/0.22 = 3.23
Speedup: 3.23 / (4 × 0.1 + 1) = 3.23 / 1.4 = 2.31×

Configuration	c	α	tokens/step	speedup	extra VRAM	pieces to maintain
Self-spec early-exit (k=8)	0.25	0.70	2.77	1.39×	0	0
Self-spec early-exit (k=4)	0.125	0.65	2.50	1.67×	0	0
Distilled external draft	0.10	0.78	3.23	2.31×	yes (+model +KV)	1 extra model

The reading is exactly what you would expect, and it is worth not dressing it up: if you have a dedicated draft, trained against your target, and it fits in memory, its higher α and lower c give it more speedup. EAGLE-3 with a well-trained draft usually wins on raw speedup. Self-spec does not compete on raw speedup; it competes on total cost. Its winning columns are the two on the right: zero extra VRAM and zero pieces to maintain. Self-spec wins when:

there is no trained draft for your model (your own SLM, an unusual fine-tune),
the draft does not fit (a 4090 already full, a long context that needs the KV),
you are on device (phone, NUC, edge), where a second model and its KV simply do not go in.

It is the same pattern as with MTP in the previous post: sometimes the best draft is the one you do not have to load.

Why it fits small models and device precisely

The regime where self-spec shines is low concurrency, memory-bandwidth-bound, with a tight memory budget, exactly that of an SLM on a single GPU or on device (the reason for the regime is in the inverted roofline). Three reasons:

Zero extra memory is decisive where there is none to spare. On an RTX 4090 (24 GB, Ada Lovelace) serving a quantised 7B–8B with a long context, every GB counts. Self-spec asks for none: it reuses weights and KV. An external draft, however small, forces you to trim context or concurrency. On device the difference is binary: with self-spec you accelerate; with an external draft there is no room, full stop.
There is no second checkpoint to version. Operationally, an SLM at the edge deployed across hundreds of boxes becomes unsustainable if each one needs two synchronised models. A single binary that does draft and verify is far simpler to maintain.
It fits MoE on device. In a fine-grained MoE for device (see native architectures for device), the shallow draft activates even fewer experts, and the memory-bound regime persists even at medium batch, precisely what the work on SSD for on-device MoE (ACM WWW 2026) exploits.

The counterpoint, repeated so it does not get forgotten: on a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8), where memory is not the bottleneck, a dedicated EAGLE-3 draft does fit and its higher α gives it more speedup. There, self-spec is plan B: you use it if the model is exotic and there is no trained draft, not because memory is tight.

Pitfalls

α depends enormously on k. Exiting too high up (small k) makes the draft cheap but sinks α; exiting too low down (k close to L) raises α but the draft costs almost a full forward pass and c → 1, killing the speedup. The optimum is empirical and model-specific. Distrust any speedup number that does not say at which k it was measured.
Training-free is not free in draft quality. SWIFT and CLaSp give lower α than LayerSkip precisely because the model’s intermediate layers were not trained to be good premature exits. The number that matters is α measured on your distribution, not the paper’s.
Sampling temperature and creative outputs lower α just as in classic speculative. At high T, the self-spec speedup erodes faster still because it starts from a lower α.
Large batch neutralises it just as it does classic speculative. As soon as decode turns compute-bound, the drafts stop being “almost free”. Self-spec is for low concurrency.

References

Elhoushi, M., et al. LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding. 2024. https://arxiv.org/abs/2404.16710
SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration. ICLR 2025. https://openreview.net/forum?id=EKJhH5D5wA
CLaSp: In-Context Layer Skip for Self-Speculative Decoding. 2025. https://arxiv.org/abs/2505.24196
ConfLayers: Confidence-Adaptive Layer Skipping for Self-Speculative Decoding. 2026. https://arxiv.org/abs/2604.14612
Self-Speculative Decoding for MoE on Device. ACM Web Conference 2026. https://doi.org/10.1145/3774904.3792218
Hugging Face blog. Faster Text Generation with Self-Speculative Decoding. https://huggingface.co/blog/layerskip
Leviathan, Y., Kalman, M., Matias, Y. Fast Inference from Transformers via Speculative Decoding. ICML 2023. https://arxiv.org/abs/2211.17192