Deepseek-V3 on lo0 — Tech Blog

MoE inference: the call centre with 256 specialists and 8 taking each call — fundamentals, expert parallel and the economics of DeepSeek-V3

Sat, 30 May 2026 08:00:00 +0200

This post goes into an architectural decision in the Deploy stage of the six-stage LLMOps pipeline. It complements those on KV cache (which does not change with MoE), FlashAttention (still the attention kernel, also dense in every current MoE), Quantization (FP8/NVFP4 over expert weights is what lets DeepSeek-V3 fit into a modest cluster) and Speculative decoding (MTP in DeepSeek-V3 is native speculative decoding).

You are here: DEPLOY

TL;DR

A Mixture of Experts (MoE) replaces the transformer’s dense FFN layer with N parallel experts plus a router that picks the k experts relevant to each token. The operational consequence is that the model’s total parameters (how many were trained and how many must be loaded into VRAM) decouple from the active parameters per token (how many take part in each forward). DeepSeek-V3 has 671 B total but only 37 B active per token; Qwen3-235B-A22B is 235 B total and 22 B active; Mixtral 8x22B is 141 B and 39 B; Llama 4 Maverick is 400 B and 17 B. Compute per token falls to 2·N_active FLOPs, half that of a dense Llama-3-70B in DeepSeek-V3’s case, but memory stays proportional to N_total: MoE sparsity is compute sparsity, not memory sparsity. Since each expert sees only batch·k/N tokens per step, the memory-bound regime persists at a much higher batch than in dense. Serving DeepSeek-V3 671 B FP8 at low latency requires expert parallel (EP) at scale, EP=32 in prefill and EP=144 in decode in DeepSeek’s production deployment, where the dominant bottleneck stops being HBM and becomes the all-to-all of dispatch and combine between GPUs. As of May 2026, the pieces that make this viable are DeepEP (FP8-native all-to-all CUDA kernels, intra-node NVLink plus inter-node RDMA), EPLB (balancing by replicating hot experts) and Wide-EP on NVL72 (72 GPUs in one coherent NVLink domain). This post takes apart the mechanism, the maths (FLOPs/token, all-to-all bytes), the updated table of MoE models as of May 2026, the parallelisms (TP / EP / DP+EP / Wide-EP), the operational pitfalls (memory wall, cold experts, imbalance) and the real reproducible numbers on H100, B200 and GB200 NVL72.

The analogy: the call centre with 256 specialists

A serious call centre handling complex technical queries runs on specialists, not generalists. You have 256 operators on the payroll, each an expert in their sub-domain: employment law, mortgages, income tax, probate, and so on. But each incoming call only needs the attention of eight specific specialists (top-8 routing): a client’s query about tax relief on a mortgage for an inherited second home involves the income tax person, the mortgage person, the capital gains person, and so on. The receptionist (the router) listens to the first second of the call, decides which eight specialists are relevant, conferences them in and combines their answers with weights.

Three immediate consequences, identical to those of MoE in production:

You pay 256 salaries (every expert is loaded in VRAM) but you only spend the time of 8 (only k experts take part in the forward of each token). Your payroll cost scales with N_total; your cost per minute of call scales with N_active.
If all the operators fit in one office (a single GPU), fine: paper just moves between desks. But if they do not fit, you have to open branches: the conference call between operators in different branches goes over a telephone line between buildings (intra-node NVLink, inter-node InfiniBand) and that line becomes the dominant bottleneck once dozens of branches are coordinating.
If one week every client asks about income tax, the three income tax operators saturate while the probate ones sit idle. You need a system that replicates the hot operators across several branches to absorb peaks, and that pairs hot with cold so that no branch goes idle.

The receptionist is the router. The branches are the GPUs. The conference call between branches is the all-to-all (dispatch plus combine). The system for replicating hot operators is EPLB. The good telephone line is DeepEP. And when someone designs a building where all 72 branches fit together with very fast internal cabling, that is NVL72.

The bare mechanism

A classic MoE layer replaces the dense FFN (y = FFN(x) = down(act(up(x) · gate(x))) in SwiGLU) with:

Router: a linear projection g(x) = W_router · x produces N scores. A softmax over those scores gives affinity probabilities.
Top-k selection: the k experts with the highest score are chosen. Their softmax_topk weights are normalised to sum to 1.
Parallel compute: each of the k selected experts runs its own FFN over x. The N - k not selected sit idle for this token.
Weighted combination: the output is y = Σ_{i ∈ topk} w_i · Expert_i(x).

Variants the market consolidated in 2025-2026:

Shared experts: on top of the k routed ones, there are s experts that always run (1 or 2). They capture general knowledge every token needs. DeepSeek-V3 uses 1 shared + 8 routed; Mixtral uses 0 shared.
Fine-grained experts: more small experts instead of a few large ones (DeepSeek: 256 vs Mixtral: 8). This allows better specialisation because the top-k combination covers more sub-domains.
Auxiliary-loss-free routing (DeepSeek-V3): instead of adding a balancing loss during training, it dynamically adjusts a bias in the routing. It keeps balance without polluting the main loss.

Attention is still dense in every current MoE model (Mixtral, DeepSeek, Qwen3-MoE, Llama 4, Kimi K2). Only the FFN is replaced. That is why FlashAttention v3/v4 is still used as-is; DeepSeek’s MLA is an optimisation orthogonal to MoE.

The relevant MoE models in May 2026

Model	Date	Total	Active	Experts	Top-k	Shared	Notes
Mixtral 8x7B	Dec 2023	47 B	13 B	8	2	0	The one that normalised MoE in open weights
Mixtral 8x22B	Apr 2024	141 B	39 B	8	2	0	—
Grok-1 (open)	Mar 2024	314 B	~86 B	8	2	0	Weights opened by xAI
DeepSeekMoE 16B	Jan 2024	16 B	2.8 B	64+2	6	2	Introduces fine-grained + shared
DeepSeek-V2	May 2024	236 B	21 B	160+2	6	2	+ MLA
Snowflake Arctic	Apr 2024	480 B	17 B	128	2	10 B dense residual	Dense-MoE hybrid
Hunyuan-Large	Nov 2024	389 B	52 B	16+1	1+shared	1	Cross-layer KV cache
DeepSeek-V3	Dec 2024	671 B	37 B	256+1	8	1	MLA + MTP, aux-loss-free, FP8 training
Llama 4 Scout	Apr 2025	109 B	17 B	16	1+shared	1	Designed for 1 H100 node
Llama 4 Maverick	Apr 2025	400 B	17 B	128	1+shared	1	—
Qwen3-235B-A22B	Apr 2025	235 B	22 B	128	8	0	—
Kimi K2 (Moonshot)	2025	1 T	32 B	384	8	(MLA)	Trained with MuonClip
DeepSeek-V3.2-Exp	Sep 2025	671 B	37 B	256+1	8	1	+ DeepSeek Sparse Attention

Three operational observations:

The active/total ratio falls with every generation: Mixtral 8x7B was 28 %, DeepSeek-V3 is 5.5 %, Llama 4 Maverick is 4.3 %. What the industry discovered is that sparsity can be very aggressive without losing quality, provided N is large enough and the routing is well learned. Fine specialisation is worth more than per-expert capacity.
The flagships of the large providers (OpenAI, Anthropic, Google) are almost certainly MoE, but the architectures are not published. The only semi-credible leak (Hotz, Jul 2023) claimed GPT-4 was an 8×~220 B MoE; no official confirmation.
DeepSeek-V3 marks the inflection point: open weights, frontier-class, MoE with aggressive sparsity and compatible with FP8 quantisation. It is the model that forced the ecosystem (vLLM, SGLang, TensorRT-LLM) to optimise wide-EP in 2025.

The maths that matters

Three numbers drive the whole architectural decision with MoE.

Compute per token. A dense transformer spends roughly 2N FLOPs per token in the forward pass (with N = total parameters). A MoE spends 2·N_active FLOPs per token (the N_total - N_active experts do not take part). For DeepSeek-V3: 2 × 37 B = 74 GFLOPs/token. For dense Llama-3-70B: 2 × 70 B = 140 GFLOPs/token. DeepSeek does half the compute per token of Llama-3-70B while offering capacity equivalent to a model almost 10 × larger.

Total memory. But all the experts must be loaded in VRAM at some point in the forward pass. The sparsity is in compute, not memory. Weight memory:

$$\text{Memory}_{\text{weights}} \approx N_{\text{total}} \cdot \text{bytes per param}$$

For DeepSeek-V3 FP8 (1 byte): 685 GB. It does not fit in 8×H100 SXM (640 GB). You need either extra quantisation or 16+ GPUs. For Mixtral 8x22B FP8: 141 GB, comfortably inside 2×H100. For Qwen3-235B-A22B FP8: 235 GB, fits in 4×H100.

All-to-all bytes per MoE layer. When experts are spread across GPUs (EP), each token has to travel to the GPUs where its k experts live. The volume per MoE layer is roughly:

$$\text{bytes a2a/layer} \approx 2 \cdot \text{batch} \cdot \text{seq len} \cdot d_{\text{model}} \cdot k \cdot \text{bytes}$$

(the factor 2 is dispatch plus combine). For DeepSeek-V3 with d_model = 7168, k = 8, batch = 4096, FP8 (1 byte): each MoE layer moves ~470 MB; multiplied by 61 MoE layers in DeepSeek-V3, the complete step adds ~29 GB of inter-GPU traffic in MoE comms alone. That is of the order of a GPU’s HBM bandwidth in a single step. This is why bandwidth and NVLink/InfiniBand topology determine how large EP can be before comms saturate.

Persistent memory-bound regime. Here is the least intuitive consequence. In dense, raising the batch moves the operation from memory-bound to compute-bound: with enough tokens per batch, the weights are amortised over more operations. In MoE, each expert sees only batch · k / N tokens. For Qwen3-235B (k=8, N=128) with batch=128, each expert processes 8 tokens per step. To reach the compute-bound regime of the tensor cores (arithmetic intensity ~100-200 FLOPs/byte) you need batches in the thousands, around 1,600 according to recent estimates for Qwen3 (Memory-Bound MoE Serving, arXiv:2512.09277). The operational consequence: MoE scales throughput with batch more linearly than dense, but it also needs far larger batches to get close to its compute ceiling.

Expert parallel and the all-to-all bottleneck

There are three main ways to parallelise a MoE across GPUs:

Tensor Parallel (TP, Megatron-style). Splits each weight matrix across GPUs by width or height. Every GPU processes all tokens and all layers. Comms per layer: 2 all-reduces. It works well for dense; for MoE it leaves the routing sparsity unexploited (each GPU holds fragments of every expert).

Expert Parallel (EP). Splits whole experts across GPUs: 256 experts / 32 GPUs = 8 experts per GPU. Each token, after routing, travels to the GPUs where its top-k k experts live. Comms per MoE layer: an all-to-all dispatch (tokens towards experts) plus an all-to-all combine (outputs back). The cost grows with EP size and becomes the bottleneck when EP > 8.

DP + EP / Wide-EP. Replicates the model across DP groups; inside each group, EP shards the experts. Wide-EP takes EP to the full dimension of a rack (NVL72, 72 GPUs in a coherent NVLink domain). Each expert receives more tokens per step, so arithmetic intensity in each expert’s GEMMs improves and per-GPU throughput rises. vLLM reports 1.8× throughput per GPU with wide-EP against smaller setups on DeepSeek-V3 (vLLM blog, Dec 2025).

Quick decision table:

Case	Recommendation
Small MoE that fits in 1 node (Mixtral 8x7B, Llama 4 Scout)	TP only or EP=2 intra-node
Medium MoE (Mixtral 8x22B, Qwen3-235B) on 1-2 H100 nodes	EP=8 intra-node, optionally TP=2
Large MoE (DeepSeek-V3) on a multi-node cluster	TP × EP cross-node (e.g. TP=4 × EP=8 on 32 GPUs)
Large MoE with NVL72 available	Wide-EP=72 (decode), EP=32 (prefill)

DeepSeek’s production deployment combines Prefill EP=32 (4 nodes × 8 GPUs) with Decode EP=144 (18 nodes × 8 GPUs). The prefill/decode split fits conceptually with Disaggregated serving: prefill is compute-bound and benefits from moderate EP; decode is memory-bound and benefits from Wide-EP, which raises the tokens per expert per step.

DeepEP, EPLB and Wide-EP: the pieces that unblocked 2025

DeepEP (DeepSeek, opened in February 2025) is a library of all-to-all CUDA kernels optimised specifically for MoE EP. Four important properties:

Native FP8 dispatch: tokens travel in FP8, halving the bandwidth consumed.
Mixing intra-node NVLink with inter-node RDMA: the kernel picks the optimal route according to topology.
CPU bypass: the GPU→GPU dispatch goes over RDMA without touching the host.
Aligned with DeepSeek-V3’s group-limited gating: the routing prefers experts in the same node group where possible, minimising inter-node traffic.

In 2025 Tencent contributed optimisations that added +30 % on the normal kernels. There is a ROCm port for AMD.

EPLB (Expert Parallelism Load Balancer) solves the runtime imbalance problem: even when the model is well balanced in global distribution, a specific batch can heavily activate certain experts (prompts in a single language, a single domain). EPLB replicates the hot experts across several GPUs (redundant experts) and packs them heuristically with cold experts to minimise load variance. It has two modes: hierarchical (for prefill with medium EP) and global (for decode with Wide-EP). It is integrated into SGLang and vLLM.

Wide-EP on NVL72 is the peak setup of 2026: 72 Blackwell GPUs in a coherent NVLink domain (1.8 TB/s bidirectional per GPU, intra-rack). Each expert receives many more tokens per step, so arithmetic intensity improves and so does throughput per GPU. Combined with NVFP4 over expert weights and FP8 over attention, SGLang reported on GB200 NVL72 26,156 tok/s/GPU in prefill and 13,386 tok/s/GPU in decode for DeepSeek-V3 (LMSYS, Sep 2025), a 3.8× prefill and 4.8× decode versus H100.

Operational pitfalls

HBM fragmentation from expert weights. With EP=8 in 1 node, each GPU holds N/8 experts, each a tuple of matrices (gate, up, down). If N=256, that is 32 FFNs per GPU. The allocator can fragment; in some cases you need a more conservative --gpu-memory-utilization than usual.

Cold experts. In real distributions some experts are activated by <1 % of tokens. EPLB compensates by replicating hot ones, but the cold ones sit occupying HBM without taking part, “dead” memory. When real demand diverges a lot from the training distribution, this shows.

Continuous batching. It works with MoE, but the effective batch size per expert is batch_total · k / N. With large N and a moderate batch, each expert sees very few tokens per step. You need a batch_total far larger than in dense to match per-GPU throughput.

Speculative decoding + MoE. They interact well. MoE sparsity keeps the memory-bound regime at high batch, which is exactly the regime where speculative decoding wins most. DeepSeek-V3 integrates MTP (Multi-Token Prediction), speculative decoding native to the model, with no external draft, with acceptance around 85-90 % on the second token (see Speculative decoding for the mechanism).

MLA and FlashMLA are not MoE. DeepSeek-V3 combines MoE with Multi-head Latent Attention (MLA), an attention variant that reduces the KV cache ~10× against standard MHA. MLA and MoE are orthogonal: MLA optimises attention, MoE optimises the FFN. For serving DeepSeek you use the specific FlashMLA kernel (not FA3 directly).

DSA (DeepSeek Sparse Attention) in DeepSeek-V3.2-Exp (Sep 2025) introduces sparsity in attention too. The first time at the frontier that sparsity is applied in both attention and FFN.

Implications for on-premise hardware

On an RTX 4090 (24 GB). Large MoEs are out of reach. What does fit: Mixtral 8x7B AWQ-INT4 (~24 GB of weights, no headroom for KV cache, so it really requires TP=2 across two 4090s); DeepSeekMoE 16B BF16 (~33 GB, does not fit whole; INT4 ~8 GB, fits with room to spare). The interesting case is llama.cpp with --n-cpu-moe: expert weights offloaded to RAM (CPU), attention and shared experts on the GPU. It lets you run DeepSeek-V3 IQ4 (~400 GB) on a workstation with 1-2 GPUs plus 512 GB of RAM at modest throughput, but it opens the door to a low-budget setup.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). These fit comfortably:

Mixtral 8x22B FP8 (~141 GB) with TP=2 or EP=4. Low latency, excellent throughput.
Qwen3-235B-A22B FP8 (~235 GB) with TP=4 or EP=4. Plenty of room.
Llama 4 Scout 109B FP8 (~110 GB) with TP=2 or TP=4. Designed by Meta to serve well on one H100 node.

What does not fit in 4×H100: DeepSeek-V3 FP8 (~685 GB) without aggressive quantisation. Options: (1) scale to 8-16 H100; (2) use AWQ INT4 over routed experts plus FP16 over activated ones (~400 GB), which fits in 8×H100; (3) wait for Blackwell or use AMD MI300X (192 GB/GPU, so DeepSeek-V3 FP8 on 4×MI300X).

The rule of thumb as of May 2026: one 4×H100 node comfortably serves MoEs up to ~200 B total in FP8; for DeepSeek-V3 you have to scale to 2 nodes or wait for B200 NVL72 (where the whole model fits with room for enormous batches).

What we have not covered

MLA and FlashMLA in detail: DeepSeek’s KV cache optimisation, orthogonal to MoE.
DeepSeek Sparse Attention (DSA) introduced in V3.2-Exp: the first production implementation of attention sparsity for frontier models.
MoE during training: load balancing losses, drop policies, auxiliary terms vs aux-loss-free, Kimi K2’s MuonClip.
MoE + LoRA: how adapter fine-tuning is done on a MoE, and what happens to the routing.
LLM-d and other open-source platforms that materialise Wide-EP on Kubernetes.

References

Shazeer, N. et al. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR 2017. https://arxiv.org/abs/1701.06538
Lepikhin, D. et al. GShard: Scaling Giant Models with Conditional Computation. ICLR 2021. https://arxiv.org/abs/2006.16668
Fedus, W., Zoph, B., Shazeer, N. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. JMLR 2022. https://arxiv.org/abs/2101.03961
Gale, T., Narayanan, D., Young, C., Zaharia, M. MegaBlocks: Efficient Sparse Training with Mixture-of-Experts. MLSys 2023. https://arxiv.org/abs/2211.15841
Mistral AI. Mixtral of Experts. 2024. https://arxiv.org/abs/2401.04088
DeepSeek-AI. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. 2024. https://arxiv.org/abs/2401.06066
DeepSeek-AI. DeepSeek-V2. 2024. https://arxiv.org/abs/2405.04434
DeepSeek-AI. DeepSeek-V3 Technical Report. 2024. https://arxiv.org/abs/2412.19437
Tencent. Hunyuan-Large. 2024. https://arxiv.org/abs/2411.02265
DeepEP repo: https://github.com/deepseek-ai/DeepEP
EPLB repo: https://github.com/deepseek-ai/EPLB
DeepSeek open-infra-index, V3/R1 Inference System Overview: https://github.com/deepseek-ai/open-infra-index
vLLM blog, Large-Scale Serving DeepSeek (Dec 2025): https://blog.vllm.ai/2025/12/17/large-scale-serving.html
vLLM blog, WideEP en GB200 (Feb 2026): https://blog.vllm.ai/2026/02/03/dsr1-gb200-part1.html
LMSYS blog, DeepSeek 96 H100 PD+EP (May 2025): https://www.lmsys.org/blog/2025-05-05-large-scale-ep/
LMSYS blog, DeepSeek GB200 NVL72 part II (Sep 2025): https://www.lmsys.org/blog/2025-09-25-gb200-part-2/
NVIDIA Dynamo + GB200 NVL72 for MoE: https://developer.nvidia.com/blog/how-nvidia-gb200-nvl72-and-nvidia-dynamo-boost-inference-performance-for-moe-models/
Tensor Economics, MoE Inference Economics from First Principles: https://www.tensoreconomics.com/p/moe-inference-economics-from-first
Cohere, Why MoE models get more from speculative decoding: https://cohere.com/blog/mixture-of-experts-models-get-more-from-speculative-decoding

Speculative decoding: the secretary who types ahead of the boss — fundamentals, maths and state of play in May 2026

Fri, 29 May 2026 11:00:00 +0200

This post complements KV cache: the working memory and Quantisation for LLM inference. Both are acceleration levers inside a single forward pass. Speculative decoding is the complementary lever: instead of making each forward pass cheaper, it tries to produce more tokens per forward pass. It is orthogonal to quantisation and to the KV cache, and it stacks multiplicatively with both.

You are here: DEPLOY

TL;DR

Autoregressive LLM inference has a structural problem: every new token needs a complete forward pass of the model, and decode forward passes are memory-bandwidth-bound, not compute-bound (the detail is in KV cache). The GPU spends most of its time waiting for the next weight to arrive from HBM. Speculative decoding uses that dead time by doing two things in parallel: a small, cheap model (draft) generates γ tokens autoregressively, fast but imprecise, and the large model (target) verifies those γ tokens in a single parallel forward pass, at almost the same cost as a single-token forward pass. An acceptance rule based on rejection sampling decides how many draft tokens are accepted, and the maths proves, it does not approximate, it proves, that the output distribution matches sampling directly from the target exactly. The technique has a ceiling: you never generate more than 1/(1-α) tokens per step, with α the acceptance rate. The five dominant families in May 2026 are vanilla SD (Leviathan 2023), Medusa (extra parallel heads), EAGLE-1/2/3 (a draft that operates at the level of hidden states), MTP (multi-token prediction native to DeepSeek-V3) and P-EAGLE (a draft that produces the γ tokens in a single forward pass, integrated in vLLM 0.16+). On low-concurrency workloads with short prompts and long outputs, the typical case of the on-premise conversational assistant, the real speedup sits between 2× and 4× on modern hardware.

The analogy: the secretary who types ahead and the boss who validates

Picture a crowded press conference with a real-time transcription setup. Two people are working:

The secretary sits at the keyboard. They know the subject, they have read the briefings and they know the boss’s verbal tics. As soon as the boss opens their mouth, the secretary starts typing what they think is coming. They are fast, three or four words ahead of the boss, but they sometimes get it wrong, especially when the boss takes an unexpected turn.

The boss periodically reads the secretary’s screen. They read a whole block at once, three or four words, and mentally compare what is written with what they would have said. If it matches their intent, they leave the text and carry on. If the secretary drifted at some point, they correct right there, and whatever the secretary typed after the divergence is discarded (it could be wrong because of the earlier error). If the whole block was fine, the boss takes the chance to add one more word while correcting.

The result: the final text is identical to what the boss would have dictated alone, but it comes out faster because the secretary works ahead.

The analogy holds on four details:

The secretary is the draft model. Small, cheap, fast, approximate.
The boss is the target model. Slow, expensive, but the only authority on output quality.
The block the boss reads at once is the target’s parallel forward pass over prompt + γ draft tokens. Verifying γ tokens costs almost the same as generating one, because the bottleneck is loading the weights from HBM, not the operations.
The “correct right where it diverged” rule is the rejection sampling that preserves the target distribution.

From here we go into the mechanism and into why quality is preserved exactly, not approximately.

The bare mechanism

Call p the target distribution (what it would produce if it were the only one speaking) and q the draft’s. The speculative decoding iteration has three steps:

Step 1 — Draft. The draft generates γ tokens autoregressively: x_1, x_2, ..., x_γ. Each one through its own forward pass. Since the draft is small, the γ steps cost little. Typically γ ∈ [4, 8].

Step 2 — Verify. The target runs a single forward pass over the complete sequence prompt + x_1 ... x_γ. Because attention is causal, that forward pass simultaneously produces the distributions p(·|prompt, x_<i) for each position i. In other words, in a single step it obtains the verification of the γ tokens and, on top of that, an extra distribution p(·|prompt, x_1...x_γ) for the next token.

Step 3 — Accept/reject token by token. For each x_i from left to right it applies:

If p(x_i) ≥ q(x_i): always accept.
If p(x_i) < q(x_i): accept with probability p(x_i)/q(x_i).
If it rejects: stop, sample a replacement token from the normalised residual distribution norm(max(0, p − q)), and discard the rest.
If it reaches the end having accepted the γ tokens: append a bonus token sampled directly from p(·|prompt, x_1...x_γ), which is already in the logits of the target’s forward pass.

Result per iteration: between 1 and γ+1 new tokens. In the best case (all accepted + bonus) γ+1 tokens are generated at the cost of a single target forward pass plus γ draft forward passes, against γ+1 target forward passes in the version without speculative.

Why quality does not degrade (the proof)

This is the part of the technique that confuses people most the first time. It looks like magic: how can sampling from an arbitrary draft and then “validating” produce exactly the same distribution as sampling from the original target?

The proof fits in two lines. For any token x, the final probability of emitting it is the sum of two disjoint events: the draft proposes it and it is accepted, or the draft proposes something else, it is rejected, and sampling from the residual yields x. Formally:

$$P(\text{emit } x) = q(x) \cdot \min\!\left(1, \frac{p(x)}{q(x)}\right) + P(\text{reject}) \cdot \frac{\max(0, p(x)-q(x))}{\sum_y \max(0, p(y)-q(y))}$$

The first term is min(p(x), q(x)). The second is max(0, p(x) − q(x)) (the total rejected mass Σ max(0, p−q) cancels the denominator exactly). Adding them: min(p(x), q(x)) + max(0, p(x) − q(x)) = p(x). The output is statistically indistinguishable from sampling directly from the target, down to floating-point numerical differences.

This matters operationally: it means speculative decoding is not perceptual compression, it is not a quality-speed trade-off, it does not require extra validation with evals. If the target would have passed a given eval, the system with speculative passes it too.

The maths that matters: ceiling and speedup

Call α the acceptance rate: the expected probability that an individual draft token is accepted. Assuming that acceptances of consecutive tokens are independent (a reasonable approximation in practice), the expected number of tokens generated per iteration is:

$$E[\text{tokens per step}] = \sum_{k=0}^{\gamma} \alpha^k + \alpha^{\gamma+1} = \frac{1 - \alpha^{\gamma+1}}{1 - \alpha}$$

And the theoretical speedup relative to generating one token per target iteration, with c = T_draft / T_target the relative cost of the draft, is:

$$\text{Speedup} = \frac{1 - \alpha^{\gamma+1}}{(1 - \alpha)(\gamma c + 1)}$$

There is a ceiling that many implementations try to beat and cannot: as γ → ∞, the speedup converges to:

$$\lim_{\gamma \to \infty} E[\text{tokens per step}] = \frac{1}{1 - \alpha}$$

It is an algorithmic ceiling, not a hardware one. With α = 0.7 you never generate more than 3.33 tokens per iteration no matter how much the hardware improves. With α = 0.8 → 5. With α = 0.9 → 10. That is why EAGLE-3, which targets α > 0.8 on many benchmarks, is not an incremental improvement over vanilla SD (α ≈ 0.5-0.6): it is a change of regime, because it raises the ceiling instead of merely getting closer to it.

A concrete numerical example, Llama 3 70B as target with a draft giving α = 0.75, γ = 5, c = 0.1:

Expected tokens per step: (1 − 0.75⁶) / (1 − 0.75) = (1 − 0.178) / 0.25 = 3.29
Speedup: 3.29 / (5 × 0.1 + 1) = 3.29 / 1.5 = 2.19×

If we raise α to 0.85 with the same configuration: tokens per step = (1 − 0.85⁶) / 0.15 = (1 − 0.377) / 0.15 = 4.16, speedup = 4.16 / 1.5 = 2.77×. A change of α = 0.75 → 0.85 (10 absolute points) raises the speedup by 27 %. That is why the research of the last two years focuses on pushing α: that is where the prize is.

The five modern families (May 2026)

Family	Year	Core idea	Draft size	Typical α / τ	Speedup
Vanilla SD	2023	Same-family target + draft pair, rejection sampling	1/10 – 1/100 of target	0.5 – 0.8	2-3×
Medusa	2024	N extra heads in parallel predict t+1, t+2, …; tree attention verifies several candidates	no draft, +1-2 % params	top-5 > 0.8	2.18-2.83×
EAGLE-1/2/3	2024-25	Autoregressive draft at the level of features (hidden states), not tokens. Reuses the target’s embedding	1 transformer block (~0.5-2 % params)	EAGLE-3: α > 0.8, τ up to 7.5	up to 6.5× peak
MTP (DeepSeek-V3)	2024	Multi-token heads trained from scratch as part of the model; at inference they act as a “free” draft	14B params in V3 671B	α > 0.8 (MTP1)	1.5-1.8×
P-EAGLE	2026	EAGLE but producing the γ drafts in a single forward pass (parallel, not autoregressive)	same as EAGLE	+30 % over EAGLE-3	4-5× vs AR

Three operational observations:

EAGLE dominates in production because its overhead is minimal (one transformer block, ~1 % of the target in parameters) and its α is high. The draft does not need its own “complete” KV cache because it shares features with the target.
Medusa mattered historically, it showed that speculative was possible without a separate draft, but EAGLE beat it on every published benchmark through 2024-2025.
MTP is special: it is not something you add to an existing model. It is something the model trained natively. If you buy DeepSeek-V3, MTP comes free and gives around 1.8× without touching anything. If you buy Llama 3, there is no MTP to be had; use EAGLE-3.

There are also two related techniques that deserve a brief mention, both without a draft model: Lookahead decoding (Fu et al. 2024), which formulates decoding as Jacobi iteration and extracts n-grams from the trajectory; and REST (He et al. 2024), which keeps an n-gram datastore and proposes drafts by longest-prefix match against the last generated tokens. Both give 1.5-2× with no extra VRAM. Useful when you have no trained draft model and do not want to maintain one.

Real implementations in May 2026

vLLM v0.16+: unified support for EAGLE-1/2/3, P-EAGLE, Medusa, native MTP (DeepSeek-V3 and variants), n-gram/suffix decoding without a draft, an arbitrary draft model and MLP speculators. The canonical flag is --speculative-config '{"method":"eagle3", "model":"...", "num_speculative_tokens": 5}'.
SGLang: native EAGLE-3 support with --speculative-algorithm EAGLE3. For DeepSeek-V3 it uses MTP through an EAGLE adapter. It has its own draft training framework (SpecForge).
TensorRT-LLM: Medusa, EAGLE (a simplified variant without a tree), ReDrafter, Lookahead. They report around 2.2× with EAGLE.
llama.cpp: basic speculative only with a draft model (--model-draft). No native EAGLE/Medusa/MTP as far as I have verified. Typical speedup 1.5-2.5× single-user.

When speculative does NOT help

The technique has three important blind spots:

Large batch. The GPU moves from memory-bound (decode at low concurrency) to compute-bound (decode at high concurrency). In the compute-bound regime the “almost free” target forward passes stop being free: γ tokens start costing γ times more instead of nearly 1. The typical crossover is at batch 16-32 for dense models; for MoE with few active parameters per token the crossover happens later. At load peaks, speculative can worsen aggregate throughput per GPU.

Prefill / TTFT. Speculative decoding does not touch prefill. The phase that processes the whole prompt stays identical. TTFT does not improve and can get marginally worse because of the draft setup. If the SLA is TTFT-bound (assistants with short outputs, search, RAG with brief answers), this is not the right tool.

Short outputs. If the model generates 10-20 tokens and the answer is over, the fixed setup overhead is not amortised. Speculative shines with long outputs (300+ tokens): generative assistants, extended code completion, drafting.

Operational pitfalls

Extra VRAM for the draft. In vanilla SD, loading a complete draft plus its KV cache is expensive. Llama 3 8B in BF16 as the draft for a 70B is about 16 GB of weights plus KV cache. On an H100 80 GB with the target already nearly filling it, that can force you to shrink the target’s KV cache and lower maximum concurrency. EAGLE solves this: the draft is one transformer block (~1 GB for a 70B) and reuses the target’s features.

Draft quantisation. Quantising the draft to INT4 with GPTQ degrades α substantially (errors in the logits accumulate in the comparison against p). AWQ holds up better but also lowers α. Common practice in 2026: target in FP8 or INT4, draft in FP16/BF16. The draft is small enough that the VRAM saved by quantising it does not compensate for the drop in α and therefore in speedup. The detail of each format is in Quantisation.

Interaction with continuous batching. It does not break it, but it creates nested raggedness: each request in the batch can accept a different number of tokens on each iteration. vLLM’s PagedAttention absorbs it, but the scheduler loses efficiency. At low QPS (conversational assistant, low simultaneous concurrency) the combination is excellent. At high QPS there is real tension, and work such as Goodput-optimized speculative decoding (Liu et al., 2024) optimises γ dynamically according to the state of the batch.

Sampling temperature. α falls with high temperatures. At T = 1.0 with creative outputs (free-form writing), α can drop 10-15 points relative to T = 0 with the same model-draft pair. The speedup scales accordingly.

Implications on on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace)

The classic use is vanilla SD with two quantised models: for instance, Llama 3 70B AWQ-INT4 as target (~35 GB → needs TP=2 across two 4090s) and Llama 3 8B AWQ-INT4 as draft. In practice, the most realistic single-card case is Llama 3 8B target + a 1B model as draft or Qwen 3 14B target + Qwen 3 0.5B as draft: they fit comfortably and give 1.8-2.5× on conversational tasks. For EAGLE on consumer cards, the official drafts for popular families (Llama 3, Qwen 3) are published on Hugging Face and take up 0.5-2 GB extra.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here EAGLE-3 shines:

Llama 3 70B FP8 + EAGLE-3 draft in FP16: the draft takes about 0.7 GB; the target about 70 GB with TP=2 (35 GB per GPU). The speedup observed in reproducible benchmarks is between 2.5× and 4× at batch 1-4, falling to almost break-even at batch 32.
DeepSeek-V3 671B FP8 + native MTP: the model ships with MTP trained in; there is nothing to add. The speedup is 1.5-1.8× with zero extra VRAM. It is the most efficient option operationally: zero additional pieces.
Combining with disaggregated serving: as Disaggregated serving explains, prefill and decode can live in separate pods. Speculative applies only in the decode pods, which fits the separation perfectly (prefill is compute-bound and would not benefit).

The rule of thumb on an H100 cluster in May 2026: if the model is DeepSeek-V3 / V4 → native MTP, nothing else; if it is Llama 3 / Qwen 3 → EAGLE-3 with the official draft; if it is exotic → vanilla SD with a draft from the same family.

What we have not covered

Training custom EAGLE drafts with SpecForge: how to collect target trajectories and train the draft on-policy.
Speculative Prefill (arXiv:2502.02789): a variant for accelerating TTFT, a different mechanism from the decode one described here.
Tree attention in detail: how Medusa and EAGLE-2 verify several candidates at once with specific attention masks.
MoE + speculative: the combination has non-trivial interactions with the expert router. Low activated params keep the memory-bound regime in place even at high batch, which changes the rules.

References

Leviathan, Y., Kalman, M., Matias, Y. Fast Inference from Transformers via Speculative Decoding. ICML 2023. https://arxiv.org/abs/2211.17192
Chen, C., Borgeaud, S., Irving, G., Lespiau, J.B., Sifre, L., Jumper, J. Accelerating Large Language Model Decoding with Speculative Sampling. DeepMind 2023. https://arxiv.org/abs/2302.01318
Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J.D., Chen, D., Dao, T. Medusa: Simple LLM Inference Acceleration with Multiple Decoding Heads. ICML 2024. https://arxiv.org/abs/2401.10774
Li, Y., Wei, F., Zhang, C., Zhang, H. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. ICML 2024. https://arxiv.org/abs/2401.15077
Li, Y., Wei, F., Zhang, C., Zhang, H. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. EMNLP 2024. https://arxiv.org/abs/2406.16858
Li, Y., Wei, F., Zhang, C., Zhang, H. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. NeurIPS 2025. https://arxiv.org/abs/2503.01840
DeepSeek-AI. DeepSeek-V3 Technical Report — Multi-Token Prediction. https://arxiv.org/abs/2412.19437
Fu, Y., Bailis, P., Stoica, I., Zhang, H. Break the Sequential Dependency of LLM Inference Using Lookahead Decoding. ICML 2024. https://arxiv.org/abs/2402.02057
He, Z., Zhong, Z., Cai, T., Lee, J.D., He, D. REST: Retrieval-Based Speculative Decoding. NAACL 2024. https://arxiv.org/abs/2311.08252
vLLM speculative decoding docs: https://docs.vllm.ai/en/latest/features/speculative_decoding/
SGLang speculative decoding docs: https://docs.sglang.ai/advanced_features/speculative_decoding.html
TensorRT-LLM speculative sampling: https://nvidia.github.io/TensorRT-LLM/advanced/speculative-decoding.html
vLLM blog Speculative Decoding in vLLM (oct 2024): https://blog.vllm.ai/2024/10/17/spec-decode.html
vLLM blog P-EAGLE: Parallel Speculative Decoding in vLLM (mar 2026): https://vllm.ai/blog/2026-03-13-p-eagle
Official EAGLE repo: https://github.com/SafeAILab/EAGLE
Official Medusa repo: https://github.com/FasterDecoding/Medusa

Deepseek-V3 on lo0 — Tech Blog

MoE inference: the call centre with 256 specialists and 8 taking each call — fundamentals, expert parallel and the economics of DeepSeek-V3

You are here: DEPLOY

TL;DR

The analogy: the call centre with 256 specialists

The bare mechanism

The relevant MoE models in May 2026

The maths that matters

Expert parallel and the all-to-all bottleneck

DeepEP, EPLB and Wide-EP: the pieces that unblocked 2025

Operational pitfalls

Implications for on-premise hardware

What we have not covered

See also

References

Speculative decoding: the secretary who types ahead of the boss — fundamentals, maths and state of play in May 2026

You are here: DEPLOY

TL;DR

The analogy: the secretary who types ahead and the boss who validates

The bare mechanism

Why quality does not degrade (the proof)

The maths that matters: ceiling and speedup

The five modern families (May 2026)

Real implementations in May 2026

When speculative does NOT help

Operational pitfalls

Implications on on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace)

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

What we have not covered

See also

References