Sglang on lo0 — Tech Blog

LLM serving engines compared on the Pareto frontier: vLLM, SGLang, TRT-LLM and Dynamo

Tue, 16 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter). Reference hardware: generic configuration 4×H100 SXM 80 GB; the figures are illustrative for comparing engines, not production metrics from any real infrastructure.

TL;DR

Four OSS engines dominate on-premise LLM serving in 2026: vLLM, SGLang, NVIDIA TensorRT-LLM and NVIDIA Dynamo. None is superior on every axis. In raw throughput with a shared prompt, SGLang beats vLLM by 29 % on Llama-3.3-70B FP8 over H100 (16,200 vs 12,500 tok/s). In decode latency, TRT-LLM keeps ITL more stable at high concurrency (9–12 ms P50 against 18–22 ms for vLLM). Dynamo is not an inference engine: it is the orchestration layer on top of the previous three for multi-node clusters, and it reports up to 7× additional throughput in disaggregated serving on Blackwell. The right choice depends on three variables: load pattern (batch/interactive), deployment scale (single node/multi-node) and operational constraints (OSS maturity, ease of operation). The Pareto decision table is at the end of the article.

Framing the article: what this B8 is and is not

This article (B8 of the benchmarking pillar) takes for granted the metrics from the introduction to benchmarking and the tool catalogue from the card-by-card piece. It applies those concepts to engine selection. The protocol from GuideLLM for SLO validation and AIPerf in depth detail how to run the harness; here the result is used. The fundamentals of continuous batching, disaggregated serving, the attention backend and quantisation are assumed known.

Card for each engine

vLLM

Attribute	Value
Reference version	v0.9.x (June 2026)
Licence	Apache 2.0
Maintainer / governance	vLLM Project (Linux Foundation AI), led by UC Berkeley + Red Hat
Supported hardware	NVIDIA (Ampere/Hopper/Blackwell), AMD ROCm, Intel Gaudi, TPU, CPU
Repository	github.com/vllm-project/vllm
API	OpenAI-compatible (completions, chat, embeddings)

Origin: PagedAttention (Kwon et al., 2023). The first implementation of continuous batching with a paged KV cache. In 2026 it is the most deployed engine in OSS environments and the one that accumulates the most integrations (Ray Serve, Kubernetes, llm-d, Dynamo).

SGLang

Attribute	Value
Reference version	v0.4.x (June 2026)
Licence	Apache 2.0
Maintainer / governance	LMSYS Org (Berkeley, CMU, UCSD)
Supported hardware	NVIDIA (Ampere/Hopper/Blackwell), AMD ROCm
Repository	github.com/sgl-project/sglang
API	OpenAI-compatible + SGLang runtime API

Origin: the SGLang paper, 2024 (Zheng et al.). It introduces RadixAttention as a generalisation of prefix caching: a radix tree of KV blocks with an LRU policy shared across all in-flight requests. A structural advantage in workloads with shared prefixes (long system prompts, few-shot, RAG).

NVIDIA TensorRT-LLM (TRT-LLM)

Attribute	Value
Reference version	v0.18.x (June 2026)
Licence	Apache 2.0
Maintainer / governance	NVIDIA (proprietary in practice, OSS in name)
Supported hardware	NVIDIA only: Ampere (A100), Hopper (H100/H200/GH200), Ada (L40/L40S), Blackwell (B200/GB200)
Repository	github.com/NVIDIA/TensorRT-LLM
API	Python + C++ runtime; integrates with Triton Inference Server

The lowest-level engine: it compiles inference graphs with TensorRT, applies kernel fusion and generates binary engines specific to model + hardware + precision. Maximum performance on NVIDIA, with no portability to other hardware. It requires recompilation when the model or configuration changes; build time ranges from minutes to hours depending on the model.

NVIDIA Dynamo

Attribute	Value
Reference version	v1.1.1 (May 2026)
Licence	Apache 2.0
Maintainer / governance	NVIDIA + OSS community (ai-dynamo org on GitHub)
Supported hardware	NVIDIA (through the underlying engine: vLLM, SGLang or TRT-LLM)
Repository	github.com/ai-dynamo/dynamo
Role	Orchestration layer, NOT an inference engine

Dynamo does not run inference directly: it is the scheduler and router that coordinates pools of workers (vLLM, SGLang, TRT-LLM) across multiple nodes. Implemented in Rust (performance) + Python (extensibility). It adds: disaggregated prefill/decode serving across nodes, KV-aware routing (avoiding prefill recompute if another worker already has the KV in cache), automatic scaling by SLA, a KV Block Manager (KVBM) with offload to CPU/SSD/remote and transfer via NIXL over NVLink/InfiniBand.

Honourable mentions: LMDeploy and HF TGI

LMDeploy (OpenMMLab/InternLM, Apache 2.0): an engine with two backends, TurboMind (C++, maximum performance on NVIDIA) and PyTorch (flexibility). KV cache blocking and persistent batching; 4-bit inference up to 2.4× faster than FP16 in its own benchmark. A narrower ecosystem; no AMD support.

HF TGI (Hugging Face, Apache 2.0): the reference engine for the HuggingFace ecosystem. TGI v3.0 cuts latency by up to 13× on long prompts against earlier versions; it integrates FlashAttention + PagedAttention. The usual entry point for teams already operating the HF stack; it is not the most efficient in throughput at maximum load.

Capability matrix

The table uses this notation: Yes = supported and stable, Beta = available but not recommended for production, No = not available or only with a manual workaround.

Capability	vLLM	SGLang	TRT-LLM	Dynamo (via backend)
Continuous batching	Yes	Yes	Yes	Yes (delegates to the engine)
PagedAttention	Yes	Yes	Yes	Yes
RadixAttention / prefix caching	Yes (chunked prefill + prefix)	Yes (native RadixAttention)	Yes (prefix caching)	Yes + KV-aware routing
Speculative decoding	Yes (n-gram, EAGLE, DFlash, suffix)	Yes (draft+verify)	Yes (EAGLE-3, MTP)	Yes
FP8 quantisation	Yes	Yes	Yes	Yes
MXFP4/NVFP4 quantisation	Yes	Beta	Yes (Blackwell)	Yes
INT4 AWQ quantisation	Yes	Yes	Yes	Yes
GPTQ quantisation	Yes	Yes	Not native	Yes (via vLLM)
INT8 quantisation (SmoothQuant)	Yes	Yes	Yes	Yes
Structured output	Yes	Yes (compressed FSM)	Beta	Yes
Dynamic multi-LoRA	Yes	Yes	Beta	Yes
Disaggregated prefill/decode	Beta (experimental)	Yes	Yes	Yes (first class)
Chunked prefill	Yes	Yes	Yes	Yes
Multimodal support (VLM)	Yes	Yes	Yes	Yes
Multi-node tensor parallelism	Yes	Yes	Yes	Yes (+ pipeline across nodes)
OpenAI API compatible	Yes	Yes	Yes (via Triton)	Yes
Non-NVIDIA hardware	Yes (AMD/Intel/TPU)	Yes (AMD)	No	No

Sources for the matrix: docs.vllm.ai, docs.sglang.io, nvidia.github.io/TensorRT-LLM, docs.nvidia.com/dynamo, github.com/ai-dynamo/dynamo (feature matrix, June 2026).

Why they are not compared lightly: the variables that break a comparison

Before the table of figures, the four decisions that invalidate any cross-engine benchmark:

1. A different harness. A benchmark run with each engine’s native micro-bench (vllm bench serve vs SGLang bench) measures the client, not the engine. The discrepancy can be up to 7.2× at high concurrency. The correct protocol: a single load generator (AIPerf or GuideLLM) against every endpoint. The engine is the variable, the tool is the constant.

2. Different ISL/OSL. Input Sequence Length and Output Sequence Length determine the prefill/decode ratio and therefore which engine is favoured. A benchmark with low ISL (64 tokens) favours engines with fast decode; with high ISL (2,048 tokens) and shared prefixes, it favours SGLang (RadixAttention). Fixing ISL and OSL is mandatory for the comparison to be valid.

3. Different precision. FP16 and FP8 are not comparable: FP8 gives between 1.5× and 2× more throughput on Hopper. Comparing vLLM FP16 with TRT-LLM FP8 says nothing about the engine.

4. Not reporting P99. Maximum throughput is reached at the point where TTFT P99 already violates the SLO. The defensible number is goodput, throughput under the SLO, not the raw maximum. The end-to-end latency formula:

$$\text{latency}_{e2e} \approx \text{TTFT} + (N_{\text{out}} - 1) \times \text{TPOT}$$

where $N_{\text{out}}$ is the number of output tokens, makes it clear that a high TTFT with low TPOT has the same effect as the inverse case only for one specific $N_{\text{out}}$: both must be reported separately.

The complete fair-comparison protocol is in the tools article.

Performance on the Pareto frontier: illustrative figures

Reference configuration for every row: 4×H100 SXM 80 GB NVLink, model Llama-3.1-70B-Instruct FP8, harness AIPerf (formerly genai-perf, the multi-process successor), dataset ShareGPT (realistic length distribution), mean ISL ~512 tok, mean OSL ~256 tok, SLO TTFT P99 < 500 ms. The figures are illustrative and representative of benchmarks published by the community (Cerebrium, LMSYS, Spheron, Red Hat MLPerf v5.1); they are not measurements from any real infrastructure. Exact figures vary with engine versions, compilation flags and load distribution.

Throughput vs latency: Pareto table

Engine	Config	Throughput (tok/s)	TTFT P50 (ms)	TTFT P99 (ms)	ITL P50 (ms)	Goodput (@SLO)
vLLM v0.9	FP8, chunked prefill	12,500	160	420	18	~96 %
SGLang v0.4	FP8, RadixAttention	16,200	140	390	21	~97 %
TRT-LLM v0.18	FP8, compiled engine	14,800	190	480	10	~93 %
vLLM + Dynamo	FP8, disagg. P/D 2+2	18,500	120	310	19	~98 %
SGLang + Dynamo	FP8, disagg. P/D 2+2	21,000	110	280	22	~99 %
TRT-LLM + Dynamo	FP8, disagg. P/D 2+2	22,500	130	350	11	~97 %

Reading notes:

“Disagg. P/D 2+2” = 2 prefill GPUs + 2 decode GPUs (the same 4×H100 budget).
Goodput falls against raw throughput when the latency tail exceeds the SLO of 500 ms P99. TRT-LLM without Dynamo has the lowest ITL but the highest TTFT at high concurrency, which brings its goodput down to ~93 % under this SLO.
Dynamo adds ~30–40 % of effective throughput over 4 GPUs thanks to disaggregated serving and KV-aware routing, in the scenario with reusable prefixes (ShareGPT).
In workloads without shared prefixes (short ISL, unique prompts), Dynamo’s advantage shrinks to ~10–15 % over the base engine.

Saturation point (concurrency sweep, vLLM FP8, 4×H100)

Concurrency	TTFT P99 (ms)	ITL P50 (ms)	Throughput (tok/s)	Goodput (tok/s)
8	210	14	5,200	5,200
16	370	18	9,800	9,800
32	420	18	12,500	12,200
48	680	26	13,800	7,400
64	1,200	41	14,100	2,100

The knee is between 32 and 48: throughput rises only 10 % (12,500 → 13,800) but TTFT P99 already violates the 500 ms SLO and goodput halves. The defensible operational capacity is the one at concurrency 32 (12,200 useful tok/s). Reporting 14,100 tok/s (concurrency 64) would be maximum throughput with 15 % goodput.

Fair-comparison methodology: the harness as the constant

The protocol that makes numbers comparable across different engines. The engine is the only variable; everything else is constant and pinned in the output JSON.

Variables that are fixed (constants of the experiment)

Variable	Value fixed in the reference experiment
Load tool	AIPerf v2.x (multi-process)
Model	Llama-3.1-70B-Instruct
Precision	FP8
Hardware	4×H100 SXM 80 GB NVLink
Dataset	ShareGPT (realistic ISL/OSL distribution)
SLO	TTFT P99 < 500 ms, ITL P50 < 30 ms
Warm-up	200 requests discarded before measuring
Sweep	concurrencies 1, 4, 8, 16, 32, 48, 64
Primary metric	goodput (useful tok/s under SLO)

Harness steps

Deploy the engine with the exact config (pinned flags, fixed version).
Warm-up: 200 requests; discard the results.
Concurrency sweep: aiperf profile --concurrency-range 1:64:step with AIPerf.
Extend the sweep past the knee (until goodput falls below 50 %).
Collect the JSON: TTFT P50/P99, ITL P50, raw throughput, goodput.
Change only the engine; repeat steps 1–5.
Compare the goodput column across rows.

The sweep with GuideLLM is equivalent for SLO validation; AIPerf is preferable for engine comparisons because its automatic estimatedCapacity normalises the saturation point. Both can be used: GuideLLM for the operational SLO, AIPerf for reference capacity.

Reproducible output format

{
 "harness": "aiperf", "harness_version": "2.x",
 "model": "Llama-3.1-70B-Instruct", "precision": "FP8",
 "hardware": "4xH100_SXM_80GB_NVLink",
 "dataset": "sharegpt",
 "slo": {"ttft_p99_ms": 500, "itl_p50_ms": 30},
 "motor": "vllm", "motor_version": "0.9.x",
 "concurrency_sweep": [8, 16, 32, 48, 64],
 "results_at_knee": {
 "concurrency": 32,
 "ttft_p50_ms": 210, "ttft_p99_ms": 420,
 "itl_p50_ms": 18, "throughput_tok_s": 12500,
 "goodput_tok_s": 12200
 }
}

This versioned JSON is the auditable datum: anyone can reproduce the figure with the same tool, version, model, hardware and load.

State of the art 2026: relative maturity

Dimension	vLLM	SGLang	TRT-LLM	Dynamo
Production maturity	High (3+ years)	Medium-high (2 years)	High (NVIDIA)	Medium (1.0 GA Mar 2026)
Release cadence	Weekly	Weekly	Monthly	Fortnightly
Integration ecosystem	Very broad (Ray, k8s, llm-d, Dynamo, NIM)	Broad (Dynamo, k8s)	NVIDIA-centric (Triton, NIM)	NVIDIA-centric
Ease of operation	High (pip install, OpenAI API)	High	Medium (requires compiling engines)	Medium-low (multi-node, etcd/NATS)
Non-NVIDIA hardware	Yes (AMD, Intel, TPU)	Yes (AMD)	No	No
Disaggregated serving	Beta	Stable	Stable	First class
Dynamic multi-LoRA	Stable	Stable	Beta	Stable (via vLLM/SGLang)
Structured output	Stable	Stable (FSM)	Beta	Stable (via the engine)
OSS community support	Very high (>30 K GitHub stars)	High (~20 K stars)	High (~10 K stars)	Growing (6.8 K stars)

Pareto decision table: when to choose each engine

No prose; the decision as a table of criteria and outcome.

Selection criterion	Recommended engine	Quantitative justification
Single node, quick prototype, team with no LLM serving experience	vLLM	pip install + OpenAI API in minutes; the broadest community support
Workload with long shared prefixes (RAG, few-shot, system prompts > 512 tok)	SGLang	RadixAttention: up to 6.4× more throughput vs a baseline without prefix caching
Maximum throughput on NVIDIA, no portability, team with TRT experience	TRT-LLM	Fused kernels + native FP8: ITL 9–12 ms vs 18–22 ms for vLLM on the same hardware
Multi-node (> 8 GPUs), mixed prefill-intensive/decode-intensive traffic	Dynamo + SGLang	Disaggregated P/D: up to 7× additional throughput on Blackwell; 2× TTFT with KV-aware routing
AMD ROCm or Intel Gaudi hardware	vLLM	The only SOTA engine with stable non-NVIDIA support
Structured output + high concurrency	SGLang	Compressed FSM: guided decode with no appreciable overhead
Stable dynamic multi-LoRA in production	vLLM or SGLang	TRT-LLM multi-LoRA still in beta (Jun 2026)
Hugging Face ecosystem, models < 13B	HF TGI v3	Direct HF Hub integration; competitive latency on small models
Maximum energy efficiency (J/token) on NVIDIA	TRT-LLM	Lower-overhead kernels → fewer J/token at iso-throughput

Quantisation: effect on throughput and quality

The four main options in order of decreasing speed and increasing quality:

Format	Relative throughput (H100)	Quality degradation (perplexity)	Engines with stable support
NVFP4 / MXFP4	~2.2× vs FP16	1–3 % on MMLU	TRT-LLM (Blackwell), vLLM (experimental)
FP8 (W8A8)	~1.7× vs FP16	< 1 % on MMLU	vLLM, SGLang, TRT-LLM
INT4 AWQ	~1.5× vs FP16	1–2 % on MMLU	vLLM, SGLang, TRT-LLM
GPTQ (INT4)	~1.4× vs FP16	1–3 % on MMLU	vLLM, SGLang
FP16 (baseline)	1.0×	0 %	All

FP8 is the de facto standard on Hopper for production: 1.7× throughput with degradation below 1 %. The article on quantisation in depth develops the quality/speed trade-off by format and model architecture.

Disaggregated serving: when the overhead is worth it

Separating prefill/decode into independent pools introduces transfer latency for KV tensors between nodes (NIXL over NVLink or InfiniBand). The overhead pays off when:

Condition	Effect of disaggregated serving
Mean ISL > 1,024 tokens	Prefill dominates; separate pools stop it blocking decode
Prefill/decode ratio > 3:1 in time	The decode pool sits idle waiting for prefill
Mixed traffic (prefill burst + decode queue)	Independent scaling of each pool by SLA
< 512 tokens ISL, unique prompts	Transfer overhead exceeds the benefit; an aggregated engine is enough

Dynamo quantifies the break-even point: disaggregated serving pays off when the transfer time for KV tensors (a function of ISL and NVLink/IB bandwidth) is lower than the time the decode worker would spend waiting for the prefill worker in aggregated mode. On 4×H100 with an ISL of 512 tok, break-even occurs at a prefill/decode ratio > 2.5:1. The article on disaggregated serving develops the full formula.

MLPerf Inference v5.1: cross-vendor reference figures

MLPerf Inference v5.1 (September 2025) is the standard for cross-vendor comparability. Server scenario (Llama-3.1-70B, tokens/s):

Submitter	Hardware	Scenario	Throughput (tok/s)
Red Hat (vLLM)	1×H100 80 GB	Server	5,103
Red Hat (vLLM)	1×L40S	Server	1,207
NVIDIA (TRT-LLM)	8×H100 SXM	Server	~52,000 (estimated)

The MLPerf figures are not directly comparable with the internal benchmarks above (a different harness, MLPerf’s Server scenario with a Poisson distribution and fixed SLOs). They are useful for comparing vendors under the same rules; not for sizing a specific case. For that, run your own sweep with AIPerf/GuideLLM against the target endpoint.

Variables the benchmark does not capture

Omitted variable	Effect in production
Network latency (gateway, load balancer)	Can add 20–100 ms to the TTFT measured on the server
Engine compilation time (TRT-LLM)	Hours per model×precision×hardware; penalises cold starts and CI
Weight loading time on the GPU	Relevant in autoscaling; ModelExpress (Dynamo) cuts it 7×
Operational cost of the system (Dynamo vs a simple engine)	Dynamo requires etcd/NATS, a Rust runtime, a more specialised team
Response quality	Performance does not imply quality; measure with lm-evaluation-harness (article B7)
Energy (J/token)	Does not come out of a throughput benchmark; requires DCGM running alongside

Sources

vLLM · documentación oficial — https://docs.vllm.ai/en/latest/
vLLM · blog de rendimiento (v0.6.0, 2,7× throughput) — https://blog.vllm.ai/2024/09/05/perf-update.html
vLLM · disaggregated prefill (experimental) — https://docs.vllm.ai/en/latest/features/disagg_prefill/
SGLang · documentación oficial — https://docs.sglang.io/
SGLang · paper RadixAttention (LMSYS, 2024) — https://arxiv.org/pdf/2312.07104
SGLang · blog LMSYS sobre Llama-3 serving — https://www.lmsys.org/blog/2024-07-25-sglang-llama3/
NVIDIA TensorRT-LLM · GitHub — https://github.com/NVIDIA/TensorRT-LLM
NVIDIA TensorRT-LLM · overview y release notes — https://nvidia.github.io/TensorRT-LLM/overview.html
NVIDIA TensorRT-LLM · quantización — https://nvidia.github.io/TensorRT-LLM/features/quantization.html
NVIDIA Dynamo · GitHub (ai-dynamo/dynamo) — https://github.com/ai-dynamo/dynamo
NVIDIA Dynamo · documentación oficial — https://docs.nvidia.com/dynamo/latest
NVIDIA Dynamo · disaggregated serving — https://docs.dynamo.nvidia.com/dynamo/design-docs/disaggregated-serving
Cerebrium · benchmark vLLM vs SGLang vs TRT-LLM (Llama 3.1-70B) — https://cerebrium.ai/blog/benchmarking-vllm-sglang-tensorrt-for-llama-3-1-api
Spheron · vLLM vs TRT-LLM vs SGLang H100 benchmarks 2026 — https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/
MLCommons · MLPerf Inference v5.1 results — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
Red Hat · MLPerf Inference v5.1 con vLLM — https://www.redhat.com/en/blog/efficient-and-reproducible-llm-inference-red-hat-mlperf-inference-v51-results
arXiv 2605.24217 · sesgo sistemático en benchmarks de inferencia LLM — https://arxiv.org/html/2605.24217
IETF Draft · LLM Benchmarking Methodology (enero 2026) — https://www.ietf.org/archive/id/draft-gaikwad-llm-benchmarking-methodology-00.html

LLM benchmark tool catalogue: the practical deep dive

Sun, 14 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

What this article covers

Second article in the benchmarking track (B2). The B1 introduction pinned the metrics and the methodology; this article is the practical manual: for each tool, what it measures, how it is actually invoked (with the command), what output format it produces and when to choose it. The goal is that, after reading it, you know what to run to measure your engine and you can reproduce the number. No universal recommendations; only the mechanics of each tool and the criterion for choosing.

How to choose: the tool map

Recalling the split from B1, the tools are ordered on two axes: micro-bench (single-process, for tuning an engine) versus load generator (multi-process, for measuring real capacity), and engine-native versus endpoint-agnostic.

Datasets: the synthetic load matters as much as the tool

Before the profiles, one point that changes results without anyone noticing: what load you feed it. The tools accept different kinds of dataset, and each one measures something different:

Dataset	What it simulates	Bias it introduces
random (fixed lengths)	controlled uniform load	unrealistic: traffic is not fixed length
sharegpt (real conversations)	realistic prompt distribution	the de facto standard for comparison
your own traces	your real traffic	the most faithful, but specific to your case

The trap: a benchmark with short, fixed-length prompts gives a very high throughput that looks nothing like production, where lengths vary and long prompts dominate the prefill cost. For a defensible figure, use sharegpt (comparability) or, better, traces from your own traffic (fidelity). And always declare the length distribution (prompt/output) alongside the number, because two benchmarks with different datasets are not comparable even if they use the same tool.

vLLM bench serve

What it measures: TTFT, TPOT, throughput and latencies of the vLLM server under a synthetic load. Class: micro-bench (single-process). It is the tool for tuning vLLM and seeing the effect of its optimisations (decode, prefill).

Typical invocation (vLLM · benchmark CLI):

vllm bench serve \
 --backend vllm \
 --model meta-llama/Llama-3.1-8B-Instruct \
 --endpoint /v1/completions \
 --dataset-name sharegpt \
 --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
 --num-prompts 1000

Key parameters: --num-prompts (load), --dataset-name (sharegpt, random, etc.), --request-rate (requests/s). The output is a console summary with TTFT, TPOT, throughput and percentiles, and it can be dumped to JSON. To sweep concurrencies there is vllm bench sweep serve, which automates the sweep (vLLM · sweep).

Limit: single-process, it saturates on the client at high concurrency; less flexible than GuideLLM in datasets and load patterns. Use it to iterate quickly on the vLLM config, not to measure maximum capacity at scale.

SGLang bench

What it measures: the equivalent for the SGLang engine (TTFT, TPOT, throughput). Class: micro-bench. Use: tuning SGLang and comparing it against itself across configurations. The mechanics are analogous to vllm bench serve: a synthetic load, console/JSON output with the same metrics. If you are evaluating SGLang against vLLM, do not compare them with their respective micro-benches (same class but different implementations): use an endpoint-agnostic load generator (GuideLLM/AIPerf) against both endpoints, so that the tool is not the variable.

AIPerf (NVIDIA, formerly genai-perf)

What it measures: TTFT, ITL, throughput and latency against any compatible endpoint (vLLM, NIM, TGI, SGLang). Class: multi-process load generator. It is the successor to genai-perf (retired on 15 Apr 2026). Its distinguishing feature: during the sweep it detects GPU saturation and returns the previous iteration as estimatedCapacity, which is why the sweep has to extend beyond the knee (AIPerf).

Output: structured metrics (JSON) with TTFT/ITL distributions and the estimatedCapacity. Use it when you want the real capacity of an endpoint, whatever the engine, with automatic detection of the saturation point.

GuideLLM (vLLM project)

What it measures: full distributions of TTFT, ITL and end-to-end behaviour, for SLO-driven evaluation. Class: multi-process load generator. It is the recommended tool for benchmarking vLLM servers in production: more flexible than vllm bench serve in dataset loading, request format and traffic patterns, with live progress and automatic report generation (Red Hat).

Typical invocation:

guidellm benchmark \
 --target "http://localhost:8000" \
 --rate-type throughput \
 --max-requests 1000 \
 --data "samples=1000,prompt_tokens=1024,output_tokens=256"

Key parameters: --rate-type (synchronous, concurrent, throughput, or rate-based), --data (the load specification: number of samples and prompt/output lengths), --target (the endpoint). It generates reproducible sweeps to find the safe operating range under SLO, with full distributions (not just means). Output: a report with percentiles and, often, exportable to JSON/HTML. It is the default option for answering “how hard can I load this engine without breaking the SLO?”.

LLMPerf (Anyscale/Ray)

What it measures: throughput and latency at the inference level. Class: load generator. Use: endpoint validation, historically very widespread in the Ray/Anyscale ecosystem. It is a solid and well-known option, though less focused on distributions and sweeps than GuideLLM/AIPerf. It fits if you already operate on Ray or want a simple tool to validate an endpoint.

inference-benchmarker (Hugging Face)

What it measures: latency and throughput of inference endpoints, aimed at producing comparable reports. Class: load generator. Use: an OSS alternative inside the Hugging Face ecosystem, useful if you already work with TGI or the HF stack. Like the others in its class, its value lies in measuring real capacity with distributed load; the choice between this one, GuideLLM and AIPerf usually comes down to the ecosystem you already operate in rather than to deep differences in what they measure.

GuideLLM versus AIPerf: which of the two load generators

They are the two serious multi-process load options, and they overlap a great deal. The practical differences that tip the choice:

Criterion	GuideLLM	AIPerf
Origin	vLLM project (Red Hat)	NVIDIA (successor to genai-perf)
Focus	SLO-driven evaluation, reproducible sweeps	real capacity, automatic estimatedCapacity
Load patterns	synchronous, concurrent, rate-based	sweep with saturation detection
Report	live progress + automatic report	structured metrics (JSON)
Ecosystem	vLLM / OpenShift AI	NVIDIA NIM / Triton / vLLM

In practice: if your question is “how hard can I load it without breaking the SLO?”, GuideLLM answers it more directly with its SLO-driven sweeps; if your question is “what is the maximum capacity of this endpoint?”, AIPerf gives it with its automatic knee detection. Many teams use both: GuideLLM for the operational SLO, AIPerf for the reference capacity. What you must not do is compare a GuideLLM result with an AIPerf one as if they were the same measurement: although both are multi-process, their sweep methodology differs; pick one for a given comparison and stick to it.

MLPerf Inference (MLCommons)

What it measures: performance under standardised scenarios (Offline, Server, Interactive) with strict rules. Unlike the previous ones, MLPerf is not “run” for your day-to-day case: it is a vendor comparison suite whose results are read (published by NVIDIA, AMD, Intel and others). Use it to compare hardware and engines against each other under identical rules, not to size your specific workload. For that, GuideLLM/AIPerf against your endpoint.

Practical comparison table

Tool	Class	Base command	Output	When
vllm bench serve	micro	`vllm bench serve`	console/JSON	tune vLLM
vllm bench sweep	micro (sweep)	`vllm bench sweep serve`	console/JSON	sweep concurrency in vLLM
SGLang bench	micro	bench from the SGLang repo	console/JSON	tune SGLang
AIPerf	load	`aiperf profile …`	JSON + estimatedCapacity	real capacity, multi-endpoint
GuideLLM	load	`guidellm benchmark …`	report + JSON/HTML	SLO, reproducible sweep
LLMPerf	load	Ray/LLMPerf script	JSON	validate endpoint (Ray)
MLPerf	suite	(results are read)	official results	compare vendors

Summary: what you run for each question

So as not to get lost in the catalogue, the direct mapping from question to tool:

Your question	Tool	Why
“Does my vLLM config change help?”	`vllm bench serve` (+ sweep)	fast, engine-native
“How hard can I load without breaking the SLO?”	GuideLLM	SLO-driven sweeps
“What is the maximum capacity of the endpoint?”	AIPerf	automatic knee detection
“vLLM or SGLang for my workload?”	GuideLLM/AIPerf against both	same tool, engine as the variable
“Which hardware/engine is better in the abstract?”	read MLPerf	cross-vendor comparability
“Is there a regression in this release?”	short sweep in CI against a baseline	continuous detection

The rule underlying the whole table: micro-bench to iterate on an engine, load generator to measure capacity and decide, MLPerf to compare vendors. And, for any comparison between systems, the same tool for every candidate. If these three categories are clear to you, the specific choice is secondary.

Step-by-step benchmark methodology

A reliable run is not “launching the command”: it is a procedure. The steps, with the load tool (GuideLLM/AIPerf) against your endpoint:

Deploy the engine with the exact config to be measured (model, precision, flags), pinned.
Warm up: send a few requests and discard them, so that the prefix cache and the autotuning do not inflate the first numbers.
Sweep: sweep rising concurrencies (1, 8, 16, 24, 32…) beyond the knee, to see where latency takes off.
Collect the output in JSON with percentiles (TTFT/ITL/throughput/goodput) and the metadata.
Compare: read the goodput under the SLO, not the maximum throughput.

Skipping step 2 (warm-up) or not going past the knee in step 3 are the two mistakes that bias the result the most.

Comparing two engines: the fair protocol

If the goal is to choose between vLLM and SGLang (or TRT-LLM), the protocol that avoids false conclusions:

Same load tool (GuideLLM or AIPerf) against both endpoints, never each engine’s own micro-bench.
Same dataset and length distribution.
Same hardware and precision.
Same SLO to compute the goodput of both.
Vary only the engine; everything else fixed.

Only then is the difference you measure down to the engine and not to the tool, the dataset or the hardware. It is the controlled experiment that supports the scorecard row (article B8).

Output format and comparability

What you record alongside the number is what makes it comparable. A useful output includes, in JSON so it can be versioned:

{
 "tool": "guidellm", "version": "x.y.z",
 "model": "Llama-3.1-70B", "precision": "FP16",
 "hardware": "8xH100 SXM NVLink",
 "load": {"prompt_tokens": 1024, "output_tokens": 256, "concurrency": 16},
 "results": {"ttft_p50_ms": 180, "ttft_p99_ms": 460,
 "itl_p50_ms": 22, "throughput_tok_s": 3400, "goodput_tok_s": 3330}
}

Saving this for every run turns a benchmark into an auditable datum: anyone can reproduce the figure with the same tool, version, model, hardware and load. It is the material of the reproducible harness (article S4), and the difference between a defensible number and a console screenshot.

Worked example: reading the output of a sweep

An illustrative output from a GuideLLM sweep on a 70B on 8×H100 (SLO: P99 TTFT < 500 ms), just as you would read it:

Concurrency	TTFT P99 (ms)	ITL P50 (ms)	Throughput (tok/s)	Goodput (tok/s)
8	240	20	2,100	2,100
16	460	22	3,400	3,330
24	980	31	3,900	2,420
32	1,800	54	4,000	800

How to read it: the knee sits between 16 and 24. At concurrency 16, P99 (460 ms) meets the SLO and goodput (3,330 tok/s) ≈ throughput. At 24, throughput rises little (3,400 → 3,900) but P99 already violates the SLO and goodput falls to 2,420. At 32, throughput is at its maximum (4,000) but goodput collapses to 800: the system “performs well” while serving requests that do not comply. The defensible capacity is the one at concurrency 16 (3,330 useful tok/s), and that is the number that goes into capacity planning and into cost per token. Anyone reporting “4,000 tok/s” is describing the point where the system no longer meets its SLO.

Automating the harness

A manual run does not scale to a benchmarking programme. The minimum automation:

An idempotent script that deploys the engine, warms it up, runs the sweep and saves the JSON with all the metadata (model, version, hardware, dataset, SLO).
Naming by date and config, to version the runs and compare over time.
A results store (a git repo of JSONs, or a bucket) so that anyone can reproduce and compare.

The goal is that reproducing a number is one command, not an afternoon. It is the basis of the harness in article S4, and what turns benchmarking from a one-off activity into a continuous platform capability.

CI integration: continuous benchmarking

The next level is measuring on every change: a CI job that, when the engine or the config is updated, launches a short sweep against a test environment and compares against the baseline. If goodput falls by more than a threshold, the pipeline fails. That way a performance regression is caught at the commit, not in production. Watch out for two things: benchmarking in CI consumes GPU-hours (budget for it) and it needs a stable environment (same hardware) for the comparison to be valid. You do not need the full sweep on every commit: a short sweep covering the knee is enough to detect regressions; the exhaustive sweep is for releases.

The cost of measuring (in euros)

Benchmarking consumes GPU-hours, and that has a cost worth budgeting for. A serious sweep on an 8×H100 node can occupy the cards for a couple of hours; at an amortised cost of ~11 €/h, that is ~22 € per full sweep, more if you sweep several models and precisions. It is not much per run, but a continuous benchmarking programme (every release, every config change) adds up. The practical rule: automate the harness so that each run is cheap and reproducible, and measure what you are going to use to decide, not for the sake of exhaustiveness. The cost of measuring is part of the cost of the platform, small next to the cost of serving but real.

Observing during the benchmark: one run, three axes

A trick that saves work and connects the series: while the sweep runs, capture the GPU metrics too. With DCGM exporting to Prometheus during the run, you record at the same time:

Axis	Source during the sweep	Metric
Performance	the tool (GuideLLM/AIPerf)	TTFT, ITL, throughput, goodput
Energy	DCGM	power (W) → J/token
Cost	node price (OpenCost)	€/hour → CPM per point

That way, from a single run you get the three numbers for the same operating point: at concurrency 16, the goodput (3,330 tok/s), the average power (from DCGM, which divided by the throughput gives J/token) and the cost per token (with the node price). Instead of three separate campaigns, you measure the three axes at once, and they end up coherent by construction because they correspond to the same instant and the same load. It is what the harness in article S4 does, and the reason to export DCGM during the benchmark even if you are only after performance: energy and cost come almost free if you capture them in the same window.

The methodological warning: align the time windows. The DCGM power figures and the tool’s metrics have to cover exactly the same interval (no warm-up, no shutdown), or the J/token does not correspond to the throughput measured. Same window, same three numbers.

The benchmark is also FinOps and energy

An idea that connects this article with the rest of the series: measuring performance is, in fact, measuring cost and energy. By the identity of the opening article, goodput is the denominator of cost per token and of energy per token. When a sweep reveals that config A gives 3,330 tok/s of goodput and config B gives 4,000, you are not only measuring speed: you are measuring that B costs fewer euros and fewer watts per token. That is why the output JSON of a benchmark should be accompanied by the iron cost (from OpenCost) to compute the real CPM of every point in the sweep: throughput × node price = cost per token. Benchmarking is not an isolated axis; it is the tool that, indirectly, moves the cost and the energy of the platform the most, and the one that fills the performance column of the scorecard with numbers that translate directly into euros.

The operational consequence: do not benchmark performance in a vacuum. Every run you save with its throughput and its goodput should be crossable with the node cost (€/hour) and the energy (J/token) to give the three faces of the same operating point. That is how the benchmarking track feeds the FinOps and energy tracks instead of living apart.

State of the art 2026 and limits

Migration to multi-process: AIPerf (formerly genai-perf) and GuideLLM consolidate the measurement of real capacity; micro-benches are left for tuning engines.
GuideLLM as the OSS standard for SLO-driven evaluation with an automatic report.
Careful with comparing across classes: a micro-bench and a load generator are not comparable; pin the class and the tool.
Version and dataset matter: the same command with a different dataset or version gives a different number; pin them.
MLPerf does not size your case: it compares vendors, it does not replace a sweep against your workload.

With the practical catalogue covered, the next article in the track (B3) goes into GuideLLM and SLO validation under load in depth. The tool is the means; the reproducible datum is the end.

Mistakes that invalidate a benchmark

To finish, the list of what turns a run into rubbish, by frequency:

Mistake	Effect	Fix
Comparing different classes	up to 7× of spurious difference	same tool for everyone
Not discarding the warm-up	artificially low TTFT	warm up and discard
Not going past the knee	you do not know the safe capacity	extend the sweep
Unrealistic dataset (fixed length)	throughput that does not apply	sharegpt or your own traces
Reporting the mean, not P99	hides the tail	percentiles always
Tokenizer foreign to the model	biased tok/s and cost per token	count with the model’s own
Not pinning version/config	irreproducible	save everything in the JSON

Any one of these is enough for the number not to be defensible. A benchmark is only as good as its methodology: the tool matters less than running it properly and recording everything.

Closing

The catalogue of benchmark tools boils down to a simple decision (micro-bench to tune an engine, load generator to measure capacity, MLPerf to compare vendors) and to a discipline that weighs more than the choice: the method. The same guidellm benchmark gives a golden datum or a useless one depending on the dataset, the warm-up, how far the sweep goes and what you record in the output. For a sovereign architecture proposal, performance only counts if it comes with its command, its version, its load and its goodput under SLO, and, crossed with cost in euros and energy per token, it becomes the scorecard column that decides which engine and which configuration hold up the platform. You choose the tool in five minutes; the methodology is what makes the number survive an audit.

Sources

vLLM · CLI de benchmark (bench serve) — https://docs.vllm.ai/en/latest/cli/bench/serve/
vLLM · bench sweep serve — https://docs.vllm.ai/en/latest/cli/bench/sweep/serve/
Red Hat · desplegar y benchmarkear vLLM con GuideLLM en Kubernetes — https://developers.redhat.com/articles/2025/12/24/how-deploy-and-benchmark-vllm-guidellm-kubernetes
GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
NVIDIA AIPerf · guía de benchmarking — https://lucaberton.com/blog/nvidia-aiperf-llm-inference-benchmarking-guide/
Medium · Benchmarking LLM Serving Performance (guía) — https://medium.com/@kimdoil1211/benchmarking-llm-serving-performance-a-comprehensive-guide-db94b1bfe8cf

Serving reasoning models: the invisible scratchpad that decides your latency and your bill

Fri, 12 Jun 2026 05:30:00 +0200

TL;DR

A reasoning model generates, before the answer you see, a thinking block, hundreds or thousands of tokens enclosed between <think> and </think>, that the user does not read but that consumes the same compute, the same VRAM and the same bill as any other token. Serving them in production is not “serving a bigger model”: it is serving a model whose cost per request is variable and, by default, uncontrolled. Three levers change the equation: choosing well between think / non-think mode (the hybrid models of 2026 allow it within the same weights), setting a reasoning budget (thinking_token_budget in vLLM, s1-style budget forcing) and measuring the impact on the KV cache (a 30k-token reasoning chain can eat ~9 GB of KV in FP16). Managed well, you gain accuracy where it matters; managed badly, you pay up to 113× more energy for an answer that did not need that much thinking.

The analogy

Picture a candidate sitting a competitive exam. Before writing the good answer on the official sheet, they fill three sheets of scratch paper: they try approaches, correct themselves, discard. Those scratch sheets are not handed in, the board only reads the final answer, but the candidate has spent ink, time and half an hour of the exam on them.

A reasoning model does exactly that. The <think>…</think> block is the scratch paper: it reasons out loud, contradicts itself, backtracks, and finally emits the “clean” answer. The production problem is that you pay for the scratch paper too: every reasoning token takes a slot in the batch, grows the KV cache and adds to the bill just like an output token. And, like any nervous candidate, the model tends to overthink: it writes five sheets when one would have done.

The operational question in this post is not “does it reason well?”, but “how much scratch paper do I let it write, and when do I take the pen away?”.

What changes when the model “thinks”

In a normal model, a request is: prompt → output tokens. In a reasoning model the sequence has two stretches:

Reasoning block (<think> … </think>): the chain of thought (CoT). Long, variable, normally hidden from the end user.
Final answer: what the user sees.

vLLM models this explicitly. When started with a reasoning parser, the output carries a reasoning_content field separate from content:

vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
 --reasoning-parser deepseek_r1

The parser detects the thinking delimiters and separates both stretches in the API response (vLLM · Reasoning Outputs). This matters for serving for one practical reason: if your gateway or your UI does not separate reasoning_content from content, you will end up showing the scratch paper, or, worse, recording it in logs and traces without meaning to.

The 2026 landscape: from the model that “always reasons” to the switchable hybrid

The first generation (o1, DeepSeek-R1, QwQ-32B) were models that always reasoned. In 2026 the dominant pattern is the hybrid: a single set of weights that switches between think and non-think.

DeepSeek-V3.1 introduced a hybrid reasoning architecture in a single model: DeepSeek-V3.1-Think for complex problems and a faster direct mode for the trivial, switchable by the user (InfoQ). Important for serving: reasoning is off by default and is turned on with thinking=True in chat_template_kwargs.
Qwen3 brings Hybrid Thinking Modes and an explicit Thinking Budget to bound the reasoning compute; in its series thinking is on by default (like QwQ-32B).
gpt-oss shows a different reasoning structure from the R1/Qwen family, which in practice means that the parser and the delimiters are not universal: each family has its own.

The architectural consequence is clear: in 2026 you no longer decide “a model that reasons” vs “one that does not”, but when you let the same model reason. That turns reasoning into a routing and budgeting decision, not a model-catalogue one. It connects directly with the L7 inference router: the “cheap” route turns thinking off; the “expensive” one turns it on with a cap.

The economics: why the scratch paper drives up the bill

Here is the heart of the problem. Reasoning tokens are invisible to the user but real in compute and in billing. Providers charge them as output tokens; API providers bill those reasoning tokens at the output price, not for free (Test-Time Compute Quietly Changed the Economics of Inference). On-premise there is no external invoice, but there is a cost: they occupy batch slots, KV VRAM and watts.

The billable cost (or the equivalent cost in GPU·hours) of a request is:

$$\text{compute tokens} = \text{prompt} + \underbrace{\text{reasoning tokens}}_{\text{hidden, variable}} + \text{visible answer}$$

The middle term is the one that runs out of control. On hard tasks the reasoning/visible-answer ratio can be enormous, and empirical analysis has measured up to 113× the energy cost against standard inference on certain coding tasks (Towards Data Science · Inference Scaling).

And it does not always buy accuracy. The overthinking phenomenon is documented: more thinking does not guarantee a better answer, and beyond a certain point it makes things worse. There is work showing that shorter chains are up to 34.5 % more accurate than the longest one sampled for the same question (Don’t Overthink it), plus studies on the “mirage” of test-time scaling (Mirage of Test-Time Scaling) and on when more thinking hurts (When More Thinking Hurts). The first large-scale study of test-time scaling, more than 30 billion tokens generated with 8 open-source LLMs from 7B to 235B across 4 reasoning datasets, systematises where it scales and where it does not (The Art of Scaling Test-Time Compute).

The operational reading: reasoning is a cost you have to budget for, not a “more quality” switch you leave on full all the time.

Controlling the thinking budget

Three mechanisms, from coarsest to finest.

1. Switching think / non-think

The cheapest thing is not to reason when it is not needed. On hybrid models, turn thinking off for classification, extraction, formatting, trivial answers:

# DeepSeek-V3.1: reasoning OFF by default; turned on explicitly
chat_template_kwargs = {"thinking": True} # only on the "hard" route

This is 80 % of the saving for 20 % of the effort: route by difficulty and reserve thinking for what actually pays for it.

2. Setting a cap: `thinking_token_budget`

vLLM ships a sampling parameter thinking_token_budget that counts reasoning tokens and, on reaching the limit, forces the block to close (reasoning_end_str, typically </think>), making the model answer straight away. If it is not specified, there is no limit other than max_tokens (PR #37112, implementation note). Underneath, a ReasoningBudgetLogitsProcessor injects something like “Let me stop thinking and answer now.</think>” when the cap is reached.

# Define the delimiters of the reasoning block
vllm serve Qwen/Qwen3-32B \
 --reasoning-parser qwen3 \
 --reasoning-config '{"reasoning_start_str":"<think>","reasoning_end_str":"</think>"}'

// per request: cut the scratch paper off at 1024 tokens
{"messages":[...], "thinking_token_budget": 1024}

3. Budget forcing (state of the art)

The technique that popularised fine-grained control is budget forcing, from the s1 paper: to spend less, thinking is terminated by forcing the end delimiter; to spend more, the end is suppressed and the word “Wait” is appended, which pushes the model to review and often correct its own reasoning. With this, s1-32B (a Qwen2.5-32B-Instruct fine-tuned on just 1,000 examples) beat o1-preview by as much as 27 % on AIME24/MATH (s1: Simple Test-Time Scaling). The serving lesson: the budget is not only a cost cap, it is a bidirectional quality lever.

$$\text{accuracy} \approx f(\text{reasoning tokens}) \quad\text{with a maximum, not monotonic}$$

The curve rises, saturates and on many tasks falls. Your job is to operate it near the knee, not at the extreme.

The impact almost nobody measures: the KV cache

This is where reasoning really bites serving. Every token of scratch paper is one more token in the KV cache. A long reasoning chain inflates the working memory just like a long input context.

The KV size grows linearly with the sequence (reasoning included):

$$\text{KV bytes} = 2 \times L \times h_{kv} \times d_{head} \times s \times b$$

where $L$ is layers, $h_{kv}$ KV heads (few, with GQA), $d_{head}$ the per-head dimension, $s$ the sequence length and $b$ the bytes per element. The term reasoning blows up is $s$: a “normal” 500-token answer and one with 8,000 tokens of thinking do not take up the same space, not by a long way.

The real numbers are frightening: for a distilled reasoning model with standard attention such as DeepSeek-R1-Distill-Llama-70B, a single 30,000-token reasoning chain consumes ~9 GB of KV in FP16, which quantising the KV to FP8 reduces to ~4.5 GB, one of the highest-impact optimisations for chain-of-thought workloads (Spheron · KV Cache Optimization). Multiply that by concurrency and you understand why a cluster that served 200 simultaneous requests of a “normal” model drops to 40 with the same model reasoning.

Two state-of-the-art nuances worth knowing:

Prefix caching is not free here. The empirical study of reasoning-model serving finds that prefix caching clearly improves speed on models of 14B or more, but hurts on 7B models (cache management costs more than it saves) (Reasoning Language Model Inference Serving Unveiled). And since the bulk of the reasoning cost sits in the decode (generating tokens one by one), prefix caching, which only speeds up the prefill, barely touches the expensive phase. It connects with prefix cache: engineering the hit rate.
Reasoning-specific KV compression. Techniques such as R-KV have appeared, compressing the KV by exploiting the redundancy typical of chains of thought (the model repeats itself and rambles), recovering memory without touching quality (R-KV).

The most profitable and available lever today is still KV in FP8 (see FP8 end to end) combined with a reasoning budget that avoids 30k-token tails in the first place.

Reasoning + structured output: order matters

If you serve structured output (JSON with a schema) and reasoning at the same time, there is a subtle trap: the grammar must not be applied during the thinking block. The model needs free text to reason; forcing the schema inside the <think> breaks the reasoning. vLLM solves it by having the structured-output engine (xgrammar) use the reasoner’s end_token_id to skip the constraint while the block lasts and apply it only to the final answer (vLLM · Reasoning Outputs). If you build this by hand on another engine, remember: first let it think, then impose the form. It links with structured output.

Reference architecture

On the blog’s example cluster, a node of 4×H100 SXM (80 GB, NVLink), a sensible topology for serving reasoning without wrecking the SLO:

A cheap difficulty classifier up front (a small model or rules) that picks the route. Most of the traffic does not need to think.
Fast route (non-think): the same hybrid model with reasoning off. Low TTFT, predictable cost.
Reasoning route (think): reasoning on with thinking_token_budget by default (say 1–2k tokens) and the option to raise it per request for the genuinely hard cases.
KV in FP8 in the engine on the reasoning route, plus per-request reasoning-length metrics exported to your observability (instrumenting vLLM with OTel).

To prototype this routing outside the cluster, an RTX 5090 (Blackwell, 32 GB) serves a 7–14B reasoning model and lets you validate parsers, budgets and the reasoning_content/content separation before touching the H100s. Do not expect to serve a 32B in think mode at high concurrency on a consumer card: the reasoning KV eats the 32 GB in no time.

Sizing starts, as always, from an SLO; the difference is that now the “answer size” in your capacity planning has to count the scratch paper, not just the visible answer.

Operational pitfalls (and honest scepticism)

Leaking reasoning_content. If the gateway does not separate it, you end up showing or logging the scratch paper, which may contain wrong attempts, sensitive data or language you do not want in your UI. Always separate it at L7.
Serving reasoning “just in case”. Overthinking is real and sometimes lowers accuracy. Do not turn think on by default: turn it on per route, with a budget.
Forgetting the KV. The most common mistake: sizing concurrency from the length of the visible answer. The KV is dictated by the total sequence, reasoning included. Measure p50/p95 reasoning tokens per endpoint.
Prefix caching as a silver bullet. On small models it can make things worse, and in any case it does not touch the decode, which is where the reasoning cost goes.
Assuming universal delimiters. R1, Qwen3 and gpt-oss do not share a reasoning structure. Use the right --reasoning-parser per family and test streaming (the parser has to separate correctly token by token).
Believing more compute = more intelligence. The state of the art (s1, short-m@k, “When More Thinking Hurts”) points the other way in many cases: there is a knee, and operating beyond it is burning watts. Test-time compute is a tool with diminishing, and sometimes negative, returns, not a linear lever.

A note of caution for June 2026: the field of reasoning control (budget forcing, reasoning-specific KV compression, thinking early-exit) moves fast and almost all of it is very recent. The specific figures, the 113×, the 34.5 %, the 9 GB, depend on model, task and configuration; take them as an order of magnitude for design, not as constants. Measure on your own workload before committing to SLOs.

Closing

Serving a reasoning model means serving a model whose cost per request you decide with the thinking budget, not the catalogue. The chain goes: route by difficulty → turn think off where it adds nothing → set thinking_token_budget by default on the route that thinks → quantise the KV to FP8 → measure reasoning length as a first-class metric. Do that, and reasoning becomes a capability you pay for only when it buys accuracy. Do not, and your cluster will serve a fifth of the users while the model fills sheets of scratch paper nobody is going to read.

Sources

vLLM · Reasoning Outputs — https://docs.vllm.ai/en/stable/features/reasoning_outputs/
vLLM PR #37112 · reasoning_budget — https://github.com/vllm-project/vllm/pull/37112
Adding Reasoning Budget to vLLM — https://shuyo.wordpress.com/2026/03/26/adding-reasoning-budget-to-vllm-reasoning-token-limit/
s1: Simple Test-Time Scaling — https://arxiv.org/abs/2501.19393
The Art of Scaling Test-Time Compute — https://arxiv.org/abs/2512.02008
Don’t Overthink it (short-m@k) — https://arxiv.org/pdf/2505.17813
Mirage of Test-Time Scaling — https://arxiv.org/pdf/2506.04210
When More Thinking Hurts — https://arxiv.org/html/2604.10739v1
Reasoning Language Model Inference Serving Unveiled — https://arxiv.org/pdf/2510.18672
R-KV: Redundancy-aware KV Cache Compression — https://arxiv.org/pdf/2505.24133
Spheron · KV Cache Optimization Guide (2026) — https://www.spheron.network/blog/kv-cache-optimization-guide/
Test-Time Compute Quietly Changed the Economics of Inference — https://medium.com/@Elongated_musk/test-time-compute-quietly-changed-the-economics-of-inference-9d0b8d77641c
Inference Scaling (Test-Time Compute) · Towards Data Science — https://towardsdatascience.com/inference-scaling-test-time-compute-why-reasoning-models-raise-your-compute-bill/
DeepSeek-V3.1 Hybrid Reasoning · InfoQ — https://www.infoq.com/news/2025/09/deepseek-v31-hybrid/

Structured output: the form with dropdowns that strikes out invalid answers before the model picks one — Outlines, XGrammar, LLGuidance and the maths of the bitmask

Sat, 30 May 2026 16:30:00 +0200

This post complements those on Continuous batching (where the scheduler lives) and Speculative decoding (another technique operating on the last mile of the sampler). Structured output is the LLM’s output contract towards the code that consumes it; without it, the integration between an LLM and applications is fragile by default.

You are here: DEPLOY

TL;DR

An LLM produces free text, but many applications (function calling, entity extraction, routing, text-to-SQL, config generation) need to parse it as JSON, as a tool call with typed args, as a valid SQL statement, or as one option from an enum. The naive solutions fail: prompt engineering (“answer in JSON”) leaves 25 % of outputs unparseable in many models; post-hoc validation plus retry costs latency and does not guarantee termination; json-repair libraries are heuristic patches. Constrained decoding guarantees conformance 100 %, by construction: at every generation step, before sampling from the softmax over the V vocabulary tokens, the tokens that would break the target structure are masked to -∞. The output is valid by mathematical contract, not by luck. The four dominant families in May 2026 are Outlines (Willard & Louf, 2023; FSM + token trie precomputed from regex/JSON Schema/CFG), XGrammar (Dong et al., 2024-25, CMU+MLC; byte-level pushdown automaton with an adaptive cache of context-independent tokens, the default in vLLM v1, SGLang, TensorRT-LLM, NIM and MLC-LLM), LLGuidance (Microsoft Research; Earley parser + regex derivatives, ~50 µs of CPU per token, underneath OpenAI Structured Outputs) and LM Format Enforcer (noamgat; aimed at JSON Schema, integrated in many engines as a fallback). XGrammar-2 (May 2026) introduces Structural Tag and a cross-grammar cache for dynamic agentic tool calling. The real cost when well integrated: <5 % in TPOT and ~40 µs of CPU per token of mask computation, partially overlappable with the model’s forward pass. The open operational question: does it degrade reasoning? The paper Let me Speak Freely? (Tam et al., EMNLP 2024) reported significant degradation in reasoning under format constraints; the dottxt rebuttal showed that the effect came from prompts that were not equivalent between experiments. The emerging consensus in May 2026: use two-pass (free reasoning with CoT in text plus structured output in a second call) for tasks that require multi-step reasoning; use single-pass constrained for extraction, classification and function calling, where forced structure improves exactness and reduces hallucinations. This post takes apart the mechanism, the maths (bitmask size = V/8 bytes, per-step latency), the comparison table of backends, the pitfalls (compile time, tokenizer-specific FSM, streaming SSE) and the production deployment pattern.

The analogy: the form with dropdowns instead of free-text fields

Imagine two ways of asking someone to fill in a form.

The first way is to hand them the paper with blank fields and say “fill it in with this exact format: the name goes here, then the ID number without spaces, then the date as YYYY-MM-DD, then mark one of these five possible reasons separated by semicolons”. You explain the format in exhaustive detail, but the person is still free to write whatever they like in each field. If they are in a hurry they may drop a zero from the ID number, put the date in the American format, tick two reasons when only one was asked for. When you receive the form, you often have to send it back: field X badly formatted, reason Y invalid, date Z impossible. This is exactly what an LLM does when you ask it to “answer in JSON with this schema”: it works most of the time, it fails a non-negligible percentage, and you have no formal guarantee of anything.

The second way is to give them an electronic form where the fields are not free text. The name accepts any text, but the ID number has a mask that only lets you type eight digits followed by a letter; the date is a picker that only lets you choose valid dates; the reasons are a dropdown with five closed options. The person can type whatever they want, but the form does not accept invalid characters at any point. The result is parseable by construction: when you receive the completed form, the ID number has exactly the expected format, the date is a real date, the reason is one of the five. This is constrained decoding.

The analogy holds on four mappings:

The person filling it in = the LLM producing logits over the vocabulary.
The characters the form lets you type in each field = the bitmask applied to the logits before sampling.
How the form knows which characters to allow depending on which field you are in = the automaton (FSM or PDA) that holds the current state of the grammar.
The fact that the dropdown is precomputed when the page loads rather than recomputed each time = the precomputed table of valid tokens per FSM state, which makes the per-step cost amortised O(1).

The problem structured output solves

The operational problem is the contract between the LLM and the code that consumes its output. There are three naive approaches, all with documented failures:

Pure prompt engineering. “Answer only with valid JSON, no comments or prose”. It works most of the time for good models; it fails between 5 % and 25 % of the time depending on the model, the temperature, the complexity of the schema and the length of the output. The model adds a trailing comma, escapes a quote badly, wraps the JSON in a ```json markdown block, hallucinates a field that was not in the schema, ignores a required field. SqueezeBits measures a ≤72 % correct rate without constraining for some models on JSON Schemas of moderate complexity.

Post-hoc validation plus retry. The server receives the output, tries to parse it, and if that fails returns the error to the model and asks it to try again. Cost: 2-3× latency in the worst case (typically 2-3 retries before giving up), no guarantee of termination, noise in the logs, hard to test deterministically.

JSON repair libraries (json_repair, the fast-json-repair Rust port). Heuristic patches for common errors: trailing commas, missing quotes, prose interleaved with JSON. Useful as a fallback for approach 2; they are not a contract.

The accumulated operational cost: latency inflated by retries, noise in production, painful debugging of intermittent parse errors, broken contracts with downstream clients that assumed clean parsing.

Constrained decoding: the principle

At each decode step, the model produces a vector logits ∈ R^V where V is the vocabulary size (Llama 3: 128,256, GPT-4o: ~200,000, Qwen 3: 152,064). The conventional sampler applies softmax plus a sampling strategy (greedy, top-k, top-p, temperature) over the V tokens.

Constrained decoding interposes one operation before the softmax: it applies a bitmask that sets to -∞ the logits of the tokens that would violate the target grammar at this step. The result: the softmax only assigns probability mass to admissible tokens; sampling, whatever its strategy, can only pick a valid one.

The two operational questions are always the same:

How do you know which tokens are valid at each step? → hold the current state of an automaton (FSM/PDA) built from the grammar; look up the table (state → set of valid tokens).
How much does it cost to compute and apply the mask? → this is where Outlines, XGrammar, LLGuidance and LM Format Enforcer compete.

The four families of backends

Backend	Origin	Algorithm	Grammar formats	Default in	Notes
Outlines	dottxt (2023)	FSM + precomputed token trie	regex, JSON Schema, Lark CFG	HF TGI	The Rust version (outlines-core) closes part of the gap with XGrammar
XGrammar	CMU + MLC (2024-25)	Byte-level PDA + adaptive ctx-indep/dep cache	regex, JSON Schema, EBNF/CFG	vLLM v1, SGLang, TensorRT-LLM, NIM, MLC-LLM	Speedup vs naive: up to 100× CFG, 3× JSON; <40 µs/token JSON
LLGuidance	Microsoft Research	Earley parser + regex derivatives	regex, JSON Schema, Lark	OpenAI Structured Outputs (internally), Chromium	~50 µs CPU/token; underneath Guidance, llama.cpp, mistral.rs
LM Format Enforcer	noamgat	char-level parser + tokenizer prefix tree	JSON Schema, regex	(deprecated default in many engines, fallback in NIM)	Slower than XGrammar (3.5× worse JSON, 10× worse CFG)

Three operational observations:

XGrammar is the de facto default in May 2026 across the open-source ecosystem (vLLM v1, SGLang, TensorRT-LLM, NIM, MLC-LLM). Its byte-level PDA with an adaptive cache gives it almost zero overhead when well integrated.
LLGuidance is the least known but most used piece on the market, because it sits underneath OpenAI Structured Outputs (confirmed in the README of the repo itself). 50 µs of CPU per token for 128k tokenizers.
Outlines was the first, it still holds the conceptual mindshare (papers, canonical blog posts), but it lost operational ground to XGrammar. The Rust version (outlines-core) closes part of the gap.

XGrammar in detail (what sits under vLLM and SGLang)

The paper by Dong, Yin, Ruan and Chen (arXiv:2411.15100, November 2024) introduces a partitioning technique that is key to understanding why XGrammar is 3-100× faster than the alternatives.

Partitioning the vocabulary at each PDA state:

Context-independent tokens (~99 % of the vocab in a typical JSON Schema): their validity is decided only by the current PDA position, without needing to inspect the full stack. These are precomputable and stored as bitmasks in a cache.
Context-dependent tokens (~1 %): they require inspecting the PDA stack at runtime. They are handled case by case at a higher cost.

That partition is the fundamental reason why XGrammar works in production at low TPOT. 99 % of the lookups go to a precomputed table in O(1); only the residual 1 % pays the real cost.

Other combined techniques: a pushdown automaton for full CFGs (not just regex), a persistent stack for fast branching/rollback, JIT compilation plus an Earley parser in XGrammar-2.

Reported speedups (paper):

Up to 100× over previous solutions on CFG.
3× on JSON Schema versus Outlines.
Latency <40 µs per token on JSON Schema; <200 µs on XML/Python DSL.
When well integrated in vLLM/SGLang/TRT-LLM: near-zero end-to-end overhead.

XGrammar-2 (May 2026, arXiv:2601.04426) adds two pieces that matter for agents:

Structural Tag: a composable JSON protocol that unifies OpenAI harmony, tool calling and reasoning channels.
Cross-Grammar Cache for reuse at sub-structure level → it allows dynamic switching between sub-grammars in agentic loops without recompiling.
6× faster compile time than XGrammar-1.

The maths of the bitmask

Three numbers drive the operational decision.

Bitmask size per step. If the vocabulary has V tokens, the bit-packed bitmask takes V/8 bytes:

$$\text{bitmask size} = \frac{V}{8} \text{ bytes}$$

Llama 3 (V=128,256): 16 KB per bitmask per request per step.
GPT-4o tokenizer o200k (V≈200,000): 25 KB.
Llama 2 (V=32,000): 4 KB.

For a batch of 32 requests with structured output active on an H100, we are talking about ~512 KB of bitmasks per step, trivial against the GBs the forward pass moves.

Cost of applying the bitmask to the logits. A simple CUDA kernel, O(V) complexity, ~5-10 µs of latency on an H100. Negligible.

Cost of computing the bitmask (CPU-side). This is where backends differ:

Backend	CPU latency per token (Llama 3, V=128k)
LLGuidance	~50 µs
XGrammar	~40 µs (JSON Schema), ~200 µs (XML/DSL)
outlines-core	comparable to XGrammar after 2024
Outlines Python (legacy)	200-1000 µs
LM Format Enforcer	intermediate, degrades with a large vocab

With a model forward pass on the order of 10-50 ms per token in decode, a 40-50 µs mask is <0.5 % overhead, invisible. The operational key is that the CPU-side mask computation can be overlapped with the GPU-side forward pass: while the GPU computes the logits for token t, the CPU precomputes the mask for token t+1 based on the FSM state after token t-1. SGLang does this explicitly; vLLM v1 improved substantially over v0.

Compile-time cost. Precomputing the FSM/PDA and the cache:

Simple schemas (1-5 fields): <100 ms.
Deep JSON Schemas with many $defs: seconds.
OpenAI structured outputs: “10s typical, up to 1 minute for complex schemas”, cached after the first call.
XGrammar-2: 6× faster than XGrammar-1.
LLGuidance: ~2 ms startup.

Operational best practice: pre-cache the known schemas when the server starts, to avoid latency spikes on the first request for each new schema.

Does it degrade the model’s reasoning?

It is the most interesting question in the field and the answer is not obvious.

The theoretical argument for degradation: forcing the structure changes the model’s probability distribution; if the structured path pushes towards low-probability paths, the model gets “stuck” in a suboptimal branch with no room to explore.

The paper Let me Speak Freely? (Tam et al., arXiv:2408.02442, EMNLP 2024 Industry Track) reported significant degradation on reasoning tasks under format constraints (JSON/XML/YAML). The stricter the format, the greater the degradation. The paper itself, however, acknowledged an improvement on classification tasks with forced structure.

The dottxt rebuttal, Say What You Mean (official blog post): a methodological critique. The prompts in the original paper were different between structured and unstructured (not apples to apples). The JSON prompts in the original experiment gave less information than the unstructured ones. Re-running with equivalent prompts (Llama-3-8B-Instruct), dottxt does not reproduce the degradation. The conclusion: the paper confused the format constraint with the prompt engineering of that constraint.

Emerging consensus in May 2026 (gathered from technical blogs and community empiricism):

For extraction, classification, function calling and routing: constrained decoding improves exactness and reduces hallucinations. It is the right tool.
For multi-step reasoning (maths, logic, code review): use two-pass:
1. First call: free reasoning with Chain-of-Thought in natural text (generous max_tokens, no constraint).
2. Second call: pass it the reasoning plus the schema, and ask it to produce structured output with constrained decoding.

The “reason then structure” pattern gives the best of both worlds: reasoning without a straitjacket plus an output guaranteed to be parseable.

Real implementations in May 2026

Engine	Default backend	Others available	API
vLLM v1	XGrammar (auto)	Outlines, Guidance (llguidance), LM Format Enforcer	`guided_json`, `guided_regex`, `guided_choice`, `guided_grammar` in the request
SGLang	XGrammar	Outlines, LLGuidance	`response_format.json_schema`, `extra_body.regex`, `extra_body.ebnf`
TensorRT-LLM	XGrammar	LLGTRT (Rust llguidance)	Official integration since Jan 2025
NVIDIA NIM	XGrammar (switched from Outlines in 2025)	LM Format Enforcer (requires `NIM_ENABLE_KV_CACHE_REUSE=0`)	Pluggable multi-backend
llama.cpp	Native GBNF	LLGuidance	Each candidate token is tested against the parse state
HF TGI	Outlines	XGrammar (experimental)	“Guidance” feature with `/generate` and `/chat/completion` with `tools`
MLC-LLM	XGrammar (native, same team)	—	Its own API

vLLM example:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")

response = client.chat.completions.create(
 model="meta-llama/Llama-3.1-70B-Instruct",
 messages=[{"role": "user", "content": "Extract name and age from: María is 34 years old"}],
 extra_body={
 "guided_json": {
 "type": "object",
 "properties": {
 "name": {"type": "string"},
 "age": {"type": "integer"}
 },
 "required": ["name", "age"]
 }
 }
)

SGLang equivalent:

response = client.chat.completions.create(
 model="meta-llama/Llama-3.1-70B-Instruct",
 messages=[...],
 response_format={
 "type": "json_schema",
 "json_schema": {
 "name": "person",
 "strict": True,
 "schema": {...}
 }
 }
)

Usage patterns in production

Function calling / tool use. The LLM produces {tool_name: enum, arguments: object} according to a schema. The dominant case today (OpenAI, Anthropic, open models). The schema guarantees that tool_name is in the set of available tools and that arguments has the right types.

Entity extraction. A schema with {name, address, phone, ...} from unstructured text. It moves schema adherence from 76% to 98% in vendor benchmarks.

Routing / classification. The LLM picks between N options (enum in the schema). The bitmask shrinks to a handful of valid tokens → almost zero overhead, maximum reliability.

Constrained SQL. An SQL grammar as GBNF or Lark → it avoids injection and syntax errors. Useful in text-to-SQL agents.

Code generation with a valid AST. A grammar for the target language (Python, Rust, a custom DSL). It guarantees that the output is compilable or parseable code.

Agentic loops. XGrammar-2 Structural Tag for dynamic switching between formats (tool call → reasoning channel → tool result) without recompiling the grammar.

Operational pitfalls

Compile time for large schemas. JSON Schemas with many $defs can take seconds to compile. This causes slow startup if they are loaded at boot, or latency spikes on the first request if they are loaded on demand. Mitigation: warm the cache at boot with the fleet’s known schemas.

Tokenizer-specific FSM/PDA. A schema precompiled for Llama 3 (128k tokenizer) does not work for Qwen (152k). The schema cache must be keyed by (tokenizer_hash, schema_hash). Changing model = invalidating the cache.

Schema changes. Version schemas explicitly. Breaking changes → rebuild plus warm cache. Do not silence compile errors in production.

Streaming SSE. Structured output with streaming requires parsers tolerant of partial output (partial-json-parser, json-stream). Strict Pydantic v1 fails; v2 with partial validation works. Older clients may not handle incremental validation, so test with the real client.

Token healing. Tokens that cross boundaries (:// as a single token versus : + //) can break pure constraints. Outlines and XGrammar mitigate this internally; llama.cpp and others require explicit token healing. If the model sneaks a “strange” character in after the structure, this is probably why.

Real known bugs in vLLM 2025-26:

vLLM 0.8.4: XGrammar rejects minItems in JSON Schema (issue #16880).
vLLM with Qwen 2.5 VL: Outlines/XGrammar does not respect the schema in some cases (issue #13038).
The vLLM apply_token_bitmask_inplace bitmask backend is always auto, not exposed to the user.

OpenAI subset limitations: if you are going to port a schema validated on OpenAI structured outputs to another backend, check that you are not using constructs that OpenAI rejects but others allow (deep $ref, pattern, default, depth >5, anyOf as root).

Implications on on-premise hardware

On an RTX 4090 (24 GB). Any model you serve with vLLM or llama.cpp can carry structured output at practically negligible cost. The CPU-side mask latency (~40 µs) is trivial compared with typical TPOT (30-100 ms on consumer hardware). The interesting case: function calling over Llama 3 8B or Qwen 3 14B INT4 → reliable tool use without retries, with no need for a hosted API.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). Here XGrammar as the vLLM v1 / SGLang default is the way:

Llama 3 70B FP8 + XGrammar JSON Schema: TPOT P95 stable under 60 ms even with structured output active across the whole batch. It supports hundreds of distinct cached schemas.
DeepSeek-V3 + XGrammar Structural Tag: agentic tool calling with native MTP and constrained decoding combined; the mask cost overlaps with the MoE forward pass.
Multi-tenant function calling: each client can have its own set of tools (that is, its own schemas); compile time is amortised through caching, runtime is invariant.

The rule of thumb for May 2026: XGrammar by default, pre-cache the fleet’s schemas at boot, two-pass for reasoning tasks.

What we have not covered

Dynamic tool routing with XGrammar-2 Structural Tag: the detail of how TagDispatch picks sub-grammars at runtime.
Constrained beam search and its interaction with grammar: theoretical quality degradation versus greedy.
Grammars for code generation with a full AST: production-grade Python/Rust grammars, performance.
JSON Schema → Pydantic → grammar pipelines: tooling to reduce human error.
Inhibition decoding (the Inhibition Decoding paper, 2025): a variant that penalises but does not forbid certain tokens, useful for soft safety constraints.

References

Willard, B., Louf, R. Efficient Guided Generation for Large Language Models (Outlines). 2023. https://arxiv.org/abs/2307.09702
Dong, Y., Yin, X., Ruan, F., Chen, T. XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. 2024. https://arxiv.org/abs/2411.15100
XGrammar-2: Dynamic Structured Generation for Agentic LLMs. 2026. https://arxiv.org/abs/2601.04426
Tam, Z. et al. Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models. EMNLP 2024 Industry. https://arxiv.org/abs/2408.02442
dottxt blog, Say What You Mean: https://blog.dottxt.ai/say-what-you-mean.html
OpenAI, Introducing Structured Outputs in the API: https://openai.com/index/introducing-structured-outputs-in-the-api/
Outlines repo: https://github.com/dottxt-ai/outlines
outlines-core (Rust): https://github.com/dottxt-ai/outlines-core
XGrammar repo: https://github.com/mlc-ai/xgrammar
LLGuidance (Microsoft Research) repo: https://github.com/guidance-ai/llguidance
LM Format Enforcer repo: https://github.com/noamgat/lm-format-enforcer
llama.cpp grammars: https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md
vLLM Structured Outputs docs: https://docs.vllm.ai/en/stable/features/structured_outputs/
SGLang Structured Outputs docs: https://docs.sglang.io/advanced_features/structured_outputs.html
TensorRT-LLM guided decoding (Triton): https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/tensorrtllm_backend/docs/guided_decoding.html
NIM Structured Generation: https://docs.nvidia.com/nim/large-language-models/1.12.0/structured-generation.html
MLC blog, XGrammar: https://blog.mlc.ai/2024/11/22/achieving-efficient-flexible-portable-structured-generation-with-xgrammar
MLC blog, XGrammar-2: https://blog.mlc.ai/2026/05/04/xgrammar-2-fast-customizable-structured-generation
SqueezeBits, Guided decoding performance: vLLM vs SGLang: https://blog.squeezebits.com/guided-decoding-performance-vllm-sglang
Red Hat, Structured outputs in vLLM: https://developers.redhat.com/articles/2025/06/03/structured-outputs-vllm-guiding-ai-responses
Aidan Cooper, Constrained Decoding guide: https://www.aidancooper.co.uk/constrained-decoding/

Continuous batching: the salon with 8 chairs that does not wait for the slow customer — Orca, vLLM, chunked prefill and goodput

Sat, 30 May 2026 16:00:00 +0200

This post complements those on KV cache (the artefact continuous batching manages), PagedAttention (the memory piece that makes it viable), Disaggregated serving (the next layer of optimisation), Speculative decoding, Multi-LoRA and MoE (the three extensions that coexist with the scheduler in production).

You are here: DEPLOY

TL;DR

The static batching of the original HuggingFace Transformers (the pre-2022 era) systematically under-used the GPU for two structural reasons. First: the scheduling unit was the complete request; the longest one in the batch blocked all the others until it finished (severe head-of-line blocking, P95 TTFT five to ten times worse than reasonable). Second: each batch slot reserved memory for max_seq_len even though the real output was much shorter; documented padding waste sat between 60 % and 80 % and sustained GPU SM utilisation on real workloads fell to 20-40 %. Orca (Yu et al., OSDI 2022, FriendliAI + Seoul National University) introduced the idea that unjammed everything: the scheduling unit stops being the request and becomes one decoder iteration, a single token. After each iteration the scheduler can add new requests to the batch and remove the finished ones. vLLM (Kwon et al., SOSP 2023, UC Berkeley) materialised it open-source and production-grade thanks to PagedAttention, which solves the KV cache fragmentation that theoretical continuous batching would cause with dynamic allocation. SARATHI / Sarathi-Serve (Microsoft Research India, OSDI 2024) closed the last gap: the prefill stalls that paused active decodes when a new request arrived, via chunked prefill (splitting a long prefill into small chunks and mixing them with decodes in the same step) and stall-free batching. DistServe (Zhong et al., OSDI 2024) reformulated the key metric: what matters is goodput (requests/s meeting the TTFT and TPOT SLOs), not raw throughput. In May 2026, vLLM v1 brings a unified scheduler with chunked prefill always-on; SGLang adds RadixAttention, which gives cross-request prefix-cache hits; TensorRT-LLM calls it in-flight batching; llama.cpp supports it natively. The three operational tensions are speculative decoding (nested raggedness), multi-LoRA (each request with its adapter) and MoE (each expert sees very few tokens per step at a typical batch size). This post takes apart the mechanism, the maths (GPU utilisation, goodput vs throughput), the three variants (Orca → vLLM → Sarathi-Serve), the pitfalls (preempt-on-OOM, starvation, inverse HoL) and the real numbers with production configurations.

The analogy: the salon with 8 chairs

A hair salon with 8 chairs and a single brilliant stylist who moves between them. Customers arrive with very different needs: some want a 15-minute cut, others a 2-hour colour with a base, others a 90-minute straightening. The question is how to organise the flow.

The traditional strategy (what HuggingFace Transformers did in its original generate()) is to seat 8 customers at once, all at the same time, and accept nobody new until the last one finishes. If one of those 8 is a 2-hour job, the 7 who wanted the 15-minute cut sit there doing nothing for 1 hour and 45 minutes. The stylist finishes with the quick ones and is left staring at empty chairs until the colour is done. When everyone is ready, another 8 come in. That is static batching, and the only thing that stops it from being worse is that the GPU does not complain like a human customer.

The continuous strategy (Orca, vLLM) changes the scheduling unit. The stylist does not think “I am going to do one whole customer and then the next”; he thinks “on each tick I take one step of work at every occupied chair and, every time a chair frees up, I call the next customer in the queue without waiting for the others to finish”. The quick-cut customer leaves after 15 minutes, his chair is filled immediately with the next one, and the slow jobs proceed at their own pace without delaying anyone. The stylist is never idle.

The continuous strategy with chunked prefill (SARATHI / Sarathi-Serve) adds a subtler distinction. Some customers need a long initial phase (a 10-minute hair analysis before the cut; in LLM terms, the prefill of the prompt). Without chunked prefill, the stylist had to stop all the other chairs to do the new customer’s analysis in one go, and that was a stall visible in the TPOT of the active ones. With chunked prefill, the analysis is split into 2-minute pieces interleaved between the other customers’ active cuts. The customers already under way no longer notice pauses; the new customer takes slightly longer to start his cut proper, but the whole salon does not freeze.

And the metric that matters: the owner does not want to maximise “customers served per hour” at the cost of some leaving furious. He wants to maximise “customers served per hour within the time SLA”, and that is goodput, DistServe’s contribution.

The problem continuous batching solves

There are two structural pathologies of static batching that deserve to be explained with concrete numbers.

Padding waste. Each batch slot reserved memory for max_seq_len (prompt + max output), even though the real output ended in far fewer tokens. For a batch of 32 with heterogeneously distributed output lengths (half of them ≤50 tokens, a long tail up to 4,000), the typical memory waste was 60-80 %. That translated directly into wasted concurrency: with the same VRAM, instead of serving 32 requests with smart allocation, you served 8.

HoL blocking (Head-of-Line). The scheduling unit was the complete request. A batch containing one request of 500 tokens and 31 of ≤50 tokens ran 450 extra “empty” iterations (the GPU executed forward passes for all 32 slots, even though 31 had already finished). Wasted compute cost: ~84 % of the tail time in the example.

Measurable result. Sustained GPU SM utilisation on real workloads under static batching: 20-40 %. That is, ~70 % of the datacenter’s compute went unused. When the first Orca and vLLM benchmarks came out showing 10-24× throughput improvement, it was not marketing exaggeration; it was recovering all that wasted compute.

Orca (OSDI ‘22): the idea that changed everything

The paper by Yu, Jeong, Kim, Kim and Chun at OSDI 2022 (“Orca: A Distributed Serving System for Transformer-Based Generative Models”, from Seoul National University + FriendliAI) introduced two contributions that have remained the basis of everything that followed.

Iteration-level scheduling. Instead of scheduling at the level of the complete request, it schedules at the level of one decoder iteration: the step that generates ONE token. After each iteration the scheduler can (a) add new requests to the batch, (b) remove requests that have generated EOS or reached max_tokens, (c) reorder priorities. The compute engine executes exactly one iteration over the current batch.

Selective batching. Here is the non-obvious subtlety. The technical problem with batching requests of different lengths and different KV cache states is that some operations (the GEMMs of the Q, K, V projections and the FFN) are insensitive to position and can be batched by concatenating tokens, whereas attention is sensitive to per-request state (each request has its own KV cache of a different length). Orca’s solution: batch the GEMMs (concatenate all the step’s tokens into a [total_tokens, hidden] tensor) and run attention sequentially per request.

Paper result: up to 36.9× throughput vs FasterTransformer on GPT-3 175B at the same latency level. Orca is not open-source, it is only documented in the paper. FriendliAI commercialises it as the Friendli Engine. But the idea was published and everyone adopted it.

vLLM (SOSP ‘23): the open-source materialisation

What Orca described as a concept, vLLM materialised in production. The paper by Kwon, Li, Zhuang, Sheng et al. (UC Berkeley Sky Computing Lab, SOSP 2023) introduces PagedAttention, covered in detail in the PagedAttention deep dive, but it also consolidates continuous batching as universal practice.

The reason PagedAttention is a prerequisite for practical continuous batching is fragmentation. If you are going to insert and remove requests from the batch dynamically, and each request has a KV cache that grows on every iteration, traditional contiguous allocation fragments the HBM until it is unusable. PagedAttention splits the KV cache into fixed-size blocks (16 tokens by default) allocated on demand from a global pool. Memory waste falls from ~60-80 % to less than 4 % (only the last partially filled block per sequence).

vLLM paper metrics (2023):

vs HuggingFace Transformers (static, no continuous batching): up to 24× throughput.
vs HuggingFace TGI (which already had primitive continuous batching): ~3.5×.
vs FasterTransformer: 2-4× at the same latency.

What operationally distinguishes vLLM from Orca: vLLM runs attention in a single fused CUDA kernel (paged_attention_kernel) over non-contiguous blocks; Orca described attention as sequential, request by request. And vLLM exposes OpenAI-compatible APIs that let you drop the engine into existing stacks without touching the client.

Chunked prefill (SARATHI / Sarathi-Serve, OSDI ‘24)

There is a detail the original Orca/vLLM continuous batching did not solve: when a new request enters the batch, its prefill (processing the whole prompt at once) can take hundreds of milliseconds. During that time, the active decodes of the other requests are essentially paused, the GPU being dedicated to the new prefill. This showed up as spikes in TPOT (“inter-token latency”) every time a long request came in, which broke strict SLAs.

SARATHI (Agrawal et al., arXiv 2308.16369, August 2023) and later Sarathi-Serve (same group at Microsoft Research India, OSDI 2024, arXiv 2403.02310) introduce two combined ideas:

Chunked prefill. A long prefill (8,192 tokens, say) is split into chunks (2,048 tokens, say) processed one per iteration. Instead of one 200 ms step processing 8K tokens, four 50 ms steps processing 2K each.

Decode-maximal batching (“stall-free”). On each iteration, the scheduler first fills the batch with the active decodes (each costing 1 token), and only the leftover space goes to new prefill chunks. The result: active decodes keep advancing 1 token per iteration without pausing, while the new request completes its prefill in small bites.

The observation that justifies it: prefill is compute-bound (it processes N tokens at once and saturates FLOPs) while decode is memory-bound (1 token per step, under-using compute, with the GPU waiting on HBM). Mixing prefill chunks with decodes in the same step exploits the arithmetic-intensity slack: the decodes piggyback on the free compute of the prefill chunk.

Numbers:

Original SARATHI (LLaMA-13B on an A6000): decode throughput +10×, end-to-end +1.33×.
Sarathi-Serve (Mistral-7B on an A100): 2.6× serving capacity vs plain vLLM. Yi-34B on 2×A100: 3.7×. Falcon-180B with pipeline parallel: 5.6×.

Adoption in May 2026: always-on in vLLM v1 (default since v0.8.0, January 2025), SGLang, TensorRT-LLM. The key setting in vLLM is --max-num-batched-tokens (the token budget per step; default 2048). Raising it prioritises throughput, lowering it prioritises low TPOT.

Goodput: the metric that matters (DistServe, OSDI ‘24)

The classic “throughput” metric (requests/s or tokens/s) has a problem when there are SLOs. A server can report 1,000 req/s while P99 TTFT is 30 seconds and the SLO is 1 second, so only about 200 req/s actually meet the contract.

DistServe (Zhong et al., OSDI 2024) formalises goodput as the right metric: goodput = max sustained request rate meeting the SLOs (TTFT bound AND TPOT bound). The practical definition is usually: maximum rate with ≥90 % of requests within both SLOs.

Why it matters for the scheduler:

Optimising raw throughput leads to maximising batch size, which inflates P99 TPOT.
Optimising goodput limits the batch size when TPOT starts violating the SLO, prefers small requests if the batch already has a tail, and leaves resources available for new requests.

DistServe result: up to 7.4× more requests served or a 12.6× tighter SLO vs vLLM at the same SLO attainment. The gain comes from disaggregating prefill and decode onto different GPUs (removing the interference between phases), but the idea of optimising for goodput is independent and applicable to any scheduler.

Operationally this translates into monitoring:

goodput_proxy = histogram_quantile(0.95, vllm:time_to_first_token_seconds_bucket) < SLO_TTFT
AND histogram_quantile(0.95, vllm:time_per_output_token_seconds_bucket) < SLO_TPOT

The iterative scheduler in action

The maths that matter

Three formulas explain much of the operational behaviour.

GPU utilisation under static batching. With a batch of size B whose seq_len_i are the real lengths and max(seq_len_i) is the length that defines the padding:

$$U_{\text{static}} = \frac{\sum_i \text{seq len}_i}{B \cdot \max_i \text{seq len}_i}$$

For B=32, 30 sequences of 50 tokens and 2 of 500: U = (30·50 + 2·500) / (32·500) = 2500/16000 = 15.6 %. Four out of every five GPU cycles wasted.

GPU utilisation under continuous batching (idealised).

$$U_{\text{continuous}} \approx 1 - \frac{T_{\text{scheduler}}}{T_{\text{iteration}}}$$

With scheduler overhead of ~50-200 µs and an iteration time of ~10-30 ms: U > 95 %. The unit of loss is no longer padding, it is scheduling overhead, and that is negligible compared with the forward pass.

Goodput vs throughput.

$$\text{Goodput}(R) = R \cdot P(\text{latency} < \text{SLO})$$

where R is the offered request rate. Typical curve: goodput grows linearly with R up to the saturation knee, then falls because P(SLO) collapses when the system congests. The optimal point is just before the knee, not at peak throughput.

Example: at R=100 req/s with P(SLO)=0.99, goodput = 99. At R=200 req/s with P(SLO)=0.4, goodput = 80. More offered load, less useful goodput.

Chunked prefill token budget. On each vLLM step with chunked prefill active:

$$\text{prefill tokens this step} = \text{max num batched tokens} - \text{num decodes active}$$

Each active decode costs 1 token of the budget; the rest is filled with new prefill chunks. If max_num_batched_tokens = 2048 and num_decodes_active = 200, there are 1,848 tokens for prefill (one chunk of 1,848 or several small chunks).

Real implementations in May 2026

Engine	Current scheduler V	Chunked prefill default	Relevant notes
vLLM v1 (default ≥0.8.0)	unified V1	always-on	EngineCore isolated in a separate process; prefix caching with O(1) eviction; preempt-mode `recompute` by default; xgrammar/outlines backends.
SGLang	own (PyTorch ecosystem)	yes	RadixAttention gives cross-request prefix-cache hits; non-blocking CPU scheduler; leader in stable latency at high concurrency.
TensorRT-LLM	proprietary “in-flight batching”	yes	Policies `GUARANTEED_NO_EVICT` (conservative, default) and `MAX_UTILIZATION` (aggressive, risk of a pause when the KV is full). Compile-time vs runtime.
Triton + tensorrtllm_backend	`gpt_model_type: inflight_fused_batching`	yes	`max_queue_delay_microseconds` to group newly arrived requests. Decoupled mode for SSE streaming.
llama.cpp (llama-server)	own	`--cont-batching` ON since 2024	`-np N` parallel slots; no PagedAttention (contiguous KV per slot), so less flexible but simpler. Endpoint `:8080/metrics`.

A typical production-ready vLLM v1 configuration:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
 --tensor-parallel-size 4 \
 --max-num-batched-tokens 4096 \
 --max-num-seqs 256 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --preemption-mode recompute \
 --scheduling-policy fcfs \
 --gpu-memory-utilization 0.92

SGLang equivalent:

python -m sglang.launch_server \
 --model meta-llama/Llama-3.1-70B-Instruct \
 --tp 4 \
 --chunked-prefill-size 4096 \
 --max-running-requests 256 \
 --enable-radix-cache

The three operational tensions

Continuous batching + speculative decoding. Speculative decoding produces 1 to γ+1 tokens per step depending on the acceptance rate. The batch stops being uniform in tokens produced per iteration: nested raggedness. PagedAttention absorbs it (the KV cache can grow at different rates per request in the same step), but the scheduler loses symmetry. At low QPS (a conversational assistant) the combination is excellent: vLLM reports up to 2.8× speedup. At high QPS, the draft consumes slots from the decode pool and can reduce aggregate goodput. Rule of thumb: disable speculative decoding when gpu_cache_usage > 0.85. Full detail in Speculative decoding.

Continuous batching + multi-LoRA. Each request in the batch can use a different adapter (via SGMV, see Multi-LoRA serving). Worst case: every request in the batch with a different adapter and a different rank, and throughput falls by up to 50 % vs the base with no LoRA. Best case: all requests on the same adapter, equivalent to the base with no LoRA. Practical mitigation: group adapters by rank in the routing ahead of the engine; set --max-lora-rank to the maximum actually served, not generously above it.

Continuous batching + MoE. Each expert sees batch · k / N tokens per step. With DeepSeek-V3 (256 experts, k=8) and batch=32 in decode, each expert processes only 1 token on average: total compute starvation. To match the per-GPU throughput of a dense model, MoE needs batches »10× larger, which puts pressure on the KV cache. Wide-EP (see MoE inference) distributes the experts across many GPUs and allows larger effective batches per expert, at the cost of all-to-all comms that add milliseconds per step.

Metrics to monitor

The Prometheus metrics exposed by vLLM (prefix vllm:, vllm_ when scraped):

vllm:time_to_first_token_seconds (Histogram) — TTFT including queue time.
vllm:time_per_output_token_seconds (Histogram) — TPOT.
vllm:e2e_request_latency_seconds (Histogram) — end-to-end.
vllm:num_requests_running (Gauge) — active batch.
vllm:num_requests_waiting (Gauge) — queue depth.
vllm:num_requests_swapped (Gauge) — preempted to CPU.
vllm:gpu_cache_usage_perc (Gauge) — fraction of KV cache occupied.
vllm:gpu_prefix_cache_hit_rate (Gauge) — prefix cache hits.
vllm:num_preemptions_total (Counter) — preemptions. Any sustained value is a red flag.

Practical operational rules:

Stable zone under sustained load: gpu_cache_usage_perc ∈ [0.7, 0.9].
Warning at >0.95 (preemption imminent).
Critical if num_requests_waiting grows faster than num_requests_running: the server is not absorbing the load; scale out.

Operational pitfalls

Preempt-on-OOM. When gpu_cache_usage reaches ~1.0 with pending requests that need to grow their KV, vLLM preempts. V1 does RECOMPUTE by default (discards the KV, regenerates it on return); V0 did SWAP (moves it to CPU). RECOMPUTE is better for short sequences (cheap to regenerate); SWAP is better for long ones. The vllm:num_preemptions_total metric should be zero or near zero in steady state.

Inverse HoL blocking (memory monopoly). A very long request occupies many KV blocks, so small requests do not fit in the batch even though compute is free. Chunked prefill mitigates the compute blocking during a new prefill, but it does not solve the memory monopoly. Partial solution: per-request limit policies (an aggressive max_tokens) or priorities.

Starvation. FCFS can leave requests pending for a long time if the active ones do not finish. vLLM supports --scheduling-policy priority with an x-priority header. Recent work (NeurIPS 2024 Efficient LLM Scheduling by Learning to Rank, arXiv:2501.14312 Locality-aware Fair Scheduling) proposes schedulers with quantum-based starvation prevention; not yet integrated into vLLM mainline.

Badly calibrated chunk size. A small chunk (512) gives low TPOT, high TTFT and memory overhead from more KV accesses. A large chunk (8192+) gives low TTFT and TPOT spikes during the chunk. Rule: start at 2,048, measure P95 TPOT, adjust.

Batch size cap. A high --max-num-seqs gives more concurrency but P99 TPOT explodes. A low one wastes throughput. Rule of thumb: max_num_seqs ≈ HBM_for_KV / (avg_seq_len × bytes_per_token_KV).

Implications for on-premise hardware

On an RTX 4090 (24 GB). llama.cpp with --cont-batching -np 4-8 is the natural pattern. Typical models: Llama 3 8B Q4_K_M with ~8 parallel slots, aggregate throughput on the order of hundreds of tok/s. vLLM also works if the weights fit (Llama 3 8B BF16 does; the 70B does not fit whole), although PagedAttention on consumer hardware gives less return than in the datacenter.

On a generic 4×H100 SXM cluster (320 GB, NVLink). Here vLLM v1 / SGLang are the de facto standard. Typical configurations:

Llama 3 70B FP8 + TP=4: dozens of concurrent sessions with P95 TPOT under 50 ms, tens of thousands of aggregate tok/s at a moderate batch size.
Llama 3 70B AWQ-INT4 + TP=2 plus the rest of the cluster for additional concurrency or multi-LoRA with SGMV.
DeepSeek-V3 needs larger setups (8-16 H100) to fit whole in FP8; with Wide-EP, continuous batching starts operating on much larger batches and the economics change (see MoE).

The rule of thumb for May 2026: vLLM v1 with chunked prefill always-on and prefix caching enabled is the sensible default configuration for any dense model that fits comfortably; SGLang offers better stable latency at high concurrency thanks to the overlap between the CPU scheduler and the GPU step; TensorRT-LLM gives peak throughput at high concurrency with the rigidity of compile-time.

What we have not covered

Locality-aware fair scheduling (arXiv:2501.14312) and learning-to-rank schedulers (NeurIPS 2024): the next generation of algorithms closing the fairness vs prefix-cache locality trade-off.
Smooth goodput (arXiv:2410.14257): a refinement of the DistServe metric using max slowdown instead of a binary SLO.
Triton tensorrtllm_backend in production: decoupled mode for streaming, ensembles with pre/post-processing, autoscaling with KServe.
vLLM speculators v0.3.0 and the vLLM-compatible drafter training framework.

References

Yu, G.-I., Jeong, J., Kim, G.-W., Kim, S., Chun, B.-G. Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022. https://www.usenix.org/system/files/osdi22-yu.pdf
Kwon, W. et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. https://arxiv.org/abs/2309.06180
Agrawal, A. et al. SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. 2023. https://arxiv.org/abs/2308.16369
Agrawal, A. et al. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. OSDI 2024. https://arxiv.org/abs/2403.02310
Zhong, Y. et al. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024. https://arxiv.org/abs/2401.09670
Sheng, Y. et al. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. MLSys 2024. https://arxiv.org/abs/2311.03285
Efficient LLM Scheduling by Learning to Rank. NeurIPS 2024.
Locality-aware Fair Scheduling. 2025. https://arxiv.org/abs/2501.14312
vLLM V1 alpha release (Jan 2025): https://blog.vllm.ai/2025/01/27/v1-alpha-release.html
vLLM Anatomy (Sep 2025): https://blog.vllm.ai/2025/09/05/anatomy-of-vllm.html
vLLM Large-Scale Serving (Dec 2025): https://blog.vllm.ai/2025/12/17/large-scale-serving.html
vLLM speculators v0.3.0 (Dec 2025): https://blog.vllm.ai/2025/12/13/speculators-v030.html
Anyscale, Continuous Batching for LLM Inference (2023): https://www.anyscale.com/blog/continuous-batching-llm-inference
vLLM metrics docs: https://docs.vllm.ai/en/latest/design/metrics/
vLLM optimization: https://docs.vllm.ai/en/stable/configuration/optimization/
SGLang: https://github.com/sgl-project/sglang
TensorRT-LLM performance tuning: https://nvidia.github.io/TensorRT-LLM/performance/performance-tuning-guide/tuning-max-batch-size-and-max-num-tokens.html
Red Hat, 5 steps to triage vLLM performance (Mar 2026): https://developers.redhat.com/articles/2026/03/09/5-steps-triage-vllm-performance
NVIDIA blog, Chunked Prefill with TensorRT-LLM: https://developer.nvidia.com/blog/streamlining-ai-inference-performance-and-deployment-with-nvidia-tensorrt-llm-chunked-prefill/

MoE inference: the call centre with 256 specialists and 8 taking each call — fundamentals, expert parallel and the economics of DeepSeek-V3

Sat, 30 May 2026 08:00:00 +0200

This post goes into an architectural decision in the Deploy stage of the six-stage LLMOps pipeline. It complements those on KV cache (which does not change with MoE), FlashAttention (still the attention kernel, also dense in every current MoE), Quantization (FP8/NVFP4 over expert weights is what lets DeepSeek-V3 fit into a modest cluster) and Speculative decoding (MTP in DeepSeek-V3 is native speculative decoding).

You are here: DEPLOY

TL;DR

A Mixture of Experts (MoE) replaces the transformer’s dense FFN layer with N parallel experts plus a router that picks the k experts relevant to each token. The operational consequence is that the model’s total parameters (how many were trained and how many must be loaded into VRAM) decouple from the active parameters per token (how many take part in each forward). DeepSeek-V3 has 671 B total but only 37 B active per token; Qwen3-235B-A22B is 235 B total and 22 B active; Mixtral 8x22B is 141 B and 39 B; Llama 4 Maverick is 400 B and 17 B. Compute per token falls to 2·N_active FLOPs, half that of a dense Llama-3-70B in DeepSeek-V3’s case, but memory stays proportional to N_total: MoE sparsity is compute sparsity, not memory sparsity. Since each expert sees only batch·k/N tokens per step, the memory-bound regime persists at a much higher batch than in dense. Serving DeepSeek-V3 671 B FP8 at low latency requires expert parallel (EP) at scale, EP=32 in prefill and EP=144 in decode in DeepSeek’s production deployment, where the dominant bottleneck stops being HBM and becomes the all-to-all of dispatch and combine between GPUs. As of May 2026, the pieces that make this viable are DeepEP (FP8-native all-to-all CUDA kernels, intra-node NVLink plus inter-node RDMA), EPLB (balancing by replicating hot experts) and Wide-EP on NVL72 (72 GPUs in one coherent NVLink domain). This post takes apart the mechanism, the maths (FLOPs/token, all-to-all bytes), the updated table of MoE models as of May 2026, the parallelisms (TP / EP / DP+EP / Wide-EP), the operational pitfalls (memory wall, cold experts, imbalance) and the real reproducible numbers on H100, B200 and GB200 NVL72.

The analogy: the call centre with 256 specialists

A serious call centre handling complex technical queries runs on specialists, not generalists. You have 256 operators on the payroll, each an expert in their sub-domain: employment law, mortgages, income tax, probate, and so on. But each incoming call only needs the attention of eight specific specialists (top-8 routing): a client’s query about tax relief on a mortgage for an inherited second home involves the income tax person, the mortgage person, the capital gains person, and so on. The receptionist (the router) listens to the first second of the call, decides which eight specialists are relevant, conferences them in and combines their answers with weights.

Three immediate consequences, identical to those of MoE in production:

You pay 256 salaries (every expert is loaded in VRAM) but you only spend the time of 8 (only k experts take part in the forward of each token). Your payroll cost scales with N_total; your cost per minute of call scales with N_active.
If all the operators fit in one office (a single GPU), fine: paper just moves between desks. But if they do not fit, you have to open branches: the conference call between operators in different branches goes over a telephone line between buildings (intra-node NVLink, inter-node InfiniBand) and that line becomes the dominant bottleneck once dozens of branches are coordinating.
If one week every client asks about income tax, the three income tax operators saturate while the probate ones sit idle. You need a system that replicates the hot operators across several branches to absorb peaks, and that pairs hot with cold so that no branch goes idle.

The receptionist is the router. The branches are the GPUs. The conference call between branches is the all-to-all (dispatch plus combine). The system for replicating hot operators is EPLB. The good telephone line is DeepEP. And when someone designs a building where all 72 branches fit together with very fast internal cabling, that is NVL72.

The bare mechanism

A classic MoE layer replaces the dense FFN (y = FFN(x) = down(act(up(x) · gate(x))) in SwiGLU) with:

Router: a linear projection g(x) = W_router · x produces N scores. A softmax over those scores gives affinity probabilities.
Top-k selection: the k experts with the highest score are chosen. Their softmax_topk weights are normalised to sum to 1.
Parallel compute: each of the k selected experts runs its own FFN over x. The N - k not selected sit idle for this token.
Weighted combination: the output is y = Σ_{i ∈ topk} w_i · Expert_i(x).

Variants the market consolidated in 2025-2026:

Shared experts: on top of the k routed ones, there are s experts that always run (1 or 2). They capture general knowledge every token needs. DeepSeek-V3 uses 1 shared + 8 routed; Mixtral uses 0 shared.
Fine-grained experts: more small experts instead of a few large ones (DeepSeek: 256 vs Mixtral: 8). This allows better specialisation because the top-k combination covers more sub-domains.
Auxiliary-loss-free routing (DeepSeek-V3): instead of adding a balancing loss during training, it dynamically adjusts a bias in the routing. It keeps balance without polluting the main loss.

Attention is still dense in every current MoE model (Mixtral, DeepSeek, Qwen3-MoE, Llama 4, Kimi K2). Only the FFN is replaced. That is why FlashAttention v3/v4 is still used as-is; DeepSeek’s MLA is an optimisation orthogonal to MoE.

The relevant MoE models in May 2026

Model	Date	Total	Active	Experts	Top-k	Shared	Notes
Mixtral 8x7B	Dec 2023	47 B	13 B	8	2	0	The one that normalised MoE in open weights
Mixtral 8x22B	Apr 2024	141 B	39 B	8	2	0	—
Grok-1 (open)	Mar 2024	314 B	~86 B	8	2	0	Weights opened by xAI
DeepSeekMoE 16B	Jan 2024	16 B	2.8 B	64+2	6	2	Introduces fine-grained + shared
DeepSeek-V2	May 2024	236 B	21 B	160+2	6	2	+ MLA
Snowflake Arctic	Apr 2024	480 B	17 B	128	2	10 B dense residual	Dense-MoE hybrid
Hunyuan-Large	Nov 2024	389 B	52 B	16+1	1+shared	1	Cross-layer KV cache
DeepSeek-V3	Dec 2024	671 B	37 B	256+1	8	1	MLA + MTP, aux-loss-free, FP8 training
Llama 4 Scout	Apr 2025	109 B	17 B	16	1+shared	1	Designed for 1 H100 node
Llama 4 Maverick	Apr 2025	400 B	17 B	128	1+shared	1	—
Qwen3-235B-A22B	Apr 2025	235 B	22 B	128	8	0	—
Kimi K2 (Moonshot)	2025	1 T	32 B	384	8	(MLA)	Trained with MuonClip
DeepSeek-V3.2-Exp	Sep 2025	671 B	37 B	256+1	8	1	+ DeepSeek Sparse Attention

Three operational observations:

The active/total ratio falls with every generation: Mixtral 8x7B was 28 %, DeepSeek-V3 is 5.5 %, Llama 4 Maverick is 4.3 %. What the industry discovered is that sparsity can be very aggressive without losing quality, provided N is large enough and the routing is well learned. Fine specialisation is worth more than per-expert capacity.
The flagships of the large providers (OpenAI, Anthropic, Google) are almost certainly MoE, but the architectures are not published. The only semi-credible leak (Hotz, Jul 2023) claimed GPT-4 was an 8×~220 B MoE; no official confirmation.
DeepSeek-V3 marks the inflection point: open weights, frontier-class, MoE with aggressive sparsity and compatible with FP8 quantisation. It is the model that forced the ecosystem (vLLM, SGLang, TensorRT-LLM) to optimise wide-EP in 2025.

The maths that matters

Three numbers drive the whole architectural decision with MoE.

Compute per token. A dense transformer spends roughly 2N FLOPs per token in the forward pass (with N = total parameters). A MoE spends 2·N_active FLOPs per token (the N_total - N_active experts do not take part). For DeepSeek-V3: 2 × 37 B = 74 GFLOPs/token. For dense Llama-3-70B: 2 × 70 B = 140 GFLOPs/token. DeepSeek does half the compute per token of Llama-3-70B while offering capacity equivalent to a model almost 10 × larger.

Total memory. But all the experts must be loaded in VRAM at some point in the forward pass. The sparsity is in compute, not memory. Weight memory:

$$\text{Memory}_{\text{weights}} \approx N_{\text{total}} \cdot \text{bytes per param}$$

For DeepSeek-V3 FP8 (1 byte): 685 GB. It does not fit in 8×H100 SXM (640 GB). You need either extra quantisation or 16+ GPUs. For Mixtral 8x22B FP8: 141 GB, comfortably inside 2×H100. For Qwen3-235B-A22B FP8: 235 GB, fits in 4×H100.

All-to-all bytes per MoE layer. When experts are spread across GPUs (EP), each token has to travel to the GPUs where its k experts live. The volume per MoE layer is roughly:

$$\text{bytes a2a/layer} \approx 2 \cdot \text{batch} \cdot \text{seq len} \cdot d_{\text{model}} \cdot k \cdot \text{bytes}$$

(the factor 2 is dispatch plus combine). For DeepSeek-V3 with d_model = 7168, k = 8, batch = 4096, FP8 (1 byte): each MoE layer moves ~470 MB; multiplied by 61 MoE layers in DeepSeek-V3, the complete step adds ~29 GB of inter-GPU traffic in MoE comms alone. That is of the order of a GPU’s HBM bandwidth in a single step. This is why bandwidth and NVLink/InfiniBand topology determine how large EP can be before comms saturate.

Persistent memory-bound regime. Here is the least intuitive consequence. In dense, raising the batch moves the operation from memory-bound to compute-bound: with enough tokens per batch, the weights are amortised over more operations. In MoE, each expert sees only batch · k / N tokens. For Qwen3-235B (k=8, N=128) with batch=128, each expert processes 8 tokens per step. To reach the compute-bound regime of the tensor cores (arithmetic intensity ~100-200 FLOPs/byte) you need batches in the thousands, around 1,600 according to recent estimates for Qwen3 (Memory-Bound MoE Serving, arXiv:2512.09277). The operational consequence: MoE scales throughput with batch more linearly than dense, but it also needs far larger batches to get close to its compute ceiling.

Expert parallel and the all-to-all bottleneck

There are three main ways to parallelise a MoE across GPUs:

Tensor Parallel (TP, Megatron-style). Splits each weight matrix across GPUs by width or height. Every GPU processes all tokens and all layers. Comms per layer: 2 all-reduces. It works well for dense; for MoE it leaves the routing sparsity unexploited (each GPU holds fragments of every expert).

Expert Parallel (EP). Splits whole experts across GPUs: 256 experts / 32 GPUs = 8 experts per GPU. Each token, after routing, travels to the GPUs where its top-k k experts live. Comms per MoE layer: an all-to-all dispatch (tokens towards experts) plus an all-to-all combine (outputs back). The cost grows with EP size and becomes the bottleneck when EP > 8.

DP + EP / Wide-EP. Replicates the model across DP groups; inside each group, EP shards the experts. Wide-EP takes EP to the full dimension of a rack (NVL72, 72 GPUs in a coherent NVLink domain). Each expert receives more tokens per step, so arithmetic intensity in each expert’s GEMMs improves and per-GPU throughput rises. vLLM reports 1.8× throughput per GPU with wide-EP against smaller setups on DeepSeek-V3 (vLLM blog, Dec 2025).

Quick decision table:

Case	Recommendation
Small MoE that fits in 1 node (Mixtral 8x7B, Llama 4 Scout)	TP only or EP=2 intra-node
Medium MoE (Mixtral 8x22B, Qwen3-235B) on 1-2 H100 nodes	EP=8 intra-node, optionally TP=2
Large MoE (DeepSeek-V3) on a multi-node cluster	TP × EP cross-node (e.g. TP=4 × EP=8 on 32 GPUs)
Large MoE with NVL72 available	Wide-EP=72 (decode), EP=32 (prefill)

DeepSeek’s production deployment combines Prefill EP=32 (4 nodes × 8 GPUs) with Decode EP=144 (18 nodes × 8 GPUs). The prefill/decode split fits conceptually with Disaggregated serving: prefill is compute-bound and benefits from moderate EP; decode is memory-bound and benefits from Wide-EP, which raises the tokens per expert per step.

DeepEP, EPLB and Wide-EP: the pieces that unblocked 2025

DeepEP (DeepSeek, opened in February 2025) is a library of all-to-all CUDA kernels optimised specifically for MoE EP. Four important properties:

Native FP8 dispatch: tokens travel in FP8, halving the bandwidth consumed.
Mixing intra-node NVLink with inter-node RDMA: the kernel picks the optimal route according to topology.
CPU bypass: the GPU→GPU dispatch goes over RDMA without touching the host.
Aligned with DeepSeek-V3’s group-limited gating: the routing prefers experts in the same node group where possible, minimising inter-node traffic.

In 2025 Tencent contributed optimisations that added +30 % on the normal kernels. There is a ROCm port for AMD.

EPLB (Expert Parallelism Load Balancer) solves the runtime imbalance problem: even when the model is well balanced in global distribution, a specific batch can heavily activate certain experts (prompts in a single language, a single domain). EPLB replicates the hot experts across several GPUs (redundant experts) and packs them heuristically with cold experts to minimise load variance. It has two modes: hierarchical (for prefill with medium EP) and global (for decode with Wide-EP). It is integrated into SGLang and vLLM.

Wide-EP on NVL72 is the peak setup of 2026: 72 Blackwell GPUs in a coherent NVLink domain (1.8 TB/s bidirectional per GPU, intra-rack). Each expert receives many more tokens per step, so arithmetic intensity improves and so does throughput per GPU. Combined with NVFP4 over expert weights and FP8 over attention, SGLang reported on GB200 NVL72 26,156 tok/s/GPU in prefill and 13,386 tok/s/GPU in decode for DeepSeek-V3 (LMSYS, Sep 2025), a 3.8× prefill and 4.8× decode versus H100.

Operational pitfalls

HBM fragmentation from expert weights. With EP=8 in 1 node, each GPU holds N/8 experts, each a tuple of matrices (gate, up, down). If N=256, that is 32 FFNs per GPU. The allocator can fragment; in some cases you need a more conservative --gpu-memory-utilization than usual.

Cold experts. In real distributions some experts are activated by <1 % of tokens. EPLB compensates by replicating hot ones, but the cold ones sit occupying HBM without taking part, “dead” memory. When real demand diverges a lot from the training distribution, this shows.

Continuous batching. It works with MoE, but the effective batch size per expert is batch_total · k / N. With large N and a moderate batch, each expert sees very few tokens per step. You need a batch_total far larger than in dense to match per-GPU throughput.

Speculative decoding + MoE. They interact well. MoE sparsity keeps the memory-bound regime at high batch, which is exactly the regime where speculative decoding wins most. DeepSeek-V3 integrates MTP (Multi-Token Prediction), speculative decoding native to the model, with no external draft, with acceptance around 85-90 % on the second token (see Speculative decoding for the mechanism).

MLA and FlashMLA are not MoE. DeepSeek-V3 combines MoE with Multi-head Latent Attention (MLA), an attention variant that reduces the KV cache ~10× against standard MHA. MLA and MoE are orthogonal: MLA optimises attention, MoE optimises the FFN. For serving DeepSeek you use the specific FlashMLA kernel (not FA3 directly).

DSA (DeepSeek Sparse Attention) in DeepSeek-V3.2-Exp (Sep 2025) introduces sparsity in attention too. The first time at the frontier that sparsity is applied in both attention and FFN.

Implications for on-premise hardware

On an RTX 4090 (24 GB). Large MoEs are out of reach. What does fit: Mixtral 8x7B AWQ-INT4 (~24 GB of weights, no headroom for KV cache, so it really requires TP=2 across two 4090s); DeepSeekMoE 16B BF16 (~33 GB, does not fit whole; INT4 ~8 GB, fits with room to spare). The interesting case is llama.cpp with --n-cpu-moe: expert weights offloaded to RAM (CPU), attention and shared experts on the GPU. It lets you run DeepSeek-V3 IQ4 (~400 GB) on a workstation with 1-2 GPUs plus 512 GB of RAM at modest throughput, but it opens the door to a low-budget setup.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). These fit comfortably:

Mixtral 8x22B FP8 (~141 GB) with TP=2 or EP=4. Low latency, excellent throughput.
Qwen3-235B-A22B FP8 (~235 GB) with TP=4 or EP=4. Plenty of room.
Llama 4 Scout 109B FP8 (~110 GB) with TP=2 or TP=4. Designed by Meta to serve well on one H100 node.

What does not fit in 4×H100: DeepSeek-V3 FP8 (~685 GB) without aggressive quantisation. Options: (1) scale to 8-16 H100; (2) use AWQ INT4 over routed experts plus FP16 over activated ones (~400 GB), which fits in 8×H100; (3) wait for Blackwell or use AMD MI300X (192 GB/GPU, so DeepSeek-V3 FP8 on 4×MI300X).

The rule of thumb as of May 2026: one 4×H100 node comfortably serves MoEs up to ~200 B total in FP8; for DeepSeek-V3 you have to scale to 2 nodes or wait for B200 NVL72 (where the whole model fits with room for enormous batches).

What we have not covered

MLA and FlashMLA in detail: DeepSeek’s KV cache optimisation, orthogonal to MoE.
DeepSeek Sparse Attention (DSA) introduced in V3.2-Exp: the first production implementation of attention sparsity for frontier models.
MoE during training: load balancing losses, drop policies, auxiliary terms vs aux-loss-free, Kimi K2’s MuonClip.
MoE + LoRA: how adapter fine-tuning is done on a MoE, and what happens to the routing.
LLM-d and other open-source platforms that materialise Wide-EP on Kubernetes.

References

Shazeer, N. et al. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR 2017. https://arxiv.org/abs/1701.06538
Lepikhin, D. et al. GShard: Scaling Giant Models with Conditional Computation. ICLR 2021. https://arxiv.org/abs/2006.16668
Fedus, W., Zoph, B., Shazeer, N. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. JMLR 2022. https://arxiv.org/abs/2101.03961
Gale, T., Narayanan, D., Young, C., Zaharia, M. MegaBlocks: Efficient Sparse Training with Mixture-of-Experts. MLSys 2023. https://arxiv.org/abs/2211.15841
Mistral AI. Mixtral of Experts. 2024. https://arxiv.org/abs/2401.04088
DeepSeek-AI. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. 2024. https://arxiv.org/abs/2401.06066
DeepSeek-AI. DeepSeek-V2. 2024. https://arxiv.org/abs/2405.04434
DeepSeek-AI. DeepSeek-V3 Technical Report. 2024. https://arxiv.org/abs/2412.19437
Tencent. Hunyuan-Large. 2024. https://arxiv.org/abs/2411.02265
DeepEP repo: https://github.com/deepseek-ai/DeepEP
EPLB repo: https://github.com/deepseek-ai/EPLB
DeepSeek open-infra-index, V3/R1 Inference System Overview: https://github.com/deepseek-ai/open-infra-index
vLLM blog, Large-Scale Serving DeepSeek (Dec 2025): https://blog.vllm.ai/2025/12/17/large-scale-serving.html
vLLM blog, WideEP en GB200 (Feb 2026): https://blog.vllm.ai/2026/02/03/dsr1-gb200-part1.html
LMSYS blog, DeepSeek 96 H100 PD+EP (May 2025): https://www.lmsys.org/blog/2025-05-05-large-scale-ep/
LMSYS blog, DeepSeek GB200 NVL72 part II (Sep 2025): https://www.lmsys.org/blog/2025-09-25-gb200-part-2/
NVIDIA Dynamo + GB200 NVL72 for MoE: https://developer.nvidia.com/blog/how-nvidia-gb200-nvl72-and-nvidia-dynamo-boost-inference-performance-for-moe-models/
Tensor Economics, MoE Inference Economics from First Principles: https://www.tensoreconomics.com/p/moe-inference-economics-from-first
Cohere, Why MoE models get more from speculative decoding: https://cohere.com/blog/mixture-of-experts-models-get-more-from-speculative-decoding

Speculative decoding: the secretary who types ahead of the boss — fundamentals, maths and state of play in May 2026

Fri, 29 May 2026 11:00:00 +0200

This post complements KV cache: the working memory and Quantisation for LLM inference. Both are acceleration levers inside a single forward pass. Speculative decoding is the complementary lever: instead of making each forward pass cheaper, it tries to produce more tokens per forward pass. It is orthogonal to quantisation and to the KV cache, and it stacks multiplicatively with both.

You are here: DEPLOY

TL;DR

Autoregressive LLM inference has a structural problem: every new token needs a complete forward pass of the model, and decode forward passes are memory-bandwidth-bound, not compute-bound (the detail is in KV cache). The GPU spends most of its time waiting for the next weight to arrive from HBM. Speculative decoding uses that dead time by doing two things in parallel: a small, cheap model (draft) generates γ tokens autoregressively, fast but imprecise, and the large model (target) verifies those γ tokens in a single parallel forward pass, at almost the same cost as a single-token forward pass. An acceptance rule based on rejection sampling decides how many draft tokens are accepted, and the maths proves, it does not approximate, it proves, that the output distribution matches sampling directly from the target exactly. The technique has a ceiling: you never generate more than 1/(1-α) tokens per step, with α the acceptance rate. The five dominant families in May 2026 are vanilla SD (Leviathan 2023), Medusa (extra parallel heads), EAGLE-1/2/3 (a draft that operates at the level of hidden states), MTP (multi-token prediction native to DeepSeek-V3) and P-EAGLE (a draft that produces the γ tokens in a single forward pass, integrated in vLLM 0.16+). On low-concurrency workloads with short prompts and long outputs, the typical case of the on-premise conversational assistant, the real speedup sits between 2× and 4× on modern hardware.

The analogy: the secretary who types ahead and the boss who validates

Picture a crowded press conference with a real-time transcription setup. Two people are working:

The secretary sits at the keyboard. They know the subject, they have read the briefings and they know the boss’s verbal tics. As soon as the boss opens their mouth, the secretary starts typing what they think is coming. They are fast, three or four words ahead of the boss, but they sometimes get it wrong, especially when the boss takes an unexpected turn.

The boss periodically reads the secretary’s screen. They read a whole block at once, three or four words, and mentally compare what is written with what they would have said. If it matches their intent, they leave the text and carry on. If the secretary drifted at some point, they correct right there, and whatever the secretary typed after the divergence is discarded (it could be wrong because of the earlier error). If the whole block was fine, the boss takes the chance to add one more word while correcting.

The result: the final text is identical to what the boss would have dictated alone, but it comes out faster because the secretary works ahead.

The analogy holds on four details:

The secretary is the draft model. Small, cheap, fast, approximate.
The boss is the target model. Slow, expensive, but the only authority on output quality.
The block the boss reads at once is the target’s parallel forward pass over prompt + γ draft tokens. Verifying γ tokens costs almost the same as generating one, because the bottleneck is loading the weights from HBM, not the operations.
The “correct right where it diverged” rule is the rejection sampling that preserves the target distribution.

From here we go into the mechanism and into why quality is preserved exactly, not approximately.

The bare mechanism

Call p the target distribution (what it would produce if it were the only one speaking) and q the draft’s. The speculative decoding iteration has three steps:

Step 1 — Draft. The draft generates γ tokens autoregressively: x_1, x_2, ..., x_γ. Each one through its own forward pass. Since the draft is small, the γ steps cost little. Typically γ ∈ [4, 8].

Step 2 — Verify. The target runs a single forward pass over the complete sequence prompt + x_1 ... x_γ. Because attention is causal, that forward pass simultaneously produces the distributions p(·|prompt, x_<i) for each position i. In other words, in a single step it obtains the verification of the γ tokens and, on top of that, an extra distribution p(·|prompt, x_1...x_γ) for the next token.

Step 3 — Accept/reject token by token. For each x_i from left to right it applies:

If p(x_i) ≥ q(x_i): always accept.
If p(x_i) < q(x_i): accept with probability p(x_i)/q(x_i).
If it rejects: stop, sample a replacement token from the normalised residual distribution norm(max(0, p − q)), and discard the rest.
If it reaches the end having accepted the γ tokens: append a bonus token sampled directly from p(·|prompt, x_1...x_γ), which is already in the logits of the target’s forward pass.

Result per iteration: between 1 and γ+1 new tokens. In the best case (all accepted + bonus) γ+1 tokens are generated at the cost of a single target forward pass plus γ draft forward passes, against γ+1 target forward passes in the version without speculative.

Why quality does not degrade (the proof)

This is the part of the technique that confuses people most the first time. It looks like magic: how can sampling from an arbitrary draft and then “validating” produce exactly the same distribution as sampling from the original target?

The proof fits in two lines. For any token x, the final probability of emitting it is the sum of two disjoint events: the draft proposes it and it is accepted, or the draft proposes something else, it is rejected, and sampling from the residual yields x. Formally:

$$P(\text{emit } x) = q(x) \cdot \min\!\left(1, \frac{p(x)}{q(x)}\right) + P(\text{reject}) \cdot \frac{\max(0, p(x)-q(x))}{\sum_y \max(0, p(y)-q(y))}$$

The first term is min(p(x), q(x)). The second is max(0, p(x) − q(x)) (the total rejected mass Σ max(0, p−q) cancels the denominator exactly). Adding them: min(p(x), q(x)) + max(0, p(x) − q(x)) = p(x). The output is statistically indistinguishable from sampling directly from the target, down to floating-point numerical differences.

This matters operationally: it means speculative decoding is not perceptual compression, it is not a quality-speed trade-off, it does not require extra validation with evals. If the target would have passed a given eval, the system with speculative passes it too.

The maths that matters: ceiling and speedup

Call α the acceptance rate: the expected probability that an individual draft token is accepted. Assuming that acceptances of consecutive tokens are independent (a reasonable approximation in practice), the expected number of tokens generated per iteration is:

$$E[\text{tokens per step}] = \sum_{k=0}^{\gamma} \alpha^k + \alpha^{\gamma+1} = \frac{1 - \alpha^{\gamma+1}}{1 - \alpha}$$

And the theoretical speedup relative to generating one token per target iteration, with c = T_draft / T_target the relative cost of the draft, is:

$$\text{Speedup} = \frac{1 - \alpha^{\gamma+1}}{(1 - \alpha)(\gamma c + 1)}$$

There is a ceiling that many implementations try to beat and cannot: as γ → ∞, the speedup converges to:

$$\lim_{\gamma \to \infty} E[\text{tokens per step}] = \frac{1}{1 - \alpha}$$

It is an algorithmic ceiling, not a hardware one. With α = 0.7 you never generate more than 3.33 tokens per iteration no matter how much the hardware improves. With α = 0.8 → 5. With α = 0.9 → 10. That is why EAGLE-3, which targets α > 0.8 on many benchmarks, is not an incremental improvement over vanilla SD (α ≈ 0.5-0.6): it is a change of regime, because it raises the ceiling instead of merely getting closer to it.

A concrete numerical example, Llama 3 70B as target with a draft giving α = 0.75, γ = 5, c = 0.1:

Expected tokens per step: (1 − 0.75⁶) / (1 − 0.75) = (1 − 0.178) / 0.25 = 3.29
Speedup: 3.29 / (5 × 0.1 + 1) = 3.29 / 1.5 = 2.19×

If we raise α to 0.85 with the same configuration: tokens per step = (1 − 0.85⁶) / 0.15 = (1 − 0.377) / 0.15 = 4.16, speedup = 4.16 / 1.5 = 2.77×. A change of α = 0.75 → 0.85 (10 absolute points) raises the speedup by 27 %. That is why the research of the last two years focuses on pushing α: that is where the prize is.

The five modern families (May 2026)

Family	Year	Core idea	Draft size	Typical α / τ	Speedup
Vanilla SD	2023	Same-family target + draft pair, rejection sampling	1/10 – 1/100 of target	0.5 – 0.8	2-3×
Medusa	2024	N extra heads in parallel predict t+1, t+2, …; tree attention verifies several candidates	no draft, +1-2 % params	top-5 > 0.8	2.18-2.83×
EAGLE-1/2/3	2024-25	Autoregressive draft at the level of features (hidden states), not tokens. Reuses the target’s embedding	1 transformer block (~0.5-2 % params)	EAGLE-3: α > 0.8, τ up to 7.5	up to 6.5× peak
MTP (DeepSeek-V3)	2024	Multi-token heads trained from scratch as part of the model; at inference they act as a “free” draft	14B params in V3 671B	α > 0.8 (MTP1)	1.5-1.8×
P-EAGLE	2026	EAGLE but producing the γ drafts in a single forward pass (parallel, not autoregressive)	same as EAGLE	+30 % over EAGLE-3	4-5× vs AR

Three operational observations:

EAGLE dominates in production because its overhead is minimal (one transformer block, ~1 % of the target in parameters) and its α is high. The draft does not need its own “complete” KV cache because it shares features with the target.
Medusa mattered historically, it showed that speculative was possible without a separate draft, but EAGLE beat it on every published benchmark through 2024-2025.
MTP is special: it is not something you add to an existing model. It is something the model trained natively. If you buy DeepSeek-V3, MTP comes free and gives around 1.8× without touching anything. If you buy Llama 3, there is no MTP to be had; use EAGLE-3.

There are also two related techniques that deserve a brief mention, both without a draft model: Lookahead decoding (Fu et al. 2024), which formulates decoding as Jacobi iteration and extracts n-grams from the trajectory; and REST (He et al. 2024), which keeps an n-gram datastore and proposes drafts by longest-prefix match against the last generated tokens. Both give 1.5-2× with no extra VRAM. Useful when you have no trained draft model and do not want to maintain one.

Real implementations in May 2026

vLLM v0.16+: unified support for EAGLE-1/2/3, P-EAGLE, Medusa, native MTP (DeepSeek-V3 and variants), n-gram/suffix decoding without a draft, an arbitrary draft model and MLP speculators. The canonical flag is --speculative-config '{"method":"eagle3", "model":"...", "num_speculative_tokens": 5}'.
SGLang: native EAGLE-3 support with --speculative-algorithm EAGLE3. For DeepSeek-V3 it uses MTP through an EAGLE adapter. It has its own draft training framework (SpecForge).
TensorRT-LLM: Medusa, EAGLE (a simplified variant without a tree), ReDrafter, Lookahead. They report around 2.2× with EAGLE.
llama.cpp: basic speculative only with a draft model (--model-draft). No native EAGLE/Medusa/MTP as far as I have verified. Typical speedup 1.5-2.5× single-user.

When speculative does NOT help

The technique has three important blind spots:

Large batch. The GPU moves from memory-bound (decode at low concurrency) to compute-bound (decode at high concurrency). In the compute-bound regime the “almost free” target forward passes stop being free: γ tokens start costing γ times more instead of nearly 1. The typical crossover is at batch 16-32 for dense models; for MoE with few active parameters per token the crossover happens later. At load peaks, speculative can worsen aggregate throughput per GPU.

Prefill / TTFT. Speculative decoding does not touch prefill. The phase that processes the whole prompt stays identical. TTFT does not improve and can get marginally worse because of the draft setup. If the SLA is TTFT-bound (assistants with short outputs, search, RAG with brief answers), this is not the right tool.

Short outputs. If the model generates 10-20 tokens and the answer is over, the fixed setup overhead is not amortised. Speculative shines with long outputs (300+ tokens): generative assistants, extended code completion, drafting.

Operational pitfalls

Extra VRAM for the draft. In vanilla SD, loading a complete draft plus its KV cache is expensive. Llama 3 8B in BF16 as the draft for a 70B is about 16 GB of weights plus KV cache. On an H100 80 GB with the target already nearly filling it, that can force you to shrink the target’s KV cache and lower maximum concurrency. EAGLE solves this: the draft is one transformer block (~1 GB for a 70B) and reuses the target’s features.

Draft quantisation. Quantising the draft to INT4 with GPTQ degrades α substantially (errors in the logits accumulate in the comparison against p). AWQ holds up better but also lowers α. Common practice in 2026: target in FP8 or INT4, draft in FP16/BF16. The draft is small enough that the VRAM saved by quantising it does not compensate for the drop in α and therefore in speedup. The detail of each format is in Quantisation.

Interaction with continuous batching. It does not break it, but it creates nested raggedness: each request in the batch can accept a different number of tokens on each iteration. vLLM’s PagedAttention absorbs it, but the scheduler loses efficiency. At low QPS (conversational assistant, low simultaneous concurrency) the combination is excellent. At high QPS there is real tension, and work such as Goodput-optimized speculative decoding (Liu et al., 2024) optimises γ dynamically according to the state of the batch.

Sampling temperature. α falls with high temperatures. At T = 1.0 with creative outputs (free-form writing), α can drop 10-15 points relative to T = 0 with the same model-draft pair. The speedup scales accordingly.

Implications on on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace)

The classic use is vanilla SD with two quantised models: for instance, Llama 3 70B AWQ-INT4 as target (~35 GB → needs TP=2 across two 4090s) and Llama 3 8B AWQ-INT4 as draft. In practice, the most realistic single-card case is Llama 3 8B target + a 1B model as draft or Qwen 3 14B target + Qwen 3 0.5B as draft: they fit comfortably and give 1.8-2.5× on conversational tasks. For EAGLE on consumer cards, the official drafts for popular families (Llama 3, Qwen 3) are published on Hugging Face and take up 0.5-2 GB extra.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here EAGLE-3 shines:

Llama 3 70B FP8 + EAGLE-3 draft in FP16: the draft takes about 0.7 GB; the target about 70 GB with TP=2 (35 GB per GPU). The speedup observed in reproducible benchmarks is between 2.5× and 4× at batch 1-4, falling to almost break-even at batch 32.
DeepSeek-V3 671B FP8 + native MTP: the model ships with MTP trained in; there is nothing to add. The speedup is 1.5-1.8× with zero extra VRAM. It is the most efficient option operationally: zero additional pieces.
Combining with disaggregated serving: as Disaggregated serving explains, prefill and decode can live in separate pods. Speculative applies only in the decode pods, which fits the separation perfectly (prefill is compute-bound and would not benefit).

The rule of thumb on an H100 cluster in May 2026: if the model is DeepSeek-V3 / V4 → native MTP, nothing else; if it is Llama 3 / Qwen 3 → EAGLE-3 with the official draft; if it is exotic → vanilla SD with a draft from the same family.

What we have not covered

Training custom EAGLE drafts with SpecForge: how to collect target trajectories and train the draft on-policy.
Speculative Prefill (arXiv:2502.02789): a variant for accelerating TTFT, a different mechanism from the decode one described here.
Tree attention in detail: how Medusa and EAGLE-2 verify several candidates at once with specific attention masks.
MoE + speculative: the combination has non-trivial interactions with the expert router. Low activated params keep the memory-bound regime in place even at high batch, which changes the rules.

References

Leviathan, Y., Kalman, M., Matias, Y. Fast Inference from Transformers via Speculative Decoding. ICML 2023. https://arxiv.org/abs/2211.17192
Chen, C., Borgeaud, S., Irving, G., Lespiau, J.B., Sifre, L., Jumper, J. Accelerating Large Language Model Decoding with Speculative Sampling. DeepMind 2023. https://arxiv.org/abs/2302.01318
Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J.D., Chen, D., Dao, T. Medusa: Simple LLM Inference Acceleration with Multiple Decoding Heads. ICML 2024. https://arxiv.org/abs/2401.10774
Li, Y., Wei, F., Zhang, C., Zhang, H. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. ICML 2024. https://arxiv.org/abs/2401.15077
Li, Y., Wei, F., Zhang, C., Zhang, H. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. EMNLP 2024. https://arxiv.org/abs/2406.16858
Li, Y., Wei, F., Zhang, C., Zhang, H. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. NeurIPS 2025. https://arxiv.org/abs/2503.01840
DeepSeek-AI. DeepSeek-V3 Technical Report — Multi-Token Prediction. https://arxiv.org/abs/2412.19437
Fu, Y., Bailis, P., Stoica, I., Zhang, H. Break the Sequential Dependency of LLM Inference Using Lookahead Decoding. ICML 2024. https://arxiv.org/abs/2402.02057
He, Z., Zhong, Z., Cai, T., Lee, J.D., He, D. REST: Retrieval-Based Speculative Decoding. NAACL 2024. https://arxiv.org/abs/2311.08252
vLLM speculative decoding docs: https://docs.vllm.ai/en/latest/features/speculative_decoding/
SGLang speculative decoding docs: https://docs.sglang.ai/advanced_features/speculative_decoding.html
TensorRT-LLM speculative sampling: https://nvidia.github.io/TensorRT-LLM/advanced/speculative-decoding.html
vLLM blog Speculative Decoding in vLLM (oct 2024): https://blog.vllm.ai/2024/10/17/spec-decode.html
vLLM blog P-EAGLE: Parallel Speculative Decoding in vLLM (mar 2026): https://vllm.ai/blog/2026-03-13-p-eagle
Official EAGLE repo: https://github.com/SafeAILab/EAGLE
Official Medusa repo: https://github.com/FasterDecoding/Medusa

LLM inference operators on Kubernetes: OME, vLLM Production Stack, NVIDIA Dynamo and llm-d

Mon, 18 May 2026 17:00:00 +0200

TL;DR

Serving an LLM in production is not running a binary: it is coordinating a model (tens of gigabytes that take minutes to load), a runtime (vLLM, SGLang, TensorRT-LLM with a hundred flags), heterogeneous GPUs (NVLink, MIG, PCIe), prefill and decode that live better apart, a KV cache that wants offloading to colder tiers, intelligent routing that exploits prefix caching, and autoscaling that reacts to metrics other than CPU%. A flat Kubernetes Deployment only covers the first 20% of that. The other 80% is covered by LLM inference operators, of which four matter in 2026: OME (LMSYS, July 2025, multi-engine with a focus on SGLang), vLLM Production Stack (a curated Helm chart from vLLM itself with LMCache for tiered KV), NVIDIA Dynamo (the official successor to Triton, multi-engine, with its own Grove scheduler) and llm-d (a CNCF donation of March 2026 by Red Hat + Google + IBM + CoreWeave + NVIDIA, built on vLLM, focused on distributed scale). Behind all four sits KServe, the CNCF parent operator that normalised the InferenceService concept and that several of them lean on. This article walks through the full hierarchy, gives a decision map and teaches you not to get lost when someone drops seven acronyms in the first meeting.

This article closes the LLM inference series. The previous ones were KV cache: the working memory that holds up LLM inference, vLLM on Kubernetes: the LLM inference piece that actually scales and PagedAttention from the inside and the state of the art of the KV cache in 2026. There we explained what happens inside one inference process. Here we explain how many inference processes are coordinated through Kubernetes.

The analogy: from `init.d` to systemd to operators

Anyone with 20 years of sysadmin behind them will recognise the pattern. Decades ago, starting a service on Linux was a shell script in /etc/init.d/: start, stop, status, reloaded by hand. When services got more complex, with dependencies between them, monitoring, restart on failure, per-user slots, it became obvious that a script was not enough. Along came systemd, which turned “a service” into a declarative unit with dependencies, resources, restart policy, sockets and timers. The script did not disappear; it moved up one level of abstraction.

Kubernetes made the same move for distributed services. A Deployment declares “I want N replicas of this container”; a Service declares “these replicas are exposed like this”; an Ingress declares “this HTTP traffic comes in here”. The controller translates the declaration into real state and keeps the system converged.

Serving LLMs in 2024 was the equivalent of /etc/init.d/: every team wrote its own Deployment/Service/HPA with custom scripts for model loading, session draining and GPU handling. We covered it in the vLLM on Kubernetes article: it can be done, and it does work, but it is repetitive, fragile and nobody is extracting the right abstractions. Serving LLMs in 2026 has gone through the same transition as services: the equivalent of systemd has appeared, the inference operators, which normalise the abstractions and let the engineer declare what matters: “this model, with this runtime, this scalable, with this routing policy”.

There are four relevant operators in 2026 and a fifth common ancestor. Let us take them in order.

Why an operator, and not just a Deployment

Listing what an inference operator adds over a flat Deployment is the best way to understand what problem it solves:

The model as a first-class citizen. In a Deployment, the model is “the thing you download in an initContainer and mount as a volume”. In an operator, the model is a CustomResource with metadata (origin, fingerprint, licence, GPU requirements). It can be shared between InferenceServices, versioned, replicated to multiple nodes. It is the difference between “a file” and “a managed artifact”.

The runtime as a first-class citizen. The same goes for the runtime (vLLM/SGLang/TRT-LLM): it is not “a Docker image with flags”; it is a ServingRuntime that declares which args it accepts, which metrics it exports, which deployment types it supports (single-node, multi-node TP, PD-disag). Changing runtime is changing a reference, not rewriting every manifest.

Declarative composition. An InferenceService (the core CRD of KServe and its descendants) references a model and a runtime, declares the scaling policy, wires up observability and configures routing. The controller composes all the pieces: Deployment(s), Service, HPA, possibly a LeaderWorkerSet, a KEDA ScaledObject, a Gateway API HTTPRoute. You declare intent; the operator emits the 8 derived resources.

Operational prefill–decode disaggregation. As we saw in the PagedAttention article, separating prefill and decode into distinct pools can give 7× goodput. Modelling that with flat Deployments is feasible, but it requires coordinating two sets of pods, a transport to move the KV cache and conditional routing. An operator models it as a single InferenceService with two sub-pools.

Autoscaling on LLM metrics. The standard HPA does not understand vllm:num_requests_waiting. An operator integrates KEDA or Prometheus Adapter automatically and exposes the right metrics as knobs on the CRD.

Multi-tenancy. Multiple models on the same cluster, with quotas, priorities and fairness. One Deployment per model scaling independently is fine up to the fifth model; beyond that, coordinating GPUs between tenants becomes operationally hostile.

Model lifecycle. Weights on a shared PVC, warm-up of the first pod, rolling updates with maxUnavailable: 0, draining of active sessions, integrated observability. Things that with a flat Deployment have to be reinvented by every team.

If your workload is one model, one node, up to three replicas, a flat Deployment is enough and an operator is overkill. If your workload is two or more models, serious scaling, disaggregation or multi-tenancy, an operator stops being optional.

KServe: the common ancestor

Before the four new ones, we have to mention KServe, the parent operator from which all the rest conceptually descend. It was born as KFServing inside the Kubeflow project in 2019, was renamed KServe when it became independent in 2021, and in 2025 was accepted into the CNCF as a project incubating towards graduation.

KServe’s conceptual contribution is the InferenceService CRD, which has become the field’s common vocabulary: a declarative K8s object that joins a model (origin + metadata) with a predictor (runtime + resources) and produces a ready HTTP service. Under the hood, the controller emits Deployments, Services, HorizontalPodAutoscalers, Knative Services if you go serverless, Istio VirtualServices if you do traffic splitting.

KServe was designed in a pre-LLM era: its first use cases were traditional scikit-learn, TensorFlow and PyTorch models served as simple REST APIs. That gives it strengths (it is mature, with 6 years in production at Bloomberg, JPMorgan and others) and weaknesses (it was not designed to manage multi-node tensor parallel, prefill–decode disaggregation, or the patterns specific to LLMs).

The way the ecosystem has reacted is elegant: the new LLM operators inherit from or take inspiration from InferenceService but extend the API with LLM-specific primitives. OME is the clearest example: it uses the name InferenceService and the idea of “model + runtime → service”, but adds BaseModel, a ServingRuntime with LLM-aware flags, and deployment modes (PD-disag, multi-node) that KServe does not cover natively.

OME (Open Model Engine)

OME was published by the LMSYS team in July 2025 (announced on their blog). It is an operator that understands SGLang deeply (that is its first-class runtime) but also supports vLLM, TensorRT-LLM and Triton.

The CRD hierarchy

OME models the domain with four main CRDs:

BaseModel and ClusterBaseModel: the model itself. It defines origin (Hugging Face, S3, URL), fingerprint and metadata. The Cluster* version is global; BaseModel is namespaced. It lets multiple InferenceService objects reference the same model without duplicating the download.
FineTunedWeight: LoRA adapters or finetuned weights served on top of a BaseModel. Critical for multi-tenant setups where each customer has their own finetune.
ServingRuntime and ClusterServingRuntime: the runtime (vLLM, SGLang, and so on) with its configuration. It declares which args it accepts, which metrics it exports, which deployment modes it supports.
InferenceService: the central declarative piece, joining BaseModel + ServingRuntime + infrastructure.

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
 name: llama3-70b-prod
 namespace: inference
spec:
 model:
 name: meta-llama-3-70b-instruct  # reference to a BaseModel
 runtime:
 name: sglang-h100  # reference to a ServingRuntime
 deploymentMode: PrefillDecodeDisaggregated  # standard | PD | MultiNode | Serverless
 prefill:
 minReplicas: 2
 maxReplicas: 8
 resources:
 requests:
 nvidia.com/gpu: 4
 decode:
 minReplicas: 4
 maxReplicas: 16
 resources:
 requests:
 nvidia.com/gpu: 1
 router:
 type: cache-aware  # SGLang router with cache awareness
 autoscaling:
 metricSource: keda
 metrics:
 - type: prometheus
 metricName: vllm_requests_waiting
 threshold: "10"

That is what the operator takes as input. The output is roughly 8 derived resources that would be horrible to declare by hand: two LeaderWorkerSets (one per prefill/decode pool), two Services, a Deployment for the router, KEDA ScaledObjects for each pool, a Gateway API HTTPRoute, and a PriorityClass that hooks into Kueue for gang scheduling.

The four deployment modes

OME materialises the InferenceService differently depending on deploymentMode:

Standard: one Deployment with N replicas; the classic. For small or single-GPU models.
PrefillDecodeDisaggregated: two coordinated pools; the SGLang router routes between them.
MultiNode: tensor parallel across multiple nodes via LeaderWorkerSet, with NCCL/InfiniBand. For models above 70B where a single node is not enough.
Serverless: Knative-style scale-to-zero. For sporadic workloads where the cost of keeping GPUs powered up does not pay off. The trade-off: the first request pays the model’s cold start cost (minutes).

Integration with the K8s ecosystem

OME does not invent primitives where they already exist. It leans on:

Kueue for gang scheduling: all the pods of a tensor parallel group have to start at once or not at all; Kueue guarantees it.
LeaderWorkerSet (LWS) for multi-node: workers join the leader’s Ray cluster, with an atomic lifecycle (one going down restarts the group).
KEDA for autoscaling on LLM-specific Prometheus metrics (queue depth, GPU cache usage, TTFT p95).
Gateway API and its Inference Extension for advanced routing (model-aware, prefix-aware, weighted canary).

The consequence: OME feels “idiomatically Kubernetes”. It does not introduce new concepts where they are not needed; it uses standard primitives and concentrates on what is specific to the LLM domain.

When to choose it

OME is the natural choice if SGLang is your main runtime and/or if you come from the KServe ecosystem and want an idiomatic evolution of it. It is mature but relatively young (a year old at the time of writing); expect rough edges on advanced features.

vLLM Production Stack

vLLM Production Stack is the official project from vLLM itself for production on Kubernetes. Its philosophy is the opposite of OME’s: instead of an operator with new CRDs, it is a curated Helm chart that deploys a coherent set of pieces.

The three pieces

The stack has three components:

Serving engines: vLLM pods, configured with the flags we have been seeing throughout the series (--enable-prefix-caching, --kv-cache-dtype fp8, and so on). The Helm chart lets you declare them as a list; it deploys the underlying Deployments and Services.
Request router: a proxy in front of the engines that decides which one each request goes to. It supports several policies:
- Round-robin: trivial, for a baseline.
- Session-based: pins each session to one replica to keep its KV cache.
- Prefix-aware: detects shared prefixes between requests and routes them to the replica that already has them cached.
- KV-aware: sees each replica’s gpu_cache_usage_perc and avoids the saturated ones.
- Disaggregated-prefill with native LMCache: separates prefill and decode, with LMCache as the transport for the KV cache between the two.
Observability stack: Prometheus + Grafana with ready-made dashboards. It measures TTFT, TBT (Time-Between-Tokens), throughput, queue depth and GPU memory.

LMCache and tiered KV

One of the most interesting pieces the stack brings is LMCache, which adds a KV cache with multiple tiers: GPU HBM as L1, CPU RAM as L2, local disk as L3, and optionally remote storage as L4. When a KV cache block does not fit in HBM, instead of evicting and recomputing it, LMCache drops it to a lower tier. For workloads with shared prefixes and multi-turn conversations, the saving is huge.

LMCache integrates as a sidecar to the engines and as part of the transport in disaggregated-prefill. Production Stack ships it enabled by default in its Helm chart.

A typical manifest (values.yaml)

servingEngineSpec:
 modelSpec:
 - name: llama3-8b
 repository: vllm/vllm-openai
 tag: v0.6.3
 modelURL: meta-llama/Meta-Llama-3-8B-Instruct
 replicaCount: 3
 requestCPU: 4
 requestMemory: 16Gi
 requestGPU: 1
 vllmConfig:
 enablePrefixCaching: true
 kvCacheDtype: fp8
 maxModelLen: 32768
 enableChunkedPrefill: true

routerSpec:
 routingLogic: prefix-aware  # round-robin | session | prefix-aware | kv-aware
 sessionKey: x-user-id  # when routingLogic=session

cacheserverSpec:
 enabled: true # LMCache for tiered KV
 storageBackends:
 - cpu
 - disk  # offload to local disk

observabilitySpec:
 prometheus:
 enabled: true
 grafana:
 enabled: true
 dashboards:
 - vllm-engine-metrics
 - lmcache-metrics

This is declarative but these are not CRDs: they are Helm chart values. The difference with OME is not semantic (both start from declaration) but operational: with Helm, changes go through helm upgrade; with CRDs, they go through kubectl apply. For teams already living in GitOps with Argo CD or Flux, both approaches integrate cleanly, but the flows are different.

When to choose it

If vLLM is your only runtime and you want the closest thing to “the happy path the project recommends”, this one. It is the productionised version maintained by the same people who write the engine. The downsides: it ties you to vLLM (it is not generic) and it does not solve some advanced cases such as multi-tenancy with strict quotas or gang scheduling, where OME or full-fledged operators are better.

NVIDIA Dynamo

NVIDIA Dynamo is the official successor to Triton Inference Server, announced at GTC 2025 and merged into the brand as Dynamo-Triton in March of that year. Triton had spent years as the most-used inference engine in “serious” NVIDIA infrastructures; Dynamo is what NVIDIA believes the new generation needs.

What it is exactly

Dynamo is a distributed inference framework, not exactly a Kubernetes operator. It has its own runtime (it can run engines), a scheduler (Grove), intelligent routing, multi-tier KV cache management and disaggregation. It supports SGLang, TensorRT-LLM and vLLM as engines, but the engines are executed by Dynamo, not the other way round: the model is “Dynamo manages, the engine executes”.

On Kubernetes, Dynamo is deployed via its own operator plus CRDs, normalised with the K8s integration NVIDIA formalised at the end of 2025 (covered by this InfoQ piece). The CRDs are product-specific: they define a DynamoCluster, a topology of prefill/decode workers and a routing policy.

The four contributions

Dynamo is sold on four pillars, with figures reported by NVIDIA:

Disaggregated serving built in, with its own scheduler.
Smart routing based on cache state: if a worker already has most of a prompt cached, the request goes there.
Multi-tier KV cache: analogous to LMCache, with HBM/RAM/SSD/NVMe.
Autoscaling integrated with Dynamo’s scheduler.

The marketing figure: up to 30× more throughput than legacy Triton on the same hardware. With all the caution a vendor benchmark deserves.

Grove: its own scheduler

A controversial decision in Dynamo is not to rely 100% on the Kubernetes scheduler and instead bring its own scheduler, called Grove, that understands GPU topologies. Grove decides which worker runs on which physical GPU, which interconnects (NVLink/InfiniBand) are relevant, and how to distribute tensor parallel across nodes. That gives it more control than the standard kube-scheduler.

Operationally: if your cluster is “pure Kubernetes” with kube-scheduler and heterogeneous workloads (not only LLMs), Grove adds one more component to operate. If your cluster is dedicated to LLM inference and there is already a team dedicated to running it, Grove gives you more levers.

When to choose it

Dynamo makes sense if:

Your infrastructure is NVIDIA-heavy (Hopper, Blackwell, GB200) and you want to exploit the latest in TensorRT-LLM with the familiar Triton integration, modernised.
You were already a Triton user for legacy inference (vision, recommendation) and want to keep the ecosystem.
You have a dedicated SRE team for inference and Grove’s extra operational complexity is not a problem.

It is the vendor-specific option of the quartet. In exchange it gives you NVIDIA’s support and first-rate integration with their hardware. If your organisation is already fighting NVIDIA for GPUs, they may even call you offering help with Dynamo.

llm-d

llm-d is the youngest and the most “political” of the four. In March 2026, at KubeCon Europe Amsterdam, Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA announced the joint donation of the project to the CNCF as a Sandbox project, with backing from AMD, Cisco, Hugging Face, Intel, Lambda, Mistral AI, UC Berkeley and the University of Chicago. An explicit coalition for vendor neutrality.

Philosophy

llm-d positions itself as the vendor-neutral “Kubernetes blueprint” for distributed inference. It is not a runtime; it is a system that sits on top of vLLM (the default engine) and orchestrates the control plane.

The primitives the project puts on the table:

Intelligent routing with prefix-cache awareness and load-aware balancing.
Tiered KV cache with offload to CPU and disk for multi-turn.
Prefill/decode disaggregation over fast interconnects.
Wide expert parallelism for serving very large Mixture-of-Experts (MoE) models, a critical pattern popularised by DeepSeek-V3 and Mixtral, where the experts live on different GPUs and tokens have to be routed to the right expert.

Numbers

The v0.5 release validates ~3.1k tok/s per decode GPU on B200, and up to 50k output tok/s on a 16×16 B200 prefill/decode topology. The most interesting benchmark: an order-of-magnitude reduction in TTFT against a round-robin baseline. In other words, intelligent routing is worth what they say it is.

CNCF and the future

Donating to the CNCF as a Sandbox project means neutral governance: no vendor is in charge. For an organisation wary of being tied to a single supplier, llm-d is probably the safest medium-term bet. The price: like any Sandbox project, it is not yet “boring” in the way vLLM is. There is API churn, features that move around, documentation that lags the code.

When to choose it

llm-d makes sense if:

You want multi-vendor portability without ties to NVIDIA, Red Hat or Google.
Your workload includes large MoE models (DeepSeek-V3, Mixtral 8x22B, Llama 4 Behemoth if the size is confirmed), where wide expert parallelism is decisive.
Your organisation is already comfortable with CNCF Sandbox (projects under active evolution, not yet a stable 1.0).
You want to bet on the project that will probably be the de facto standard in 2-3 years.

The common ancestor is still there: KServe

It is worth reconnecting before the comparison: KServe is still alive and heavily used in organisations that serve both LLMs and traditional models (scikit-learn, XGBoost, PyTorch CV). Its InferenceService is generic enough to serve any model, including vLLM or SGLang as a ServingRuntime. What it does not do well is the LLM-specific part: disaggregation, multi-node tensor parallel, routing with KV cache awareness. If your organisation already has KServe in production for other models, adding an LLM-specific operator alongside it (OME, vLLM Stack or llm-d) is reasonable. Fighting it all out from pure KServe is not.

Decision map

Dimension	OME	vLLM Prod Stack	NVIDIA Dynamo	llm-d
Philosophy	Classic K8s-idiomatic operator	Curated Helm chart	Framework with its own scheduler	Vendor-neutral CNCF blueprint
Own CRDs	Yes (BaseModel, ServingRuntime, InferenceService…)	No (Helm values)	Yes (DynamoCluster)	Yes (KServe-derived + extensions)
Primary runtime	SGLang (first class), also vLLM/TRT-LLM/Triton	vLLM only	TensorRT-LLM (first class), also SGLang/vLLM	vLLM (first class)
PD-disaggregation	Yes, declarative	Yes, with LMCache	Yes, own scheduler	Yes, native
Multi-node TP	Yes, via LWS	Limited	Yes, via Grove	Yes, via LWS and MoE EP
Multi-model in a cluster	Yes, mature multi-tenant	Yes (list of models in values)	Yes	Yes
Multi-LoRA	Yes, first class (FineTunedWeight CRD)	Limited	Yes	On the roadmap
Tiered KV cache	Via LMCache (external integration)	Native LMCache	Own multi-tier	Yes, native
Intelligent routing	Cache-aware via the SGLang router	Prefix-aware / KV-aware / session-based	Own smart routing	Prefix-cache + load-aware
GPU scheduler	kube-scheduler + Kueue	kube-scheduler	Grove (its own)	kube-scheduler + Kueue
Hardware	NVIDIA, AMD ROCm, Intel	NVIDIA, AMD ROCm	NVIDIA only (with emphasis)	NVIDIA, AMD, Intel — neutral
Maturity (mid-2026)	Young, evolving	Stable	Stable, vendor-driven	CNCF Sandbox, evolving fast
Governance	LMSYS (academic-industrial)	vLLM project (academic)	NVIDIA (vendor)	CNCF (neutral)
Learning curve	Medium (4 new CRDs)	Low (familiar Helm values)	Medium-high (Grove + own CRDs)	Medium (similar to extended KServe)

When to choose each one

Choose OME if:

SGLang is your main engine.
You need multi-LoRA serving in production.
The hierarchical abstraction (BaseModel → ServingRuntime → InferenceService) fits you and you come from, or live alongside, KServe.
You have an appetite for a young and very active project.

Choose vLLM Production Stack if:

vLLM is your only engine and you want to align with what the project recommends.
Your team already lives in Helm and does not want to learn new CRDs.
LMCache + advanced routing inside a single Helm chart is exactly what you need.
Your scale is medium (tens of replicas), not extreme.

Choose NVIDIA Dynamo if:

Your infrastructure is NVIDIA-heavy and you want the most optimised path for Hopper/Blackwell.
You already ran Triton for legacy inference and the transition is natural.
You accept vendor lock-in in exchange for direct NVIDIA support.
Your organisation has a dedicated SRE team for inference.

Choose llm-d if:

You want to bet on the future CNCF standard, neutral between vendors.
Your workload includes large MoE models with wide expert parallelism.
You operate multi-cloud or multi-hardware and portability is valuable.
You accept the immaturity of a Sandbox project in exchange for the bet on the future.

Choose pure KServe if:

You already serve non-LLM models and want to unify; LLMs are a minority of your workload.
You need the most conservative and mature use case.
You accept that advanced LLM features (disaggregation, MoE EP, smart routing) are yours to add with external pieces.

Concrete scenarios

Scenario A — Small startup, 1-2 models, 1-3 GPU nodes. You probably do not need an operator. Deployment + Service + HPA with KEDA metrics, as in the vLLM on Kubernetes article. When you grow to 5+ models, reassess.

Scenario B — Mid-sized company, 5-15 models, internal multi-tenant. vLLM Production Stack or OME are the reasonable options. Production Stack if vLLM is all you are going to use; OME if you want runtime flexibility and idiomatic CRDs.

Scenario C — Internal corporate platform or an external service for end customers. llm-d or Dynamo. llm-d if you value vendor neutrality; Dynamo if you live on NVIDIA infrastructure and want the path they recommend.

Scenario D — Mixed cluster of LLMs plus traditional models. KServe as the base, with an LLM operator alongside (OME is the most natural given its conceptual kinship).

Common traps

“I’ll start with pure KServe because it’s mature”. For mid-sized LLMs and up, pure KServe leaves a lot of optimisation on the table. The sensible route is KServe as the base if you live alongside other models, but with an LLM-specific operator next to it.

“I’ll wire everything by hand to understand it”. Reasonable in a PoC, suicidal in production. There are 8 derived resources per model. Multiply by 10 models. You are writing 80 YAMLs and maintaining them. Use an operator.

“I’ll pick the one I like best and pivot if I’m wrong”. Pivoting between operators is not free: although the InferenceService abstraction is converging, the details (how LoRA is modelled, how routing is configured, how metrics are exposed) vary. Migrating from OME to Dynamo is a project of weeks, not days.

“I’ll go with Dynamo because it’s NVIDIA’s and therefore better”. Only if your organisation is already aligned with its operational philosophy (its own scheduler, acceptable vendor lock-in). For many cases, vLLM Production Stack or llm-d give 95% of the value with less friction.

“Helm chart vs operator is a technical decision”. It is a cultural and operational decision. If your team ships via Argo CD with Helm values in Git, Production Stack fits without friction. If your team lives in direct kubectl apply -f and the idea of operators feels natural, OME or llm-d.

What we have not covered

Mooncake: the KV cache system shared between instances that Kimi/Moonshot runs in production across hundreds of millions of queries. It is a primitive (not a complete operator), but it integrates as a cache tier with several of the above.
Ray Serve LLM: Anyscale’s offering, on Kubernetes through KubeRay. More tied to the Ray ecosystem than to native K8s CRDs. Useful if Ray is already part of your infrastructure.
Fireworks AI, Modular MAX: commercial platforms with similar primitives, but hosted. They are not K8s operators; they are competitors at another layer.
Gateway API Inference Extension: the sigwg proposal to extend Gateway API with LLM primitives (model-aware routing, sticky sessions, fairness). In 2026 it is in alpha; the operators above are already starting to support it. When it matures, routing will stop being each operator’s problem and become part of the Kubernetes standard.
A generic inference observability stack: Prometheus + Grafana is standardising around the vllm:* metrics we covered in the vLLM article. There is an OpenTelemetry effort for LLMs (gen-ai semantic conventions) that is probably the next link in the chain.

Closing the series

This series of four articles has walked through LLM inference in production from the bottom up:

KV cache: the working memory that holds up LLM inference — why each token consumes VRAM, and how much.
vLLM on Kubernetes: the LLM inference piece that actually scales — how a model is served in production with a serious Deployment.
PagedAttention from the inside and the state of the art of the KV cache in 2026 — what happens inside the engine at block level, and what has arrived since.
This one — how many models are orchestrated in a cluster.

If you have got this far, you have the vocabulary and the map to sit in a meeting where five people throw acronyms around and place each one correctly. And, more importantly, to start making reasoned decisions about where to begin.

References

Operators and projects covered:

OME — Open Model Engine (GitHub) — LMSYS operator for LLM serving with SGLang/vLLM/TRT-LLM/Triton.
Introducing OME (LMSYS Blog, jul 2025) — announcement and architecture.
vLLM Production Stack (GitHub) — official vLLM Helm chart for K8s.
vLLM Production Stack docs — installation and configuration.
LMCache (GitHub) — KV cache with tiers.
NVIDIA Dynamo — successor to Triton.
NVIDIA Dynamo Addresses Multi-Node LLM Inference Challenges (InfoQ, dic 2025) — K8s integration.
llm-d (GitHub) — CNCF Sandbox project.
IBM, Red Hat, and Google donated llm-d to CNCF (The New Stack) — KubeCon EU 2026 announcement.
Red Hat bets big on Kubernetes inference with llm-d (SiliconANGLE, mar 2026) — coverage of the announcement.

Ancestors and primitives:

KServe (site) and KServe joins CNCF (The New Stack).
Kueue — gang scheduling.
LeaderWorkerSet — coordinated workloads such as multi-pod tensor parallel.
KEDA — autoscaling on external metrics.
Gateway API — the successor to Ingress.

Analysis and perspectives:

Building Efficient LLM Inference with the Cloud Native Quartet: KServe, vLLM, llm-d, and WG Serving (Jimmy Song) — an integrating view.
Complete Guide to llm-d CNCF Sandbox (DEV Community) — an operational walkthrough.
Previous articles on this blog: KV cache, vLLM on Kubernetes, PagedAttention deep dive.

Sglang on lo0 — Tech Blog

LLM serving engines compared on the Pareto frontier: vLLM, SGLang, TRT-LLM and Dynamo

TL;DR

Framing the article: what this B8 is and is not

Card for each engine

vLLM

SGLang

NVIDIA TensorRT-LLM (TRT-LLM)

NVIDIA Dynamo

Honourable mentions: LMDeploy and HF TGI

Capability matrix

Why they are not compared lightly: the variables that break a comparison

Performance on the Pareto frontier: illustrative figures

Throughput vs latency: Pareto table

Saturation point (concurrency sweep, vLLM FP8, 4×H100)

Fair-comparison methodology: the harness as the constant

Variables that are fixed (constants of the experiment)

Harness steps

Reproducible output format

State of the art 2026: relative maturity

Pareto decision table: when to choose each engine

Quantisation: effect on throughput and quality

Disaggregated serving: when the overhead is worth it

MLPerf Inference v5.1: cross-vendor reference figures

Variables the benchmark does not capture

Sources

LLM benchmark tool catalogue: the practical deep dive

What this article covers

How to choose: the tool map

Datasets: the synthetic load matters as much as the tool

vLLM bench serve

SGLang bench

AIPerf (NVIDIA, formerly genai-perf)

GuideLLM (vLLM project)

LLMPerf (Anyscale/Ray)

inference-benchmarker (Hugging Face)

GuideLLM versus AIPerf: which of the two load generators

MLPerf Inference (MLCommons)

Practical comparison table

Summary: what you run for each question

Step-by-step benchmark methodology

Comparing two engines: the fair protocol

Output format and comparability

Worked example: reading the output of a sweep

Automating the harness

CI integration: continuous benchmarking

The cost of measuring (in euros)

Observing during the benchmark: one run, three axes

The benchmark is also FinOps and energy

State of the art 2026 and limits

Mistakes that invalidate a benchmark

Closing

See also

Sources

Serving reasoning models: the invisible scratchpad that decides your latency and your bill

TL;DR

The analogy

What changes when the model “thinks”

The 2026 landscape: from the model that “always reasons” to the switchable hybrid

The economics: why the scratch paper drives up the bill

Controlling the thinking budget

1. Switching think / non-think

2. Setting a cap: thinking_token_budget

3. Budget forcing (state of the art)

The impact almost nobody measures: the KV cache

Reasoning + structured output: order matters

Reference architecture

Operational pitfalls (and honest scepticism)

Closing

Sources

Structured output: the form with dropdowns that strikes out invalid answers before the model picks one — Outlines, XGrammar, LLGuidance and the maths of the bitmask

You are here: DEPLOY

TL;DR

The analogy: the form with dropdowns instead of free-text fields

The problem structured output solves

Constrained decoding: the principle

The four families of backends

XGrammar in detail (what sits under vLLM and SGLang)

The maths of the bitmask

Does it degrade the model’s reasoning?

2. Setting a cap: `thinking_token_budget`

The analogy: from `init.d` to systemd to operators