Tensorrt-Llm on lo0 — Tech Blog

LLM serving engines compared on the Pareto frontier: vLLM, SGLang, TRT-LLM and Dynamo

Tue, 16 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter). Reference hardware: generic configuration 4×H100 SXM 80 GB; the figures are illustrative for comparing engines, not production metrics from any real infrastructure.

TL;DR

Four OSS engines dominate on-premise LLM serving in 2026: vLLM, SGLang, NVIDIA TensorRT-LLM and NVIDIA Dynamo. None is superior on every axis. In raw throughput with a shared prompt, SGLang beats vLLM by 29 % on Llama-3.3-70B FP8 over H100 (16,200 vs 12,500 tok/s). In decode latency, TRT-LLM keeps ITL more stable at high concurrency (9–12 ms P50 against 18–22 ms for vLLM). Dynamo is not an inference engine: it is the orchestration layer on top of the previous three for multi-node clusters, and it reports up to 7× additional throughput in disaggregated serving on Blackwell. The right choice depends on three variables: load pattern (batch/interactive), deployment scale (single node/multi-node) and operational constraints (OSS maturity, ease of operation). The Pareto decision table is at the end of the article.

Framing the article: what this B8 is and is not

This article (B8 of the benchmarking pillar) takes for granted the metrics from the introduction to benchmarking and the tool catalogue from the card-by-card piece. It applies those concepts to engine selection. The protocol from GuideLLM for SLO validation and AIPerf in depth detail how to run the harness; here the result is used. The fundamentals of continuous batching, disaggregated serving, the attention backend and quantisation are assumed known.

Card for each engine

vLLM

Attribute	Value
Reference version	v0.9.x (June 2026)
Licence	Apache 2.0
Maintainer / governance	vLLM Project (Linux Foundation AI), led by UC Berkeley + Red Hat
Supported hardware	NVIDIA (Ampere/Hopper/Blackwell), AMD ROCm, Intel Gaudi, TPU, CPU
Repository	github.com/vllm-project/vllm
API	OpenAI-compatible (completions, chat, embeddings)

Origin: PagedAttention (Kwon et al., 2023). The first implementation of continuous batching with a paged KV cache. In 2026 it is the most deployed engine in OSS environments and the one that accumulates the most integrations (Ray Serve, Kubernetes, llm-d, Dynamo).

SGLang

Attribute	Value
Reference version	v0.4.x (June 2026)
Licence	Apache 2.0
Maintainer / governance	LMSYS Org (Berkeley, CMU, UCSD)
Supported hardware	NVIDIA (Ampere/Hopper/Blackwell), AMD ROCm
Repository	github.com/sgl-project/sglang
API	OpenAI-compatible + SGLang runtime API

Origin: the SGLang paper, 2024 (Zheng et al.). It introduces RadixAttention as a generalisation of prefix caching: a radix tree of KV blocks with an LRU policy shared across all in-flight requests. A structural advantage in workloads with shared prefixes (long system prompts, few-shot, RAG).

NVIDIA TensorRT-LLM (TRT-LLM)

Attribute	Value
Reference version	v0.18.x (June 2026)
Licence	Apache 2.0
Maintainer / governance	NVIDIA (proprietary in practice, OSS in name)
Supported hardware	NVIDIA only: Ampere (A100), Hopper (H100/H200/GH200), Ada (L40/L40S), Blackwell (B200/GB200)
Repository	github.com/NVIDIA/TensorRT-LLM
API	Python + C++ runtime; integrates with Triton Inference Server

The lowest-level engine: it compiles inference graphs with TensorRT, applies kernel fusion and generates binary engines specific to model + hardware + precision. Maximum performance on NVIDIA, with no portability to other hardware. It requires recompilation when the model or configuration changes; build time ranges from minutes to hours depending on the model.

NVIDIA Dynamo

Attribute	Value
Reference version	v1.1.1 (May 2026)
Licence	Apache 2.0
Maintainer / governance	NVIDIA + OSS community (ai-dynamo org on GitHub)
Supported hardware	NVIDIA (through the underlying engine: vLLM, SGLang or TRT-LLM)
Repository	github.com/ai-dynamo/dynamo
Role	Orchestration layer, NOT an inference engine

Dynamo does not run inference directly: it is the scheduler and router that coordinates pools of workers (vLLM, SGLang, TRT-LLM) across multiple nodes. Implemented in Rust (performance) + Python (extensibility). It adds: disaggregated prefill/decode serving across nodes, KV-aware routing (avoiding prefill recompute if another worker already has the KV in cache), automatic scaling by SLA, a KV Block Manager (KVBM) with offload to CPU/SSD/remote and transfer via NIXL over NVLink/InfiniBand.

Honourable mentions: LMDeploy and HF TGI

LMDeploy (OpenMMLab/InternLM, Apache 2.0): an engine with two backends, TurboMind (C++, maximum performance on NVIDIA) and PyTorch (flexibility). KV cache blocking and persistent batching; 4-bit inference up to 2.4× faster than FP16 in its own benchmark. A narrower ecosystem; no AMD support.

HF TGI (Hugging Face, Apache 2.0): the reference engine for the HuggingFace ecosystem. TGI v3.0 cuts latency by up to 13× on long prompts against earlier versions; it integrates FlashAttention + PagedAttention. The usual entry point for teams already operating the HF stack; it is not the most efficient in throughput at maximum load.

Capability matrix

The table uses this notation: Yes = supported and stable, Beta = available but not recommended for production, No = not available or only with a manual workaround.

Capability	vLLM	SGLang	TRT-LLM	Dynamo (via backend)
Continuous batching	Yes	Yes	Yes	Yes (delegates to the engine)
PagedAttention	Yes	Yes	Yes	Yes
RadixAttention / prefix caching	Yes (chunked prefill + prefix)	Yes (native RadixAttention)	Yes (prefix caching)	Yes + KV-aware routing
Speculative decoding	Yes (n-gram, EAGLE, DFlash, suffix)	Yes (draft+verify)	Yes (EAGLE-3, MTP)	Yes
FP8 quantisation	Yes	Yes	Yes	Yes
MXFP4/NVFP4 quantisation	Yes	Beta	Yes (Blackwell)	Yes
INT4 AWQ quantisation	Yes	Yes	Yes	Yes
GPTQ quantisation	Yes	Yes	Not native	Yes (via vLLM)
INT8 quantisation (SmoothQuant)	Yes	Yes	Yes	Yes
Structured output	Yes	Yes (compressed FSM)	Beta	Yes
Dynamic multi-LoRA	Yes	Yes	Beta	Yes
Disaggregated prefill/decode	Beta (experimental)	Yes	Yes	Yes (first class)
Chunked prefill	Yes	Yes	Yes	Yes
Multimodal support (VLM)	Yes	Yes	Yes	Yes
Multi-node tensor parallelism	Yes	Yes	Yes	Yes (+ pipeline across nodes)
OpenAI API compatible	Yes	Yes	Yes (via Triton)	Yes
Non-NVIDIA hardware	Yes (AMD/Intel/TPU)	Yes (AMD)	No	No

Sources for the matrix: docs.vllm.ai, docs.sglang.io, nvidia.github.io/TensorRT-LLM, docs.nvidia.com/dynamo, github.com/ai-dynamo/dynamo (feature matrix, June 2026).

Why they are not compared lightly: the variables that break a comparison

Before the table of figures, the four decisions that invalidate any cross-engine benchmark:

1. A different harness. A benchmark run with each engine’s native micro-bench (vllm bench serve vs SGLang bench) measures the client, not the engine. The discrepancy can be up to 7.2× at high concurrency. The correct protocol: a single load generator (AIPerf or GuideLLM) against every endpoint. The engine is the variable, the tool is the constant.

2. Different ISL/OSL. Input Sequence Length and Output Sequence Length determine the prefill/decode ratio and therefore which engine is favoured. A benchmark with low ISL (64 tokens) favours engines with fast decode; with high ISL (2,048 tokens) and shared prefixes, it favours SGLang (RadixAttention). Fixing ISL and OSL is mandatory for the comparison to be valid.

3. Different precision. FP16 and FP8 are not comparable: FP8 gives between 1.5× and 2× more throughput on Hopper. Comparing vLLM FP16 with TRT-LLM FP8 says nothing about the engine.

4. Not reporting P99. Maximum throughput is reached at the point where TTFT P99 already violates the SLO. The defensible number is goodput, throughput under the SLO, not the raw maximum. The end-to-end latency formula:

$$\text{latency}_{e2e} \approx \text{TTFT} + (N_{\text{out}} - 1) \times \text{TPOT}$$

where $N_{\text{out}}$ is the number of output tokens, makes it clear that a high TTFT with low TPOT has the same effect as the inverse case only for one specific $N_{\text{out}}$: both must be reported separately.

The complete fair-comparison protocol is in the tools article.

Performance on the Pareto frontier: illustrative figures

Reference configuration for every row: 4×H100 SXM 80 GB NVLink, model Llama-3.1-70B-Instruct FP8, harness AIPerf (formerly genai-perf, the multi-process successor), dataset ShareGPT (realistic length distribution), mean ISL ~512 tok, mean OSL ~256 tok, SLO TTFT P99 < 500 ms. The figures are illustrative and representative of benchmarks published by the community (Cerebrium, LMSYS, Spheron, Red Hat MLPerf v5.1); they are not measurements from any real infrastructure. Exact figures vary with engine versions, compilation flags and load distribution.

Throughput vs latency: Pareto table

Engine	Config	Throughput (tok/s)	TTFT P50 (ms)	TTFT P99 (ms)	ITL P50 (ms)	Goodput (@SLO)
vLLM v0.9	FP8, chunked prefill	12,500	160	420	18	~96 %
SGLang v0.4	FP8, RadixAttention	16,200	140	390	21	~97 %
TRT-LLM v0.18	FP8, compiled engine	14,800	190	480	10	~93 %
vLLM + Dynamo	FP8, disagg. P/D 2+2	18,500	120	310	19	~98 %
SGLang + Dynamo	FP8, disagg. P/D 2+2	21,000	110	280	22	~99 %
TRT-LLM + Dynamo	FP8, disagg. P/D 2+2	22,500	130	350	11	~97 %

Reading notes:

“Disagg. P/D 2+2” = 2 prefill GPUs + 2 decode GPUs (the same 4×H100 budget).
Goodput falls against raw throughput when the latency tail exceeds the SLO of 500 ms P99. TRT-LLM without Dynamo has the lowest ITL but the highest TTFT at high concurrency, which brings its goodput down to ~93 % under this SLO.
Dynamo adds ~30–40 % of effective throughput over 4 GPUs thanks to disaggregated serving and KV-aware routing, in the scenario with reusable prefixes (ShareGPT).
In workloads without shared prefixes (short ISL, unique prompts), Dynamo’s advantage shrinks to ~10–15 % over the base engine.

Saturation point (concurrency sweep, vLLM FP8, 4×H100)

Concurrency	TTFT P99 (ms)	ITL P50 (ms)	Throughput (tok/s)	Goodput (tok/s)
8	210	14	5,200	5,200
16	370	18	9,800	9,800
32	420	18	12,500	12,200
48	680	26	13,800	7,400
64	1,200	41	14,100	2,100

The knee is between 32 and 48: throughput rises only 10 % (12,500 → 13,800) but TTFT P99 already violates the 500 ms SLO and goodput halves. The defensible operational capacity is the one at concurrency 32 (12,200 useful tok/s). Reporting 14,100 tok/s (concurrency 64) would be maximum throughput with 15 % goodput.

Fair-comparison methodology: the harness as the constant

The protocol that makes numbers comparable across different engines. The engine is the only variable; everything else is constant and pinned in the output JSON.

Variables that are fixed (constants of the experiment)

Variable	Value fixed in the reference experiment
Load tool	AIPerf v2.x (multi-process)
Model	Llama-3.1-70B-Instruct
Precision	FP8
Hardware	4×H100 SXM 80 GB NVLink
Dataset	ShareGPT (realistic ISL/OSL distribution)
SLO	TTFT P99 < 500 ms, ITL P50 < 30 ms
Warm-up	200 requests discarded before measuring
Sweep	concurrencies 1, 4, 8, 16, 32, 48, 64
Primary metric	goodput (useful tok/s under SLO)

Harness steps

Deploy the engine with the exact config (pinned flags, fixed version).
Warm-up: 200 requests; discard the results.
Concurrency sweep: aiperf profile --concurrency-range 1:64:step with AIPerf.
Extend the sweep past the knee (until goodput falls below 50 %).
Collect the JSON: TTFT P50/P99, ITL P50, raw throughput, goodput.
Change only the engine; repeat steps 1–5.
Compare the goodput column across rows.

The sweep with GuideLLM is equivalent for SLO validation; AIPerf is preferable for engine comparisons because its automatic estimatedCapacity normalises the saturation point. Both can be used: GuideLLM for the operational SLO, AIPerf for reference capacity.

Reproducible output format

{
 "harness": "aiperf", "harness_version": "2.x",
 "model": "Llama-3.1-70B-Instruct", "precision": "FP8",
 "hardware": "4xH100_SXM_80GB_NVLink",
 "dataset": "sharegpt",
 "slo": {"ttft_p99_ms": 500, "itl_p50_ms": 30},
 "motor": "vllm", "motor_version": "0.9.x",
 "concurrency_sweep": [8, 16, 32, 48, 64],
 "results_at_knee": {
 "concurrency": 32,
 "ttft_p50_ms": 210, "ttft_p99_ms": 420,
 "itl_p50_ms": 18, "throughput_tok_s": 12500,
 "goodput_tok_s": 12200
 }
}

This versioned JSON is the auditable datum: anyone can reproduce the figure with the same tool, version, model, hardware and load.

State of the art 2026: relative maturity

Dimension	vLLM	SGLang	TRT-LLM	Dynamo
Production maturity	High (3+ years)	Medium-high (2 years)	High (NVIDIA)	Medium (1.0 GA Mar 2026)
Release cadence	Weekly	Weekly	Monthly	Fortnightly
Integration ecosystem	Very broad (Ray, k8s, llm-d, Dynamo, NIM)	Broad (Dynamo, k8s)	NVIDIA-centric (Triton, NIM)	NVIDIA-centric
Ease of operation	High (pip install, OpenAI API)	High	Medium (requires compiling engines)	Medium-low (multi-node, etcd/NATS)
Non-NVIDIA hardware	Yes (AMD, Intel, TPU)	Yes (AMD)	No	No
Disaggregated serving	Beta	Stable	Stable	First class
Dynamic multi-LoRA	Stable	Stable	Beta	Stable (via vLLM/SGLang)
Structured output	Stable	Stable (FSM)	Beta	Stable (via the engine)
OSS community support	Very high (>30 K GitHub stars)	High (~20 K stars)	High (~10 K stars)	Growing (6.8 K stars)

Pareto decision table: when to choose each engine

No prose; the decision as a table of criteria and outcome.

Selection criterion	Recommended engine	Quantitative justification
Single node, quick prototype, team with no LLM serving experience	vLLM	pip install + OpenAI API in minutes; the broadest community support
Workload with long shared prefixes (RAG, few-shot, system prompts > 512 tok)	SGLang	RadixAttention: up to 6.4× more throughput vs a baseline without prefix caching
Maximum throughput on NVIDIA, no portability, team with TRT experience	TRT-LLM	Fused kernels + native FP8: ITL 9–12 ms vs 18–22 ms for vLLM on the same hardware
Multi-node (> 8 GPUs), mixed prefill-intensive/decode-intensive traffic	Dynamo + SGLang	Disaggregated P/D: up to 7× additional throughput on Blackwell; 2× TTFT with KV-aware routing
AMD ROCm or Intel Gaudi hardware	vLLM	The only SOTA engine with stable non-NVIDIA support
Structured output + high concurrency	SGLang	Compressed FSM: guided decode with no appreciable overhead
Stable dynamic multi-LoRA in production	vLLM or SGLang	TRT-LLM multi-LoRA still in beta (Jun 2026)
Hugging Face ecosystem, models < 13B	HF TGI v3	Direct HF Hub integration; competitive latency on small models
Maximum energy efficiency (J/token) on NVIDIA	TRT-LLM	Lower-overhead kernels → fewer J/token at iso-throughput

Quantisation: effect on throughput and quality

The four main options in order of decreasing speed and increasing quality:

Format	Relative throughput (H100)	Quality degradation (perplexity)	Engines with stable support
NVFP4 / MXFP4	~2.2× vs FP16	1–3 % on MMLU	TRT-LLM (Blackwell), vLLM (experimental)
FP8 (W8A8)	~1.7× vs FP16	< 1 % on MMLU	vLLM, SGLang, TRT-LLM
INT4 AWQ	~1.5× vs FP16	1–2 % on MMLU	vLLM, SGLang, TRT-LLM
GPTQ (INT4)	~1.4× vs FP16	1–3 % on MMLU	vLLM, SGLang
FP16 (baseline)	1.0×	0 %	All

FP8 is the de facto standard on Hopper for production: 1.7× throughput with degradation below 1 %. The article on quantisation in depth develops the quality/speed trade-off by format and model architecture.

Disaggregated serving: when the overhead is worth it

Separating prefill/decode into independent pools introduces transfer latency for KV tensors between nodes (NIXL over NVLink or InfiniBand). The overhead pays off when:

Condition	Effect of disaggregated serving
Mean ISL > 1,024 tokens	Prefill dominates; separate pools stop it blocking decode
Prefill/decode ratio > 3:1 in time	The decode pool sits idle waiting for prefill
Mixed traffic (prefill burst + decode queue)	Independent scaling of each pool by SLA
< 512 tokens ISL, unique prompts	Transfer overhead exceeds the benefit; an aggregated engine is enough

Dynamo quantifies the break-even point: disaggregated serving pays off when the transfer time for KV tensors (a function of ISL and NVLink/IB bandwidth) is lower than the time the decode worker would spend waiting for the prefill worker in aggregated mode. On 4×H100 with an ISL of 512 tok, break-even occurs at a prefill/decode ratio > 2.5:1. The article on disaggregated serving develops the full formula.

MLPerf Inference v5.1: cross-vendor reference figures

MLPerf Inference v5.1 (September 2025) is the standard for cross-vendor comparability. Server scenario (Llama-3.1-70B, tokens/s):

Submitter	Hardware	Scenario	Throughput (tok/s)
Red Hat (vLLM)	1×H100 80 GB	Server	5,103
Red Hat (vLLM)	1×L40S	Server	1,207
NVIDIA (TRT-LLM)	8×H100 SXM	Server	~52,000 (estimated)

The MLPerf figures are not directly comparable with the internal benchmarks above (a different harness, MLPerf’s Server scenario with a Poisson distribution and fixed SLOs). They are useful for comparing vendors under the same rules; not for sizing a specific case. For that, run your own sweep with AIPerf/GuideLLM against the target endpoint.

Variables the benchmark does not capture

Omitted variable	Effect in production
Network latency (gateway, load balancer)	Can add 20–100 ms to the TTFT measured on the server
Engine compilation time (TRT-LLM)	Hours per model×precision×hardware; penalises cold starts and CI
Weight loading time on the GPU	Relevant in autoscaling; ModelExpress (Dynamo) cuts it 7×
Operational cost of the system (Dynamo vs a simple engine)	Dynamo requires etcd/NATS, a Rust runtime, a more specialised team
Response quality	Performance does not imply quality; measure with lm-evaluation-harness (article B7)
Energy (J/token)	Does not come out of a throughput benchmark; requires DCGM running alongside

Sources

vLLM · documentación oficial — https://docs.vllm.ai/en/latest/
vLLM · blog de rendimiento (v0.6.0, 2,7× throughput) — https://blog.vllm.ai/2024/09/05/perf-update.html
vLLM · disaggregated prefill (experimental) — https://docs.vllm.ai/en/latest/features/disagg_prefill/
SGLang · documentación oficial — https://docs.sglang.io/
SGLang · paper RadixAttention (LMSYS, 2024) — https://arxiv.org/pdf/2312.07104
SGLang · blog LMSYS sobre Llama-3 serving — https://www.lmsys.org/blog/2024-07-25-sglang-llama3/
NVIDIA TensorRT-LLM · GitHub — https://github.com/NVIDIA/TensorRT-LLM
NVIDIA TensorRT-LLM · overview y release notes — https://nvidia.github.io/TensorRT-LLM/overview.html
NVIDIA TensorRT-LLM · quantización — https://nvidia.github.io/TensorRT-LLM/features/quantization.html
NVIDIA Dynamo · GitHub (ai-dynamo/dynamo) — https://github.com/ai-dynamo/dynamo
NVIDIA Dynamo · documentación oficial — https://docs.nvidia.com/dynamo/latest
NVIDIA Dynamo · disaggregated serving — https://docs.dynamo.nvidia.com/dynamo/design-docs/disaggregated-serving
Cerebrium · benchmark vLLM vs SGLang vs TRT-LLM (Llama 3.1-70B) — https://cerebrium.ai/blog/benchmarking-vllm-sglang-tensorrt-for-llama-3-1-api
Spheron · vLLM vs TRT-LLM vs SGLang H100 benchmarks 2026 — https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/
MLCommons · MLPerf Inference v5.1 results — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
Red Hat · MLPerf Inference v5.1 con vLLM — https://www.redhat.com/en/blog/efficient-and-reproducible-llm-inference-red-hat-mlperf-inference-v51-results
arXiv 2605.24217 · sesgo sistemático en benchmarks de inferencia LLM — https://arxiv.org/html/2605.24217
IETF Draft · LLM Benchmarking Methodology (enero 2026) — https://www.ietf.org/archive/id/draft-gaikwad-llm-benchmarking-methodology-00.html

Continuous batching: the salon with 8 chairs that does not wait for the slow customer — Orca, vLLM, chunked prefill and goodput

Sat, 30 May 2026 16:00:00 +0200

This post complements those on KV cache (the artefact continuous batching manages), PagedAttention (the memory piece that makes it viable), Disaggregated serving (the next layer of optimisation), Speculative decoding, Multi-LoRA and MoE (the three extensions that coexist with the scheduler in production).

You are here: DEPLOY

TL;DR

The static batching of the original HuggingFace Transformers (the pre-2022 era) systematically under-used the GPU for two structural reasons. First: the scheduling unit was the complete request; the longest one in the batch blocked all the others until it finished (severe head-of-line blocking, P95 TTFT five to ten times worse than reasonable). Second: each batch slot reserved memory for max_seq_len even though the real output was much shorter; documented padding waste sat between 60 % and 80 % and sustained GPU SM utilisation on real workloads fell to 20-40 %. Orca (Yu et al., OSDI 2022, FriendliAI + Seoul National University) introduced the idea that unjammed everything: the scheduling unit stops being the request and becomes one decoder iteration, a single token. After each iteration the scheduler can add new requests to the batch and remove the finished ones. vLLM (Kwon et al., SOSP 2023, UC Berkeley) materialised it open-source and production-grade thanks to PagedAttention, which solves the KV cache fragmentation that theoretical continuous batching would cause with dynamic allocation. SARATHI / Sarathi-Serve (Microsoft Research India, OSDI 2024) closed the last gap: the prefill stalls that paused active decodes when a new request arrived, via chunked prefill (splitting a long prefill into small chunks and mixing them with decodes in the same step) and stall-free batching. DistServe (Zhong et al., OSDI 2024) reformulated the key metric: what matters is goodput (requests/s meeting the TTFT and TPOT SLOs), not raw throughput. In May 2026, vLLM v1 brings a unified scheduler with chunked prefill always-on; SGLang adds RadixAttention, which gives cross-request prefix-cache hits; TensorRT-LLM calls it in-flight batching; llama.cpp supports it natively. The three operational tensions are speculative decoding (nested raggedness), multi-LoRA (each request with its adapter) and MoE (each expert sees very few tokens per step at a typical batch size). This post takes apart the mechanism, the maths (GPU utilisation, goodput vs throughput), the three variants (Orca → vLLM → Sarathi-Serve), the pitfalls (preempt-on-OOM, starvation, inverse HoL) and the real numbers with production configurations.

The analogy: the salon with 8 chairs

A hair salon with 8 chairs and a single brilliant stylist who moves between them. Customers arrive with very different needs: some want a 15-minute cut, others a 2-hour colour with a base, others a 90-minute straightening. The question is how to organise the flow.

The traditional strategy (what HuggingFace Transformers did in its original generate()) is to seat 8 customers at once, all at the same time, and accept nobody new until the last one finishes. If one of those 8 is a 2-hour job, the 7 who wanted the 15-minute cut sit there doing nothing for 1 hour and 45 minutes. The stylist finishes with the quick ones and is left staring at empty chairs until the colour is done. When everyone is ready, another 8 come in. That is static batching, and the only thing that stops it from being worse is that the GPU does not complain like a human customer.

The continuous strategy (Orca, vLLM) changes the scheduling unit. The stylist does not think “I am going to do one whole customer and then the next”; he thinks “on each tick I take one step of work at every occupied chair and, every time a chair frees up, I call the next customer in the queue without waiting for the others to finish”. The quick-cut customer leaves after 15 minutes, his chair is filled immediately with the next one, and the slow jobs proceed at their own pace without delaying anyone. The stylist is never idle.

The continuous strategy with chunked prefill (SARATHI / Sarathi-Serve) adds a subtler distinction. Some customers need a long initial phase (a 10-minute hair analysis before the cut; in LLM terms, the prefill of the prompt). Without chunked prefill, the stylist had to stop all the other chairs to do the new customer’s analysis in one go, and that was a stall visible in the TPOT of the active ones. With chunked prefill, the analysis is split into 2-minute pieces interleaved between the other customers’ active cuts. The customers already under way no longer notice pauses; the new customer takes slightly longer to start his cut proper, but the whole salon does not freeze.

And the metric that matters: the owner does not want to maximise “customers served per hour” at the cost of some leaving furious. He wants to maximise “customers served per hour within the time SLA”, and that is goodput, DistServe’s contribution.

The problem continuous batching solves

There are two structural pathologies of static batching that deserve to be explained with concrete numbers.

Padding waste. Each batch slot reserved memory for max_seq_len (prompt + max output), even though the real output ended in far fewer tokens. For a batch of 32 with heterogeneously distributed output lengths (half of them ≤50 tokens, a long tail up to 4,000), the typical memory waste was 60-80 %. That translated directly into wasted concurrency: with the same VRAM, instead of serving 32 requests with smart allocation, you served 8.

HoL blocking (Head-of-Line). The scheduling unit was the complete request. A batch containing one request of 500 tokens and 31 of ≤50 tokens ran 450 extra “empty” iterations (the GPU executed forward passes for all 32 slots, even though 31 had already finished). Wasted compute cost: ~84 % of the tail time in the example.

Measurable result. Sustained GPU SM utilisation on real workloads under static batching: 20-40 %. That is, ~70 % of the datacenter’s compute went unused. When the first Orca and vLLM benchmarks came out showing 10-24× throughput improvement, it was not marketing exaggeration; it was recovering all that wasted compute.

Orca (OSDI ‘22): the idea that changed everything

The paper by Yu, Jeong, Kim, Kim and Chun at OSDI 2022 (“Orca: A Distributed Serving System for Transformer-Based Generative Models”, from Seoul National University + FriendliAI) introduced two contributions that have remained the basis of everything that followed.

Iteration-level scheduling. Instead of scheduling at the level of the complete request, it schedules at the level of one decoder iteration: the step that generates ONE token. After each iteration the scheduler can (a) add new requests to the batch, (b) remove requests that have generated EOS or reached max_tokens, (c) reorder priorities. The compute engine executes exactly one iteration over the current batch.

Selective batching. Here is the non-obvious subtlety. The technical problem with batching requests of different lengths and different KV cache states is that some operations (the GEMMs of the Q, K, V projections and the FFN) are insensitive to position and can be batched by concatenating tokens, whereas attention is sensitive to per-request state (each request has its own KV cache of a different length). Orca’s solution: batch the GEMMs (concatenate all the step’s tokens into a [total_tokens, hidden] tensor) and run attention sequentially per request.

Paper result: up to 36.9× throughput vs FasterTransformer on GPT-3 175B at the same latency level. Orca is not open-source, it is only documented in the paper. FriendliAI commercialises it as the Friendli Engine. But the idea was published and everyone adopted it.

vLLM (SOSP ‘23): the open-source materialisation

What Orca described as a concept, vLLM materialised in production. The paper by Kwon, Li, Zhuang, Sheng et al. (UC Berkeley Sky Computing Lab, SOSP 2023) introduces PagedAttention, covered in detail in the PagedAttention deep dive, but it also consolidates continuous batching as universal practice.

The reason PagedAttention is a prerequisite for practical continuous batching is fragmentation. If you are going to insert and remove requests from the batch dynamically, and each request has a KV cache that grows on every iteration, traditional contiguous allocation fragments the HBM until it is unusable. PagedAttention splits the KV cache into fixed-size blocks (16 tokens by default) allocated on demand from a global pool. Memory waste falls from ~60-80 % to less than 4 % (only the last partially filled block per sequence).

vLLM paper metrics (2023):

vs HuggingFace Transformers (static, no continuous batching): up to 24× throughput.
vs HuggingFace TGI (which already had primitive continuous batching): ~3.5×.
vs FasterTransformer: 2-4× at the same latency.

What operationally distinguishes vLLM from Orca: vLLM runs attention in a single fused CUDA kernel (paged_attention_kernel) over non-contiguous blocks; Orca described attention as sequential, request by request. And vLLM exposes OpenAI-compatible APIs that let you drop the engine into existing stacks without touching the client.

Chunked prefill (SARATHI / Sarathi-Serve, OSDI ‘24)

There is a detail the original Orca/vLLM continuous batching did not solve: when a new request enters the batch, its prefill (processing the whole prompt at once) can take hundreds of milliseconds. During that time, the active decodes of the other requests are essentially paused, the GPU being dedicated to the new prefill. This showed up as spikes in TPOT (“inter-token latency”) every time a long request came in, which broke strict SLAs.

SARATHI (Agrawal et al., arXiv 2308.16369, August 2023) and later Sarathi-Serve (same group at Microsoft Research India, OSDI 2024, arXiv 2403.02310) introduce two combined ideas:

Chunked prefill. A long prefill (8,192 tokens, say) is split into chunks (2,048 tokens, say) processed one per iteration. Instead of one 200 ms step processing 8K tokens, four 50 ms steps processing 2K each.

Decode-maximal batching (“stall-free”). On each iteration, the scheduler first fills the batch with the active decodes (each costing 1 token), and only the leftover space goes to new prefill chunks. The result: active decodes keep advancing 1 token per iteration without pausing, while the new request completes its prefill in small bites.

The observation that justifies it: prefill is compute-bound (it processes N tokens at once and saturates FLOPs) while decode is memory-bound (1 token per step, under-using compute, with the GPU waiting on HBM). Mixing prefill chunks with decodes in the same step exploits the arithmetic-intensity slack: the decodes piggyback on the free compute of the prefill chunk.

Numbers:

Original SARATHI (LLaMA-13B on an A6000): decode throughput +10×, end-to-end +1.33×.
Sarathi-Serve (Mistral-7B on an A100): 2.6× serving capacity vs plain vLLM. Yi-34B on 2×A100: 3.7×. Falcon-180B with pipeline parallel: 5.6×.

Adoption in May 2026: always-on in vLLM v1 (default since v0.8.0, January 2025), SGLang, TensorRT-LLM. The key setting in vLLM is --max-num-batched-tokens (the token budget per step; default 2048). Raising it prioritises throughput, lowering it prioritises low TPOT.

Goodput: the metric that matters (DistServe, OSDI ‘24)

The classic “throughput” metric (requests/s or tokens/s) has a problem when there are SLOs. A server can report 1,000 req/s while P99 TTFT is 30 seconds and the SLO is 1 second, so only about 200 req/s actually meet the contract.

DistServe (Zhong et al., OSDI 2024) formalises goodput as the right metric: goodput = max sustained request rate meeting the SLOs (TTFT bound AND TPOT bound). The practical definition is usually: maximum rate with ≥90 % of requests within both SLOs.

Why it matters for the scheduler:

Optimising raw throughput leads to maximising batch size, which inflates P99 TPOT.
Optimising goodput limits the batch size when TPOT starts violating the SLO, prefers small requests if the batch already has a tail, and leaves resources available for new requests.

DistServe result: up to 7.4× more requests served or a 12.6× tighter SLO vs vLLM at the same SLO attainment. The gain comes from disaggregating prefill and decode onto different GPUs (removing the interference between phases), but the idea of optimising for goodput is independent and applicable to any scheduler.

Operationally this translates into monitoring:

goodput_proxy = histogram_quantile(0.95, vllm:time_to_first_token_seconds_bucket) < SLO_TTFT
AND histogram_quantile(0.95, vllm:time_per_output_token_seconds_bucket) < SLO_TPOT

The iterative scheduler in action

The maths that matter

Three formulas explain much of the operational behaviour.

GPU utilisation under static batching. With a batch of size B whose seq_len_i are the real lengths and max(seq_len_i) is the length that defines the padding:

$$U_{\text{static}} = \frac{\sum_i \text{seq len}_i}{B \cdot \max_i \text{seq len}_i}$$

For B=32, 30 sequences of 50 tokens and 2 of 500: U = (30·50 + 2·500) / (32·500) = 2500/16000 = 15.6 %. Four out of every five GPU cycles wasted.

GPU utilisation under continuous batching (idealised).

$$U_{\text{continuous}} \approx 1 - \frac{T_{\text{scheduler}}}{T_{\text{iteration}}}$$

With scheduler overhead of ~50-200 µs and an iteration time of ~10-30 ms: U > 95 %. The unit of loss is no longer padding, it is scheduling overhead, and that is negligible compared with the forward pass.

Goodput vs throughput.

$$\text{Goodput}(R) = R \cdot P(\text{latency} < \text{SLO})$$

where R is the offered request rate. Typical curve: goodput grows linearly with R up to the saturation knee, then falls because P(SLO) collapses when the system congests. The optimal point is just before the knee, not at peak throughput.

Example: at R=100 req/s with P(SLO)=0.99, goodput = 99. At R=200 req/s with P(SLO)=0.4, goodput = 80. More offered load, less useful goodput.

Chunked prefill token budget. On each vLLM step with chunked prefill active:

$$\text{prefill tokens this step} = \text{max num batched tokens} - \text{num decodes active}$$

Each active decode costs 1 token of the budget; the rest is filled with new prefill chunks. If max_num_batched_tokens = 2048 and num_decodes_active = 200, there are 1,848 tokens for prefill (one chunk of 1,848 or several small chunks).

Real implementations in May 2026

Engine	Current scheduler V	Chunked prefill default	Relevant notes
vLLM v1 (default ≥0.8.0)	unified V1	always-on	EngineCore isolated in a separate process; prefix caching with O(1) eviction; preempt-mode `recompute` by default; xgrammar/outlines backends.
SGLang	own (PyTorch ecosystem)	yes	RadixAttention gives cross-request prefix-cache hits; non-blocking CPU scheduler; leader in stable latency at high concurrency.
TensorRT-LLM	proprietary “in-flight batching”	yes	Policies `GUARANTEED_NO_EVICT` (conservative, default) and `MAX_UTILIZATION` (aggressive, risk of a pause when the KV is full). Compile-time vs runtime.
Triton + tensorrtllm_backend	`gpt_model_type: inflight_fused_batching`	yes	`max_queue_delay_microseconds` to group newly arrived requests. Decoupled mode for SSE streaming.
llama.cpp (llama-server)	own	`--cont-batching` ON since 2024	`-np N` parallel slots; no PagedAttention (contiguous KV per slot), so less flexible but simpler. Endpoint `:8080/metrics`.

A typical production-ready vLLM v1 configuration:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
 --tensor-parallel-size 4 \
 --max-num-batched-tokens 4096 \
 --max-num-seqs 256 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --preemption-mode recompute \
 --scheduling-policy fcfs \
 --gpu-memory-utilization 0.92

SGLang equivalent:

python -m sglang.launch_server \
 --model meta-llama/Llama-3.1-70B-Instruct \
 --tp 4 \
 --chunked-prefill-size 4096 \
 --max-running-requests 256 \
 --enable-radix-cache

The three operational tensions

Continuous batching + speculative decoding. Speculative decoding produces 1 to γ+1 tokens per step depending on the acceptance rate. The batch stops being uniform in tokens produced per iteration: nested raggedness. PagedAttention absorbs it (the KV cache can grow at different rates per request in the same step), but the scheduler loses symmetry. At low QPS (a conversational assistant) the combination is excellent: vLLM reports up to 2.8× speedup. At high QPS, the draft consumes slots from the decode pool and can reduce aggregate goodput. Rule of thumb: disable speculative decoding when gpu_cache_usage > 0.85. Full detail in Speculative decoding.

Continuous batching + multi-LoRA. Each request in the batch can use a different adapter (via SGMV, see Multi-LoRA serving). Worst case: every request in the batch with a different adapter and a different rank, and throughput falls by up to 50 % vs the base with no LoRA. Best case: all requests on the same adapter, equivalent to the base with no LoRA. Practical mitigation: group adapters by rank in the routing ahead of the engine; set --max-lora-rank to the maximum actually served, not generously above it.

Continuous batching + MoE. Each expert sees batch · k / N tokens per step. With DeepSeek-V3 (256 experts, k=8) and batch=32 in decode, each expert processes only 1 token on average: total compute starvation. To match the per-GPU throughput of a dense model, MoE needs batches »10× larger, which puts pressure on the KV cache. Wide-EP (see MoE inference) distributes the experts across many GPUs and allows larger effective batches per expert, at the cost of all-to-all comms that add milliseconds per step.

Metrics to monitor

The Prometheus metrics exposed by vLLM (prefix vllm:, vllm_ when scraped):

vllm:time_to_first_token_seconds (Histogram) — TTFT including queue time.
vllm:time_per_output_token_seconds (Histogram) — TPOT.
vllm:e2e_request_latency_seconds (Histogram) — end-to-end.
vllm:num_requests_running (Gauge) — active batch.
vllm:num_requests_waiting (Gauge) — queue depth.
vllm:num_requests_swapped (Gauge) — preempted to CPU.
vllm:gpu_cache_usage_perc (Gauge) — fraction of KV cache occupied.
vllm:gpu_prefix_cache_hit_rate (Gauge) — prefix cache hits.
vllm:num_preemptions_total (Counter) — preemptions. Any sustained value is a red flag.

Practical operational rules:

Stable zone under sustained load: gpu_cache_usage_perc ∈ [0.7, 0.9].
Warning at >0.95 (preemption imminent).
Critical if num_requests_waiting grows faster than num_requests_running: the server is not absorbing the load; scale out.

Operational pitfalls

Preempt-on-OOM. When gpu_cache_usage reaches ~1.0 with pending requests that need to grow their KV, vLLM preempts. V1 does RECOMPUTE by default (discards the KV, regenerates it on return); V0 did SWAP (moves it to CPU). RECOMPUTE is better for short sequences (cheap to regenerate); SWAP is better for long ones. The vllm:num_preemptions_total metric should be zero or near zero in steady state.

Inverse HoL blocking (memory monopoly). A very long request occupies many KV blocks, so small requests do not fit in the batch even though compute is free. Chunked prefill mitigates the compute blocking during a new prefill, but it does not solve the memory monopoly. Partial solution: per-request limit policies (an aggressive max_tokens) or priorities.

Starvation. FCFS can leave requests pending for a long time if the active ones do not finish. vLLM supports --scheduling-policy priority with an x-priority header. Recent work (NeurIPS 2024 Efficient LLM Scheduling by Learning to Rank, arXiv:2501.14312 Locality-aware Fair Scheduling) proposes schedulers with quantum-based starvation prevention; not yet integrated into vLLM mainline.

Badly calibrated chunk size. A small chunk (512) gives low TPOT, high TTFT and memory overhead from more KV accesses. A large chunk (8192+) gives low TTFT and TPOT spikes during the chunk. Rule: start at 2,048, measure P95 TPOT, adjust.

Batch size cap. A high --max-num-seqs gives more concurrency but P99 TPOT explodes. A low one wastes throughput. Rule of thumb: max_num_seqs ≈ HBM_for_KV / (avg_seq_len × bytes_per_token_KV).

Implications for on-premise hardware

On an RTX 4090 (24 GB). llama.cpp with --cont-batching -np 4-8 is the natural pattern. Typical models: Llama 3 8B Q4_K_M with ~8 parallel slots, aggregate throughput on the order of hundreds of tok/s. vLLM also works if the weights fit (Llama 3 8B BF16 does; the 70B does not fit whole), although PagedAttention on consumer hardware gives less return than in the datacenter.

On a generic 4×H100 SXM cluster (320 GB, NVLink). Here vLLM v1 / SGLang are the de facto standard. Typical configurations:

Llama 3 70B FP8 + TP=4: dozens of concurrent sessions with P95 TPOT under 50 ms, tens of thousands of aggregate tok/s at a moderate batch size.
Llama 3 70B AWQ-INT4 + TP=2 plus the rest of the cluster for additional concurrency or multi-LoRA with SGMV.
DeepSeek-V3 needs larger setups (8-16 H100) to fit whole in FP8; with Wide-EP, continuous batching starts operating on much larger batches and the economics change (see MoE).

The rule of thumb for May 2026: vLLM v1 with chunked prefill always-on and prefix caching enabled is the sensible default configuration for any dense model that fits comfortably; SGLang offers better stable latency at high concurrency thanks to the overlap between the CPU scheduler and the GPU step; TensorRT-LLM gives peak throughput at high concurrency with the rigidity of compile-time.

What we have not covered

Locality-aware fair scheduling (arXiv:2501.14312) and learning-to-rank schedulers (NeurIPS 2024): the next generation of algorithms closing the fairness vs prefix-cache locality trade-off.
Smooth goodput (arXiv:2410.14257): a refinement of the DistServe metric using max slowdown instead of a binary SLO.
Triton tensorrtllm_backend in production: decoupled mode for streaming, ensembles with pre/post-processing, autoscaling with KServe.
vLLM speculators v0.3.0 and the vLLM-compatible drafter training framework.

References

Yu, G.-I., Jeong, J., Kim, G.-W., Kim, S., Chun, B.-G. Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022. https://www.usenix.org/system/files/osdi22-yu.pdf
Kwon, W. et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. https://arxiv.org/abs/2309.06180
Agrawal, A. et al. SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. 2023. https://arxiv.org/abs/2308.16369
Agrawal, A. et al. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. OSDI 2024. https://arxiv.org/abs/2403.02310
Zhong, Y. et al. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024. https://arxiv.org/abs/2401.09670
Sheng, Y. et al. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. MLSys 2024. https://arxiv.org/abs/2311.03285
Efficient LLM Scheduling by Learning to Rank. NeurIPS 2024.
Locality-aware Fair Scheduling. 2025. https://arxiv.org/abs/2501.14312
vLLM V1 alpha release (Jan 2025): https://blog.vllm.ai/2025/01/27/v1-alpha-release.html
vLLM Anatomy (Sep 2025): https://blog.vllm.ai/2025/09/05/anatomy-of-vllm.html
vLLM Large-Scale Serving (Dec 2025): https://blog.vllm.ai/2025/12/17/large-scale-serving.html
vLLM speculators v0.3.0 (Dec 2025): https://blog.vllm.ai/2025/12/13/speculators-v030.html
Anyscale, Continuous Batching for LLM Inference (2023): https://www.anyscale.com/blog/continuous-batching-llm-inference
vLLM metrics docs: https://docs.vllm.ai/en/latest/design/metrics/
vLLM optimization: https://docs.vllm.ai/en/stable/configuration/optimization/
SGLang: https://github.com/sgl-project/sglang
TensorRT-LLM performance tuning: https://nvidia.github.io/TensorRT-LLM/performance/performance-tuning-guide/tuning-max-batch-size-and-max-num-tokens.html
Red Hat, 5 steps to triage vLLM performance (Mar 2026): https://developers.redhat.com/articles/2026/03/09/5-steps-triage-vllm-performance
NVIDIA blog, Chunked Prefill with TensorRT-LLM: https://developer.nvidia.com/blog/streamlining-ai-inference-performance-and-deployment-with-nvidia-tensorrt-llm-chunked-prefill/

Multi-LoRA serving: the single translator with a thousand glossaries — shared base, thousands of concurrent adapters and the SGMV kernel

Sat, 30 May 2026 14:00:00 +0200

This post complements Continuous fine-tuning in production. Continuous fine-tuning is the producer of the adapters; multi-LoRA serving is the consumer that puts them to work. Without this layer, the whole feedback cycle breaks in the last mile. It also crosses over with Modern alignment (DPO/KTO/ORPO/SimPO) (each alignment policy can live as a separate adapter) and Quantisation (a quantised base frees memory for many more adapters).

You are here: DEPLOY

TL;DR

The dominant pattern in 2026 is not one model per customer but a single general-purpose base model plus N fine LoRA adapters per task, customer, language or domain. The reason is obvious: a rank-16 LoRA over Llama-3-70B takes about 400 MB; a full fine-tune takes about 140 GB. Dozens or hundreds of adapters per base is manageable; dozens or hundreds of bases is prohibitive. What is not obvious is how to serve them concurrently without reloading weights every time the adapter changes (killing batching) or replicating the base (killing memory). The answer crystallised in 2024 with two complementary papers: S-LoRA (Sheng et al., Stanford + UC Berkeley, MLSys 2024) introduced unified paging, where adapter weights live in the same memory pool as the KV cache, both pageable, and heterogeneous batching, where a batch can hold requests with different adapters and different ranks without padding; Punica (Chen et al., UW + Duke, MLSys 2024) introduced the CUDA kernel that has become the de facto standard: SGMV (Segmented Gather Matrix-Vector multiplication), which computes Y += Σ_i X_i · A_i · B_i in a single pass, grouping requests by adapter. SGMV sits today underneath vLLM, LoRAX (Predibase), SGLang and TGI. The measurable operational result: up to 2,000 concurrent adapters on a single GPU (S-LoRA paper), up to 4× throughput vs naive vLLM and up to 30× vs HuggingFace PEFT. The price: typical overhead of 10-30 % latency per layer with an active adapter in a heterogeneous batch, practically zero when every request in the batch uses the same adapter, and 20-40 % in the worst case. This post takes apart the mechanism, the maths (memory per adapter, overhead per rank), the comparison table of implementations, the pitfalls (cold start, mismatched rank, fragmentation) and the real economics on an H100 with a Llama-3-70B FP8 base + 200 adapters.

The analogy: the single translator with a thousand glossaries

Imagine a specialised translation agency with a single senior translator, brilliant, fluent in fifteen languages and across all the general technical domains. That translator is expensive to hire and expensive to train: it took years of formation and experience that is not easily replicated. But the agency receives texts from very different clients: a law firm using legal terminology specific to its jurisdiction, a manufacturer with an internal parts nomenclature, a hospital with its own clinical abbreviations. Every client has its own jargon.

The agency does not hire one translator per client; that would be ridiculous, 90 % of the work is shared. What it does is keep one glossary per client: a small notebook, easy to update, containing the specific terms and how they are translated for that client. When the translator receives a text, they open the glossary for the client in question and work with it alongside. As they translate each word, they first check whether it is in the glossary; if it is, they use the specific version; if not, they use their general knowledge.

The glossaries live on a shared shelf, ordered by recent use: the most consulted ones within reach, the old ones in the archive. When a new client arrives, their glossary is brought from the archive to the shelf. When the desk fills up, the least used glossary goes back to the archive.

And the most important part: the translator can have several glossaries open at once because they are working in parallel on five texts from five clients. It is not one glossary per document; it is one glossary per client, and client A’s documents use A’s glossary, client B’s use B’s, all on the same desk.

The analogy holds on five mappings:

The single translator = the base model (Llama-3-70B, Qwen2.5-72B). Expensive to train, a single copy in VRAM.
Each glossary = a LoRA adapter. Small (~150-400 MB), specific, easy to update.
The shelf with the glossaries within reach = the pool of adapters cached in VRAM (typically 50-200 at a time with an FP8 base on an H100 80 GB).
The archive = adapter storage on MinIO/S3/HF Hub. Hundreds or thousands, fetched on demand.
Working in parallel with several glossaries open = a heterogeneous batch with SGMV. The kernel does the grouped lookup into the right glossary for each word in the batch.

The bare mechanism: what a LoRA does and why it can be served multi-tenant

A LoRA adapter modifies a matrix W of the base model by adding a low-rank product to it:

$$W' = W + B A$$

where W ∈ R^{d_out × d_in} is the original matrix (the base weights), A ∈ R^{r × d_in} and B ∈ R^{d_out × r} are the adapter’s trainable matrices, and r is the rank (typically 8, 16, 32 or 64, always much smaller than d_in and d_out).

In a forward pass, instead of computing y = W' x, it computes:

$$y = W x + B(Ax)$$

That is: the base computation (Wx) happens exactly as before; the adapter adds two cheap matmuls (Ax and then B(·)) that supply the correction. The matrix BA is never explicitly materialised.

What this enables in serving: if you have the base loaded and N different adapters, Wx is computed only once for all the tokens in the batch (the base is the same). What changes between tokens is only the delta B_i(A_i x). If the tokens in the batch use different adapters, different deltas have to be applied per token, and that is what the SGMV kernel does in a single pass.

Without a specialised kernel this degenerates: you need to launch N separate matmuls (one per adapter), pay kernel launch overhead N times and lose the batching. With a specialised kernel (SGMV), every delta is computed in a single pass grouped by adapter.

SGMV: the kernel that holds it all up

SGMV (Segmented Gather Matrix-Vector multiplication) is the CUDA kernel Punica introduced and that vLLM, LoRAX, SGLang and TGI have adopted as their multi-LoRA engine.

Its job is to compute, given a batch of tokens with mixed adapters:

$$y_t = W x_t + B_{a(t)} A_{a(t)} x_t \quad \forall t \in \text{batch}$$

where a(t) is the adapter assigned to token t. SGMV operates in two phases:

SGMV-shrink: projection d_in → r with the corresponding A_{a(t)} matrix.
SGMV-expand: projection r → d_out with the corresponding B_{a(t)} matrix.

Internally, SGMV groups the batch’s tokens by adapter (it segments them), and for each segment it uses the optimal kernel for the size: for large segments (several requests with the same adapter) it goes through tensor cores; for small segments (one request per adapter) it uses the batched gather path that minimises launch overhead.

The result, in a single kernel pass, is the correct delta for every token in the batch, whatever its rank or its adapter. Punica reported up to 12× throughput vs vLLM/FasterTransformer/HF Transformers/DeepSpeed in heterogeneous multi-tenant scenarios; when every request uses the same adapter, SGMV is practically equivalent to the base without LoRA because it reduces to the “single large segment” case that is optimal for tensor cores.

S-LoRA refined SGMV with two kernels specific to different serving phases: MBGMM (Multi-size Batched Gather Matrix-Matrix) for prefill, MBGMV for decode. Both support different ranks among requests in the same batch, which was a limitation in the original SGMV.

The maths that matters

Three numbers drive every operational decision with multi-LoRA.

Memory per adapter. For a d_in × d_out matrix with rank r and b bytes per parameter (BF16/FP16 = 2):

$$\text{bytes per matrix} = (d_{\text{in}} \cdot r + r \cdot d_{\text{out}}) \cdot b$$

Summing over all the target matrices in each layer and multiplying by the number of layers gives the adapter size. A concrete calculation for Llama-3-70B, rank 16, BF16, all the matrices (Q, K, V, O, gate, up, down):

Matrix	Dimension	Bytes
Q (8192→8192)	8192·16 + 16·8192	524,288
O (8192→8192)	8192·16 + 16·8192	524,288
K (8192→1024)	8192·16 + 16·1024	294,912
V (8192→1024)	8192·16 + 16·1024	294,912
gate (8192→28,672)	8192·16 + 16·28,672	1,179,648
up (8192→28,672)	8192·16 + 16·28,672	1,179,648
down (28,672→8192)	28,672·16 + 16·8192	1,179,648
Sum per layer		5,177,344 ≈ 4.94 MB
Adapter total (80 layers)		~395 MB

If it is limited to attention-only (Q, O, V, K): about 125 MB per adapter. The choice of which matrices get LoRA belongs to whoever trains it; in serving it is inherited and it determines the cost.

Memory per rank. Linear: rank 8 → ~200 MB; rank 16 → ~400 MB; rank 32 → ~800 MB; rank 64 → ~1.6 GB. The simple sizing rule is: max_lora_rank should be the maximum rank you are going to serve, no more. Setting it higher wastes reserved memory in every slot.

How many adapters fit. For an H100 SXM 80 GB with a Llama-3-70B FP8 base (~70 GB), about 10 GB are left after a minimal KV cache → around 25 fully targeted r=16 adapters or around 80 attention-only. With an AWQ INT4 base (~35 GB), about 45 GB are left → hundreds of adapters. The rule: quantising the base does not just free memory, it multiplies the economics of the platform.

Latency overhead per adapter. Under real reported conditions:

Case	Typical overhead
All requests in the batch on the same adapter	~0 % (equivalent to a static merge)
Heterogeneous batch, similar ranks (e.g. all r=16)	10-30 % per layer
Heterogeneous batch, mismatched ranks (r=8 with r=128)	up to +84 % P95 TTFT for the lower rank
Naive PEFT LoRA (no SGMV)	250-950 % extra

It scales linearly with rank: rank 8 ≈ baseline; rank 64 ≈ 3-4 × overhead.

The real implementations in May 2026

Implementation	Base kernel	Hot-swap	Quantised base + LoRA	Operational notes
vLLM	SGMV + ext	Yes (LoRAResolver, S3/HF/FS plugins)	AWQ/GPTQ yes, bnb 4-bit offline only	The de facto default. `--enable-lora --max-loras N --max-lora-rank R --max-cpu-loras M`
LoRAX (Predibase)	Optimised SGMV	Yes (dynamic loading)	Yes	Designed specifically for multi-LoRA. Supports Medusa adapters per adapter (spec-dec per adapter).
SGLang	SGMV / csgmv	Yes	Yes	`--enable-lora-overlap-loading` cuts TTFT by up to 78 % on LoRA-heavy workloads
TensorRT-LLM	LoRA Executor (C++)	Pre-compile at build time	INT4 + LoRA common	Peak throughput on H100/B200, less flexible than vLLM
HF TGI	Punica fork	Yes (`LORA_ADAPTERS=...`)	Yes	In maintenance mode as of May 2026; HF recommends vLLM or SGLang
NVIDIA NIM	TRT-LLM under the hood	Static or dynamic (`NIM_PEFT_REFRESH_INTERVAL`)	Yes	Adapter store per model; polling for hot-add/remove

Three operational observations:

vLLM dominates open-source serving in 2026 thanks to the combination of a mature SGMV + LoRAResolver plugins + support for a quantised base. The critical parameter is --max-lora-rank: many installations set it to the maximum “just in case” and waste memory silently.
LoRAX wins in production operations with thousands of rarely used adapters thanks to its dynamic loading that does not block concurrent requests. A public case: Convirza with 60+ concurrent adapters and sub-2s P95.
SGLang wins on latency when cold starts are frequent thanks to --enable-lora-overlap-loading (async H2D during the compute of the previous request).

Combined pattern with quantisation and disaggregated serving

With quantisation (Quantisation). The canonical stack in May 2026 is base in FP8 (Hopper/Blackwell) or INT4 AWQ + adapters in BF16/FP16. The adapters are not quantised: they are small, the memory saved is irrelevant, and quantisation noise would accumulate badly with the delta. Quantising the base frees massive amounts of memory for more adapters with no significant loss (<1 % in typical MMLU with AWQ INT4, somewhat more in math/code/reasoning). An adapter trained with a BF16 base works with an FP8/INT4 base at inference with marginal loss, and that is what makes QLoRA operationally trivial: train with a 4-bit base and deploy with a consistent 4-bit base.

With disaggregated serving (Disaggregated serving). Multi-LoRA + prefill/decode disaggregation adds a management layer: each pod needs access to the request’s active adapter. The 2026 strategy: replicate hot adapters across every pod (prefill and decode), evict the cold ones. Cold adapters not present in the target pod are transferred on demand, accepting the extra cost in TTFT. Recent work (InfiniLoRA, FASTLIBRA, LoRAServe) automates this balancing, but the simple rule of thumb works in most deployments.

Operational pitfalls

Cold start. The first request to a sleeping adapter involves a fetch (from S3/MinIO/HF Hub) → CPU load → H2D copy. For an adapter of around 400 MB: 0.5-5 s typically, depending on bandwidth. Under high concurrency it can be up to 35 % of E2E latency (Predictive-LoRA paper, arXiv:2512.20210). Established mitigations: SGLang --enable-lora-overlap-loading (cuts TTFT by 35-78 %); vLLM pre-warming with a dummy request when an adapter is registered; predictive prefetching based on patterns (Predictive-LoRA cuts cold start by 68 %).

Heterogeneous worst case. If every request in the batch has a different adapter and a different rank, SGMV loses its advantage because each segment has size 1. Throughput can fall by up to 50 % vs the base without LoRA. Practical mitigation: group adapters by rank in the upstream routing, trying to get requests of the same rank into the same batch step.

Mismatched rank. Co-batching rank 8 with rank 128 penalises the smaller one: +84 % P95 TTFT for the rank-8 requests (Serving Heterogeneous LoRA Adapters paper, arXiv:2511.22880). Practice: normalise the rank across the fleet whenever possible (train every adapter at the same rank, or at least within a narrow range).

Eviction. LRU is the default. If you have more adapters than fit in CPU RAM, the evicted ones get fetched again from cold storage. Monitoring cold_starts_per_minute and cache_hit_ratio per endpoint is basic hygiene.

Base versioning. Every adapter is tied to an exact base version (Llama-3-70B-Instruct ≠ Llama-3.1-70B-Instruct). The routing must validate the base+adapter pair before serving, or the output will be silent rubbish.

Memory fragmentation. Without paged management (the pre-S-LoRA / pre-vLLM case), evicting and inserting adapters fragments HBM until it becomes unusable. Unified Paging solves it: the LoRA weights and the KV cache live in the same pool of blocks, interchangeable.

Implications on on-premise hardware

On an RTX 4090 (24 GB). The classic case: base Llama-3-8B FP8 (~8 GB) or Llama-3-8B AWQ-INT4 (~5 GB) + dozens of r=16 adapters (~80 MB each, around 3 MB per adapter for Llama-3-8B with r=16 attention-only). 50-100 active adapters fit easily on the 4090. It is the natural setup for multi-tenant demos, fine-tuning over an 8B base and platform prototypes.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). This is where the serious setup comes in:

Llama-3-70B FP8 (~70 GB) fits on 2 H100s; the other 2 GPUs are available for batch + adapters.
Llama-3-70B AWQ-INT4 (~35 GB) fits on 1 H100; the rest of the cluster serves more concurrency or more adapters.
~200 fully targeted r=16 adapters fit with a comfortable budget on the cluster, enough for a SaaS platform with dozens of tenants and simultaneous A/B testing.
QLoRA training + consistent serving: the train→deploy adapter cycle takes hours, not days, because the adapter is around 400 MB instead of around 140 GB.

The rule of thumb on an H100 cluster in May 2026: an FP8 or INT4 base + 100-500 adapters per cluster is operationally trivial with vLLM or LoRAX; going past a thousand concurrent adapters starts to require serious tuning of eviction and prefetch.

Typical production stack

[API Gateway]
↓ (JWT with tenant_id / API key)
[Router]
↓ injects adapter_id into the request
[vLLM / LoRAX with --enable-lora]
--max-loras 16
--max-lora-rank 32
--max-cpu-loras 200
+ LoRAResolver → s3://adapters/{tenant_id}/{version}/
+ Base: Llama-3-70B-FP8 loaded once on 4×H100 TP=4
↓
[GPU]
FP8 base (70 GB) + ~150 BF16 adapters hot in HBM + ~1,000 warm in RAM
[MinIO / S3]
Cold storage for thousands of adapters
[CI pipeline]
→ trains new QLoRA adapter → push MinIO → notify server → warm-up
[Observability]
Prom: active_adapters, cache_hit_ratio, cold_starts_per_minute,
per_adapter_throughput, P99_with_lora vs P99_base

Routing pattern: Authorization: Bearer <key> → middleware extracts tenant_id → maps to adapter_id (Postgres or Redis) → POST /v1/completions with model: "<adapter_id>".

What we have not covered

DoRA (Weight-Decomposed LoRA): decomposes the update into magnitude + direction, closing part of the quality gap with full fine-tuning. Supported by TensorRT-LLM and others, but the serving pattern is identical to LoRA.
MoE + LoRA: how adapter fine-tuning is done over an MoE, what happens to the routing. Not trivial, an active research area in 2026.
Activated LoRA (arXiv:2512.17910): a variant that reuses the KV cache between compatible adapters, cutting the cost of cold start with shared prefixes.
LoRA for speculative decoding: each adapter brings its own Medusa head, supported by LoRAX as “Turbo LoRA”.
Compress-then-Serve (arXiv:2407.00066): quantising the adapters themselves to serve even more concurrently. Still a marginal practice in production as of May 2026.

References

Hu, E. et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. https://arxiv.org/abs/2106.09685
Sheng, Y. et al. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. MLSys 2024. https://arxiv.org/abs/2311.03285
Chen, L. et al. Punica: Multi-Tenant LoRA Serving. MLSys 2024. https://arxiv.org/abs/2310.18547
Dettmers, T. et al. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. https://arxiv.org/abs/2305.14314
Brüel-Gabrielsson, R. et al. Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead. 2024. https://arxiv.org/abs/2407.00066
Serving Heterogeneous LoRA Adapters in Distributed LLM Inference. 2025. https://arxiv.org/abs/2511.22880
InfiniLoRA: Disaggregated Multi-LoRA Serving. 2026. https://arxiv.org/abs/2604.07173
Predictive-LoRA: Proactive Fragmentation-Aware Serverless LoRA Serving. https://arxiv.org/abs/2512.20210
S-LoRA repo: https://github.com/S-LoRA/S-LoRA
Punica repo: https://github.com/punica-ai/punica
LoRAX repo (Predibase): https://github.com/predibase/lorax
vLLM LoRA docs: https://docs.vllm.ai/en/stable/features/lora/
SGLang LoRA docs: https://docs.sglang.io/advanced_features/lora.html
TensorRT-LLM LoRA docs: https://nvidia.github.io/TensorRT-LLM/advanced/lora.html
NVIDIA NIM PEFT: https://docs.nvidia.com/nim/large-language-models/latest/peft.html
HF TGI Multi-LoRA blog: https://huggingface.co/blog/multi-lora-serving
LMSYS S-LoRA recipe blog: https://www.lmsys.org/blog/2023-11-15-slora/
Predibase LoRAX blog: https://predibase.com/blog/lorax-the-open-source-framework-for-serving-100s-of-fine-tuned-llms-in

Speculative decoding: the secretary who types ahead of the boss — fundamentals, maths and state of play in May 2026

Fri, 29 May 2026 11:00:00 +0200

This post complements KV cache: the working memory and Quantisation for LLM inference. Both are acceleration levers inside a single forward pass. Speculative decoding is the complementary lever: instead of making each forward pass cheaper, it tries to produce more tokens per forward pass. It is orthogonal to quantisation and to the KV cache, and it stacks multiplicatively with both.

You are here: DEPLOY

TL;DR

Autoregressive LLM inference has a structural problem: every new token needs a complete forward pass of the model, and decode forward passes are memory-bandwidth-bound, not compute-bound (the detail is in KV cache). The GPU spends most of its time waiting for the next weight to arrive from HBM. Speculative decoding uses that dead time by doing two things in parallel: a small, cheap model (draft) generates γ tokens autoregressively, fast but imprecise, and the large model (target) verifies those γ tokens in a single parallel forward pass, at almost the same cost as a single-token forward pass. An acceptance rule based on rejection sampling decides how many draft tokens are accepted, and the maths proves, it does not approximate, it proves, that the output distribution matches sampling directly from the target exactly. The technique has a ceiling: you never generate more than 1/(1-α) tokens per step, with α the acceptance rate. The five dominant families in May 2026 are vanilla SD (Leviathan 2023), Medusa (extra parallel heads), EAGLE-1/2/3 (a draft that operates at the level of hidden states), MTP (multi-token prediction native to DeepSeek-V3) and P-EAGLE (a draft that produces the γ tokens in a single forward pass, integrated in vLLM 0.16+). On low-concurrency workloads with short prompts and long outputs, the typical case of the on-premise conversational assistant, the real speedup sits between 2× and 4× on modern hardware.

The analogy: the secretary who types ahead and the boss who validates

Picture a crowded press conference with a real-time transcription setup. Two people are working:

The secretary sits at the keyboard. They know the subject, they have read the briefings and they know the boss’s verbal tics. As soon as the boss opens their mouth, the secretary starts typing what they think is coming. They are fast, three or four words ahead of the boss, but they sometimes get it wrong, especially when the boss takes an unexpected turn.

The boss periodically reads the secretary’s screen. They read a whole block at once, three or four words, and mentally compare what is written with what they would have said. If it matches their intent, they leave the text and carry on. If the secretary drifted at some point, they correct right there, and whatever the secretary typed after the divergence is discarded (it could be wrong because of the earlier error). If the whole block was fine, the boss takes the chance to add one more word while correcting.

The result: the final text is identical to what the boss would have dictated alone, but it comes out faster because the secretary works ahead.

The analogy holds on four details:

The secretary is the draft model. Small, cheap, fast, approximate.
The boss is the target model. Slow, expensive, but the only authority on output quality.
The block the boss reads at once is the target’s parallel forward pass over prompt + γ draft tokens. Verifying γ tokens costs almost the same as generating one, because the bottleneck is loading the weights from HBM, not the operations.
The “correct right where it diverged” rule is the rejection sampling that preserves the target distribution.

From here we go into the mechanism and into why quality is preserved exactly, not approximately.

The bare mechanism

Call p the target distribution (what it would produce if it were the only one speaking) and q the draft’s. The speculative decoding iteration has three steps:

Step 1 — Draft. The draft generates γ tokens autoregressively: x_1, x_2, ..., x_γ. Each one through its own forward pass. Since the draft is small, the γ steps cost little. Typically γ ∈ [4, 8].

Step 2 — Verify. The target runs a single forward pass over the complete sequence prompt + x_1 ... x_γ. Because attention is causal, that forward pass simultaneously produces the distributions p(·|prompt, x_<i) for each position i. In other words, in a single step it obtains the verification of the γ tokens and, on top of that, an extra distribution p(·|prompt, x_1...x_γ) for the next token.

Step 3 — Accept/reject token by token. For each x_i from left to right it applies:

If p(x_i) ≥ q(x_i): always accept.
If p(x_i) < q(x_i): accept with probability p(x_i)/q(x_i).
If it rejects: stop, sample a replacement token from the normalised residual distribution norm(max(0, p − q)), and discard the rest.
If it reaches the end having accepted the γ tokens: append a bonus token sampled directly from p(·|prompt, x_1...x_γ), which is already in the logits of the target’s forward pass.

Result per iteration: between 1 and γ+1 new tokens. In the best case (all accepted + bonus) γ+1 tokens are generated at the cost of a single target forward pass plus γ draft forward passes, against γ+1 target forward passes in the version without speculative.

Why quality does not degrade (the proof)

This is the part of the technique that confuses people most the first time. It looks like magic: how can sampling from an arbitrary draft and then “validating” produce exactly the same distribution as sampling from the original target?

The proof fits in two lines. For any token x, the final probability of emitting it is the sum of two disjoint events: the draft proposes it and it is accepted, or the draft proposes something else, it is rejected, and sampling from the residual yields x. Formally:

$$P(\text{emit } x) = q(x) \cdot \min\!\left(1, \frac{p(x)}{q(x)}\right) + P(\text{reject}) \cdot \frac{\max(0, p(x)-q(x))}{\sum_y \max(0, p(y)-q(y))}$$

The first term is min(p(x), q(x)). The second is max(0, p(x) − q(x)) (the total rejected mass Σ max(0, p−q) cancels the denominator exactly). Adding them: min(p(x), q(x)) + max(0, p(x) − q(x)) = p(x). The output is statistically indistinguishable from sampling directly from the target, down to floating-point numerical differences.

This matters operationally: it means speculative decoding is not perceptual compression, it is not a quality-speed trade-off, it does not require extra validation with evals. If the target would have passed a given eval, the system with speculative passes it too.

The maths that matters: ceiling and speedup

Call α the acceptance rate: the expected probability that an individual draft token is accepted. Assuming that acceptances of consecutive tokens are independent (a reasonable approximation in practice), the expected number of tokens generated per iteration is:

$$E[\text{tokens per step}] = \sum_{k=0}^{\gamma} \alpha^k + \alpha^{\gamma+1} = \frac{1 - \alpha^{\gamma+1}}{1 - \alpha}$$

And the theoretical speedup relative to generating one token per target iteration, with c = T_draft / T_target the relative cost of the draft, is:

$$\text{Speedup} = \frac{1 - \alpha^{\gamma+1}}{(1 - \alpha)(\gamma c + 1)}$$

There is a ceiling that many implementations try to beat and cannot: as γ → ∞, the speedup converges to:

$$\lim_{\gamma \to \infty} E[\text{tokens per step}] = \frac{1}{1 - \alpha}$$

It is an algorithmic ceiling, not a hardware one. With α = 0.7 you never generate more than 3.33 tokens per iteration no matter how much the hardware improves. With α = 0.8 → 5. With α = 0.9 → 10. That is why EAGLE-3, which targets α > 0.8 on many benchmarks, is not an incremental improvement over vanilla SD (α ≈ 0.5-0.6): it is a change of regime, because it raises the ceiling instead of merely getting closer to it.

A concrete numerical example, Llama 3 70B as target with a draft giving α = 0.75, γ = 5, c = 0.1:

Expected tokens per step: (1 − 0.75⁶) / (1 − 0.75) = (1 − 0.178) / 0.25 = 3.29
Speedup: 3.29 / (5 × 0.1 + 1) = 3.29 / 1.5 = 2.19×

If we raise α to 0.85 with the same configuration: tokens per step = (1 − 0.85⁶) / 0.15 = (1 − 0.377) / 0.15 = 4.16, speedup = 4.16 / 1.5 = 2.77×. A change of α = 0.75 → 0.85 (10 absolute points) raises the speedup by 27 %. That is why the research of the last two years focuses on pushing α: that is where the prize is.

The five modern families (May 2026)

Family	Year	Core idea	Draft size	Typical α / τ	Speedup
Vanilla SD	2023	Same-family target + draft pair, rejection sampling	1/10 – 1/100 of target	0.5 – 0.8	2-3×
Medusa	2024	N extra heads in parallel predict t+1, t+2, …; tree attention verifies several candidates	no draft, +1-2 % params	top-5 > 0.8	2.18-2.83×
EAGLE-1/2/3	2024-25	Autoregressive draft at the level of features (hidden states), not tokens. Reuses the target’s embedding	1 transformer block (~0.5-2 % params)	EAGLE-3: α > 0.8, τ up to 7.5	up to 6.5× peak
MTP (DeepSeek-V3)	2024	Multi-token heads trained from scratch as part of the model; at inference they act as a “free” draft	14B params in V3 671B	α > 0.8 (MTP1)	1.5-1.8×
P-EAGLE	2026	EAGLE but producing the γ drafts in a single forward pass (parallel, not autoregressive)	same as EAGLE	+30 % over EAGLE-3	4-5× vs AR

Three operational observations:

EAGLE dominates in production because its overhead is minimal (one transformer block, ~1 % of the target in parameters) and its α is high. The draft does not need its own “complete” KV cache because it shares features with the target.
Medusa mattered historically, it showed that speculative was possible without a separate draft, but EAGLE beat it on every published benchmark through 2024-2025.
MTP is special: it is not something you add to an existing model. It is something the model trained natively. If you buy DeepSeek-V3, MTP comes free and gives around 1.8× without touching anything. If you buy Llama 3, there is no MTP to be had; use EAGLE-3.

There are also two related techniques that deserve a brief mention, both without a draft model: Lookahead decoding (Fu et al. 2024), which formulates decoding as Jacobi iteration and extracts n-grams from the trajectory; and REST (He et al. 2024), which keeps an n-gram datastore and proposes drafts by longest-prefix match against the last generated tokens. Both give 1.5-2× with no extra VRAM. Useful when you have no trained draft model and do not want to maintain one.

Real implementations in May 2026

vLLM v0.16+: unified support for EAGLE-1/2/3, P-EAGLE, Medusa, native MTP (DeepSeek-V3 and variants), n-gram/suffix decoding without a draft, an arbitrary draft model and MLP speculators. The canonical flag is --speculative-config '{"method":"eagle3", "model":"...", "num_speculative_tokens": 5}'.
SGLang: native EAGLE-3 support with --speculative-algorithm EAGLE3. For DeepSeek-V3 it uses MTP through an EAGLE adapter. It has its own draft training framework (SpecForge).
TensorRT-LLM: Medusa, EAGLE (a simplified variant without a tree), ReDrafter, Lookahead. They report around 2.2× with EAGLE.
llama.cpp: basic speculative only with a draft model (--model-draft). No native EAGLE/Medusa/MTP as far as I have verified. Typical speedup 1.5-2.5× single-user.

When speculative does NOT help

The technique has three important blind spots:

Large batch. The GPU moves from memory-bound (decode at low concurrency) to compute-bound (decode at high concurrency). In the compute-bound regime the “almost free” target forward passes stop being free: γ tokens start costing γ times more instead of nearly 1. The typical crossover is at batch 16-32 for dense models; for MoE with few active parameters per token the crossover happens later. At load peaks, speculative can worsen aggregate throughput per GPU.

Prefill / TTFT. Speculative decoding does not touch prefill. The phase that processes the whole prompt stays identical. TTFT does not improve and can get marginally worse because of the draft setup. If the SLA is TTFT-bound (assistants with short outputs, search, RAG with brief answers), this is not the right tool.

Short outputs. If the model generates 10-20 tokens and the answer is over, the fixed setup overhead is not amortised. Speculative shines with long outputs (300+ tokens): generative assistants, extended code completion, drafting.

Operational pitfalls

Extra VRAM for the draft. In vanilla SD, loading a complete draft plus its KV cache is expensive. Llama 3 8B in BF16 as the draft for a 70B is about 16 GB of weights plus KV cache. On an H100 80 GB with the target already nearly filling it, that can force you to shrink the target’s KV cache and lower maximum concurrency. EAGLE solves this: the draft is one transformer block (~1 GB for a 70B) and reuses the target’s features.

Draft quantisation. Quantising the draft to INT4 with GPTQ degrades α substantially (errors in the logits accumulate in the comparison against p). AWQ holds up better but also lowers α. Common practice in 2026: target in FP8 or INT4, draft in FP16/BF16. The draft is small enough that the VRAM saved by quantising it does not compensate for the drop in α and therefore in speedup. The detail of each format is in Quantisation.

Interaction with continuous batching. It does not break it, but it creates nested raggedness: each request in the batch can accept a different number of tokens on each iteration. vLLM’s PagedAttention absorbs it, but the scheduler loses efficiency. At low QPS (conversational assistant, low simultaneous concurrency) the combination is excellent. At high QPS there is real tension, and work such as Goodput-optimized speculative decoding (Liu et al., 2024) optimises γ dynamically according to the state of the batch.

Sampling temperature. α falls with high temperatures. At T = 1.0 with creative outputs (free-form writing), α can drop 10-15 points relative to T = 0 with the same model-draft pair. The speedup scales accordingly.

Implications on on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace)

The classic use is vanilla SD with two quantised models: for instance, Llama 3 70B AWQ-INT4 as target (~35 GB → needs TP=2 across two 4090s) and Llama 3 8B AWQ-INT4 as draft. In practice, the most realistic single-card case is Llama 3 8B target + a 1B model as draft or Qwen 3 14B target + Qwen 3 0.5B as draft: they fit comfortably and give 1.8-2.5× on conversational tasks. For EAGLE on consumer cards, the official drafts for popular families (Llama 3, Qwen 3) are published on Hugging Face and take up 0.5-2 GB extra.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here EAGLE-3 shines:

Llama 3 70B FP8 + EAGLE-3 draft in FP16: the draft takes about 0.7 GB; the target about 70 GB with TP=2 (35 GB per GPU). The speedup observed in reproducible benchmarks is between 2.5× and 4× at batch 1-4, falling to almost break-even at batch 32.
DeepSeek-V3 671B FP8 + native MTP: the model ships with MTP trained in; there is nothing to add. The speedup is 1.5-1.8× with zero extra VRAM. It is the most efficient option operationally: zero additional pieces.
Combining with disaggregated serving: as Disaggregated serving explains, prefill and decode can live in separate pods. Speculative applies only in the decode pods, which fits the separation perfectly (prefill is compute-bound and would not benefit).

The rule of thumb on an H100 cluster in May 2026: if the model is DeepSeek-V3 / V4 → native MTP, nothing else; if it is Llama 3 / Qwen 3 → EAGLE-3 with the official draft; if it is exotic → vanilla SD with a draft from the same family.

What we have not covered

Training custom EAGLE drafts with SpecForge: how to collect target trajectories and train the draft on-policy.
Speculative Prefill (arXiv:2502.02789): a variant for accelerating TTFT, a different mechanism from the decode one described here.
Tree attention in detail: how Medusa and EAGLE-2 verify several candidates at once with specific attention masks.
MoE + speculative: the combination has non-trivial interactions with the expert router. Low activated params keep the memory-bound regime in place even at high batch, which changes the rules.

References

Leviathan, Y., Kalman, M., Matias, Y. Fast Inference from Transformers via Speculative Decoding. ICML 2023. https://arxiv.org/abs/2211.17192
Chen, C., Borgeaud, S., Irving, G., Lespiau, J.B., Sifre, L., Jumper, J. Accelerating Large Language Model Decoding with Speculative Sampling. DeepMind 2023. https://arxiv.org/abs/2302.01318
Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J.D., Chen, D., Dao, T. Medusa: Simple LLM Inference Acceleration with Multiple Decoding Heads. ICML 2024. https://arxiv.org/abs/2401.10774
Li, Y., Wei, F., Zhang, C., Zhang, H. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. ICML 2024. https://arxiv.org/abs/2401.15077
Li, Y., Wei, F., Zhang, C., Zhang, H. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. EMNLP 2024. https://arxiv.org/abs/2406.16858
Li, Y., Wei, F., Zhang, C., Zhang, H. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. NeurIPS 2025. https://arxiv.org/abs/2503.01840
DeepSeek-AI. DeepSeek-V3 Technical Report — Multi-Token Prediction. https://arxiv.org/abs/2412.19437
Fu, Y., Bailis, P., Stoica, I., Zhang, H. Break the Sequential Dependency of LLM Inference Using Lookahead Decoding. ICML 2024. https://arxiv.org/abs/2402.02057
He, Z., Zhong, Z., Cai, T., Lee, J.D., He, D. REST: Retrieval-Based Speculative Decoding. NAACL 2024. https://arxiv.org/abs/2311.08252
vLLM speculative decoding docs: https://docs.vllm.ai/en/latest/features/speculative_decoding/
SGLang speculative decoding docs: https://docs.sglang.ai/advanced_features/speculative_decoding.html
TensorRT-LLM speculative sampling: https://nvidia.github.io/TensorRT-LLM/advanced/speculative-decoding.html
vLLM blog Speculative Decoding in vLLM (oct 2024): https://blog.vllm.ai/2024/10/17/spec-decode.html
vLLM blog P-EAGLE: Parallel Speculative Decoding in vLLM (mar 2026): https://vllm.ai/blog/2026-03-13-p-eagle
Official EAGLE repo: https://github.com/SafeAILab/EAGLE
Official Medusa repo: https://github.com/FasterDecoding/Medusa

Tensorrt-Llm on lo0 — Tech Blog

LLM serving engines compared on the Pareto frontier: vLLM, SGLang, TRT-LLM and Dynamo

TL;DR

Framing the article: what this B8 is and is not

Card for each engine

vLLM

SGLang

NVIDIA TensorRT-LLM (TRT-LLM)

NVIDIA Dynamo

Honourable mentions: LMDeploy and HF TGI

Capability matrix

Why they are not compared lightly: the variables that break a comparison

Performance on the Pareto frontier: illustrative figures

Throughput vs latency: Pareto table

Saturation point (concurrency sweep, vLLM FP8, 4×H100)

Fair-comparison methodology: the harness as the constant

Variables that are fixed (constants of the experiment)

Harness steps

Reproducible output format

State of the art 2026: relative maturity

Pareto decision table: when to choose each engine

Quantisation: effect on throughput and quality

Disaggregated serving: when the overhead is worth it

MLPerf Inference v5.1: cross-vendor reference figures

Variables the benchmark does not capture

Sources

Continuous batching: the salon with 8 chairs that does not wait for the slow customer — Orca, vLLM, chunked prefill and goodput

You are here: DEPLOY

TL;DR

The analogy: the salon with 8 chairs

The problem continuous batching solves

Orca (OSDI ‘22): the idea that changed everything

vLLM (SOSP ‘23): the open-source materialisation

Chunked prefill (SARATHI / Sarathi-Serve, OSDI ‘24)

Goodput: the metric that matters (DistServe, OSDI ‘24)

The iterative scheduler in action

The maths that matter

Real implementations in May 2026

The three operational tensions

Metrics to monitor

Operational pitfalls

Implications for on-premise hardware

What we have not covered

See also

References

Multi-LoRA serving: the single translator with a thousand glossaries — shared base, thousands of concurrent adapters and the SGMV kernel

You are here: DEPLOY

TL;DR

The analogy: the single translator with a thousand glossaries

The bare mechanism: what a LoRA does and why it can be served multi-tenant

SGMV: the kernel that holds it all up

The maths that matters

The real implementations in May 2026

Combined pattern with quantisation and disaggregated serving

Operational pitfalls

Implications on on-premise hardware

Typical production stack

What we have not covered

See also

References

Speculative decoding: the secretary who types ahead of the boss — fundamentals, maths and state of play in May 2026

You are here: DEPLOY

TL;DR

The analogy: the secretary who types ahead and the boss who validates

The bare mechanism

Why quality does not degrade (the proof)

The maths that matters: ceiling and speedup

The five modern families (May 2026)

Real implementations in May 2026

When speculative does NOT help

Operational pitfalls

Implications on on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace)

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

What we have not covered

See also

References