Decode on lo0 — Tech Blog

The signature-dish specialist: vLLM's attention backend (FlashAttention, FlashInfer and the prefill/decode asymmetry)

Mon, 08 Jun 2026 05:40:00 +0200

Part of the under the engine series. The PagedAttention post explained where the KV lives (in paged blocks). This one explains who reads it and how: the attention kernel. And it connects with FlashAttention v1-v4, which took that kernel apart from the inside; here we look at the level above, how vLLM chooses between several kernels and why it needs more than one.

TL;DR

An LLM forward is, for the most part, standard matrix multiplications that any library does well. The exception that decides performance is attention, and having one good kernel is not enough: you need two, because the two phases of inference are physically opposite problems. Prefill processes the whole prompt: many queries against many keys, dense and compute-bound, the home ground of FlashAttention’s IO-aware tiling. Decode generates one token: a single query against all the accumulated KV, thin and memory-bound, where the only thing that matters is saturating HBM bandwidth while reading the paged KV. That is why vLLM does not have “the attention kernel” but a switchable backend (FLASH_ATTN, FLASHINFER, TRITON_ATTN and others) plus logic that picks according to the GPU: by default FA4 on Blackwell (SM100), FA3 on Hopper (SM90), FA2 elsewhere, with FlashInfer as the alternative that compiles bespoke kernels (JIT) and knows how to do cascade attention for shared prefixes. This post explains why prefill and decode are opposites (with arithmetic intensity), how the backend reads paged KV, how the engine chooses, what FlashInfer brings, the 10 knobs and the trap of pinning a backend blindly. On the generic 4×H100 SXM cluster.

Where you are: the specialist, not the commis

In the kitchen, almost all the work is chopping and sautéing: standard operations any competent commis chef executes, the matrix multiplications of the feed-forward layers and the projections. There is one dish that is not delegated: the signature dish, the one that defines the restaurant. That dish is attention, and it has a quirk: it is cooked in two radically different ways depending on the moment of service.

During prefill, when a new order arrives with its whole prompt, you cook big: a lot of raw material at once, a lot of fire, an intense operation that fills the stoves. During decode, when a table asks for “one more dish”, you cook à la carte: a single dish, but you have to go to the larder and bring back every ingredient that table has accumulated over the whole meal. One is a problem of firepower; the other, of larder speed. The same specialist does not do both well. That is why vLLM has several, and a chef who decides which one steps up depending on the GPU and the phase. That is the attention backend.

Why prefill and decode are opposite problems

This is the central idea, and one sum proves it: arithmetic intensity (FLOPs per byte read). An operation with high intensity is limited by compute; one with low intensity, by memory.

Prefill. We serve $N$ queries (the whole prompt) against $N$ keys. The $QK^\top$ operation and the $\text{softmax}\cdot V$ do on the order of $N^2 d$ FLOPs and read on the order of $N d$ of data. Intensity grows with $N$:

$$I_\text{prefill} \sim \frac{N^2 d}{N d} = N$$

With large $N$ (a prompt of thousands of tokens), intensity is high: compute-bound. This is where FlashAttention’s tiling squeezes the tensor cores and where you get close to the GPU’s peak TFLOPS.

Decode. We serve one query (the new token) against $L$ keys (all the accumulated KV). FLOPs on the order of $L d$; bytes read on the order of $L d s$ (the entire KV has to be read from HBM). Intensity is:

$$I_\text{decode} \sim \frac{L d}{L d s} = \frac{1}{s} \quad (\approx 0.5 \text{ FLOP/byte in FP16})$$

Constant and tiny: memory-bound. The decode kernel is not limited by how much the GPU can compute but by how fast it reads the KV from HBM. It makes no difference that the H100 has 132 idle SMs (see the SM post): the bottleneck is the 3.35 TB/s of bandwidth, and the decode kernel exists so as not to waste a single one of those bytes/s.

The design consequence: a kernel optimised for prefill (dense tiling, maximum tensor-core occupancy) is not the optimum for decode (coalesced reads of the paged KV, minimum latency). Serious servers have different kernels, or one kernel with two paths. In models with MLA (multi-head latent attention), vLLM goes as far as using separate backends for prefill and decode, selectable independently (attention backends, vLLM).

The scheduler’s trick: prefill and decode in the same forward

Here the circle closes with the scheduler post. Because vLLM V1 mixes requests in prefill and in decode in every step, one single forward has to serve both. The backend receives metadata telling it, for each sequence in the batch, how many queries it brings and how much KV it has to read, and applies the right path to each one. That is why the attention backend and the scheduler are coupled: the first has to digest the heterogeneous batch the second builds.

How the backend reads the paged KV

The kernel does not receive a contiguous KV tensor: it receives the block table from the block manager and does a gather over the physical blocks. This imposes a real constraint on the backend: it has to support the paged layout and vLLM’s block_size. Not every kernel in the world does; the ones vLLM integrates (FlashAttention, FlashInfer, Triton) are adapted to read KV in fixed-size blocks scattered across HBM. That is why you cannot plug in any attention kernel from a paper: it has to speak the language of the pigeonhole larder.

The backends and how the engine chooses

vLLM exposes a backend abstraction with several implementations (vLLM deepwiki):

FLASH_ATTN — the FlashAttention family. By default the version is picked by architecture: FA4 on SM100 (Blackwell), FA3 on SM90 (Hopper), FA2 elsewhere, configurable with flash_attn_version.
FLASHINFER — an attention engine with JIT compilation and specialisable kernels; strong on heterogeneous KV and shared prefixes.
TRITON_ATTN — written in Triton, portable and with no dependency on precompiled CUDA binaries (Triton backend deep dive, vLLM, mar-2026).
Specific backends for MLA and for non-NVIDIA hardware.

Selection is automatic unless you force it with VLLM_ATTENTION_BACKEND. The heuristic tries FlashAttention first; on Blackwell (SM100) the fallback order for MLA is TRT-LLM Ragged → FlashInfer → others; on other GPUs only FlashAttention is considered for the main path (attention backends, vLLM). The decision depends on: architecture (SM), dtype (FP16/BF16/FP8), head dimension, and whether the load needs a feature only one backend has (cascade attention, certain soft caps, FP8 in KV).

What FlashInfer brings: JIT and cascade attention

FlashInfer does not compete with FlashAttention on “being a bit faster”; it attacks a different problem: the heterogeneity of the KV in real serving (FlashInfer, arXiv 2501.01005). Two ideas:

JIT compilation. Instead of a monolithic kernel, FlashInfer generates bespoke kernels for the attention variant, the problem shape and the KV layout you have, injecting functors (query/key/logit transformations, masks). It specialises rather than generalises.

Cascade attention. Here is the jewel for serving with shared prefixes. If $R$ requests share a prefix of $P$ tokens (a common system prompt), naive attention would read that prefix $R$ times. Cascade attention computes it once against the shared prefix and then combines with each request’s own suffix:

$$\text{reads: } \underbrace{R \cdot (P + s_i)}_{\text{naive}} \;\longrightarrow\; \underbrace{P + \textstyle\sum_i s_i}_{\text{cascade}}$$

With $R=50$ requests and a prefix $P=1000$, that is reading 50,000 prefix tokens versus 1,000. It is the natural complement to prefix caching: the block manager shares the prefix’s memory, and cascade attention shares the compute of attending over it.

The arithmetic that matters: when switching backend gives you something

The backend only moves the needle where attention is the bottleneck. In memory-bound decode, a kernel that exploits HBM bandwidth better gives a real improvement; in compute-bound prefill with long sequences, FA3/FA4 getting close to the tensor-core peak gives a real improvement. But if your bottleneck is in another layer, launch overhead, a badly sized scheduler, the cold start, then switching backend does not touch that part. The rule, once again: measure the regime before optimising.

The 10 knobs

#	Knob	What it controls	Cost / risk
1	`VLLM_ATTENTION_BACKEND`	force a backend	mismatch with hardware/feature
2	`flash_attn_version` (2/3/4)	FA version	version unsupported on your SM
3	enable FlashInfer	JIT + cascade	initial JIT compilation time
4	cascade attention	prefix compute reuse	only helps with a heavily shared prefix
5	`kv_cache_dtype` (FP8)	FP8 support in the kernel	not every backend/SM supports it
6	`block_size`	layout the kernel must read	consistency with PagedAttention
7	MLA prefill backend	kernel for the dense phase	MLA models only
8	MLA decode backend	kernel for the thin phase	MLA models only
9	soft cap / sliding window	features that restrict backends	fewer kernel options
10	head_dim / variant	which kernels are eligible	exotic models with no support

How it connects to the rest of the stack

With FlashAttention. The FA post explains the kernel from the inside (tiling, online softmax, FA1-4); this is the level above, how vLLM chooses between kernels and why it needs more than one.

With PagedAttention. The backend reads the KV that the block manager places in blocks; it has to speak block-table.

With the scheduler. The scheduler builds mixed prefill+decode batches; the backend has to serve both regimes in a single forward.

With CUDA graphs. Attention kernels are captured in the CUDA graphs; a backend that launches many small kernels benefits more from capture.

With prefix caching. Cascade attention is the compute side of what prefix caching does in memory.

With FP8. Attending over KV in FP8 requires the backend to have the FP8 path; not all of them do on every architecture.

Traps and things that are not what they look like

“FlashInfer is always faster than FlashAttention.” No. FlashInfer wins when its specialisation (cascade, heterogeneous KV, a particular attention variant) applies to your load; in classic dense prefill, FA3/FA4 usually does as well or better. It depends on the regime, there is no universal winner.

“A good attention kernel works for everything.” The underlying error of this post. Prefill and decode are compute-bound and memory-bound respectively; a kernel tuned for one wastes effort on the other. That is why there are separate paths (and separate backends in MLA).

“Decode is compute-bound because the GPU is at 100%.” nvidia-smi at 100% is misleading (see the SM post): decode is memory-bound, the GPU is moving KV, not computing. Optimising decode’s compute is polishing what is not the bottleneck.

“I pin VLLM_ATTENTION_BACKEND and forget about it.” Pinning a backend by hand can leave you on a suboptimal one when you change GPU or version, or force a slow fallback if your hardware does not support what you asked for. Autoselection usually gets it right; pin it only with a measurement that justifies it.

“Cascade attention always helps.” Only with a heavily shared prefix across many concurrent requests. If each request has its own context, there is nothing to share and the overhead of organising the cascade is not amortised.

“The attention backend is the bottleneck, that is why I am slow.” Almost always the bottleneck is higher up (launch, scheduling, memory) or lower down (bandwidth). The backend matters where attention dominates; measure it with nsys/DCGM before changing it.

Conclusion

Of everything an LLM does while generating text, almost all of it is matrix multiplications that any library resolves. Performance is decided in a single kernel, attention, and the surprise is that it is not even one kernel: it is two opposite problems wearing the same name. Prefill wants fire, dense compute over thousands of tokens, and decode wants a fast larder, reading a token’s entire KV with the minimum waste of bandwidth. That is why vLLM did not pick a winning kernel but an abstraction that switches: FlashAttention tuned to each architecture for the general case, FlashInfer compiling bespoke kernels when there is heterogeneity or prefixes to share, Triton for portability. The head chef does not cook the signature dish just one way: he looks at who is ordering and at what point of service, and sends out the right specialist. The lesson for whoever is tuning is the usual one in this series: before changing specialist, make sure the signature dish really is what is holding you back.

References

vLLM, Attention Backends (selección, FA2/3/4 por arquitectura, MLA): https://docs.vllm.ai/en/latest/design/attention_backends/.
vLLM / DeepWiki, FlashAttention and FlashInfer: https://deepwiki.com/vllm-project/vllm/8.2-flashattention-and-flashinfer.
vLLM, Triton Attention Backend Deep Dive (mar-2026): https://vllm.ai/blog/2026-03-04-vllm-triton-backend-deep-dive.
Z. Ye et al., FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving (arXiv 2501.01005): https://arxiv.org/abs/2501.01005.
T. Dao, FlashAttention-2 / FlashAttention-3 (kernel IO-aware, async Hopper): https://github.com/Dao-AILab/flash-attention.

The chef who calls out every order: SMs, CUDA streams and CUDA graphs, or why the GPU gets bored generating tokens

Sun, 07 Jun 2026 09:00:00 +0200

This closes the “outside the API” pair. The previous post moved the weights from disk into HBM; here we look at what happens once they are inside, in the silicon that executes them. It is the floor below the kernel launch that the NUMA post mentioned without opening: who launches those kernels, how, and why in decode the GPU spends more time waiting for orders than computing.

TL;DR

An H100 has ~132 streaming multiprocessors (SMs), the “hobs” that execute the compute, and occupancy measures how many warps (groups of 32 threads) it has active to hide latency. But the decode bottleneck is rarely the power of those SMs. Each decode step launches hundreds of tiny kernels (several projections per layer × ~80 layers), and each kernel launch costs 5-10 µs of serial CPU. Since decode kernels are small (small batch, a single token), the GPU finishes them before the CPU calls out the next one: bubbles appear and the GPU sits bored waiting for orders. That regime is called launch-bound, and it is the deep reason, not power, not memory, why --enforce-eager delivers 54 tok/s where with optimisations you reach 89-140. The solution is CUDA graphs: record the entire kernel sequence once and replay it as a single submission, removing the per-launch overhead (~28% of the latency per iteration). vLLM captures ~102 graphs at startup and pads the real batch to the nearest bucket so it can replay a graph with fixed shapes. This post explains SM, occupancy, streams, launch overhead with the maths, CUDA graphs, the 10 knobs, and the trap that this capture is the second half of the cold start from the previous post. With scepticism about what actually moves the needle. On the generic 4×H100 SXM cluster.

Where you are: the silicon, below the kernel launch

The analogy: the chef who calls out every order

Last scene in the restaurant of the series. The kitchen is set up, the pantry is stocked (the previous post). Now the food has to be plated. The hobs are the SMs: 132 stations cooking in parallel. The head chef is the CPU: he calls out the orders, each kernel launch is a shout of “one matrix multiplication, coming up!”. The cooks (the SMs) execute what the chef calls out.

In prefill, processing the whole prompt, each order is a huge dish: a giant matmul over hundreds of tokens at once. The chef calls out an order and the hobs take a good while to produce it. The chef has plenty of time to call out the next one. The hobs are flat out: compute-bound.

In decode, generating one token at a time, each order is minuscule: a matmul over a single token. The hob finishes it in an instant… and sits staring at the chef waiting for the next one. But the chef can only call out one order every 5-10 µs, and there are hundreds of orders per token. The hobs, lightning fast, get bored between shouts. The restaurant is not slow because the cooks are bad: it is slow because the chef does not call out fast enough. That is the launch-bound regime.

The solution is not more hobs or faster cooks. It is to stop calling out order by order. If the chef prints the whole night’s sequence on a single sheet and hands it to the line (“do this, in this order, without waiting for me”), the hobs run without pauses. That is a CUDA graph: record the kernel sequence once and replay it in one go, without the CPU calling out each one. And --enforce-eager is exactly the opposite: forcing the chef to call out order by order, all night long.

The mechanism: SM, warps and occupancy

An H100 SXM has ~132 SMs. Each SM executes threads in groups of 32 called warps, and can have several warps “in flight” at once. Occupancy is the fraction of active warps against the maximum the SM supports. What is the point of having many active warps? To hide latency: while one warp waits for data from HBM (hundreds of cycles), the SM executes another ready warp. With few warps, the SM runs out of anyone to give a turn to and stalls.

But, and this is key, occupancy is a necessary, not sufficient condition, and it only matters if the SM has work to do. In decode, the typical problem is not low occupancy inside a kernel: it is that between kernels the SM has nothing, because the CPU has not launched the next one yet. Raising the occupancy of a kernel that lasts 8 µs does not help if the GPU spends 6 µs waiting for it to be launched.

The mechanism: streams, the order queue

A CUDA stream is a queue of operations that the GPU executes in order. Operations in the same stream are sequential; operations in different streams can overlap. That is what allows, for instance, copying data H2D in one stream while another stream computes, the compute/copy overlap. vLLM uses streams to overlap work, but a stream on its own does not remove the cost of launching each kernel: it only decides ordering and parallelism. The launch cost is still there, order by order, until the graphs come in.

The maths that matters: when the GPU is left waiting

The number that governs everything: a kernel launch costs 5-10 µs of CPU, serially. Take a Llama-70B with ~80 layers. Each layer, without fusion, launches on the order of ~10 kernels (Q/K/V projections, attention, output projection, the two or three MLP matmuls, the normalisations, RoPE and so on). That gives:

$$ N_{\text{kernels}} \approx 80 \text{ layers} \times 10 \approx 800 \text{ launches per token} $$

At 5 µs per launch, serially:

$$ T_{\text{launch}} \approx 800 \times 5\,\mu s = 4{.}0 \text{ ms per token} $$

Those 4 ms are CPU calling out orders only, not counting how long the SMs take to cook. If the GPU could compute instantaneously, the launch ceiling would be ~250 tok/s, and with synchronisation points between kernels, worse. Now compare with the memory ceiling of decode: each token reads the 140 GB of weights once from HBM:

$$ T_{\text{mem}} = \frac{140 \text{ GB}}{3{.}35 \text{ TB/s}} \approx 42 \text{ ms} \;\Rightarrow\; \approx 24 \text{ tok/s (one sequence, no batch)} $$

Here is the subtlety almost nobody has in mind. For a single sequence, decode is memory-bound at ~24 tok/s, and the 4 ms of launch fit inside the 42 ms of reading: the launch hides. But batching changes everything. When serving a batch of B sequences, the weights are read once and serve all B, so the memory cost per token amortises and falls. The GPU stops being memory-bound… and what was underneath emerges: the launch cost, which does not amortise with batch because you have to launch the same kernel sequence regardless. The result: the better you batch, the more launch-bound you become, and the more CUDA graphs pay off. That is why the raw measurement confirms it, --enforce-eager gives 54 tok/s where graphs give 89, and up to 8× in configurations where decode is very small and launch dominates completely.

vLLM’s CUDA graphs, specifically

vLLM does not capture a single graph: it captures ~102 at startup, on the order of 51 piecewise (for the mixed prefill+decode steps) and 51 full (for pure decode). Each one is recorded for a fixed batch size (a bucket: 1, 2, 4, 8… up to a maximum). In service the real batch almost never falls exactly on a bucket, so vLLM pads with zeros up to the next bucket up, replays that graph, and trims the output to the real size. That is the price of graphs: they need static shapes, and padding is what makes them static.

This has two consequences that show up in the knobs and the traps:

Capture costs time and memory. Recording 102 graphs at startup adds seconds to the cold start, the second half of the startup that the previous post left pending, and consumes HBM (each graph retains its buffers). The FULL_AND_PIECEWISE mode (the default) is the fastest in service but the one that asks for the most memory and the longest capture; FULL_DECODE_ONLY saves both in exchange for not accelerating the mixed steps.

Padding wastes some compute. Padding a batch of 33 up to the bucket of 64 computes 31 phantom sequences. It is a small cost against what removing the launch overhead saves, but it exists, and it grows if the buckets are badly chosen.

The 10 knobs worth touching

Knob 1 — Measure whether decode is launch-bound

Before touching anything: is the GPU computing or waiting? With nsys (Nsight Systems) you can see the gaps between kernels on the timeline; if there are gaps in decode, it is launch-bound and graphs will help. If the GPU is at 100% with no gaps, the bottleneck is something else (memory or compute) and graphs will not work miracles. nvidia-smi dmon showing low utilisation in decode but poor TPS is the cheap signal.

Knob 2 — Do not use `--enforce-eager` in production

--enforce-eager disables CUDA graphs. It is a debugging tool (to isolate which kernel fails), not a production one. Leaving it on “because it used to start up” throws away 26-50% of decode throughput. If it is in your production command, take it out and measure.

Knob 3 — Capture buckets (`cudagraph_capture_sizes`)

Which batch sizes to capture. Buckets too far apart make padding expensive; too many of them make capture slow and eat HBM. Tuning them to the real distribution of batch sizes you see in production is the fine adjustment, but only after measuring that distribution.

Knob 4 — CUDA graph mode

FULL_AND_PIECEWISE (default, fastest, more memory/capture), FULL_DECODE_ONLY (saves memory and capture, ideal for pure decode pods in disaggregated serving), PIECEWISE, or NONE (= eager). The right mode depends on whether the pod does pure or mixed decode.

Knob 5 — `torch.compile`

vLLM leans on torch.compile to fuse and optimise kernels before capturing them in graphs. Fewer kernels (fusion) = fewer launches = less dependence on the graph and better decode even in eager. The compilation level is a knob, with its startup time cost.

Knob 6 — Batch size: fill the hobs

Memory-bound decode amortises with batching (as we saw in continuous batching): read the weights once for B sequences. More batch = more SM occupancy and more memory amortisation. The limit is set by the HBM available for the KV cache. It is the knob that moves aggregate throughput most.

Knob 7 — Do not break stream overlap

vLLM overlaps compute and copy with streams. Patching the code to “simplify” can serialise what was overlapped. If you do not know why there are several streams, do not collapse them.

Knob 8 — Persistence mode + locked clocks

nvidia-smi -pm 1 keeps the driver resident (avoiding reinitialisations that add launch latency). Locking clocks to the boost frequency stops the GPU dropping P-state between tiny decode kernels and paying the ramp-up latency. It is the same anti-jitter spirit as the NUMA post, applied to the GPU.

Knob 9 — Fused kernels (FlashAttention, FP8 kernels)

Fewer kernels = fewer orders to call out. FlashAttention fuses attention into one kernel instead of several; fused FP8 kernels reduce the count. Fusion attacks the problem at the root: it does not speed up the launch, it removes launches.

Knob 10 — Accept the capture cost in the cold start

Graph capture adds seconds to startup. In a pod that lives for hours, it amortises easily. In one that scales up and down every minute, that cost is paid over and over; there, FULL_DECODE_ONLY (shorter capture) or accepting slightly less throughput can be worth it. It is the same warm-vs-elastic tension as in the cold start.

Summary table

#	Knob	What it attacks	Risk / cost
1	`nsys` / `dmon`	knowing whether it is launch-bound	none; do it first
2	remove `--enforce-eager`	graphs disabled	it was for debugging; reactivate if a bug returns
3	capture buckets	expensive padding / slow capture	requires measuring the real distribution
4	graph mode	memory and capture	less coverage on mixed steps
5	`torch.compile`	unfused kernels	startup time
6	batch size	occupancy + memory	HBM for KV cache
7	streams	broken overlap	do not touch if not understood
8	persistence + clocks	jitter / P-states	electrical consumption
9	fused kernels	number of launches	kernel compatibility
10	capture vs cold start	slower startup	less throughput if trimmed

How this connects with the rest of the stack

With the cold start. CUDA graph capture is the second half of the startup that the previous post opened: loading weights + capturing graphs = the complete cold start.

With continuous batching. Continuous batching is what makes decode launch-bound (it amortises memory and leaves the launch exposed), and that is why graphs and batching reinforce each other.

With the KV cache. The KV cache decides how much batch fits in HBM, and the batch decides SM occupancy and how much launch overhead matters. Everything is coupled through memory.

With the interconnect. In TP, between the compute kernels there are all-reduces (NVLink/NCCL) that also get launched and synchronised. vLLM’s custom all-reduce is integrated into the same graph so the sequence is not broken by a CPU synchronisation.

With NUMA. Who launches the kernels is the CPU from the host post; if that thread suffers jitter or lands on the wrong socket, launch overhead gets worse. Graphs reduce the dependence on that thread, which is another reason they help.

With disaggregated serving. The pure decode pods of disaggregated serving are the ideal case for FULL_DECODE_ONLY: they maximise the graph benefit exactly in the most launch-bound phase.

Traps and things that are not what they look like

“Raising occupancy will fix slow decode.” Not if the problem is launch-bound. Occupancy matters inside a kernel that has work; if the GPU is idle between kernels waiting for the CPU, more occupancy does not touch that bubble. Measure before optimising what is not the bottleneck.

“CUDA graphs always speed things up.” They speed things up when decode is launch-bound. If the GPU is already at 100% (compute-bound in prefill, or memory saturated with a huge batch), graphs add little. Their territory is decode with small kernels.

"--enforce-eager gives more stable results." It gives slower results. The stability it appears to give is that it avoids graph capture bugs on new hardware (for example a newly supported architecture). It is a temporary patch, not a production configuration.

Capturing too many buckets “just in case”. Each bucket adds capture time and HBM. Capturing 30 sizes when in production you only see 4 means paying cold start and memory for graphs that are never replayed. Tune to the real distribution.

Confusing utilisation with efficiency. nvidia-smi at 100% “utilisation” only says there is a kernel running, not that the SM is full of useful work. A low-occupancy kernel keeps “utilisation” high while wasting the SM. nvidia-smi utilisation is a coarse thermometer; to know whether the silicon is performing you need nsys/DCGM and to look at real occupancy and gaps.

Optimising the silicon before memory. If decode is limited by HBM bandwidth (large batch, large model), fighting with graphs and occupancy is polishing what is not the bottleneck. The right order: measure the regime (memory / compute / launch) and attack the one in charge.

Conclusion

Intuition says that a GPU generating slow tokens is “working hard”. Almost never: in decode it is waiting for orders. The 132 SMs cook a tiny token in an instant and then sit staring at the CPU, which can only call out one order every 5-10 µs and has hundreds to call out per token. That bottleneck, neither power nor memory but launch, is invisible on any dashboard that looks at “GPU utilisation”, and it is the real reason why --enforce-eager performs at half speed. CUDA graphs solve it with a simple idea: stop calling out order by order and hand over the whole sheet for the night, so the silicon runs without pauses. And there is an uncomfortable truth that reorders the optimisation priority: the better you batch, the more launch-bound you become, because batching kills the memory bottleneck and exposes the launch one. That is why graphs and batching are not separate optimisations: they are the same lever seen from two sides. The chef who learns not to call out every dish is what finally makes the kitchen run as fast as the hobs always could.

References

vLLM, CUDA Graphs (diseño, modos FULL/PIECEWISE, captura): https://docs.vllm.ai/en/stable/design/cuda_graphs/.
vLLM, Inside vLLM: Anatomy of a High-Throughput LLM Inference System: https://blog.vllm.ai/2025/09/05/anatomy-of-vllm.html.
NVIDIA, Getting Started with CUDA Graphs: https://developer.nvidia.com/blog/cuda-graphs/.
NVIDIA, Achieved Occupancy (ocupación de SM): https://archive.docs.nvidia.com/gameworks/content/developertools/desktop/analysis/report/cudaexperiments/kernellevel/achievedoccupancy.htm.
PyTorch, torch.compile y CUDA Graphs para inferencia LLM: https://docs.vllm.ai/en/stable/design/cuda_graphs/.
Understanding the Overheads of Launching CUDA Kernels (ICPP 2019): https://www.hpcs.cs.tsukuba.ac.jp/icpp2019/data/posters/Poster17-abst.pdf.

Optimising decode in vLLM: squeezing every token out of small hardware

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

Decode is the phase in which vLLM generates output tokens one at a time. It is memory-bound, not compute-bound: the GPU spends more time waiting for the weights to arrive from VRAM than doing arithmetic. On small hardware, an RTX 4090 (24 GB) or an L40 (48 GB), a badly configured decode wastes half the card’s capacity. Five vLLM parameters change the equation: gpu-memory-utilization, max-num-seqs, speculative decoding, FP8 KV cache and a swap-space correctly set to zero. Well calibrated, the difference is real: from 15 tokens/s to 35–50 tokens/s on the same hardware.

The analogy

An assembly-line worker putting cars together. Every car needs exactly the same process: walk to the store for the part, come back, bolt it on, repeat. The travel time to the store, the VRAM latency, is fixed and cannot be eliminated. But there are ways to make it less painful:

Have several cars in parallel on the line (more concurrency, the same travel time amortised).
Have an assistant prefabricate common parts (speculative decoding: the draft model proposes, the verifier confirms).
Keep only the most-used parts in the workshop (quantised KV cache: more contexts fit in the same space).

Those three strategies are exactly the three axes of decode optimisation in vLLM.

Why decode is memory-bound

During prefill, the GPU processes N tokens in parallel: the attention operation is a large matmul and the compute units are busy. During decode, it processes 1 token per step: the matmul becomes a vector-matrix product, an operation that underuses the tensor cores.

The typical compute utilisation ratio during decode on an RTX 4090:

$$\text{MFU}_{decode} \approx 5–15\% \quad \text{(vs 40–60\% in prefill)}$$

The bottleneck is not compute power, it is bandwidth. To generate each token, the model has to read its complete weights from VRAM:

$$\text{time per token} \approx \frac{\text{weight size in bytes}}{\text{VRAM bandwidth}}$$

For Qwen2.5-7B in BF16 (14 GB of weights) on an RTX 4090 (1,008 GB/s):

$$t \approx \frac{14 \times 10^9}{1.008 \times 10^{12}} \approx 13.9 \text{ ms/token} \approx 72 \text{ tokens/s theoretical maximum}$$

The real value is lower (~30–50 tok/s) because of scheduler overhead, attention over the growing KV cache and other latencies. But the theoretical limit sets the ceiling.

With Q4_K_M (weights ~4 GB):

$$t \approx \frac{4 \times 10^9}{1.008 \times 10^{12}} \approx 3.97 \text{ ms/token} \approx 252 \text{ tokens/s theoretical}$$

Quantising the model is the most direct way to improve decode throughput on memory-bound hardware. Everything else optimises on top of that ceiling.

The five levers

1. Give vLLM all the VRAM you can — `--gpu-memory-utilization`

--gpu-memory-utilization (short form --gpu-mem-util) defines the fraction of available VRAM that vLLM may use for the KV cache, once the model weights are loaded. The rest is reserved for activations during the forward pass and for the CUDA context.

vllm serve my-model \
 --gpu-memory-utilization 0.92

The default value is 0.90. On bare metal where no other process uses the GPU, 0.92–0.95 is safe. Do not go above 0.95: vLLM needs headroom for activations during batch peaks, and running out of VRAM in the middle of an inference results in a process crash, not a clean error.

Why it matters: more KV cache available = more simultaneous requests in flight = better GPU utilisation during decode. PagedAttention allocates the KV cache in fixed-size blocks (16 tokens per block by default), and vLLM manages them like virtual memory pages. The more blocks available, the more requests it can serve without any of them waiting for space.

RTX 4090, Qwen2.5-7B-BF16 (14 GB weights):
Total VRAM: 24 GB
Weights: 14 GB
Available for KV cache: 10 GB
gpu-memory-utilization 0.90 → 0.90 × 10 GB = 9 GB for KV cache
gpu-memory-utilization 0.94 → 0.94 × 10 GB = 9.4 GB → ~4% more tokens in flight

The impact is modest with models that fit comfortably, but it is amplified with models that push VRAM to its limit.

2. Real concurrency — `--max-num-seqs`

--max-num-seqs is the maximum number of requests vLLM can have in process simultaneously (counting prefill and decode together). It is the parameter that controls the system’s effective concurrency.

vllm serve my-model \
 --max-num-seqs 128

The effect is direct: more requests in simultaneous decode = better amortisation of the fixed cost of reading the weights. When the decode batch grows from 1 to 8, the time to generate 8 tokens is almost the same as generating 1 (the weights are read once for all of them). Aggregate throughput scales almost linearly until the KV cache or the activation VRAM becomes the bottleneck.

$$\text{aggregate throughput}(B) \approx B \times \text{throughput}(1) \quad \text{for } B \ll B_{max}$$

Common mistake: raising --max-num-seqs without making sure there is enough KV cache in VRAM for all the requests. If vLLM cannot hold the KV caches of 128 simultaneous requests, it does preemption (pauses some request and frees its KV cache) at a latency cost. Monitor vllm:num_preemptions_total.

Interaction with --max-num-batched-tokens: the vLLM scheduler processes up to max-num-batched-tokens tokens per step. If you have 128 requests in decode generating 1 token each, that is 128 decode tokens. The decode budget consumes 128 tokens of the total budget; the rest goes to prefill in chunks. Tune both values together.

# For an RTX 4090 serving ~50 concurrent users with responses of up to 512 tokens
vllm serve my-model \
 --max-num-seqs 64 \
 --max-num-batched-tokens 8192
# 64 decode tokens per step + up to 8128 chunked prefill tokens

3. Speculative decoding — `--speculative-model` + `--num-speculative-tokens`

Speculative decoding is the highest-impact change for decode on small hardware. The idea is simple: a small draft model proposes several tokens at once, and the verifier model validates or rejects them in a single forward pass.

vllm serve Qwen/Qwen2.5-7B-Instruct \
 --speculative-model Qwen/Qwen2.5-0.5B-Instruct \
 --num-speculative-tokens 5 \
 --speculative-draft-tensor-parallel-size 1

Why it works: the 7B verifier has to read 14 GB of weights per step. With 5 proposed tokens, if the acceptance rate is 80%, an average of 4–5 tokens are generated per verifier step instead of 1. Effective throughput goes up without the GPU working any harder.

The acceptance rate (α) depends on how well the draft predicts the verifier’s distribution. Within the same domain, models from the same family usually have α > 0.75:

$$\text{speedup} \approx \frac{1 + \alpha \cdot k}{1 + \alpha \cdot k / \text{cost ratio}}$$

Where $k$ is the number of proposed tokens and cost_ratio is the draft/verifier cost ratio. For a 0.5B draft and a 7B verifier (ratio ~14×):

$$\text{speedup} \approx 1 + 0.8 \times 5 \approx 5 \text{ (theoretical maximum, not reachable)}$$

In practice, with α = 0.75 and k = 5 on hardware without NVLink: 1.8–2.5× more tokens/s compared with decode alone.

EAGLE-3 in 2026: the best drafters today are not small versions of the same model, but networks specialised in predicting the verifier’s distribution. EAGLE-3 reports a 3–6.5× speedup over vanilla decode in public benchmarks. In production with mixed batches the real speedup is more conservative (1.5–3×). vLLM supports EAGLE/EAGLE-2 via --speculative-model:

# With an EAGLE drafter (requires a drafter trained specifically for the base model)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
 --speculative-model yuhuili/EAGLE3-LLaMA3.1-Instruct-8B \
 --num-speculative-tokens 6

When speculative decoding does NOT help:

Very large batches (>32 requests): the acceptance rate varies between requests and the batch spends more time re-drafting than accepting.
High-entropy tasks (brainstorming, very creative code): the draft predicts worse, α falls below 0.5 and the draft overhead outweighs the gain.
If the draft model does not fit in the available VRAM alongside the verifier.

On an RTX 4090 with a 7B verifier and a 0.5B draft (BF16): 14 + 1 GB = 15 GB. That leaves 9 GB for the KV cache. It works.

4. Quantised KV cache — `--kv-cache-dtype fp8`

This was already covered in the prefill article for its effect on context capacity. From the decode point of view, the benefit is different: more tokens fit in the KV cache → more simultaneous requests without preemption → better aggregate throughput.

vllm serve my-model \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales

Precision warning in decode: the KV cache is read on every attention step of the decode. Quantisation introduces noise into the attention activations. For long texts (>4K tokens of context) it can accumulate. In quality benchmarks (MMLU, HellaSwag) the degradation with FP8 KV and --calculate-kv-scales is <0.5% on modern models. Without --calculate-kv-scales, the degradation can be larger because the scales are fixed statically.

Optimal combination for RTX 4090:

vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \
 --quantization awq \  # weights in INT4: 4 GB model
 --kv-cache-dtype fp8 \  # KV cache at half the size
 --calculate-kv-scales \
 --gpu-memory-utilization 0.94

# Available VRAM: 24 - 4 = 20 GB for KV cache
# With FP8: ~40 KB/token (vs 80 KB BF16) → 20 GB / 40 KB = 500,000 tokens of total context
# With max-num-seqs 64 and a 4K ctx: 64 × 4096 × 40KB = 10 GB → fits with room to spare

5. Remove the swap — `--swap-space 0`

--swap-space defines how much system RAM (not VRAM) vLLM may use to preempt KV caches. When vLLM has more active requests than fit in VRAM, it can “pause” some by moving their KV cache to RAM and reactivating them later.

The problem: moving a 4K-token KV cache from VRAM to RAM and back has a latency of tens of milliseconds over PCIe. For a system where you want predictable latency, the swap introduces unacceptable jitter.

vllm serve my-model \
 --swap-space 0

With --swap-space 0, when vLLM cannot hold any more requests in VRAM, it simply queues them instead of doing preemption. The queue adds waiting latency, but it is predictable and it does not interrupt the requests already in flight.

When should you have swap? If your workload has short demand peaks and you can tolerate occasional jitter in exchange for not rejecting requests, a swap of 4–8 GB can be useful. In ENS deployments where latency is a contractual SLA, --swap-space 0 is the right option.

The reference configuration per hardware

RTX 4090 (24 GB) — 7B model, internal use

vllm serve Qwen/Qwen2.5-7B-Instruct \
 --gpu-memory-utilization 0.92 \
 --max-model-len 8192 \
 --max-num-seqs 64 \
 --max-num-batched-tokens 8192 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales \
 --swap-space 0 \
 --speculative-model Qwen/Qwen2.5-0.5B-Instruct \
 --num-speculative-tokens 5 \
 --speculative-draft-tensor-parallel-size 1 \
 --dtype bfloat16

Expected throughput: 35–55 tokens/s per user, up to 64 simultaneous, TTFT <500ms for prompts <1K tokens.

L40 (48 GB) — 14B model, multi-user

vllm serve Qwen/Qwen2.5-14B-Instruct \
 --gpu-memory-utilization 0.90 \
 --max-model-len 16384 \
 --max-num-seqs 128 \
 --max-num-batched-tokens 16384 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales \
 --swap-space 0 \
 --speculative-model Qwen/Qwen2.5-1.5B-Instruct \
 --num-speculative-tokens 5 \
 --dtype bfloat16

Expected throughput: 25–40 tokens/s per user, up to 128 simultaneous with speculative decoding active, TTFT <800ms for prompts <2K tokens.

How to measure that decode is optimised

# Key metrics at vllm:8000/metrics

vllm:generation_tokens_total # total tokens generated → trend
vllm:e2e_request_latency_seconds_* # end-to-end latency per percentile
vllm:time_per_output_token_seconds_* # ITL (inter-token latency)
vllm:num_preemptions_total # if it rises, the KV cache is filling up
vllm:spec_decode_draft_acceptance_rate # speculative decoding hit rate

If spec_decode_draft_acceptance_rate < 0.6, the drafter is not helping: turn speculative decoding off or find a drafter better trained for your model or domain.

If num_preemptions_total grows, you have too many simultaneous requests for the available KV cache. Options: lower max-num-seqs, enable FP8 KV cache, lower max-model-len, or quantise the model further.

Implications for on-premise inference

In a sovereign deployment with fixed hardware, you cannot buy more GPUs at will. Every well-calibrated tenth of gpu-memory-utilization, every point of speculative decoding acceptance rate and every MB of KV cache freed by FP8 is real capacity you do not have to provision with another node.

The combination of quantised weights (AWQ/GPTQ), FP8 KV cache and speculative decoding lets a 14B serve on one L40 what without optimisations would need two L40s in tensor parallel. That is the economic argument for investing time in these parameters.

Decode cannot be accelerated indefinitely on memory-bound hardware: the theoretical limit is set by VRAM bandwidth. But the difference between the minimum and the maximum achievable on that hardware can be 3–4× with the right levers.

References

Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023
EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees — Li et al., 2024
Fast Inference from Transformers via Speculative Decoding — Leviathan et al., 2022
vLLM Optimization and Tuning — documentación oficial
vLLM Speculative Decoding
FP8 KV Cache en vLLM

Disaggregated serving: prefill and decode in specialised pods

Fri, 22 May 2026 01:00:00 +0200

TL;DR

LLM inference has two phases with opposite profiles: prefill (processing the whole prompt in one go) is compute-bound, decode (generating token by token) is memory-bandwidth-bound. Running them on the same GPU forces a choice between two incompatible optimal hardware profiles, and leaves between 60 % and 80 % of peak capacity unused. The industry consolidated the pattern in 2026: disaggregated serving, separate pods for each phase, connected by a KV cache transfer channel (NIXL over UCX, RDMA, or NCCL as a fallback). DistServe demonstrated 7.4× more request rate at the same SLO; NVIDIA Dynamo 1.0 (GA at GTC 2026) takes the pattern to production at datacenter scale. Mixing heterogeneous hardware, H100 for prefill and commodity GPUs for decode, cuts up to 48 % off the cost per token. This article explains the why, the how, and the numbers that matter for a typical on-premise infrastructure.

You are here: Deploy

Disaggregated serving is an architectural decision in the Deploy stage of the six-stage LLMOps pipeline. It does not change the model, it does not change the data, it does not change the evals. It only changes how inference pods are spread over the GPU hardware. But that change moves aggregate throughput between 2× and 7×.

The analogy: the kitchen with two brigades

A serious industrial kitchen, any kitchen serving more than 50 covers a night, runs with two distinct brigades and two physically separate spaces.

The prep brigade starts at dawn. Its job is the mise en place: chopping, marinating, blanching, boiling stocks, preparing complex components. Equipment: good knives, big burners, convection ovens, 40-litre pots. It is capacity-intensive work and it is done in one batch. When it finishes, everything is left in labelled trays ready to use.

The service brigade comes in mid-afternoon. Its job is service: taking the trays from prep, heating portions, plating, running the pass. Equipment: salamanders, small griddles, thin spatulas, plenty of crockery. It is wrist work, rhythm work, not-failing-the-customer-who-has-the-plate-in-front-of-them work. Capacity per hour matters less than latency per plate.

If you make the same person do prep and service, both suffer. The cook is idle while doing mise en place in the middle of the afternoon. They have to stop and plate up when five orders land at once. Their working kit is designed for one or the other, not for both.

Serious kitchens solved this decades ago: separate brigades, separate spaces, separate equipment. The only thing that crosses between them is the trays of mise en place.

The trays are the KV cache. The separation is disaggregated serving. The handover from prep to service is the KV cache transfer, solved today with NIXL over RDMA. And the specialised pods are the two brigades with their optimal kit.

Quick recap: prefill and decode

A request to an LLM always goes through two phases:

Prefill. Take the complete prompt (say, 4,000 tokens) and process it in a single pass through all the layers of the model. The result is the KV cache of those 4,000 tokens (see the previous article on KV cache if you want to recall exactly what it holds). This step is massively parallel: all the tokens go through the attention matrices at once, which translates into huge, dense matrix multiplications. The GPU sits at 90-95 % compute utilisation. TTFT (time to first token) is determined by this phase.

Decode. Once the KV cache is ready, the model generates tokens one by one. Each new token is a pass through all the layers with a single query vector, reading the whole accumulated KV cache to compute attention. There is no parallelism between tokens (each one depends on the previous). What limits you here is not compute but bandwidth: every step has to read the model’s full weights from HBM. The GPU sits at 20-40 % compute utilisation, but at 90 % HBM utilisation. TBT (time between tokens) is determined by this phase.

Phase	Characteristic	Bottleneck	Key metric
Prefill	Massive parallel compute over N tokens at once	TFLOPS (compute)	TTFT
Decode	Streaming weights from HBM, 1 token at a time	HBM bandwidth	TBT (inter-token latency)

The asymmetry is structural: prefill burns compute and leaves memory half idle, decode does the opposite. A GPU designed to be excellent at both at once is a GPU designed to be badly used all the time.

Why putting them on the same GPU is a bad deal

Until 2023, the universal assumption was to run prefill and decode in the same inference process, on the same GPU. The engine scheduler (vLLM, TGI, Triton) decided on each cycle whether to prefill a new request or decode the ones already in flight. The intuition was that sharing hardware saves money.

The intuition is wrong. The problem has three faces:

Latency interference. When the engine decides to prefill a new request, it interrupts every decode in progress. That pushes up the TBT of the other requests. The user who was watching tokens flow smoothly down the screen notices a stall of several hundred milliseconds. This is known as prefill-decode interference and it degrades the experience visibly as concurrency rises.

Sub-optimal hardware for each phase. An H100 SXM has 989 TFLOPS BF16 of compute and 3.35 TB/s of HBM3. It is excellent for prefill, where compute is the limit. For decode, where the only thing that matters is bandwidth, those 989 TFLOPS are 60-70 % wasted. Conversely, a GPU with less compute but similar relative bandwidth (RTX 4090, L40S) would handle decode just as well for a fraction of the price.

Low aggregate utilisation. In real workloads with Llama 3 70B and 512-token outputs, around 80 % of the wall-clock is spent in decode. That means 80 % of your H100 cluster budget is doing memory reads, not calculations. It is like paying for a Ferrari to use it in a car park queue.

The idea: specialised pods, KV cache as the deliverable

Disaggregated serving breaks the inference cycle into two distinct services:

Prefill pod. Receives the prompt, runs the prefill, produces the KV cache. Hardware: high-compute GPUs (H100, H200, B200). Optimised for aggressive batching and throughput, not for individual latency: if 32 prompts arrive within 100 ms, it processes them together.

Decode pod. Receives the already-built KV cache, runs the token-by-token generation, streams to the client. Hardware: GPUs with good bandwidth but ideally cheaper per TFLOPS (RTX 4090, L40S, A100, even A30 depending on the case). Optimised for per-token latency (low TBT).

Between the two: a KV cache transfer over the network, which can be node-local (shared memory, NVLink), intra-rack (RDMA with InfiniBand or RoCE) or cross-rack (NIXL over UCX). The cost of this transfer scales linearly with context length, and it is the economic crux of the scheme.

The transfer protocol: the economics of movement

The KV cache transferred for a Llama 3 70B with 4K of context weighs roughly 2.6 GB (80 layers × 8 KV heads × 128 dim × 4,096 tokens × 2 (K and V) × 2 bytes in BF16). Moving 2.6 GB between two GPUs is not trivial:

Channel	Effective bandwidth	Time for 2.6 GB
Intra-node NVLink (NVSwitch)	~450 GB/s	~6 ms
Shared memory (same node, PCIe 5)	~60 GB/s	~45 ms
RDMA InfiniBand 400 Gbps	~50 GB/s	~55 ms
RDMA RoCE 200 Gbps	~25 GB/s	~105 ms
TCP/IP 10 GbE	~1 GB/s	~2.6 s

The immediate reading: above InfiniBand grade, the transfer is comfortable. Below it, it ruins the very TTFT we are trying to improve. Disaggregated serving is only viable with a decent interconnect. It is not a pattern for clusters built out of consumer Ethernet switches.

NVIDIA answered this with NIXL (NVIDIA Inference Transfer Library), released in mid-2025: a library that abstracts the transport (UCX, NCCL, direct RDMA verbs, shared memory) and picks the best available path automatically. vLLM has integrated it since late 2025 through the NixlConnector. It is now the de facto default for new deployments.

Real implementations in May 2026

The pattern’s path over two years:

2024 Jan · DistServe (HKU + UCSD): 7.4× requests at the same SLO
2024 May · SplitWise (Microsoft): variant with heterogeneous hardware
2024 Dec · experimental vLLM disagg (SharedStorage + PyNcclConnector)
2025 Mar · NIXL release (NVIDIA): unified transfer library
2025 Jul · stable vLLM NixlConnector
2025 Nov · SGLang, llm-d, MoonCake adopt the pattern
2026 Mar · NVIDIA Dynamo 1.0 GA (GTC 2026): production-ready at datacenter scale

As of today, the pattern is the default in any serious serving framework. The ones still monolithic are the small ones or the educational ones.

Three realistic options for an on-premise infrastructure:

vLLM disagg with NixlConnector. The most open route, requiring two sets of vLLM pods to be deployed (one with --kv-transfer-config '{"kv_role":"producer"}', another with "kv_role":"consumer") plus a proxy router. Enough for clusters of 4-16 GPUs.
SGLang with disagg. Conceptually equivalent, better performance on some MoE workloads.
NVIDIA Dynamo 1.0. The one taking over at datacenter scale. It covers routing, KV cache management, monitoring and scheduling in a single control plane. Heavier, but the reference solution if your cluster grows beyond 32 GPUs.

The numbers that matter

What disaggregation unlocks, in direct terms:

Metric	Aggregated (monolithic)	Disaggregated	Improvement
Goodput (req/s at the SLO)	baseline	1.4 – 2×	up to 2×
TTFT under high load	rises sharply from QPS 4	stable up to QPS 7+	~2×
Request rate at the same SLO (DistServe paper)	baseline	7.4×	7.4×
MoE throughput on Blackwell (Dynamo, GB300 NVL72)	baseline (Hopper)	up to 50×	depends on the model
Cost per token (heterogeneous H100 + commodity)	baseline (all H100)	-48 %	almost half

These numbers have to be read carefully: the most spectacular ones (7× and 50×) need specific hardware (Blackwell GB200/GB300 NVL72) and specific models (large MoE). The realistic range for a typical on-premise setup is 1.4-2× in goodput and -30 to -50 % in cost per token, depending on how heterogeneous the GPU mix is and how optimised the KV cache transfer is.

Heterogeneity: the radical version

The logical next step, proposed by SplitWise in 2024 and matured in 2025-2026 (Cronus, Tessera and others), is to mix GPU types: expensive high-compute GPUs for prefill, commodity GPUs with good bandwidth for decode.

Indicative cost (typical market prices in mid-2026):

H100 SXM: ~30-40 kUSD capex, ~3-4 USD/h amortised. Compute-heavy profile.
L40S: ~8-10 kUSD capex, ~1.5 USD/h. Intermediate profile, 864 GB/s of bandwidth.
RTX 4090: ~1.5 kUSD capex, ~0.30 USD/h. Modest compute profile but 1 TB/s of GDDR6X bandwidth, enough for decode of models up to ~30B parameters.

A realistic mixed cluster for serving an 8B model:

2× RTX 4090 (prefill batch) → ~3,000 USD capex, ~0.60 USD/h
4× RTX 4090 (decode pool) → ~6,000 USD capex, ~1.20 USD/h
TOTAL → ~9,000 USD capex, ~1.80 USD/h

Against the monolithic alternative with equivalent throughput:

2× H100 SXM (all in one) → ~70,000 USD capex, ~7 USD/h

The same throughput at a fraction of the capex and a quarter of the hourly cost, at the price of operational complexity: you now have two pools to coordinate, a transfer network to look after, and a scheduler that is not trivial.

For larger models (Llama 3 70B), the decode pool no longer fits on a single 4090 (the model does not fit in 24 GB, not even quantised to INT4 with headroom). There the sensible mix is H100 for prefill plus L40S or A100 80GB for decode, with a typical saving of 30-40 % over the all-H100 option.

Applied to typical on-premise hardware

Case 1 — One or two RTX 4090s: monolithic still wins

With a single GPU there is no disaggregation to speak of: the pattern needs at least two GPUs in separate pods. With two 4090s you can technically try it (one for prefill, one for decode with the KV cache transferred over PCIe 5 or basic RDMA), but the transfer overhead eats the gain for small models where prefill is already fast.

Recommendation: stay monolithic (traditional vLLM, properly configured with a quantised KV cache). The next justifiable level of complexity is a cluster with a fast interconnect.

Case 2 — 4×H100 SXM cluster (320 GB, NVLink): the sweet spot

The minimum realistic configuration for serious disaggregation, serving a 70B model in production:

2× H100 (TP=2) → 2 prefill pods
2× H100 (TP=2) → decode pods with several instances sharing TP
NIXL over NVLink → KV cache transfer <6 ms
Router (vLLM or Dynamo) → prompt distribution and streaming

Realistic expected result: goodput 1.6-1.9× relative to the same cluster running monolithic, with TTFT stable up to loads of QPS 7-8 (against the QPS 4 at which the monolithic setup starts to degrade).

If the heterogeneous mix is possible (adding 4-8 L40S to the cluster to make the decode pool), the cost per token drops a further 25 % to 35 %, while still serving the 70B model whole.

Position within the architecture

Disaggregated serving is a cross-cutting layer over almost everything discussed in previous articles. It touches:

The KV cache because it is the artefact transferred between pods. Without a solid grasp of how much the cache weighs and how it grows with context, you cannot size the transfer.
Continuous fine-tuning because multi-LoRA hot-swap keeps its semantics: each pod (prefill or decode) loads the adapters separately, and the router decides which adapter to apply in each phase.
Cluster topology: it changes the recommended HW, the networking required and the cost model.

If you are designing an inference infrastructure for 2026 from scratch, disaggregation stops being optional for any cluster beyond 4 GPUs of capacity. If you are modernising an existing one, it is the upgrade with the best return per euro invested, provided the networking between pods is decent (intra-node NVLink or intra-rack RDMA as a minimum).

What we have not covered (upcoming articles)

NIXL in detail: how it picks the optimal transport, how UCX is configured, what happens when RDMA fails and you have to fall back to TCP.
Routing scheduler: how the orchestrator decides which pod gets which request, dynamic batching, priority handling.
Multi-tenant disagg: KV cache isolation between tenants, per-adapter ACLs, multi-LoRA over specialised pods.
Disagg + prefix caching: how it combines with the KV cache reuse pattern when several prompts share a prefix (a common system prompt).
Disagg at the edge / local inference: viability on home hardware (4090 + Mac Studio, for example), where the transfer depends on Thunderbolt or residential Ethernet.

References

Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (OSDI 2024).
Patel et al., SplitWise: Efficient Generative LLM Inference Using Phase Splitting (ISCA 2024).
NVIDIA, NVIDIA Dynamo 1.0: Production-Ready Disaggregated Inference (GTC 2026, March): https://developer.nvidia.com/blog/nvidia-dynamo-1-production-ready/.
NVIDIA, NIXL: NVIDIA Inference Transfer Library — official documentation.
vLLM, Disaggregated Prefilling: https://docs.vllm.ai/en/stable/features/disagg_prefill/.
vLLM, NixlConnector Usage Guide: https://docs.vllm.ai/en/stable/features/nixl_connector_usage/.
Hao AI Lab, Disaggregated Inference: 18 Months Later (UCSD, 2025) — technical retrospective on the DistServe paper.

Decode on lo0 — Tech Blog

The signature-dish specialist: vLLM's attention backend (FlashAttention, FlashInfer and the prefill/decode asymmetry)

TL;DR

Where you are: the specialist, not the commis

Why prefill and decode are opposite problems

The scheduler’s trick: prefill and decode in the same forward

How the backend reads the paged KV

The backends and how the engine chooses

What FlashInfer brings: JIT and cascade attention

The arithmetic that matters: when switching backend gives you something

The 10 knobs

How it connects to the rest of the stack

Traps and things that are not what they look like

Conclusion

See also

References

The chef who calls out every order: SMs, CUDA streams and CUDA graphs, or why the GPU gets bored generating tokens

TL;DR

Where you are: the silicon, below the kernel launch

The analogy: the chef who calls out every order

The mechanism: SM, warps and occupancy

The mechanism: streams, the order queue

The maths that matters: when the GPU is left waiting

vLLM’s CUDA graphs, specifically

The 10 knobs worth touching

Knob 1 — Measure whether decode is launch-bound

Knob 2 — Do not use --enforce-eager in production

Knob 3 — Capture buckets (cudagraph_capture_sizes)

Knob 4 — CUDA graph mode

Knob 5 — torch.compile

Knob 6 — Batch size: fill the hobs

Knob 7 — Do not break stream overlap

Knob 8 — Persistence mode + locked clocks

Knob 9 — Fused kernels (FlashAttention, FP8 kernels)

Knob 10 — Accept the capture cost in the cold start

Summary table

How this connects with the rest of the stack

Traps and things that are not what they look like

Conclusion

See also

References

Optimising decode in vLLM: squeezing every token out of small hardware

TL;DR

The analogy

Why decode is memory-bound

The five levers

1. Give vLLM all the VRAM you can — --gpu-memory-utilization

2. Real concurrency — --max-num-seqs

3. Speculative decoding — --speculative-model + --num-speculative-tokens

4. Quantised KV cache — --kv-cache-dtype fp8

5. Remove the swap — --swap-space 0

The reference configuration per hardware

RTX 4090 (24 GB) — 7B model, internal use

L40 (48 GB) — 14B model, multi-user

How to measure that decode is optimised

Implications for on-premise inference

See also

References

Disaggregated serving: prefill and decode in specialised pods

TL;DR

You are here: Deploy

The analogy: the kitchen with two brigades

Quick recap: prefill and decode

Why putting them on the same GPU is a bad deal

The idea: specialised pods, KV cache as the deliverable

The transfer protocol: the economics of movement

Real implementations in May 2026

The numbers that matter

Heterogeneity: the radical version

Applied to typical on-premise hardware

Case 1 — One or two RTX 4090s: monolithic still wins

Case 2 — 4×H100 SXM cluster (320 GB, NVLink): the sweet spot

Position within the architecture

What we have not covered (upcoming articles)

See also

References

Knob 2 — Do not use `--enforce-eager` in production

Knob 3 — Capture buckets (`cudagraph_capture_sizes`)

Knob 5 — `torch.compile`

1. Give vLLM all the VRAM you can — `--gpu-memory-utilization`

2. Real concurrency — `--max-num-seqs`

3. Speculative decoding — `--speculative-model` + `--num-speculative-tokens`

4. Quantised KV cache — `--kv-cache-dtype fp8`

5. Remove the swap — `--swap-space 0`