Prefill on lo0 — Tech Blog

Aggressive RAG in small models: trading parameters for retrieval

Tue, 09 Jun 2026 02:20:00 +0000

This post belongs to the series on inference performance in small models. Its sibling piece, The roofline inverts in small models, explains why the compute-bound prefill is the bottleneck that shapes the whole discussion here. It is worth reading first: here we assume that adding more context is not free.

TL;DR

An SLM (say 1B–8B parameters) knows fewer facts than a 70B–700B model, simply because it has fewer weights in which to memorise them. But its ability to reason over text in front of it, following instructions, extracting, synthesising, comparing, degrades far less with size than its encyclopaedic knowledge does. The operational consequence is direct: use the SLM as a reasoning engine over curated context, not as a database. Move knowledge from the weights into the context via retrieval. The problem is that “aggressive retrieval” is often read as “put in lots of chunks”, and that collides head-on with three facts about SLMs: shorter context windows, worse use of long context (the lost in the middle effect is more severe the smaller the model) and a compute-bound prefill whose cost grows with the context length $C$, linearly in the projections and quadratically in attention. You cannot simply add tokens. Each retrieved token is paid for twice, in degraded quality and in TTFT, and the SLM is the worst placed to absorb either cost. The way out is not to retrieve less, but to retrieve better: reranking for precision over recall, compressing the context before injecting it, prefix caching the stable documents, semantic caching of answers, and structured output with external tools that stand in for internal knowledge. This post works through the maths and gives a TTFT number before and after compressing a context from 4,000 to 1,000 tokens on an RTX 4090.

The analogy: the open-book exam

Two students sit the same exam. The first has a prodigious memory: they have memorised the entire syllabus, paragraph by paragraph. The second has an ordinary memory, forgetting dates, confusing names, but is allowed to bring in a crib sheet.

If the second student’s crib sheet is a chaos of piled-up photocopies, they lose: they take too long to find what they are after, get distracted by irrelevant pages and run out of time. But if their crib sheet is excellent, trimmed to the essentials, reordered by relevance, with the important things at the top and no filler, then they not only do not lose: they often win, because they reason just as well as the first student and, on top of that, work from verified material instead of from blurry recollections they may be inventing.

The moral has three layers, and each maps to an engineering decision:

Memorising everything is expensive. The first student invested months. A large model invests parameters, and VRAM, and inference FLOPs, in memorising facts.
The crib sheet matters more than its size. A well-made one-page crib sheet beats ten badly organised pages. More retrieved context is not better context: the material’s precision beats volume.
Knowing how to search and synthesise is a different skill from knowing. It is the one the SLM retains. The whole strategy consists of leaning on that skill and subcontracting the memory.

The rest of the post is, essentially, how to build an excellent crib sheet under the constraint that the student (the SLM) reads slowly and gets tired with long texts.

The capacity argument: how many facts fit in the weights

Let us start by justifying the thesis with an order of magnitude, not with faith. How much factual knowledge really fits in a model’s weights?

There is a recurring empirical estimate in the interpretability and memorisation literature: a dense model can store on the order of 2 bits of memorised information per parameter before saturating (the exact figure varies by study and training regime; take it as an order of magnitude, not as a law). An 8B-parameter model then has an information storage ceiling on the order of:

$$8 \times 10^9 \text{ params} \times 2 \text{ bits/param} = 1.6 \times 10^{10} \text{ bits} \approx 2 \text{ GB of information}$$

And that budget is not only for facts: the vast majority goes on grammar, syntax, reasoning ability, code, formatting, and only a fraction is left for encyclopaedic knowledge. Compare it with the other side: a retrievable corpus of several million documents, a corporate wiki, a document repository, a technical knowledge base, easily takes up hundreds of GB to terabytes of text, indexed and queryable with millisecond latency. The asymmetry is two or three orders of magnitude in favour of the external corpus.

The conclusion is not that the weights are useless, they are where reasoning lives, which is the expensive thing to replicate, but that competing with an external index on fact capacity is losing by construction. A 70B model has ~9× more memorisation budget than an 8B one, but it is still negligible against the corpus. That is why the large model also does RAG in production. The difference is that the SLM needs it: without retrieval, its factual knowledge is too sparse and, worse, prone to hallucinating exactly in the gaps it did not memorise.

The central tension: retrieving more is not putting in more

This is where most naive designs break. “Aggressive retrieval” sounds like a large top-k: if retrieval helps, retrieve 20 chunks instead of 5. But in an SLM that fails for two independent reasons, one of quality and one of cost.

(a) SLMs make worse use of long context

The lost in the middle effect (Liu et al., 2023) is well known: LLMs retrieve information placed at the beginning and end of the context better, and the information in the middle worse. What gets less emphasis is that the effect is more severe the smaller the model. An SLM has fewer attention heads, fewer layers and poorer internal representations for “tracking” a relevant fact buried at position 14 out of 20 chunks. Its nominal context window also tends to be shorter (4K–32K against the 128K+ of the large ones), and the effective window, the length beyond which quality collapses, is smaller still. Putting in 20 chunks does not mean the model reads all 20: it means it will probably ignore or misread the ones in the middle, while paying the cost of all of them.

(b) Prefill grows with the context and is compute-bound

This is the blow people underestimate. Prefill, processing the full prompt before emitting the first token, is the compute-bound phase of inference (unlike decode, which is memory-bound; the detail lives in The roofline inverts). Its cost grows with the context length $C$, and it determines TTFT (time to first token). More chunks → more prefill tokens → more TTFT and more compute cost per request. In an SLM, where prefill is proportionally more expensive relative to the model, this hurts especially.

The operational conclusion is uncomfortable but clear: you cannot compensate for fewer parameters simply by putting in more context. Each retrieved token is paid for twice, in degraded quality and in TTFT, and the SLM is the worst placed to absorb both costs. The way out is to retrieve less but better, and to compress what you retrieve.

The maths of prefill

Let us put numbers on “prefill grows with the context”. For a context of $C$ tokens, a transformer layer does two classes of work:

Linear projections (QKV, attention output, FFN): each token is multiplied by fixed-size weight matrices. The cost is $O(C)$ in FLOPs, linear in the number of tokens.
Attention ($QK^\top$ and the multiplication by $V$): each token attends to all the others. The cost is $O(C^2)$, quadratic in the number of tokens.

The total prefill cost per layer has the form:

$$\text{FLOPs}_{\text{prefill}} \approx \underbrace{a \cdot C}_{\text{projections}} + \underbrace{b \cdot C^2}_{\text{attention}}$$

with $a$ and $b$ constants that depend on the model dimension. For moderate contexts (a few thousand tokens) in an SLM, the linear term still dominates or is comparable to the quadratic one; the quadratic term becomes dominant at long contexts. What matters: if you compress the context $C \to C/k$, the linear term falls $\times k$ and the quadratic one falls $\times k^2$. Compression is the only lever that attacks both terms at once, and it attacks the worse one disproportionately.

Numerical example: TTFT before and after compressing, RTX 4090

Let us model TTFT as the time to process the prefill tokens at a given prefill throughput. Take an RTX 4090 (24 GB, Ada Lovelace) serving a quantised SLM, with a prefill throughput of ~5,000 tok/s (an illustrative figure; the real value depends on the model, the quantisation and the batch, so measure it, do not assume it).

Let there be a retrieved context of 4,000 tokens (8 chunks of ~500 tokens). Approximating TTFT as dominated by the context prefill:

$$\text{TTFT}_{\text{before}} \approx \frac{4000 \text{ tok}}{5000 \text{ tok/s}} = 0.80 \text{ s}$$

Now we compress that context to 1,000 tokens ($k = 4$). Prefill throughput is not constant with $C$, it drops a little at long contexts because of the quadratic term, but taking the conservative linear tokens/throughput approximation:

$$\text{TTFT}_{\text{after}} \approx \frac{1000 \text{ tok}}{5000 \text{ tok/s}} = 0.20 \text{ s}$$

TTFT falls from 0.80 s to 0.20 s, a $4\times$ reduction in the linear part. But the FLOPs arithmetic is even more favourable in the attention component: that part of the work falls $\sim k^2 = 16\times$. In practice total TTFT does not fall 16× because the cost is not purely quadratic at this scale, but the real reduction lies between 4× and something larger depending on how much attention weighed, and the aggregate compute saving (what pays the electricity bill and frees the GPU for another request) is substantially bigger than the plain 4× of the token count.

The argument generalises: compressing the context by a factor $k$ reduces TTFT by at least $\sim k\times$ and the attention cost by $\sim k^2\times$. For an SLM, where TTFT is often the SLA that matters, this is the difference between an assistant that answers instantly and one that feels slow.

The five levers to resolve the tension

The strategy is not “retrieve less and settle”. It is retrieve aggressively from the index and then distil aggressively what you retrieved before it reaches the SLM. Five levers, in order of application within the pipeline.

1. Aggressive reranking: precision over recall

The initial retriever (dense, sparse or hybrid) optimises recall: it brings back 50–100 candidates so as not to leave anything out. The reranker, a cross-encoder that sees the query and the document together, optimises precision: it reorders those candidates and you keep the 3–5 best. For an SLM this is not a luxury, it is structural: since the model uses long context badly, every chunk that goes in must earn its place. Better 4 chunks of very high relevance than 15 mediocre ones. The detail of hybrid retrieval and reranking is in Reranking and hybrid retrieval; here the rule is enough: maximise recall in the retriever, maximise precision in the reranker, and inject few.

2. Context compression: distilling the crib sheet

Once you have the best chunks, they still contain filler: padding sentences, redundancy, context irrelevant to the specific query. Context compression trims them before injection:

Extractive compression (LLMLingua / LongLLMLingua style, Jiang et al. 2023): a small model scores the perplexity or relevance of each token/sentence with respect to the query and removes the low-information ones, keeping the densest extractive subset. It reduces tokens without a second large generative model in the loop. LongLLMLingua adds position-aware reordering to mitigate lost in the middle.
Abstractive compression: a model summarises the retrieved chunks into shorter text. More aggressive in token reduction, but it introduces a generative step (cost and possible loss of fidelity).
Soft prompts / context distillation: compressing the retrieved context not into text, but into a handful of embeddings/soft tokens that the model consumes directly. It reduces the number of prefill tokens to a minimum, at the price of a trained, model-specific component.

The key point connects with the maths above: compressing what you retrieved by a factor $k$ reduces prefill tokens $\times k$, and therefore TTFT $\sim\times k$ and the attention cost $\sim\times k^2$. It is the lever with the best return when long context is the bottleneck.

3. Prefix caching of stable context

Not all the context changes between requests. System instructions, definitions, recurring reference documents, schemas: these are stable prefixes. Prefix caching stores the already-computed KV cache of those prefixes and reuses it, so that prefill only processes the new part (the query and the specific chunks). If 60 % of your context is stable, you save 60 % of that segment’s prefill on every hit. For it to work, the stable context must go at the start of the prompt (the KV cache is prefix-dependent) and it pays to maximise the hit rate; the hit-rate engineering detail is in Prefix cache hit rate. It combines especially well with RAG: retrieved documents that repeat across sessions get cached once.

4. Semantic caching of answers

A layer in front of the model: if a query is semantically equivalent to one answered before (embedding similarity above a threshold), return the cached answer and skip the model entirely, retrieval, prefill and decode included. In real workloads with long tails of repeated or near-repeated questions (FAQ, support), the saving is enormous because it eliminates the full cost, not just the prefill one. The trap is the threshold: too lax and you serve wrong answers to similar-but-different questions. The design is in Semantic caching for RAG.

5. Structured output and function calling: lean on tools, not on memory

The last lever changes what the SLM depends on. Instead of asking it to know a fact (its weak point), make it call a tool that knows it: a database query, an API, a calculator, a validator. Structured output (forcing JSON conforming to a schema) and function calling turn the SLM into an orchestrator that extracts arguments from the context and delegates the computation or the query. A reasonably capable SLM emits a well-formed tool call far more reliably than it recalls a specific fact. This reduces the pressure on parametric knowledge and on retrieval: for structured, fresh data (prices, inventory, states), querying beats retrieving text and beats memorising. The fundamentals are in Structured output and Function calling.

The complete pipeline

The five levers are not alternatives: they chain together. The flow, with the token counter falling at each step:

The order matters. Retrieving aggressively (high recall) before filtering guarantees that the right material is among the candidates; reranking and compressing afterwards guarantees that only the dense and relevant material pays the prefill toll; caching wraps everything so as not to repeat work. The SLM only sees the final crib sheet, short and ordered.

Implications for on-premise inference

The mental trap to avoid: treating the SLM as a large model with less quality. It is not. It is a different cost profile that rewards a different design. Three practical consequences:

The token budget is a first-class resource. With a large model with a 128K window, “putting in a bit more” is cheap relative to the model. With an SLM, every context token shows up in TTFT and in quality. Treat context size as a quantity to minimise subject to covering the answer, not to maximise.
The investment is worth it precisely because the model is cheap. Reranker, compressor and caches add complexity, but the model they serve is economical enough to run many replicas. The bottleneck shifts from the model to the data pipeline, which is exactly where you want it.
Retrieving does not replace adapting; they combine. For deep, recurring domain knowledge, adapting the SLM with LoRA (see the sibling QLoRA and aggressive multi-LoRA) can put part of the knowledge “into the weights” cheaply, reducing what has to be retrieved. Aggressive RAG and aggressive adaptation do not compete: the first gives freshness and citability, the second gives fluency and domain formatting. A good design uses both.

On the RTX 4090 (24 GB, Ada Lovelace)

The canonical scenario: a quantised SLM (4B–8B in INT4/FP8) fits with room to spare, leaving VRAM for a generous KV cache, essential for prefix caching, and for the reranker (a cross-encoder of a few hundred MB). The LLMLingua-style extractive compressor runs on a separate small model or on CPU. The TTFT calculation above (0.80 s → 0.20 s compressing 4× at ~5,000 tok/s) is representative of this card. The rule of thumb: if TTFT goes above your SLA, the first adjustment is to compress the context, not to change the model.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

With 320 GB and native FP8 prefill is much faster, so the temptation is to relax the token discipline. That is not quite advisable: the lever changes from TTFT to aggregate throughput. Compressing the context not only speeds up each request but frees prefill compute to serve more requests per GPU; compute-bound prefill is exactly the resource that saturates first under load. Here prefix caching and semantic caching, shared across replicas, are what pay off most: at high QPS, the prefill work you avoid by caching is pure throughput you gain. The SLM is still the cheap reasoning engine; the difference is that you now run many in parallel and the data pipeline is what decides how many requests fit.

What we have not covered

Evaluating compression: how to measure that compressing $k=4$ does not throw away correct answers (faithfulness, answer recall over a set of questions with ground truth).
Query-aware versus query-agnostic compression: compressing before or after knowing the question changes what can be cached and what can be discarded.
Chunking and granularity: chunk size interacts with reranking and compression; that is left for the corpus curation post.
Multi-hop and agents: when a question requires several retrieval rounds, the token budget is split across hops and compression discipline becomes critical.

References

Lewis, P., et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020. https://arxiv.org/abs/2005.11401
Liu, N.F., et al. Lost in the Middle: How Language Models Use Long Contexts. TACL 2024. https://arxiv.org/abs/2307.03172
Jiang, H., et al. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. EMNLP 2023. https://arxiv.org/abs/2310.05736
Jiang, H., et al. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression. ACL 2024. https://arxiv.org/abs/2310.06839

The signature-dish specialist: vLLM's attention backend (FlashAttention, FlashInfer and the prefill/decode asymmetry)

Mon, 08 Jun 2026 05:40:00 +0200

Part of the under the engine series. The PagedAttention post explained where the KV lives (in paged blocks). This one explains who reads it and how: the attention kernel. And it connects with FlashAttention v1-v4, which took that kernel apart from the inside; here we look at the level above, how vLLM chooses between several kernels and why it needs more than one.

TL;DR

An LLM forward is, for the most part, standard matrix multiplications that any library does well. The exception that decides performance is attention, and having one good kernel is not enough: you need two, because the two phases of inference are physically opposite problems. Prefill processes the whole prompt: many queries against many keys, dense and compute-bound, the home ground of FlashAttention’s IO-aware tiling. Decode generates one token: a single query against all the accumulated KV, thin and memory-bound, where the only thing that matters is saturating HBM bandwidth while reading the paged KV. That is why vLLM does not have “the attention kernel” but a switchable backend (FLASH_ATTN, FLASHINFER, TRITON_ATTN and others) plus logic that picks according to the GPU: by default FA4 on Blackwell (SM100), FA3 on Hopper (SM90), FA2 elsewhere, with FlashInfer as the alternative that compiles bespoke kernels (JIT) and knows how to do cascade attention for shared prefixes. This post explains why prefill and decode are opposites (with arithmetic intensity), how the backend reads paged KV, how the engine chooses, what FlashInfer brings, the 10 knobs and the trap of pinning a backend blindly. On the generic 4×H100 SXM cluster.

Where you are: the specialist, not the commis

In the kitchen, almost all the work is chopping and sautéing: standard operations any competent commis chef executes, the matrix multiplications of the feed-forward layers and the projections. There is one dish that is not delegated: the signature dish, the one that defines the restaurant. That dish is attention, and it has a quirk: it is cooked in two radically different ways depending on the moment of service.

During prefill, when a new order arrives with its whole prompt, you cook big: a lot of raw material at once, a lot of fire, an intense operation that fills the stoves. During decode, when a table asks for “one more dish”, you cook à la carte: a single dish, but you have to go to the larder and bring back every ingredient that table has accumulated over the whole meal. One is a problem of firepower; the other, of larder speed. The same specialist does not do both well. That is why vLLM has several, and a chef who decides which one steps up depending on the GPU and the phase. That is the attention backend.

Why prefill and decode are opposite problems

This is the central idea, and one sum proves it: arithmetic intensity (FLOPs per byte read). An operation with high intensity is limited by compute; one with low intensity, by memory.

Prefill. We serve $N$ queries (the whole prompt) against $N$ keys. The $QK^\top$ operation and the $\text{softmax}\cdot V$ do on the order of $N^2 d$ FLOPs and read on the order of $N d$ of data. Intensity grows with $N$:

$$I_\text{prefill} \sim \frac{N^2 d}{N d} = N$$

With large $N$ (a prompt of thousands of tokens), intensity is high: compute-bound. This is where FlashAttention’s tiling squeezes the tensor cores and where you get close to the GPU’s peak TFLOPS.

Decode. We serve one query (the new token) against $L$ keys (all the accumulated KV). FLOPs on the order of $L d$; bytes read on the order of $L d s$ (the entire KV has to be read from HBM). Intensity is:

$$I_\text{decode} \sim \frac{L d}{L d s} = \frac{1}{s} \quad (\approx 0.5 \text{ FLOP/byte in FP16})$$

Constant and tiny: memory-bound. The decode kernel is not limited by how much the GPU can compute but by how fast it reads the KV from HBM. It makes no difference that the H100 has 132 idle SMs (see the SM post): the bottleneck is the 3.35 TB/s of bandwidth, and the decode kernel exists so as not to waste a single one of those bytes/s.

The design consequence: a kernel optimised for prefill (dense tiling, maximum tensor-core occupancy) is not the optimum for decode (coalesced reads of the paged KV, minimum latency). Serious servers have different kernels, or one kernel with two paths. In models with MLA (multi-head latent attention), vLLM goes as far as using separate backends for prefill and decode, selectable independently (attention backends, vLLM).

The scheduler’s trick: prefill and decode in the same forward

Here the circle closes with the scheduler post. Because vLLM V1 mixes requests in prefill and in decode in every step, one single forward has to serve both. The backend receives metadata telling it, for each sequence in the batch, how many queries it brings and how much KV it has to read, and applies the right path to each one. That is why the attention backend and the scheduler are coupled: the first has to digest the heterogeneous batch the second builds.

How the backend reads the paged KV

The kernel does not receive a contiguous KV tensor: it receives the block table from the block manager and does a gather over the physical blocks. This imposes a real constraint on the backend: it has to support the paged layout and vLLM’s block_size. Not every kernel in the world does; the ones vLLM integrates (FlashAttention, FlashInfer, Triton) are adapted to read KV in fixed-size blocks scattered across HBM. That is why you cannot plug in any attention kernel from a paper: it has to speak the language of the pigeonhole larder.

The backends and how the engine chooses

vLLM exposes a backend abstraction with several implementations (vLLM deepwiki):

FLASH_ATTN — the FlashAttention family. By default the version is picked by architecture: FA4 on SM100 (Blackwell), FA3 on SM90 (Hopper), FA2 elsewhere, configurable with flash_attn_version.
FLASHINFER — an attention engine with JIT compilation and specialisable kernels; strong on heterogeneous KV and shared prefixes.
TRITON_ATTN — written in Triton, portable and with no dependency on precompiled CUDA binaries (Triton backend deep dive, vLLM, mar-2026).
Specific backends for MLA and for non-NVIDIA hardware.

Selection is automatic unless you force it with VLLM_ATTENTION_BACKEND. The heuristic tries FlashAttention first; on Blackwell (SM100) the fallback order for MLA is TRT-LLM Ragged → FlashInfer → others; on other GPUs only FlashAttention is considered for the main path (attention backends, vLLM). The decision depends on: architecture (SM), dtype (FP16/BF16/FP8), head dimension, and whether the load needs a feature only one backend has (cascade attention, certain soft caps, FP8 in KV).

What FlashInfer brings: JIT and cascade attention

FlashInfer does not compete with FlashAttention on “being a bit faster”; it attacks a different problem: the heterogeneity of the KV in real serving (FlashInfer, arXiv 2501.01005). Two ideas:

JIT compilation. Instead of a monolithic kernel, FlashInfer generates bespoke kernels for the attention variant, the problem shape and the KV layout you have, injecting functors (query/key/logit transformations, masks). It specialises rather than generalises.

Cascade attention. Here is the jewel for serving with shared prefixes. If $R$ requests share a prefix of $P$ tokens (a common system prompt), naive attention would read that prefix $R$ times. Cascade attention computes it once against the shared prefix and then combines with each request’s own suffix:

$$\text{reads: } \underbrace{R \cdot (P + s_i)}_{\text{naive}} \;\longrightarrow\; \underbrace{P + \textstyle\sum_i s_i}_{\text{cascade}}$$

With $R=50$ requests and a prefix $P=1000$, that is reading 50,000 prefix tokens versus 1,000. It is the natural complement to prefix caching: the block manager shares the prefix’s memory, and cascade attention shares the compute of attending over it.

The arithmetic that matters: when switching backend gives you something

The backend only moves the needle where attention is the bottleneck. In memory-bound decode, a kernel that exploits HBM bandwidth better gives a real improvement; in compute-bound prefill with long sequences, FA3/FA4 getting close to the tensor-core peak gives a real improvement. But if your bottleneck is in another layer, launch overhead, a badly sized scheduler, the cold start, then switching backend does not touch that part. The rule, once again: measure the regime before optimising.

The 10 knobs

#	Knob	What it controls	Cost / risk
1	`VLLM_ATTENTION_BACKEND`	force a backend	mismatch with hardware/feature
2	`flash_attn_version` (2/3/4)	FA version	version unsupported on your SM
3	enable FlashInfer	JIT + cascade	initial JIT compilation time
4	cascade attention	prefix compute reuse	only helps with a heavily shared prefix
5	`kv_cache_dtype` (FP8)	FP8 support in the kernel	not every backend/SM supports it
6	`block_size`	layout the kernel must read	consistency with PagedAttention
7	MLA prefill backend	kernel for the dense phase	MLA models only
8	MLA decode backend	kernel for the thin phase	MLA models only
9	soft cap / sliding window	features that restrict backends	fewer kernel options
10	head_dim / variant	which kernels are eligible	exotic models with no support

How it connects to the rest of the stack

With FlashAttention. The FA post explains the kernel from the inside (tiling, online softmax, FA1-4); this is the level above, how vLLM chooses between kernels and why it needs more than one.

With PagedAttention. The backend reads the KV that the block manager places in blocks; it has to speak block-table.

With the scheduler. The scheduler builds mixed prefill+decode batches; the backend has to serve both regimes in a single forward.

With CUDA graphs. Attention kernels are captured in the CUDA graphs; a backend that launches many small kernels benefits more from capture.

With prefix caching. Cascade attention is the compute side of what prefix caching does in memory.

With FP8. Attending over KV in FP8 requires the backend to have the FP8 path; not all of them do on every architecture.

Traps and things that are not what they look like

“FlashInfer is always faster than FlashAttention.” No. FlashInfer wins when its specialisation (cascade, heterogeneous KV, a particular attention variant) applies to your load; in classic dense prefill, FA3/FA4 usually does as well or better. It depends on the regime, there is no universal winner.

“A good attention kernel works for everything.” The underlying error of this post. Prefill and decode are compute-bound and memory-bound respectively; a kernel tuned for one wastes effort on the other. That is why there are separate paths (and separate backends in MLA).

“Decode is compute-bound because the GPU is at 100%.” nvidia-smi at 100% is misleading (see the SM post): decode is memory-bound, the GPU is moving KV, not computing. Optimising decode’s compute is polishing what is not the bottleneck.

“I pin VLLM_ATTENTION_BACKEND and forget about it.” Pinning a backend by hand can leave you on a suboptimal one when you change GPU or version, or force a slow fallback if your hardware does not support what you asked for. Autoselection usually gets it right; pin it only with a measurement that justifies it.

“Cascade attention always helps.” Only with a heavily shared prefix across many concurrent requests. If each request has its own context, there is nothing to share and the overhead of organising the cascade is not amortised.

“The attention backend is the bottleneck, that is why I am slow.” Almost always the bottleneck is higher up (launch, scheduling, memory) or lower down (bandwidth). The backend matters where attention dominates; measure it with nsys/DCGM before changing it.

Conclusion

Of everything an LLM does while generating text, almost all of it is matrix multiplications that any library resolves. Performance is decided in a single kernel, attention, and the surprise is that it is not even one kernel: it is two opposite problems wearing the same name. Prefill wants fire, dense compute over thousands of tokens, and decode wants a fast larder, reading a token’s entire KV with the minimum waste of bandwidth. That is why vLLM did not pick a winning kernel but an abstraction that switches: FlashAttention tuned to each architecture for the general case, FlashInfer compiling bespoke kernels when there is heterogeneity or prefixes to share, Triton for portability. The head chef does not cook the signature dish just one way: he looks at who is ordering and at what point of service, and sends out the right specialist. The lesson for whoever is tuning is the usual one in this series: before changing specialist, make sure the signature dish really is what is holding you back.

References

vLLM, Attention Backends (selección, FA2/3/4 por arquitectura, MLA): https://docs.vllm.ai/en/latest/design/attention_backends/.
vLLM / DeepWiki, FlashAttention and FlashInfer: https://deepwiki.com/vllm-project/vllm/8.2-flashattention-and-flashinfer.
vLLM, Triton Attention Backend Deep Dive (mar-2026): https://vllm.ai/blog/2026-03-04-vllm-triton-backend-deep-dive.
Z. Ye et al., FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving (arXiv 2501.01005): https://arxiv.org/abs/2501.01005.
T. Dao, FlashAttention-2 / FlashAttention-3 (kernel IO-aware, async Hopper): https://github.com/Dao-AILab/flash-attention.

Optimising prefill in vLLM: the knobs your TTFT will not forgive

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

Prefill is the phase in which vLLM processes your input prompt and produces the initial KV cache. It is compute-bound (unlike decode, which is memory-bound), it takes longer the longer the prompt is, and it blocks the decode of every other queued request. There are four levers in vLLM that radically change its behaviour: chunked prefill, prefix caching, FP8 KV cache and the per-batch token budget. With modest hardware, a 24 GB RTX 4090 or a 48 GB L40, the difference between ignoring them and using them well can be a 3× lower TTFT and 40% more aggregate throughput.

The analogy

Picture a printing shop from the early twentieth century. Setting the lead type (preparing the forme) is slow and blocks the press. Printing the pages already set is fast, but it needs the forme ready before it can start.

Prefill is setting the type. Decode is printing. A press that can only do one thing at a time, either set or print, leaves the machinery idle half the time. The historical solution was to have one worker setting the next page while the previous one was already on the press. That is, exactly, chunked prefill.

What prefill really is

When a request arrives at vLLM, the engine has to process every token of the prompt at once before it can emit the first token of the response. During that processing it computes, for each prompt token, its attention Key and Value vectors. The result, the initial KV cache, is stored in VRAM and used throughout the subsequent decode.

Unlike decode, where the model processes one new token per step, in prefill it processes N tokens in one go. That makes it far more efficient in FLOPs/token (GPUs are good at large matmuls), but it carries a quadratic cost in attention:

$$\text{prefill attention FLOPs} \approx 4 \cdot N^2 \cdot d_{model}$$

With a prompt of 1,000 tokens and $d_{model} = 4096$ (Qwen2.5-7B): $4 \cdot 10^6 \cdot 4096 \approx 16 \times 10^9$ FLOPs in attention alone. With 4,000 tokens, 256× more, because of the quadratic nature.

Prefill (compute-bound):
prompt tokens → [attention O(N²)] → [FFN] → initial KV cache
Decode (memory-bound):
1 new token → [cross-attention over KV cache] → next token

Why prefill is a problem on small hardware

On an H100 with 3.35 TB/s of bandwidth, a long prefill amortises quickly. On an RTX 4090 (1.008 TB/s) or an L40 (864 GB/s), the bottleneck shows up sooner and has concrete consequences:

The head-of-line blocking problem. By default, vLLM processes a complete prefill before doing any decode. If you have 10 requests queued, 9 in decode and 1 with an 8,000-token prompt, those 9 requests stall while the GPU chews through the long prefill. Their users see the streaming freeze. This is called head-of-line blocking and it is the number one enemy of TTFT in production.

The four levers

1. Chunked prefill — `--enable-chunked-prefill` + `--max-num-batched-tokens`

Chunked prefill splits a long prefill into pieces (chunks) and interleaves them with decode steps in the same batch. In vLLM V1 (≥ 0.6) it is on by default.

vllm serve my-model \
 --enable-chunked-prefill \
 --max-num-batched-tokens 4096

--max-num-batched-tokens is the total token budget vLLM can process in a single engine step, counting prefill and decode together. It is the most important parameter for controlling the trade-off:

`max-num-batched-tokens`	Effect
Low (512–2048)	More decode steps per cycle → better ITL, worse TTFT
High (8192–32768)	Large prefill chunks → better TTFT and throughput, worse ITL

For an RTX 4090 serving 7B–13B models with mixed contexts (256–4096 tokens):

--max-num-batched-tokens 8192 # a reasonable balance point

For an L40 (48 GB) with larger models and longer prompts:

--max-num-batched-tokens 16384

How it works internally: with a budget of 4,096 tokens and a prefill of 10,000, vLLM splits it into 3 chunks (4,096 + 4,096 + 1,808). Between chunks, it processes the pending decode steps. The requests in decode keep advancing; the long prefill takes longer to finish, but it freezes nothing.

Without chunked prefill:
t=0 [prefill 10k tokens]─────────────────────────────┐
t=1 └─[decode r1,r2...r9]
With chunked prefill (budget 4096):
t=0 [prefill chunk 4096][decode r1..r9]
t=1 [prefill chunk 4096][decode r1..r9]
t=2 [prefill chunk 1808][decode r1..r9]
t=3 [decode all, including the new one]

The TTFT of the long request rises slightly (3 steps instead of 1), but the ITL of the other 9 requests is not interrupted.

2. Prefix caching — `--enable-prefix-caching`

If several requests share the same prefix, a system prompt, a few-shot block, a context document, vLLM can compute the KV cache of that prefix once and reuse it.

vllm serve my-model \
 --enable-prefix-caching

This is called Automatic Prefix Caching (APC). Internally, vLLM splits the KV cache into fixed-size blocks (16 tokens per block by default) and assigns them a content-based SHA hash. When a new request arrives, it checks whether any initial block is already in the cache. On a hit, that prefill is skipped.

The numerical impact: suppose a 512-token system prompt that appears in 80% of your requests, and a rate of 100 req/min:

Without APC: 80 req/min × 512 tokens × prefill_cost = 41,000 tokens/min of redundant prefill
With APC (80% hit rate): 20 req/min × 512 = 10,240 tokens/min of prefill → a 75% reduction

The TTFT of those 80 requests falls to whatever it costs to process only the new suffix.

Limitation with chunked prefill: when chunked prefill is active, only the first chunk of the prefill benefits from APC in the current vLLM implementation. For workloads where the cache hit rate is very high and the suffixes are short, consider lowering --max-num-batched-tokens so that the first chunk covers more of the shared prefix.

# Configuration optimised for a high prefix cache hit rate
vllm serve my-model \
 --enable-prefix-caching \
 --enable-chunked-prefill \
 --max-num-batched-tokens 4096 # smaller chunks = prefix fits in chunk 1

3. FP8 KV cache — `--kv-cache-dtype fp8`

The KV cache takes up VRAM. The more VRAM it consumes, the fewer concurrent requests you can keep in flight. On a 24 GB RTX 4090, the Qwen2.5-14B model in BF16 already takes ~28 GB, so it does not fit. In Q4 it takes ~9 GB, leaving ~14 GB for the KV cache.

How many context tokens fit in 14 GB of BF16 KV cache for a 14B with GQA?

$$\text{KV size per token} = 2 \cdot n_{kv\_heads} \cdot d_{head} \cdot n_{layers} \cdot 2 \text{ bytes}$$

For Qwen2.5-14B: $n_{kv\_heads}=8$, $d_{head}=128$, $n_{layers}=40$, BF16 → $2 \cdot 8 \cdot 128 \cdot 40 \cdot 2 = 163{,}840$ bytes ≈ 160 KB/token.

14 GB / 160 KB ≈ 87,500 tokens of total context. With 8 users in parallel and 4,096 tokens of context each: 32,768 tokens used out of 87,500. There is headroom, but it is finite.

Moving to FP8 (1 byte instead of 2):

$$\text{KV FP8} = 80 \text{ KB/token} \implies 14 \text{ GB} / 80 \text{ KB} = 175{,}000 \text{ tokens}$$

Twice the context capacity with the same VRAM. That buys either more concurrency or longer contexts.

vllm serve my-model \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales # calibrates the scales dynamically; without this there is degradation

Warning for RTX 4090 and L40: Ada Lovelace has FP8 instructions at the CUDA level but without Hopper’s (H100’s) dedicated scaling hardware. The memory reduction is real; the compute speedup is smaller than on an H100. Do not expect the same speedup as in a Hopper datacenter. On the L40S (the variant with optimised FP8 tensor cores) the benefit is larger than on an RTX 4090.

4. Context budget — `--max-model-len`

--max-model-len defines the maximum number of tokens vLLM can handle in a single request (prompt plus generation). It is the hard limit that determines how much VRAM is reserved for the KV cache in the worst case.

On small hardware, lowering it frees VRAM for more concurrency:

# 7B model on RTX 4090, typical context of 4K but the model supports 128K
vllm serve my-model \
 --max-model-len 8192 \  # instead of 131072
 --gpu-memory-utilization 0.92

With the context cut to 8,192 tokens, vLLM does not reserve KV cache for 131,072 potential tokens and can fit more simultaneous requests. The risk is obvious: requests above 8,192 tokens fail with an error. Tune it to the P99 of your real length distribution.

Interaction between parameters

The four parameters are not independent. A common mistake is enabling prefix caching without tuning the block size, or raising max-num-batched-tokens without checking that max-num-seqs allows it to be filled:

max-num-batched-tokens = 8192
max-num-seqs = 4
average prompt = 512 tokens → 4 × 512 = 2048 prefill tokens < 8192
The 8192 budget is never filled because max-num-seqs caps it first.
Fix: raise max-num-seqs or lower max-num-batched-tokens.

A balanced configuration for RTX 4090 + a 7B model:

vllm serve Qwen/Qwen2.5-7B-Instruct \
 --gpu-memory-utilization 0.92 \
 --max-model-len 8192 \
 --enable-chunked-prefill \
 --max-num-batched-tokens 8192 \
 --max-num-seqs 64 \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales

Configuration for an L40 (48 GB) + a 14B model:

vllm serve Qwen/Qwen2.5-14B-Instruct-AWQ \
 --gpu-memory-utilization 0.90 \
 --max-model-len 16384 \
 --enable-chunked-prefill \
 --max-num-batched-tokens 16384 \
 --max-num-seqs 128 \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales

How to measure that it is working

The metrics that confirm prefill is optimised:

# In vLLM's Prometheus metrics (port 8000/metrics):
vllm:time_to_first_token_seconds_bucket → TTFT distribution
vllm:gpu_cache_usage_perc → KV cache utilisation
vllm:prefix_cache_hit_rate → APC hit rate (if enabled)
vllm:num_running_seqs → simultaneous in-flight requests

A prefix_cache_hit_rate below 30% on workloads with a fixed system prompt means something in the hash is not working (a system prompt that varies by timestamp, a date format in the prompt, and so on).

Implications for on-premise inference

Chunked prefill and prefix caching are zero cost: they are enabled with flags and require no extra hardware. FP8 KV cache requires the model to be compatible (almost every modern transformer is) and that you are on Ada Lovelace or newer.

For sovereign ENS deployments where the hardware is fixed and you cannot scale horizontally on demand, a well-configured prefill is the difference between needing 4 nodes and needing 2 for the same load.

The second article in this series covers the decode optimisations: speculative decoding, KV cache tuning to maximise concurrency and how to configure gpu-memory-utilization without vLLM running out of VRAM at midnight.

References

Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023
Sarathi-Serve: Chunked Prefill and Stall-Free Scheduling — Agrawal et al., 2024
vLLM Optimization and Tuning — documentación oficial
Quantized KV Cache en vLLM

Disaggregated serving: prefill and decode in specialised pods

Fri, 22 May 2026 01:00:00 +0200

TL;DR

LLM inference has two phases with opposite profiles: prefill (processing the whole prompt in one go) is compute-bound, decode (generating token by token) is memory-bandwidth-bound. Running them on the same GPU forces a choice between two incompatible optimal hardware profiles, and leaves between 60 % and 80 % of peak capacity unused. The industry consolidated the pattern in 2026: disaggregated serving, separate pods for each phase, connected by a KV cache transfer channel (NIXL over UCX, RDMA, or NCCL as a fallback). DistServe demonstrated 7.4× more request rate at the same SLO; NVIDIA Dynamo 1.0 (GA at GTC 2026) takes the pattern to production at datacenter scale. Mixing heterogeneous hardware, H100 for prefill and commodity GPUs for decode, cuts up to 48 % off the cost per token. This article explains the why, the how, and the numbers that matter for a typical on-premise infrastructure.

You are here: Deploy

Disaggregated serving is an architectural decision in the Deploy stage of the six-stage LLMOps pipeline. It does not change the model, it does not change the data, it does not change the evals. It only changes how inference pods are spread over the GPU hardware. But that change moves aggregate throughput between 2× and 7×.

The analogy: the kitchen with two brigades

A serious industrial kitchen, any kitchen serving more than 50 covers a night, runs with two distinct brigades and two physically separate spaces.

The prep brigade starts at dawn. Its job is the mise en place: chopping, marinating, blanching, boiling stocks, preparing complex components. Equipment: good knives, big burners, convection ovens, 40-litre pots. It is capacity-intensive work and it is done in one batch. When it finishes, everything is left in labelled trays ready to use.

The service brigade comes in mid-afternoon. Its job is service: taking the trays from prep, heating portions, plating, running the pass. Equipment: salamanders, small griddles, thin spatulas, plenty of crockery. It is wrist work, rhythm work, not-failing-the-customer-who-has-the-plate-in-front-of-them work. Capacity per hour matters less than latency per plate.

If you make the same person do prep and service, both suffer. The cook is idle while doing mise en place in the middle of the afternoon. They have to stop and plate up when five orders land at once. Their working kit is designed for one or the other, not for both.

Serious kitchens solved this decades ago: separate brigades, separate spaces, separate equipment. The only thing that crosses between them is the trays of mise en place.

The trays are the KV cache. The separation is disaggregated serving. The handover from prep to service is the KV cache transfer, solved today with NIXL over RDMA. And the specialised pods are the two brigades with their optimal kit.

Quick recap: prefill and decode

A request to an LLM always goes through two phases:

Prefill. Take the complete prompt (say, 4,000 tokens) and process it in a single pass through all the layers of the model. The result is the KV cache of those 4,000 tokens (see the previous article on KV cache if you want to recall exactly what it holds). This step is massively parallel: all the tokens go through the attention matrices at once, which translates into huge, dense matrix multiplications. The GPU sits at 90-95 % compute utilisation. TTFT (time to first token) is determined by this phase.

Decode. Once the KV cache is ready, the model generates tokens one by one. Each new token is a pass through all the layers with a single query vector, reading the whole accumulated KV cache to compute attention. There is no parallelism between tokens (each one depends on the previous). What limits you here is not compute but bandwidth: every step has to read the model’s full weights from HBM. The GPU sits at 20-40 % compute utilisation, but at 90 % HBM utilisation. TBT (time between tokens) is determined by this phase.

Phase	Characteristic	Bottleneck	Key metric
Prefill	Massive parallel compute over N tokens at once	TFLOPS (compute)	TTFT
Decode	Streaming weights from HBM, 1 token at a time	HBM bandwidth	TBT (inter-token latency)

The asymmetry is structural: prefill burns compute and leaves memory half idle, decode does the opposite. A GPU designed to be excellent at both at once is a GPU designed to be badly used all the time.

Why putting them on the same GPU is a bad deal

Until 2023, the universal assumption was to run prefill and decode in the same inference process, on the same GPU. The engine scheduler (vLLM, TGI, Triton) decided on each cycle whether to prefill a new request or decode the ones already in flight. The intuition was that sharing hardware saves money.

The intuition is wrong. The problem has three faces:

Latency interference. When the engine decides to prefill a new request, it interrupts every decode in progress. That pushes up the TBT of the other requests. The user who was watching tokens flow smoothly down the screen notices a stall of several hundred milliseconds. This is known as prefill-decode interference and it degrades the experience visibly as concurrency rises.

Sub-optimal hardware for each phase. An H100 SXM has 989 TFLOPS BF16 of compute and 3.35 TB/s of HBM3. It is excellent for prefill, where compute is the limit. For decode, where the only thing that matters is bandwidth, those 989 TFLOPS are 60-70 % wasted. Conversely, a GPU with less compute but similar relative bandwidth (RTX 4090, L40S) would handle decode just as well for a fraction of the price.

Low aggregate utilisation. In real workloads with Llama 3 70B and 512-token outputs, around 80 % of the wall-clock is spent in decode. That means 80 % of your H100 cluster budget is doing memory reads, not calculations. It is like paying for a Ferrari to use it in a car park queue.

The idea: specialised pods, KV cache as the deliverable

Disaggregated serving breaks the inference cycle into two distinct services:

Prefill pod. Receives the prompt, runs the prefill, produces the KV cache. Hardware: high-compute GPUs (H100, H200, B200). Optimised for aggressive batching and throughput, not for individual latency: if 32 prompts arrive within 100 ms, it processes them together.

Decode pod. Receives the already-built KV cache, runs the token-by-token generation, streams to the client. Hardware: GPUs with good bandwidth but ideally cheaper per TFLOPS (RTX 4090, L40S, A100, even A30 depending on the case). Optimised for per-token latency (low TBT).

Between the two: a KV cache transfer over the network, which can be node-local (shared memory, NVLink), intra-rack (RDMA with InfiniBand or RoCE) or cross-rack (NIXL over UCX). The cost of this transfer scales linearly with context length, and it is the economic crux of the scheme.

The transfer protocol: the economics of movement

The KV cache transferred for a Llama 3 70B with 4K of context weighs roughly 2.6 GB (80 layers × 8 KV heads × 128 dim × 4,096 tokens × 2 (K and V) × 2 bytes in BF16). Moving 2.6 GB between two GPUs is not trivial:

Channel	Effective bandwidth	Time for 2.6 GB
Intra-node NVLink (NVSwitch)	~450 GB/s	~6 ms
Shared memory (same node, PCIe 5)	~60 GB/s	~45 ms
RDMA InfiniBand 400 Gbps	~50 GB/s	~55 ms
RDMA RoCE 200 Gbps	~25 GB/s	~105 ms
TCP/IP 10 GbE	~1 GB/s	~2.6 s

The immediate reading: above InfiniBand grade, the transfer is comfortable. Below it, it ruins the very TTFT we are trying to improve. Disaggregated serving is only viable with a decent interconnect. It is not a pattern for clusters built out of consumer Ethernet switches.

NVIDIA answered this with NIXL (NVIDIA Inference Transfer Library), released in mid-2025: a library that abstracts the transport (UCX, NCCL, direct RDMA verbs, shared memory) and picks the best available path automatically. vLLM has integrated it since late 2025 through the NixlConnector. It is now the de facto default for new deployments.

Real implementations in May 2026

The pattern’s path over two years:

2024 Jan · DistServe (HKU + UCSD): 7.4× requests at the same SLO
2024 May · SplitWise (Microsoft): variant with heterogeneous hardware
2024 Dec · experimental vLLM disagg (SharedStorage + PyNcclConnector)
2025 Mar · NIXL release (NVIDIA): unified transfer library
2025 Jul · stable vLLM NixlConnector
2025 Nov · SGLang, llm-d, MoonCake adopt the pattern
2026 Mar · NVIDIA Dynamo 1.0 GA (GTC 2026): production-ready at datacenter scale

As of today, the pattern is the default in any serious serving framework. The ones still monolithic are the small ones or the educational ones.

Three realistic options for an on-premise infrastructure:

vLLM disagg with NixlConnector. The most open route, requiring two sets of vLLM pods to be deployed (one with --kv-transfer-config '{"kv_role":"producer"}', another with "kv_role":"consumer") plus a proxy router. Enough for clusters of 4-16 GPUs.
SGLang with disagg. Conceptually equivalent, better performance on some MoE workloads.
NVIDIA Dynamo 1.0. The one taking over at datacenter scale. It covers routing, KV cache management, monitoring and scheduling in a single control plane. Heavier, but the reference solution if your cluster grows beyond 32 GPUs.

The numbers that matter

What disaggregation unlocks, in direct terms:

Metric	Aggregated (monolithic)	Disaggregated	Improvement
Goodput (req/s at the SLO)	baseline	1.4 – 2×	up to 2×
TTFT under high load	rises sharply from QPS 4	stable up to QPS 7+	~2×
Request rate at the same SLO (DistServe paper)	baseline	7.4×	7.4×
MoE throughput on Blackwell (Dynamo, GB300 NVL72)	baseline (Hopper)	up to 50×	depends on the model
Cost per token (heterogeneous H100 + commodity)	baseline (all H100)	-48 %	almost half

These numbers have to be read carefully: the most spectacular ones (7× and 50×) need specific hardware (Blackwell GB200/GB300 NVL72) and specific models (large MoE). The realistic range for a typical on-premise setup is 1.4-2× in goodput and -30 to -50 % in cost per token, depending on how heterogeneous the GPU mix is and how optimised the KV cache transfer is.

Heterogeneity: the radical version

The logical next step, proposed by SplitWise in 2024 and matured in 2025-2026 (Cronus, Tessera and others), is to mix GPU types: expensive high-compute GPUs for prefill, commodity GPUs with good bandwidth for decode.

Indicative cost (typical market prices in mid-2026):

H100 SXM: ~30-40 kUSD capex, ~3-4 USD/h amortised. Compute-heavy profile.
L40S: ~8-10 kUSD capex, ~1.5 USD/h. Intermediate profile, 864 GB/s of bandwidth.
RTX 4090: ~1.5 kUSD capex, ~0.30 USD/h. Modest compute profile but 1 TB/s of GDDR6X bandwidth, enough for decode of models up to ~30B parameters.

A realistic mixed cluster for serving an 8B model:

2× RTX 4090 (prefill batch) → ~3,000 USD capex, ~0.60 USD/h
4× RTX 4090 (decode pool) → ~6,000 USD capex, ~1.20 USD/h
TOTAL → ~9,000 USD capex, ~1.80 USD/h

Against the monolithic alternative with equivalent throughput:

2× H100 SXM (all in one) → ~70,000 USD capex, ~7 USD/h

The same throughput at a fraction of the capex and a quarter of the hourly cost, at the price of operational complexity: you now have two pools to coordinate, a transfer network to look after, and a scheduler that is not trivial.

For larger models (Llama 3 70B), the decode pool no longer fits on a single 4090 (the model does not fit in 24 GB, not even quantised to INT4 with headroom). There the sensible mix is H100 for prefill plus L40S or A100 80GB for decode, with a typical saving of 30-40 % over the all-H100 option.

Applied to typical on-premise hardware

Case 1 — One or two RTX 4090s: monolithic still wins

With a single GPU there is no disaggregation to speak of: the pattern needs at least two GPUs in separate pods. With two 4090s you can technically try it (one for prefill, one for decode with the KV cache transferred over PCIe 5 or basic RDMA), but the transfer overhead eats the gain for small models where prefill is already fast.

Recommendation: stay monolithic (traditional vLLM, properly configured with a quantised KV cache). The next justifiable level of complexity is a cluster with a fast interconnect.

Case 2 — 4×H100 SXM cluster (320 GB, NVLink): the sweet spot

The minimum realistic configuration for serious disaggregation, serving a 70B model in production:

2× H100 (TP=2) → 2 prefill pods
2× H100 (TP=2) → decode pods with several instances sharing TP
NIXL over NVLink → KV cache transfer <6 ms
Router (vLLM or Dynamo) → prompt distribution and streaming

Realistic expected result: goodput 1.6-1.9× relative to the same cluster running monolithic, with TTFT stable up to loads of QPS 7-8 (against the QPS 4 at which the monolithic setup starts to degrade).

If the heterogeneous mix is possible (adding 4-8 L40S to the cluster to make the decode pool), the cost per token drops a further 25 % to 35 %, while still serving the 70B model whole.

Position within the architecture

Disaggregated serving is a cross-cutting layer over almost everything discussed in previous articles. It touches:

The KV cache because it is the artefact transferred between pods. Without a solid grasp of how much the cache weighs and how it grows with context, you cannot size the transfer.
Continuous fine-tuning because multi-LoRA hot-swap keeps its semantics: each pod (prefill or decode) loads the adapters separately, and the router decides which adapter to apply in each phase.
Cluster topology: it changes the recommended HW, the networking required and the cost model.

If you are designing an inference infrastructure for 2026 from scratch, disaggregation stops being optional for any cluster beyond 4 GPUs of capacity. If you are modernising an existing one, it is the upgrade with the best return per euro invested, provided the networking between pods is decent (intra-node NVLink or intra-rack RDMA as a minimum).

What we have not covered (upcoming articles)

NIXL in detail: how it picks the optimal transport, how UCX is configured, what happens when RDMA fails and you have to fall back to TCP.
Routing scheduler: how the orchestrator decides which pod gets which request, dynamic batching, priority handling.
Multi-tenant disagg: KV cache isolation between tenants, per-adapter ACLs, multi-LoRA over specialised pods.
Disagg + prefix caching: how it combines with the KV cache reuse pattern when several prompts share a prefix (a common system prompt).
Disagg at the edge / local inference: viability on home hardware (4090 + Mac Studio, for example), where the transfer depends on Thunderbolt or residential Ethernet.

References

Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (OSDI 2024).
Patel et al., SplitWise: Efficient Generative LLM Inference Using Phase Splitting (ISCA 2024).
NVIDIA, NVIDIA Dynamo 1.0: Production-Ready Disaggregated Inference (GTC 2026, March): https://developer.nvidia.com/blog/nvidia-dynamo-1-production-ready/.
NVIDIA, NIXL: NVIDIA Inference Transfer Library — official documentation.
vLLM, Disaggregated Prefilling: https://docs.vllm.ai/en/stable/features/disagg_prefill/.
vLLM, NixlConnector Usage Guide: https://docs.vllm.ai/en/stable/features/nixl_connector_usage/.
Hao AI Lab, Disaggregated Inference: 18 Months Later (UCSD, 2025) — technical retrospective on the DistServe paper.

Prefill on lo0 — Tech Blog

Aggressive RAG in small models: trading parameters for retrieval

TL;DR

The analogy: the open-book exam

The capacity argument: how many facts fit in the weights

The central tension: retrieving more is not putting in more

(a) SLMs make worse use of long context

(b) Prefill grows with the context and is compute-bound

The maths of prefill

Numerical example: TTFT before and after compressing, RTX 4090

The five levers to resolve the tension

1. Aggressive reranking: precision over recall

2. Context compression: distilling the crib sheet

3. Prefix caching of stable context

4. Semantic caching of answers

5. Structured output and function calling: lean on tools, not on memory

The complete pipeline

Implications for on-premise inference

On the RTX 4090 (24 GB, Ada Lovelace)

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

What we have not covered

See also

References

The signature-dish specialist: vLLM's attention backend (FlashAttention, FlashInfer and the prefill/decode asymmetry)

TL;DR

Where you are: the specialist, not the commis

Why prefill and decode are opposite problems

The scheduler’s trick: prefill and decode in the same forward

How the backend reads the paged KV

The backends and how the engine chooses

What FlashInfer brings: JIT and cascade attention

The arithmetic that matters: when switching backend gives you something

The 10 knobs

How it connects to the rest of the stack

Traps and things that are not what they look like

Conclusion

See also

References

Optimising prefill in vLLM: the knobs your TTFT will not forgive

TL;DR

The analogy

What prefill really is

Why prefill is a problem on small hardware

The four levers

1. Chunked prefill — --enable-chunked-prefill + --max-num-batched-tokens

2. Prefix caching — --enable-prefix-caching

3. FP8 KV cache — --kv-cache-dtype fp8

4. Context budget — --max-model-len

Interaction between parameters

How to measure that it is working

Implications for on-premise inference

See also

References

Disaggregated serving: prefill and decode in specialised pods

TL;DR

You are here: Deploy

The analogy: the kitchen with two brigades

Quick recap: prefill and decode

Why putting them on the same GPU is a bad deal

The idea: specialised pods, KV cache as the deliverable

The transfer protocol: the economics of movement

Real implementations in May 2026

The numbers that matter

Heterogeneity: the radical version

Applied to typical on-premise hardware

Case 1 — One or two RTX 4090s: monolithic still wins

Case 2 — 4×H100 SXM cluster (320 GB, NVLink): the sweet spot

Position within the architecture

What we have not covered (upcoming articles)

See also

References

1. Chunked prefill — `--enable-chunked-prefill` + `--max-num-batched-tokens`

2. Prefix caching — `--enable-prefix-caching`

3. FP8 KV cache — `--kv-cache-dtype fp8`

4. Context budget — `--max-model-len`