Disaggregated-Serving on lo0 — Tech Blog

Long context and KV offloading: when the notebook does not fit on the desk

Fri, 12 Jun 2026 05:00:00 +0200

TL;DR

Serving long context is not serving a smarter model, it is managing a notebook that does not fit on the desk. The context window is set by the model (and extended with RoPE scaling / YaRN), but the cost of serving it is set by the KV cache, which grows linearly with sequence length and, in production, explodes: a 300k-token contract on Llama 3 70B eats ~93 GB of KV, more than an entire H100, and a million tokens asks for ~125 GB. When the KV does not fit in HBM, there are only two ways out: recompute (extremely expensive, attention is quadratic) or offload the KV to a cheaper memory hierarchy: DRAM, NVMe, network. The 2026 OSS state of the art, LMCache, Mooncake (Kimi’s platform) and NVIDIA Dynamo/KVBM, turns that offload into a first-class KV layer, with reuse across requests, prefill/decode disaggregation and cache-aware routing. Reported results: 3×–10× less latency with LMCache and up to +525 % throughput in long-context scenarios with Mooncake. The price: every memory hop adds transfer latency, so offloading only pays off when what you save by not recomputing beats what it costs to move the bytes.

The analogy

A researcher works at a small desk. On it fit the papers they are pulling from right now, which is the GPU’s HBM, blazingly fast but tiny. When the case is long (a thousand-page file, a million-token context), the papers do not fit. There are three options:

Throw papers away and ask the archive for them again every time they are needed. That is recomputing the KV: correct, but agonisingly slow, because “asking the archive” in an LLM means running the whole prompt through attention again, and attention is quadratic.
Put a shelf next to the desk (the CPU’s DRAM) and, further out, a store room in the basement (NVMe) and a depot in another building (network/object). Moving papers between the desk and the shelf costs seconds, not hours. That is hierarchical KV offloading.
Have two researchers split the work: one reads and marks up the whole file (prefill), then passes their notes to the second, who only drafts (decode). That is the KVCache-centric disaggregated architecture.

This post is about options 2 and 3, the ones that make long context viable in production. Option 1 is what you pay by default if you do nothing.

Part 1 · Why long context is a memory problem

The KV cache grows with the sequence

For every token that goes in or out, the model stores its key and value vectors in each layer so it does not have to recompute them. The size is:

$$\text{KV bytes} = 2 \times L \times h_{kv} \times d_{head} \times s \times b$$

with $L$ layers, $h_{kv}$ KV heads (few, with GQA), $d_{head}$ the per-head dimension, $s$ the sequence length and $b$ the bytes per element. Everything is a model constant except $s$: the KV is linear in context length. Doubling the context doubles the KV. That is the basis of KV cache: the working memory of inference.

The numbers are frightening

At production scale, that linear term turns brutal:

A 300k-token contract on Llama 3 70B consumes ~93 GB of KV alone, more than the 80 GB of an entire H100 (DigitalOcean · Long-Context Inference Cost).
A 1M-token context needs ~125 GB of KV, which exceeds both an RTX 4090 (24 GB) and an 80 GB A100 (Introl · Long-Context LLM Infrastructure).
Even a 7B at 128k climbs to ~14 GB of KV (against ~6 GB at 4k).

The KV stops being an implementation detail and becomes the resource that dictates your concurrency.

And on top of that, attention is quadratic

The KV is linear, but the compute of attention is quadratic in length. When the prompt reaches 1M tokens, generating each token can take on the order of 1.765 seconds, with more than 96 % of the latency spent in attention (Introl). The aggregate effect is a throughput collapse of 10×–100× against short contexts. This is why long context is not “fixed” with more VRAM alone: there is a memory problem (KV) and a compute problem (attention) at the same time.

Extending the window ≠ serving it cheaply

A common confusion: “my model supports 1M of context” does not mean “I can serve 1M cheaply”. The window is extended with RoPE scaling techniques such as YaRN, the practical option for fine-tuning open source models to long context: it needs 10× fewer training tokens and 2.5× fewer steps than naive RoPE interpolation, and it took LLaMA-2 from 4k to 32k and 128k (YaRN/LongRoPE). But that solves the model attending to long context, not the KV fitting on your GPU. According to the 2026 round-ups, May 2026 marked the first generation of open million-token models (with families supporting 256k extensible to 1M via YaRN) (letsdatascience). Treat it as a trend, not a closed spec, and verify capabilities per concrete model.

Part 2 · The KV memory hierarchy

The core idea of offloading is old in systems: hierarchical caching. “Hot” KV (what is in use) lives in HBM; “warm” KV drops to DRAM; “cold” KV to NVMe; “shared across nodes” to network or object storage. Every hop multiplies capacity and divides bandwidth.

Working out when offloading pays off

Offloading is not magic: moving KV from DRAM/NVMe back to HBM costs time. The rule is simple. Offloading pays off when the cost of recomputing exceeds the cost of transferring:

$$t_{recompute}(s) \;>\; t_{transfer} = \frac{\text{KV bytes}}{BW_{enlace}}$$

Since $t_{recompute}$ grows with attention (quadratic) and $t_{transfer}$ grows only with the size of the KV (linear) divided by the link bandwidth, the longer the context, the more the balance tips in favour of offloading. That is why offloading is the lever for long context: it is exactly where recomputing becomes unbearable. The LMCache documentation says it without hedging: offloading only helps when the recomputation it avoids is larger than the overhead it introduces (LMCache benchmark).

Part 3 · KV offloading in practice (OSS)

The limit of out-of-the-box prefix caching

vLLM already caches prefixes in HBM (see prefix cache: hit rate engineering and PagedAttention). The problem: it only lives in HBM, which is small. When the useful KV exceeds HBM, it is evicted and, on the next request, has to be recomputed. There is no speedup even with prefix caching enabled, because the cache is no longer there (vLLM vs LMCache benchmark). Native prefix caching is necessary but insufficient for long context.

LMCache: the persistent KV layer

LMCache adds persistent storage backends to vLLM’s prefix cache (and also supports SGLang and NVIDIA Dynamo as engines). It pulls the KV out of HBM and shares it across engines and across queries, covering a three-level hierarchy: GPU HBM, CPU DRAM and NVMe SSD, with additional backends for Redis/Valkey, Mooncake, InfiniStore, S3 and NIXL/GDS (GitHub LMCache, arXiv 2510.09665). Because DRAM/NVMe hold far more KV than HBM, the hit ratio rises, and LMCache reports 3×–10× less latency combined with vLLM. Against vLLM’s basic CPU offload, LMCache loads the KV at chunk level with high-performance CUDA kernels, which cuts transfer overhead.

A minimal start, offloading to CPU:

# vLLM with LMCache as the KV layer (offload to DRAM)
pip install lmcache
LMCACHE_CONFIG_FILE=lmcache.yaml \
vllm serve meta-llama/Llama-3.1-70B-Instruct \
 --tensor-parallel-size 4 \
 --kv-transfer-config '{"kv_connector":"LMCacheConnector","kv_role":"kv_both"}'

# lmcache.yaml — hierarchy HBM -> DRAM -> NVMe
chunk_size: 256
local_cpu: true # warm KV in DRAM
max_local_cpu_size: 200 # GB
local_disk: "file:///nvme/lmcache" # cold KV on NVMe
max_local_disk_size: 2000 # GB

Offloading to CPU adds moderate overhead but retains a good part of the benefit against recomputing (LMCache docs). Mind the physical path: that HBM↔DRAM↔NVMe traffic goes over PCIe and across NUMA nodes. It is worth reading PCIe topology, GPUDirect and ACS and NUMA, hugepages and CPU isolation, because a badly wired offload can eat its own gain.

Combine it with FP8

Offloading reduces where the KV lives; quantization reduces how much it weighs. They are orthogonal and they add up: KV in FP8 halves the size (see FP8 end-to-end), which means half the bytes to transfer on every offload. In long context, FP8 + offload is the default combination.

Part 4 · KVCache-centric / disaggregated architecture

The next level is not just offloading KV, but redesigning serving around the KV.

Mooncake (Kimi’s platform)

Mooncake, the serving platform behind Kimi (Moonshot AI), is the reference KVCache-centric disaggregated architecture: it separates the prefill and decode clusters, and exploits the underused CPU, DRAM and SSD resources of the GPU cluster to build a distributed KV cache. Its core is a KVCache-centric scheduler that maximises throughput while respecting SLOs, with a prediction-based early rejection policy for overloaded scenarios (arXiv 2407.00079, USENIX FAST'25). The long-context numbers are striking: up to +525 % throughput in certain simulated scenarios while respecting the SLO, and in production Kimi handles 115 % and 107 % more requests on A800 and H800 clusters respectively against previous systems. The title of their FAST talk sums it up: “Trading More Storage for Less Computation”, exactly the rule from Part 2.

NVIDIA Dynamo and KVBM

NVIDIA Dynamo is the open source distributed serving framework with prefill/decode disaggregation, dynamic GPU scheduling and KV-aware routing: the PrefillRouter computes overlap scores between the incoming request and the KV blocks already cached, and routes to the GPUs that already have the relevant KV, avoiding recomputation (Dynamo · Disaggregated Serving, Dynamo · KV-aware routing). KV transfer between prefill and decode is handled by NIXL directly GPU-to-GPU over the best available transport (NVLink, InfiniBand). And its offload manager, KVBM (KV Block Manager), has a three-layer architecture (LLM runtime, logical block management, NIXL transport) that offloads cold KV to CPU RAM, NVMe or network storage to free up HBM (Dynamo · KV Cache Offloading). This connects with disaggregated serving: prefill and decode in specialised pods.

The llm-d project walks the same road, from vLLM’s prefix caching to KV-aware distributed scheduling (llm-d · KV-Cache Wins You Can See).

Part 5 · Reference architecture

On the blog’s example cluster, a node with 4×H100 SXM (80 GB, NVLink), local NVMe and plenty of DRAM, a sensible stack to serve long context without recomputing pointlessly:

KV in FP8 in the engine (vLLM), to start with half the bytes.
LMCache as the KV layer with a DRAM→NVMe hierarchy, so that long prefixes (a manual, a code base, a case file) are cached once and reused across sessions without recomputation.
KV-aware routing (Dynamo or llm-d) if you have several nodes: send the request to the GPU that already holds the prefix, not to any GPU at all.
Prefill/decode disaggregation once the prefill of enormous contexts starts stealing decode from the other requests. Prefilling 300k tokens monopolises the GPU and kills everyone’s latency.
Metrics for KV hit ratio, bytes transferred per level and prefill/decode ratio, exported to your observability stack (instrumenting vLLM with OTel). Without those three metrics, offloading is faith, not engineering.

To prototype the stack, validating the LMCache config, the backend formats, the hit/miss behaviour, an RTX 5090 (Blackwell, 32 GB) with NVMe is more than enough to serve a 7–14B and watch offloading work at contexts of 32–128k. Do not expect to serve 1M on a consumer card: the long-context problem is precisely that neither the KV nor the attention fit on small hardware. Real sizing starts from the SLO and from the context-length distribution of your traffic, not from the model’s maximum window, and that is where it links to capacity planning.

Part 6 · Operational pitfalls (and honest scepticism)

Believing that native prefix caching is enough. It only lives in HBM; in long context it is evicted and you recompute. You need a persistent layer (LMCache/KVBM) for the cache to survive.
Offloading without measuring the break-even. If your prefixes are not reused, offloading only adds transfer latency without saving recomputation. Offloading shines with shared prefixes (RAG over the same corpus, agents with the same system prompt, long sessions), not with one-off, never-repeated requests.
Forgetting that attention is still quadratic. Offloading fixes the memory of the KV, not the compute of attention. For real 1M tokens you also need efficient attention, sparse attention or retrieval approaches (RetrievalAttention, ShadowKV); offloading on its own does not bring down the 1.765 s/token.
Wiring the offload badly. HBM↔DRAM↔NVMe crosses PCIe and NUMA. An offload that crosses the wrong NUMA node or saturates a PCIe lane can be slower than recomputing. Measure the link’s real bandwidth, not the datasheet figure.
Treating the figures as constants. The 93 GB, the +525 %, the 3–10× all depend on model, length, hardware and reuse pattern. They are orders of magnitude for design; measure on your own load.
Trusting million-token model specs without verifying. The landscape of open 1M models is very recent (2026) and moves fast; the specific names and limits change between versions. Verify the window and the cost of serving it per model, do not take the headline on trust.

A note of caution for June 2026: the distributed KV layer (LMCache, Mooncake, KVBM, NIXL, llm-d) is setting right now, with APIs and backends that change release by release. It is exactly the kind of piece worth assembling behind an abstraction (vLLM’s kv_connector) rather than coupling to a single vendor.

Closing

Long context is a problem of memory and compute at the same time, and the KV cache is the bottleneck of both. The window is a gift from the model; serving it cheaply is systems engineering: quantise the KV (FP8), offload it by tiers (HBM→DRAM→NVMe→network) with a persistent layer, and reorganise serving around the KV (aware routing, separated prefill/decode) when scale demands it. The rule that governs it all is Mooncake’s: trade storage for computation. You store more KV in cheap places so you do not recompute expensive attention. Do it well and a 4×H100 node serves contexts that, by recomputing, would not even get started. Do it badly, or not at all, and every long request reads the whole case file from scratch again, while the GPU drowns in a quadratic attention you had already paid for once.

Sources

LMCache · GitHub — https://github.com/LMCache/LMCache
LMCache: An Efficient KV Cache Layer (arXiv 2510.09665) — https://arxiv.org/pdf/2510.09665
LMCache · Offload KV cache to CPU (docs) — https://docs.lmcache.ai/getting_started/quickstart/offload_kv_cache.html
vLLM Prefix Caching vs. LMCache: Benchmarking KV Reuse Tradeoffs — https://levelup.gitconnected.com/vllm-prefix-caching-vs-lmcache-benchmarking-kv-reuse-tradeoffs-944fbaf98b56
Mooncake: A KVCache-centric Disaggregated Architecture (arXiv 2407.00079) — https://arxiv.org/abs/2407.00079
Mooncake · USENIX FAST'25 (Trading More Storage for Less Computation) — https://www.usenix.org/conference/fast25/presentation/qin
Mooncake · GitHub — https://github.com/kvcache-ai/Mooncake/
NVIDIA Dynamo · Disaggregated Serving (docs) — https://docs.dynamo.nvidia.com/dynamo/design-docs/disaggregated-serving
NVIDIA Dynamo · KV-aware routing (docs) — https://docs.nvidia.com/dynamo/latest/user-guides/kv-cache-aware-routing
NVIDIA Dynamo · KV Cache Offloading / KVBM (docs) — https://docs.nvidia.com/dynamo/backends/v-llm/kv-cache-offloading
llm-d · KV-Cache Wins You Can See — https://llm-d.ai/blog/kvcache-wins-you-can-see
Long-Context Inference at Scale: The Hidden Infrastructure Cost · DigitalOcean — https://www.digitalocean.com/community/tutorials/long-context-inference-production-cost
Long-Context LLM Infrastructure · Introl — https://introl.com/blog/long-context-llm-infrastructure-million-token-windows-guide
YaRN / LongRoPE (arXiv 2402.13753) — https://arxiv.org/html/2402.13753v1

Disaggregated serving: prefill and decode in specialised pods

Fri, 22 May 2026 01:00:00 +0200

TL;DR

LLM inference has two phases with opposite profiles: prefill (processing the whole prompt in one go) is compute-bound, decode (generating token by token) is memory-bandwidth-bound. Running them on the same GPU forces a choice between two incompatible optimal hardware profiles, and leaves between 60 % and 80 % of peak capacity unused. The industry consolidated the pattern in 2026: disaggregated serving, separate pods for each phase, connected by a KV cache transfer channel (NIXL over UCX, RDMA, or NCCL as a fallback). DistServe demonstrated 7.4× more request rate at the same SLO; NVIDIA Dynamo 1.0 (GA at GTC 2026) takes the pattern to production at datacenter scale. Mixing heterogeneous hardware, H100 for prefill and commodity GPUs for decode, cuts up to 48 % off the cost per token. This article explains the why, the how, and the numbers that matter for a typical on-premise infrastructure.

You are here: Deploy

Disaggregated serving is an architectural decision in the Deploy stage of the six-stage LLMOps pipeline. It does not change the model, it does not change the data, it does not change the evals. It only changes how inference pods are spread over the GPU hardware. But that change moves aggregate throughput between 2× and 7×.

The analogy: the kitchen with two brigades

A serious industrial kitchen, any kitchen serving more than 50 covers a night, runs with two distinct brigades and two physically separate spaces.

The prep brigade starts at dawn. Its job is the mise en place: chopping, marinating, blanching, boiling stocks, preparing complex components. Equipment: good knives, big burners, convection ovens, 40-litre pots. It is capacity-intensive work and it is done in one batch. When it finishes, everything is left in labelled trays ready to use.

The service brigade comes in mid-afternoon. Its job is service: taking the trays from prep, heating portions, plating, running the pass. Equipment: salamanders, small griddles, thin spatulas, plenty of crockery. It is wrist work, rhythm work, not-failing-the-customer-who-has-the-plate-in-front-of-them work. Capacity per hour matters less than latency per plate.

If you make the same person do prep and service, both suffer. The cook is idle while doing mise en place in the middle of the afternoon. They have to stop and plate up when five orders land at once. Their working kit is designed for one or the other, not for both.

Serious kitchens solved this decades ago: separate brigades, separate spaces, separate equipment. The only thing that crosses between them is the trays of mise en place.

The trays are the KV cache. The separation is disaggregated serving. The handover from prep to service is the KV cache transfer, solved today with NIXL over RDMA. And the specialised pods are the two brigades with their optimal kit.

Quick recap: prefill and decode

A request to an LLM always goes through two phases:

Prefill. Take the complete prompt (say, 4,000 tokens) and process it in a single pass through all the layers of the model. The result is the KV cache of those 4,000 tokens (see the previous article on KV cache if you want to recall exactly what it holds). This step is massively parallel: all the tokens go through the attention matrices at once, which translates into huge, dense matrix multiplications. The GPU sits at 90-95 % compute utilisation. TTFT (time to first token) is determined by this phase.

Decode. Once the KV cache is ready, the model generates tokens one by one. Each new token is a pass through all the layers with a single query vector, reading the whole accumulated KV cache to compute attention. There is no parallelism between tokens (each one depends on the previous). What limits you here is not compute but bandwidth: every step has to read the model’s full weights from HBM. The GPU sits at 20-40 % compute utilisation, but at 90 % HBM utilisation. TBT (time between tokens) is determined by this phase.

Phase	Characteristic	Bottleneck	Key metric
Prefill	Massive parallel compute over N tokens at once	TFLOPS (compute)	TTFT
Decode	Streaming weights from HBM, 1 token at a time	HBM bandwidth	TBT (inter-token latency)

The asymmetry is structural: prefill burns compute and leaves memory half idle, decode does the opposite. A GPU designed to be excellent at both at once is a GPU designed to be badly used all the time.

Why putting them on the same GPU is a bad deal

Until 2023, the universal assumption was to run prefill and decode in the same inference process, on the same GPU. The engine scheduler (vLLM, TGI, Triton) decided on each cycle whether to prefill a new request or decode the ones already in flight. The intuition was that sharing hardware saves money.

The intuition is wrong. The problem has three faces:

Latency interference. When the engine decides to prefill a new request, it interrupts every decode in progress. That pushes up the TBT of the other requests. The user who was watching tokens flow smoothly down the screen notices a stall of several hundred milliseconds. This is known as prefill-decode interference and it degrades the experience visibly as concurrency rises.

Sub-optimal hardware for each phase. An H100 SXM has 989 TFLOPS BF16 of compute and 3.35 TB/s of HBM3. It is excellent for prefill, where compute is the limit. For decode, where the only thing that matters is bandwidth, those 989 TFLOPS are 60-70 % wasted. Conversely, a GPU with less compute but similar relative bandwidth (RTX 4090, L40S) would handle decode just as well for a fraction of the price.

Low aggregate utilisation. In real workloads with Llama 3 70B and 512-token outputs, around 80 % of the wall-clock is spent in decode. That means 80 % of your H100 cluster budget is doing memory reads, not calculations. It is like paying for a Ferrari to use it in a car park queue.

The idea: specialised pods, KV cache as the deliverable

Disaggregated serving breaks the inference cycle into two distinct services:

Prefill pod. Receives the prompt, runs the prefill, produces the KV cache. Hardware: high-compute GPUs (H100, H200, B200). Optimised for aggressive batching and throughput, not for individual latency: if 32 prompts arrive within 100 ms, it processes them together.

Decode pod. Receives the already-built KV cache, runs the token-by-token generation, streams to the client. Hardware: GPUs with good bandwidth but ideally cheaper per TFLOPS (RTX 4090, L40S, A100, even A30 depending on the case). Optimised for per-token latency (low TBT).

Between the two: a KV cache transfer over the network, which can be node-local (shared memory, NVLink), intra-rack (RDMA with InfiniBand or RoCE) or cross-rack (NIXL over UCX). The cost of this transfer scales linearly with context length, and it is the economic crux of the scheme.

The transfer protocol: the economics of movement

The KV cache transferred for a Llama 3 70B with 4K of context weighs roughly 2.6 GB (80 layers × 8 KV heads × 128 dim × 4,096 tokens × 2 (K and V) × 2 bytes in BF16). Moving 2.6 GB between two GPUs is not trivial:

Channel	Effective bandwidth	Time for 2.6 GB
Intra-node NVLink (NVSwitch)	~450 GB/s	~6 ms
Shared memory (same node, PCIe 5)	~60 GB/s	~45 ms
RDMA InfiniBand 400 Gbps	~50 GB/s	~55 ms
RDMA RoCE 200 Gbps	~25 GB/s	~105 ms
TCP/IP 10 GbE	~1 GB/s	~2.6 s

The immediate reading: above InfiniBand grade, the transfer is comfortable. Below it, it ruins the very TTFT we are trying to improve. Disaggregated serving is only viable with a decent interconnect. It is not a pattern for clusters built out of consumer Ethernet switches.

NVIDIA answered this with NIXL (NVIDIA Inference Transfer Library), released in mid-2025: a library that abstracts the transport (UCX, NCCL, direct RDMA verbs, shared memory) and picks the best available path automatically. vLLM has integrated it since late 2025 through the NixlConnector. It is now the de facto default for new deployments.

Real implementations in May 2026

The pattern’s path over two years:

2024 Jan · DistServe (HKU + UCSD): 7.4× requests at the same SLO
2024 May · SplitWise (Microsoft): variant with heterogeneous hardware
2024 Dec · experimental vLLM disagg (SharedStorage + PyNcclConnector)
2025 Mar · NIXL release (NVIDIA): unified transfer library
2025 Jul · stable vLLM NixlConnector
2025 Nov · SGLang, llm-d, MoonCake adopt the pattern
2026 Mar · NVIDIA Dynamo 1.0 GA (GTC 2026): production-ready at datacenter scale

As of today, the pattern is the default in any serious serving framework. The ones still monolithic are the small ones or the educational ones.

Three realistic options for an on-premise infrastructure:

vLLM disagg with NixlConnector. The most open route, requiring two sets of vLLM pods to be deployed (one with --kv-transfer-config '{"kv_role":"producer"}', another with "kv_role":"consumer") plus a proxy router. Enough for clusters of 4-16 GPUs.
SGLang with disagg. Conceptually equivalent, better performance on some MoE workloads.
NVIDIA Dynamo 1.0. The one taking over at datacenter scale. It covers routing, KV cache management, monitoring and scheduling in a single control plane. Heavier, but the reference solution if your cluster grows beyond 32 GPUs.

The numbers that matter

What disaggregation unlocks, in direct terms:

Metric	Aggregated (monolithic)	Disaggregated	Improvement
Goodput (req/s at the SLO)	baseline	1.4 – 2×	up to 2×
TTFT under high load	rises sharply from QPS 4	stable up to QPS 7+	~2×
Request rate at the same SLO (DistServe paper)	baseline	7.4×	7.4×
MoE throughput on Blackwell (Dynamo, GB300 NVL72)	baseline (Hopper)	up to 50×	depends on the model
Cost per token (heterogeneous H100 + commodity)	baseline (all H100)	-48 %	almost half

These numbers have to be read carefully: the most spectacular ones (7× and 50×) need specific hardware (Blackwell GB200/GB300 NVL72) and specific models (large MoE). The realistic range for a typical on-premise setup is 1.4-2× in goodput and -30 to -50 % in cost per token, depending on how heterogeneous the GPU mix is and how optimised the KV cache transfer is.

Heterogeneity: the radical version

The logical next step, proposed by SplitWise in 2024 and matured in 2025-2026 (Cronus, Tessera and others), is to mix GPU types: expensive high-compute GPUs for prefill, commodity GPUs with good bandwidth for decode.

Indicative cost (typical market prices in mid-2026):

H100 SXM: ~30-40 kUSD capex, ~3-4 USD/h amortised. Compute-heavy profile.
L40S: ~8-10 kUSD capex, ~1.5 USD/h. Intermediate profile, 864 GB/s of bandwidth.
RTX 4090: ~1.5 kUSD capex, ~0.30 USD/h. Modest compute profile but 1 TB/s of GDDR6X bandwidth, enough for decode of models up to ~30B parameters.

A realistic mixed cluster for serving an 8B model:

2× RTX 4090 (prefill batch) → ~3,000 USD capex, ~0.60 USD/h
4× RTX 4090 (decode pool) → ~6,000 USD capex, ~1.20 USD/h
TOTAL → ~9,000 USD capex, ~1.80 USD/h

Against the monolithic alternative with equivalent throughput:

2× H100 SXM (all in one) → ~70,000 USD capex, ~7 USD/h

The same throughput at a fraction of the capex and a quarter of the hourly cost, at the price of operational complexity: you now have two pools to coordinate, a transfer network to look after, and a scheduler that is not trivial.

For larger models (Llama 3 70B), the decode pool no longer fits on a single 4090 (the model does not fit in 24 GB, not even quantised to INT4 with headroom). There the sensible mix is H100 for prefill plus L40S or A100 80GB for decode, with a typical saving of 30-40 % over the all-H100 option.

Applied to typical on-premise hardware

Case 1 — One or two RTX 4090s: monolithic still wins

With a single GPU there is no disaggregation to speak of: the pattern needs at least two GPUs in separate pods. With two 4090s you can technically try it (one for prefill, one for decode with the KV cache transferred over PCIe 5 or basic RDMA), but the transfer overhead eats the gain for small models where prefill is already fast.

Recommendation: stay monolithic (traditional vLLM, properly configured with a quantised KV cache). The next justifiable level of complexity is a cluster with a fast interconnect.

Case 2 — 4×H100 SXM cluster (320 GB, NVLink): the sweet spot

The minimum realistic configuration for serious disaggregation, serving a 70B model in production:

2× H100 (TP=2) → 2 prefill pods
2× H100 (TP=2) → decode pods with several instances sharing TP
NIXL over NVLink → KV cache transfer <6 ms
Router (vLLM or Dynamo) → prompt distribution and streaming

Realistic expected result: goodput 1.6-1.9× relative to the same cluster running monolithic, with TTFT stable up to loads of QPS 7-8 (against the QPS 4 at which the monolithic setup starts to degrade).

If the heterogeneous mix is possible (adding 4-8 L40S to the cluster to make the decode pool), the cost per token drops a further 25 % to 35 %, while still serving the 70B model whole.

Position within the architecture

Disaggregated serving is a cross-cutting layer over almost everything discussed in previous articles. It touches:

The KV cache because it is the artefact transferred between pods. Without a solid grasp of how much the cache weighs and how it grows with context, you cannot size the transfer.
Continuous fine-tuning because multi-LoRA hot-swap keeps its semantics: each pod (prefill or decode) loads the adapters separately, and the router decides which adapter to apply in each phase.
Cluster topology: it changes the recommended HW, the networking required and the cost model.

If you are designing an inference infrastructure for 2026 from scratch, disaggregation stops being optional for any cluster beyond 4 GPUs of capacity. If you are modernising an existing one, it is the upgrade with the best return per euro invested, provided the networking between pods is decent (intra-node NVLink or intra-rack RDMA as a minimum).

What we have not covered (upcoming articles)

NIXL in detail: how it picks the optimal transport, how UCX is configured, what happens when RDMA fails and you have to fall back to TCP.
Routing scheduler: how the orchestrator decides which pod gets which request, dynamic batching, priority handling.
Multi-tenant disagg: KV cache isolation between tenants, per-adapter ACLs, multi-LoRA over specialised pods.
Disagg + prefix caching: how it combines with the KV cache reuse pattern when several prompts share a prefix (a common system prompt).
Disagg at the edge / local inference: viability on home hardware (4090 + Mac Studio, for example), where the transfer depends on Thunderbolt or residential Ethernet.

References

Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (OSDI 2024).
Patel et al., SplitWise: Efficient Generative LLM Inference Using Phase Splitting (ISCA 2024).
NVIDIA, NVIDIA Dynamo 1.0: Production-Ready Disaggregated Inference (GTC 2026, March): https://developer.nvidia.com/blog/nvidia-dynamo-1-production-ready/.
NVIDIA, NIXL: NVIDIA Inference Transfer Library — official documentation.
vLLM, Disaggregated Prefilling: https://docs.vllm.ai/en/stable/features/disagg_prefill/.
vLLM, NixlConnector Usage Guide: https://docs.vllm.ai/en/stable/features/nixl_connector_usage/.
Hao AI Lab, Disaggregated Inference: 18 Months Later (UCSD, 2025) — technical retrospective on the DistServe paper.

Disaggregated-Serving on lo0 — Tech Blog

Long context and KV offloading: when the notebook does not fit on the desk

TL;DR

The analogy

Part 1 · Why long context is a memory problem

The KV cache grows with the sequence

The numbers are frightening

And on top of that, attention is quadratic

Extending the window ≠ serving it cheaply

Part 2 · The KV memory hierarchy

Working out when offloading pays off

Part 3 · KV offloading in practice (OSS)

The limit of out-of-the-box prefix caching

LMCache: the persistent KV layer

Combine it with FP8

Part 4 · KVCache-centric / disaggregated architecture

Mooncake (Kimi’s platform)

NVIDIA Dynamo and KVBM

Part 5 · Reference architecture

Part 6 · Operational pitfalls (and honest scepticism)

Closing

Sources

Disaggregated serving: prefill and decode in specialised pods

TL;DR

You are here: Deploy

The analogy: the kitchen with two brigades

Quick recap: prefill and decode

Why putting them on the same GPU is a bad deal

The idea: specialised pods, KV cache as the deliverable

The transfer protocol: the economics of movement

Real implementations in May 2026

The numbers that matter

Heterogeneity: the radical version

Applied to typical on-premise hardware

Case 1 — One or two RTX 4090s: monolithic still wins

Case 2 — 4×H100 SXM cluster (320 GB, NVLink): the sweet spot

Position within the architecture

What we have not covered (upcoming articles)

See also

References