Kv-Cache on lo0 — Tech Blog

Prefix routing: what LiteLLM does not do, who does, and how much it really improves

Fri, 11 Sep 2026 00:30:00 +0200

Fifth article in the operational track of the control layer. The pair with Langfuse covered observability, the proxy’s day 2 covered availability, virtual keys covered governance and humans and agents covered the coexistence of two classes of traffic. Here a claim of mine from June gets taken apart. Everything that follows is verified against LiteLLM 1.102.0 (commit of 10 September 2026), vLLM 0.29.0 and the main branches of production-stack, llm-d and Dynamo from the second week of September 2026.

TL;DR

LiteLLM does not do prefix-aware routing, neither in the open version nor in the paid one. It is not a licensing question. A search across the whole tree for kv_aware, cache_aware, radix, prefix_cache and block_hash returns zero matches in the router. Nor is there any call from the router to /metrics or to any vLLM endpoint. In June I wrote the opposite in the inference router article and I was wrong.

What LiteLLM has is affinity, which is a different thing. Four pre-call checks pin a request to a replica by session identifier, by key hash, by previous response identifier or by exact hash of a marked prefix. None is active by default. Affinity distributes conversations; prefix routing distributes prefixes shared between different conversations, which is the case that multiplies throughput.

The only one that looks at the prompt is fragile in three ways. If the client does not send cache_control, the key comes out null and the check does nothing. If the cut point goes in the last turn, which is what the automatic injection does by default, the key changes on every turn and it never hits. And the pin’s TTL is 300 seconds hardcoded in two lines, while the provider offers one-hour caches.

The engine does not answer questions, it publishes events. There is no vLLM endpoint you can ask whether it has a prefix. What there is, is a ZMQ socket that emits stored and removed blocks in msgpack, switched off by default, with silent discarding when the consumer does not keep up. A router that wants to be exact has to replicate the engine’s hash in full: chained SHA256, deterministic seed, block granularity and the extra keys in their order.

And by default it does not publish the hits, only the additions. The index of a router that only listens drifts away from the engine’s real LRU without noticing. To see the hits you have to ask for it request by request with kv_cache_report_mode: full.

The three routers that do do it are very different from each other. The vLLM production-stack one indexes chunks of 128 raw characters in a trie that is never purged, models no eviction and dies on every restart. The llm-d one offers an approximate mode and an exact event-driven one, all its plugins are in alpha or beta, and its default tokeniser does not produce real tokens. NVIDIA Dynamo’s keeps an event-driven radix tree with a weighted cost function, and ships with the parameter that avoids starving a freshly scaled replica switched off.

The big improvements only appear in the best possible case. The two- and three-digit numbers are published by the projects themselves, with synthetic shared prefixes and a round-robin balancer as the reference. The most complete independent measurement, from Meta, obtains 2.3 times more throughput and also identifies load regimes where affinity leaves capacity at between half and two thirds of the reference. Another, on a multi-region deployment with 89 % reuse, measures single-digit improvements.

The correct architecture has two layers, not one. LiteLLM on top for identity, budget and audit, pointing at a single destination per model group. The KV-aware router underneath, inside the engine’s domain. Mixing it into a single piece is not an option that exists today.

You are here: between the gateway and the engine

This article occupies the gap between two layers that already have an article of their own. Above, the L7 switchboard and the gateway that has been operated since the first article in this track. Below, the fundamentals of the KV cache and hit rate engineering.

The gap is the piece that decides which replica each request goes to, knowing what each replica has in memory. It is the only function in the stack that cannot be resolved in the gateway or in the engine separately.

The analogy: the library with four counters

A library with four counters. Each librarian has in front of them a pile of the books they have consulted lately. When a reader comes in asking for the third volume of an encyclopaedia, if that volume is in the pile at counter two, serving them there takes seconds. At any other counter someone has to go down to the store room.

A normal balancer sends the reader to the counter with the shortest queue. It is the right decision if all the counters are interchangeable, and it is the wrong decision here, because they are not: what distinguishes them is not the queue, it is the pile.

Session affinity resolves one specific case of this. The returning reader is always sent to the same counter, because they will probably ask for the fourth volume of the same encyclopaedia. It works, and it does not resolve the general case: when a hundred different readers ask for the same third volume, session affinity spreads them across the four counters and four librarians go down to the store room for the same book.

Prefix routing is looking at what the reader asks for before assigning them a counter. It has two costs that the analogy makes visible straight away. The first is that somebody has to keep track of what is in each pile, and that bookkeeping takes work. The second is that if everybody asks for the third volume, counter two becomes the library’s only desk while the other three look on.

The correction: what I wrote in June

In the inference router article, on 2 June 2026, I closed the prefix routing section with this sentence: “vLLM Production Stack router implements it natively. NVIDIA Dynamo too. LiteLLM in its enterprise version has a beta.”

The first two hold up. The third is false and was no less false in June. While preparing the humans and agents article the first hint appeared, and full verification against 1.102.0 confirms it.

Case-insensitive search over litellm/, enterprise/ and gateway/, for the terms kv_aware, kv-aware, cache_aware, cache-aware, radix, prefix_cache, prefix-cache, kv_cache, block_hash, prefix_hash, prefix_score and longest_prefix: zero matches. Search for vllm inside litellm/router.py, litellm/router_strategy/ and litellm/router_utils/: zero. Search for num_requests_waiting, gpu_cache_usage or any scraping of the backends’ /metrics from the router: zero. LiteLLM’s router never asks the engine anything.

Search for premium_user in the files where those checks would live: a single result, and it is about something else. There is no paid version with this feature. The only routing piece that is behind a licence is per-tag budgets, in litellm/router_strategy/budget_limiter.py.

Where the error came from. LiteLLM’s documentation has an entry on dynamic TPM and RPM allocation marked as beta and enterprise, and a pre-call check called prompt_caching. Skim-read and added together, they give the impression of prefix routing in beta. Read in the code, prompt_caching is something else, and it deserves a section of its own because it is still recommended for this problem.

What LiteLLM does have: four affinities

The optional_pre_call_checks field accepts eight values. None is enabled by default: apply_default_settings passes an empty list. One of the eight, forward_client_headers_by_model_group, is declared in the type and in the interface schema and is consumed by no branch of the code. It is a dead value.

The four that pin a request to a replica are these.

Check	What it pins	Where it stores it	TTL
`session_affinity`	Session identifier	Redis with a Lua script, mirrored in memory	`deployment_affinity_ttl_seconds`, 3600 by default, refreshed
`deployment_affinity`	API key hash	Same	Same
`responses_api_deployment_check`	`previous_response_id`	Same	Same
`prompt_caching`	Exact hash of a marked prefix	`DualCache`	300 s, hardcoded

The other four are router_budget_limiting, encrypted_content_affinity, enforce_model_rate_limits and the dead value.

`prompt_caching`: why it is no use for this

The key is a SHA256 over the serialisation of the cacheable prefix, and “cacheable prefix” means everything up to and including the last block marked with cache_control: {"type": "ephemeral"}. Three consequences, checked by running the functions in isolation over the repository’s code.

If the request carries no cache_control, the extraction returns an empty list, the key comes out null and the check pins nothing. Automatic injection of cut points is only enabled with litellm.enable_anthropic_prompt_caching, which defaults to false, and only for Anthropic and Bedrock providers.

If the cut point is in the system prompt, the key does not change when turns are added and the pin works. This is the good case and it requires the client to mark it deliberately.

If the cut point is in the last message, the key changes on every turn and the pin never hits. And this is exactly the default case of the automatic injection, which places one point on the system and another at index -1. Since the second is the last, the cacheable prefix becomes the whole conversation.

On top of that there is the TTL. Two lines with ttl=300, one of them with the comment # store for 5 minutes. There is no parameter to change it. Issue 28427 has had it open for months with no response from the maintainers, and it points to the mismatch with the one-hour ephemeral caches that LiteLLM itself knows how to request via cache_control.ttl: "1h".

There is a nuance the issue does not capture and that makes diagnosis worse: in the default case the pin does not even last five minutes, because the key changes first.

`session_affinity`: the one to use

For conversations, this is the right piece and it is reasonably built. The session identifier is looked for in this exact order of headers:

x-litellm-trace-id
x-litellm-session-id
Any x-<vendor>-session-id, with the regular expression ^x-.+-session-id$ and a value of at least eight alphanumeric characters
Only if the User-Agent is Codex’s: session-id, session_id, thread-id, conversation_id
Plain x-session-id, for opencode

And if nothing matches, two fallbacks: a metadata.user_id in Anthropic format, and the W3C baggage with session.id=. The third point is the one that captures Claude Code with nothing configured.

Storage is a Lua script in Redis that does GET, SET NX EX and EXPIRE, so the first writer wins and the TTL is refreshed on every request that confirms the pin. It is an inactivity TTL, not a total session duration. Without Redis it degrades to a pod-local reservation, which is atomic but not shared.

A detail of the pipeline order that already came up in the previous article and still holds: the affinity checks run inside async_get_healthy_deployments before tag filtering. If the pin returns a single deployment and that deployment does not satisfy the request’s tag, the later filter removes it and the request fails. Affinity gets no vote on the tags.

The routers that do read the prompt, and why they are no good

LiteLLM has four pieces that read message content: AutoRouter with semantic matching, RequestComplexityRouter, AdaptiveRouter and QualityRouter. They all live in async_pre_routing_hook, which runs before looking for healthy deployments.

The reason they do not solve this problem is in what they choose. All four choose a model group, not a replica. They decide whether an easy question goes to the thirty-billion-parameter model or to the small one. None has access to the replica list or knows what is in each one’s memory.

What did land: admission control

A minor correction to the previous article, which placed this in the 1.101 branch as an upcoming feature. In 1.102.0 it is already there, in litellm/proxy/middleware/admission_control_middleware.py, with the three settings in general_settings: max_in_flight_requests_per_worker, max_queued_requests_per_worker and admission_queue_timeout_seconds, the last with a default value of 1.0 seconds.

It stays inactive as long as the first one is not declared. It exposes an authenticated /health/backlog and the three metrics litellm_admission_admitted_requests, litellm_admission_queued_requests and litellm_admission_rejected_requests_total, the last with a reason label. It is a per-process semaphore, not coordinated across pods.

What the engine exposes upwards

Before looking at the routers that do do this, you have to understand what raw material they work with. The natural question would be whether vLLM has an endpoint you can ask about a prefix. It does not.

The only cache router in the entry points is vllm/entrypoints/serve/dev/cache/api_router.py, and it only exposes destructive writes: POST /reset_prefix_cache, /reset_mm_cache and /reset_encoder_cache. All three are behind VLLM_SERVER_DEV_MODE=1 and do not exist in a normal deployment. There is no read.

What there is, is an event stream.

The ZMQ stream

It is enabled with --kv-events-config, and it is off by default. The configuration accepts a publisher (null or zmq), an endpoint (by default tcp://*:5557, which binds), an optional replay endpoint, the number of batches retained for that replay (10,000), and a high water mark limit of 100,000.

That limit is the one that matters operationally: above it, ZMQ discards events if the consumer does not keep up. Silently.

The transport is a PUB socket with msgpack serialisation and three-part frames: topic, eight-byte sequence number and payload. The sequence number lets a subscriber ask for replay from where it left off through the optional ROUTER socket. With data parallelism the port shifts by rank, so you have to subscribe to 5557+rank.

Three event types. BlockStored carries the hashes, the parent hash, the token identifiers, the block size, the LoRA adapter name, the extra keys, the medium (GPU, CPU or storage) and a session identifier. BlockRemoved carries only hashes. AllBlocksCleared carries no payload and is emitted when the prefix cache is reset.

They are published once per scheduler step, only if there are events. With the feature switched off the cost is zero. With it switched on, the full token identifiers of each block travel over the wire, so bandwidth scales with tokens per second and not with requests per second.

An example in vLLM’s own repository declares itself experimental on its eighth line.

The hash that has to be replicated

An exact router does not receive the engine’s state, it reconstructs it. For that it has to compute the same hashes, and the engine computes them like this:

# vllm/v1/core/kv_cache_utils.py
def hash_block_tokens(hash_function, parent_block_hash, curr_block_token_ids, extra_keys):
 if not parent_block_hash:
 parent_block_hash = NONE_HASH
 return hash_function((parent_block_hash, tuple(curr_block_token_ids), extra_keys))

It is chained: each hash identifies the complete prefix up to that boundary, not the isolated block. The default algorithm is SHA256, configurable with --prefix-caching-hash-algo among four values.

Five things a router has to get right for its hashes to match the engine’s:

The seed. Since 0.29, NONE_HASH is deterministic from a fixed seed, so two different processes produce the same hashes with nothing configured. It is a silent change of semantics from earlier versions, which required setting PYTHONHASHSEED. With the xxhash algorithms it is still needed.

The block size. By default 16 tokens, but the attention backend can override it if the user did not pass --block-size: 256 with sliding-window sparse attention, 64 with ROCm’s AITER, 64 or more with FlashAttention on XPU. A router must not assume 16, it has to read it from the event.

The hashing granularity, which is not necessarily the physical block size. There is --prefix-match-unit, which with several cache groups resolves to the greatest common divisor of the cacheable groups’ sizes.

The extra keys, in their order: LoRA adapter name (by name, not by identifier), multimodal identifiers with their relative offset inside the block, cache_salt only in block zero, and the hash of the prompt embeddings. Sampling parameters do not go in, which is correct: the prefix KV does not depend on the temperature.

And truncation. By default the hashes travel over the wire as 64-bit integers truncated from a SHA256, not as the full bytes. It can be disabled with VLLM_KV_EVENTS_USE_INT_BLOCK_HASHES=0. With the default value, a router has a theoretical collision risk that the engine does not have.

The hits are invisible by default

This is the detail that takes the most effort to discover and has the most consequences. The events are incremental: only what gets newly cached is announced. A reused block generates no event.

A router that only listens sees the additions and the removals, but does not see the hits. Its notion of what is recent and what is cold drifts away from the engine’s real LRU with nothing to warn it.

A full mode exists, and it is per request. sampling_params.extra_args["kv_cache_report_mode"] = "full" makes BlockStored events be emitted for blocks that were hits too. The function’s docstring says so plainly: it generates events so that external consumers, “for example a gateway”, know which blocks were reused. That is to say: for this to work, the router has to inject a parameter into every request it forwards.

The metrics, and what broke

For aggregate measurement two counters remain: vllm:prefix_cache_queries_total and vllm:prefix_cache_hits_total, both in tokens and not in requests, with model and engine labels. There are two equivalents for the external connector, which allow a hit in HBM to be distinguished from one brought in from CPU or NVMe.

And there are two removals that break existing configurations. vllm:gpu_prefix_cache_hit_rate no longer exists: zero appearances in the repository. It was deprecated in 0.8, hidden in 0.9 and removed in 0.10. And vllm:gpu_cache_usage_perc was renamed to vllm:kv_cache_usage_perc, which breaks Grafana panels and autoscalers that depended on the old name.

The hit rate has to be derived in PromQL with rate() over the two counters. The one that appears in the text log is something else: it uses a window of the last thousand requests.

And it has a bias you need to know about before taking decisions with it. When a request is evicted, its blocks return to the free queue keeping the hash, so the later reprefill finds them again and counts as a hit. A high hit rate can mean useful reuse between users or it can mean the engine is evicting and recomputing in a loop. The internal statistics separate the two things with a preempted field, which is not exposed as a Prometheus metric.

The last token

A small detail with an effect on long conversations: the maximum hit length is the request’s token count minus one. There is always something to recompute in order to get logits. Since the reservation requires alignment to the block size, in practice that can force recomputing the whole final block. With blocks of 16 it is noise. With a backend that prefers 256, it is up to 256 tokens recomputed per turn.

The ones that do do it

vLLM production-stack

Eight routing logics in a single --routing-logic selector: roundrobin, session, kvaware, loadaware, prefixaware, disaggregated_prefill, disaggregated_prefill_orchestrated and priority. Two of them, loadaware and priority, are not in the Helm chart’s enum and are only enabled through extraArgs.

The naming trap is that prefixaware and kvaware are radically different things.

prefixaware is a hash trie in the router’s process. It chunks the prompt string every 128 characters, not tokens, and hashes each chunk with 64-bit xxhash. Its own docstring says so: the chunk size is “in number of characters”. That 128 is hardcoded, there is no flag or Helm value to change it.

Four consequences come out of that. --prefix-min-match-length is measured in characters and is quantised to multiples of 128. A shared prefix that differs by one character inside the first chunk breaks the whole match, because there is no partial match inside a chunk. For chat, it concatenates the content of all the messages with newlines and without applying the chat template, so the indexed string is not what the engine tokenises. And the tie-break between tied replicas is random.choice, without looking at load.

The trie has no notion of saturation. A popular prefix concentrates all the traffic on the same replica indefinitely. The docstring assumes the simplified model and declares it: “we assume there is no prefix cache eviction”.

It has no pruning either. HashTrie only has insertion and longest prefix lookup; there is no deletion method. When a replica disappears, its URLs stay forever in every node’s sets, and filtering happens at query time. It works, and the trie grows monotonically: each distinct prompt adds its length divided by 128 permanent nodes, with no bound, no TTL and no LRU. The router’s memory limit in the chart is 1,000 MiB. By contrast, the round-robin router in the same file does bound its caches to 1,024 entries.

And it dies on every restart: it is in-process memory state, with no snapshot and no initial load.

kvaware and loadaware are better and have a different price. They do not index themselves: they ask the LMCache controller, which knows about real tokens. That requires lmcache==0.3.11 and vllm==0.13.0 pinned exactly, and the official router image drags in the whole of vLLM. The limitation to read before planning anything: they only look at request_json.get("prompt", ""), with an open TODO for chat completions. With /v1/chat/completions, both tokenise an empty string.

loadaware is the only one of the three that breaks affinity on saturation, with the formula score = relative_match - beta * relative_load and beta = 1.0 by default. The authors’ own reading: a replica at twice the average load loses the equivalent of a full cache hit. The guide suggests 0.25 to favour cache and 2.0 to favour balancing.

On published results, the documentation’s benchmarking file literally says that the measurement platform is coming soon. The project blog’s two posts claim between three and ten times lower response latency, and only one declares a partial configuration: Llama 3.1 70B on four nodes with tensor parallelism 2 over 80 GB A100s, inputs of 9K tokens and outputs of 10, against AIBrix 0.2.0 and a flat deployment. Charts with no labelled points, no absolute figures and no declaration of which routing logic was used in each.

llm-d

There has been a reorganisation here that invalidates a good part of the third-party documentation. The Endpoint Picker code left the Gateway API Inference Extension repository and was merged into llm-d’s, which has been renamed to llm-d Router. The Kubernetes project keeps the InferencePool API and a lightweight reference EPP that does nothing about prefixes. If you look for KV-aware routing in the extension’s repository, today it is not there.

The InferencePool is at v1 and is GA. The other two APIs moved to the llm-d.ai group at v1alpha2, and when they coexist with the old ones the EPP prefers the new ones and ignores the old.

Scoring is a weighted sum. The default weights, when no configuration is passed: queue 2, KV utilisation 2, prefix 3. Any scorer referenced without a weight counts as 1.0. There is no configuration through environment variables: the old ENABLE_*_SCORER no longer exist, everything goes in an EndpointPickerConfig in YAML.

The prefix scorer consumes an attribute published by a producer, and there are two producers.

The approximate one computes the hashes in the EPP itself without talking to the engine, and keeps a local LRU per pod and block pair that is populated after the decision. It is an approximation by construction. It has a hard floor: a configured block size below 64 tokens is raised to 64 at request time, because the index stores one entry per pod and block and going down to 16 would multiply memory by four.

The exact one builds the real index from the engine’s ZMQ events, with one subscriber per pod that is installed and torn down with the endpoint’s registration and deregistration. It has adapters for vLLM and SGLang, deduplicates evictions by reference counting, and the replay buffer that allows the index to be rebuilt after an EPP restart requires vLLM 0.26.0 or above.

The trap is in the tokeniser. The EPP tokenises through a plugin, and the default backend is estimate: byte packing with no tokeniser, roughly 4 bytes per token, with identifiers that do not correspond to real engine tokens. The exact scorer needs real tokens, and for that you have to explicitly configure the vllm backend, which does HTTP against the engine’s /v1/completions/render endpoints. If it is omitted, the auto-created estimate satisfies the dependency and the system degrades silently.

On maturity, the file itself declares it: all in-tree plugins are in alpha or beta, and they will be promoted to stable when the project approaches 1.0. An alpha plugin configured without the corresponding flag makes the EPP fail to start.

And on replicas, the warning is explicit: active-active mode should be avoided with approximate prefix routing, because the EPP replicas do not share state and each one only sees the traffic it has served itself, which significantly degrades the hit rate. The exact index is safe in high availability, and even so the reference guides fix a single replica because the accounting of in-flight requests is local to the process.

The cost is the best documented in the whole field, and it deserves a table. Simulator with Qwen3 8B, 100 pods, 100K input tokens:

Requests/s	Cap on tokens to match	Peak CPU	Peak memory
5.0	4,096	1.19 cores	0.26 GiB
5.0	100,000	3.82 cores	0.65 GiB
98.7	4,096	35.17 cores	2.46 GiB
98.8	100,000	46.50 cores	3.41 GiB

With outputs of 10K tokens at 50 requests per second, 32.53 cores and 12.54 GiB. The maintainers’ sizing rule is half a core to one core per request per second on large agentic loads. And idle consumption scales with the number of pods because of continuous scraping: with 100 pods, about 7.5 cores with no traffic.

NVIDIA Dynamo

A precise event-driven index, with a global radix tree in each frontend’s process. Since all the replicas consume the same event plane, there is no need to synchronise routers with each other for the prefix state.

The cost function is explicit and weighted. It discounts from the prefill the blocks that overlap in GPU, host and disk with different weights (1.0 for the overlap credit, 0.75 for host, 0.25 for disk), and adds the potential decode blocks and the active requests.

There is a parameter to look at before deploying: --router-kv-overlap-score-credit-decay is at 0 by default, that is, disabled. Its description says what it is for: preventing loaded, cache-rich replicas from winning again and again while the freshly autoscaled ones receive too little traffic. With the default value, a new pod has no cache, therefore never wins the scoring, therefore never generates cache.

The approximate mode exists (--no-router-kv-events, with a TTL of 120 seconds) and the documentation says it is not the recommended route in production.

Dynamo is the most honest project in its measurement methodology: its comparison guide recommends contrasting --router-mode random against kv over the public Mooncake trace from FAST'25, with a declared cache ratio of 59 %, and it generates the table locally instead of publishing a number.

The rest of the field

SGLang has renamed its router to sgl-model-gateway, version 0.3.2 within SGLang 0.5.19. Its cache_aware policy is an approximate radix tree over raw characters, with switching to shortest queue when the system is unbalanced according to two combined thresholds. One detail that can cost an afternoon: the Rust defaults and the Python CLI defaults do not match, and on the maximum tree size they differ by four orders of magnitude (10,000 nodes against 2^26).

AIBrix 0.7.0 offers a standard mode with a local hash table and a KV event synchronisation mode behind a flag and a build tag, with vLLM 0.7.0 or above.

Ray Serve LLM has PrefixCacheAffinityRouter with an explicit alpha API warning, a character-based prefix tree in a decoupled actor, and its imbalance threshold at infinity by default, that is, out of the box it never breaks affinity on load.

Envoy AI Gateway 1.1.0 implements none of this and delegates to InferencePool and the EPP. KServe the same, with its LLMInferenceService at v1alpha1. Mooncake Conductor is still an unimplemented proposal.

Comparison table

Project	Version and date	Index	Where it lives	Kubernetes	Declared maturity
production-stack `prefixaware`	main branch, 09-Sep-2026	Approximate, character trie, no eviction	Router process	Optional	WIP in the README
production-stack `kvaware`/`loadaware`	same	Exact via LMCache controller	LMCache controller	Optional	No chat completions
llm-d Router approximate	v0.10.0, 17-Aug-2026	Approximate, LRU per pod	EPP process	Yes	Beta, active-active discouraged
llm-d Router exact	same	Exact through ZMQ events	EPP process	Yes	Beta
NVIDIA Dynamo	v1.4.2, 27-Aug-2026	Exact, event-driven radix tree	Frontend process	No	GA with experimental subsystems
SGLang `sgl-model-gateway`	0.3.2, 03-Sep-2026	Approximate, character radix	Gateway process	No	Default policy
AIBrix	v0.7.0, 16-Jun-2026	Both	Gateway plugin	Yes	Synchronisation behind a flag
Ray Serve LLM	Ray 2.58	Approximate, characters	Ray actor	No	Declared alpha
Envoy AI Gateway	v1.1.0, 21-Aug-2026	None of its own	Delegates to EPP	Yes	Stable 1.x API

How much it really improves

Here two types of source have to be separated.

The projects’ own numbers. Red Hat published the llm-d case in May: Qwen3 32B, eight vLLM pods over 16 H100s with tensor parallelism 2, synthetic shared-prefix load with 150 groups of five prompts, 6,000 tokens of system prompt, 1,200 of question and 1,000 of output. Result: up to 109 % more throughput and up to 99 % less TTFT, around 200 concurrent users within SLO against around 20. The repository’s result files give more detail: peak throughput from 6,986 to 14,892 output tokens per second, p90 TTFT from 135.5 seconds to 0.26.

That same file publishes the regression, and that is why it is worth reading: p50 inter-token latency rises by 22.4 % with vLLM and by 45.4 % with SGLang. The trade-off is declared: routing by affinity concentrates more concurrent work on the pods with a warm cache.

It is the best possible case. A synthetic shared prefix and a reference that is a flat Kubernetes Service.

The independent measurements. There are three and they say different things.

CacheRoute, from Meta, August 2026, is the most complete. Llama 3.3 70B in fp8, 30 targets with tensor parallelism 2 over 60 H100s, a semi-synthetic trace from production telemetry with 128,824 business keys, and all the alternatives reimplemented in the same harness instead of comparing against a dumb balancer. It obtains 176 requests per second with a p99 SLO of 3.5 seconds against 76 for the best competitor, and a hit rate of 93.2 % against 72.0 %.

And it publishes two negative findings that nobody else publishes. There are load regimes where affinity reduces capacity to between half and two thirds of the reference, to the point that the authors make a mirrored-traffic trial mandatory before deploying. And growing the fleet can cool a prefix even though total cache capacity grows, because the time between revisits stretches beyond the eviction window. Scaling horizontally can make the hit rate worse.

GORGO, June 2026, measures on a multi-region deployment with a dataset where intra-user prefix reuse is 89.4 % and the average prompt is almost 18,000 tokens. An improvement of 6.9 to 15.5 % in p95 TTFT. With extremely high reuse, the improvement is single-digit, because network latency and queuing dominate.

LAAR, from IBM Research Tokyo, April 2026, is the uncomfortable counterpoint: in its comparison, session affinity routing was the worst of the baselines, and at 64K contexts a load-based routing gives lower absolute latency at the cost of fewer hits.

The joint reading fits in three conditions. The big improvements require high, concentrated prefix reuse, long prompts, and a weak reference. Take away any of the three and the numbers deflate.

The five failure modes

Hot spot from affinity. The busiest bucket inherits the load’s skew, and tail latency starts following the hottest target instead of the fleet average. Mitigations exist in the four main products and in three of the four they ship switched off: Dynamo’s credit decay at 0, Ray’s imbalance threshold at infinity, and production-stack’s prefixaware with none at all.

Starvation of the freshly scaled pod. A new pod has no cache, so it never wins the scoring, so it never warms up. Dynamo documents it as the reason for the decay parameter. I found no primary documentation from any of the projects about the specific interaction with KEDA.

Cost of the index. The llm-d numbers in the previous table are the best public reference that exists. Raising the cap on tokens to match from 16,384 to 400,000 can more than double the EPP’s CPU with little traffic.

Traffic with unique prefixes. Each project degrades to something different. SGLang routes to the smallest tree, which is not the same as the least loaded. Ray falls back to power of two choices. Dynamo has an explicit preset, --load-aware, to say “I want the load model without reuse”. And production-stack’s prefixaware keeps inserting into the trie, that is, keeps paying the cost while receiving nothing.

Sibling bursts. Parallel sampling, best of N, or an agent that opens five branches at once. With event-driven routing, no engine has yet emitted the block stored event, all the siblings score zero overlap, and the prefix is precomputed on every replica. Dynamo has a prediction TTL for this; the others do not.

The architecture that comes out of all of the above

Two layers, with the responsibilities separated.

On top, LiteLLM, with what it does do well: non-forgeable identity through the virtual key, budgets, spend logging, concurrency limits, fallback between providers and API unification. For the gateway, each model group has a single destination: the URL of the router below.

model_list:
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://kv-router.inferencia.svc:8000/v1
 api_key: os.environ/VLLM_KEY

router_settings:
 routing_strategy: simple-shuffle
 num_retries: 1
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600

general_settings:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

Below, the KV-aware router, inside the engine’s domain, talking to the vLLM replicas and consuming their events.

Four consequences of this separation that have to be accepted before building it.

Per-replica visibility in LiteLLM is lost. For the gateway there is a single deployment, so its cooldowns, its per-deployment counters and its strategies stop meaning anything. Observability of which replica served what goes downstairs.

Retries stack up. The client SDK retries, LiteLLM retries, and the router below may have its own failover. In the previous article 45 calls per turn came out with three layers; with four it is worse. Lowering num_retries in the gateway to 1, or to 0, is the sensible decision when there is a router below that already retries.

LiteLLM’s session affinity stops making sense for the KV, because it no longer chooses a replica. It is kept if it is used for something else, such as pinning a conversation to a specific provider.

And the choice of the router below conditions the rest of the cluster far more than the gateway’s does. llm-d requires Kubernetes 1.32 or above, Gateway API, the inference extension and a conformant Gateway. Dynamo does not require Kubernetes. production-stack has a Helm chart and works outside it. That decision is the expensive one, not the gateway’s.

When not to build this

Three cases where the correct answer is not to do it, at least not yet.

A single replica per model. With one replica, the engine’s prefix cache already does all the work and there is nothing to route. That is the case for a good part of the four H100 deployments with one large model and tensor parallelism.

Low or scattered prefix reuse. Before building anything, the measurement is rate(vllm:prefix_cache_hits_total[5m]) / rate(vllm:prefix_cache_queries_total[5m]) per replica, with a mental correction for the bias from reprefills after eviction. If it is already high with blind distribution, reuse is concentrated inside each conversation, and there LiteLLM’s session affinity costs one line of configuration and resolves the case.

Traffic where the network dominates. The GORGO result is the warning: with enormous prompts and a distributed deployment, the improvement is eaten up by queuing and network latency.

The case where it does pay off is specific: several replicas of the same model, a large system prompt shared between different users, or a RAG with a corpus of documents that repeat across requests. There the prefill saved is real and it is most of the work.

Checklist

Measure the current hit rate per replica with the two vLLM counters before touching anything, and check whether the engine is evicting with vllm:num_preemptions_total.
Check that prefix caching is switched on. It is no longer --enable-prefix-caching that switches it on: the default value is derived from the model and the log of the decision is at DEBUG level. What is actionable today is --no-enable-prefix-caching to switch it off.
Audit the prompts before routing: a timestamp or a session identifier at the start of the system prompt breaks the match from block zero and no routing layer fixes it.
If the KV event stream is enabled, size the bandwidth by tokens per second and watch discarding at the high water mark. With data parallelism, subscribe to one port per rank.
If an exact router is chosen, verify the vLLM version against the one the replay buffer requires.
If llm-d is chosen, explicitly configure the vllm tokenisation backend. The default estimate does not fail, it degrades.
If production-stack is chosen with kvaware or loadaware, check first whether the traffic goes through /v1/chat/completions, because today it does not work there.
Review the load-based affinity-breaking parameter, because in three of the four main projects it ships disabled.
Lower num_retries in LiteLLM when there is a router below that already does failover.
Trial with mirrored traffic before changing production. It is the explicit recommendation of the only independent measurement that went looking for bad regimes, and found them.

Traps and things that are not what they seem

LiteLLM has no prefix routing in any version. Neither the open one nor the paid one. I got it wrong in June.
prompt_caching does nothing if the client does not send cache_control, and with the default automatic injection it fails on every turn because it marks the last message.
That pin’s TTL is 300 hardcoded seconds, incompatible with the one-hour caches that LiteLLM itself knows how to request.
production-stack’s prefixaware indexes characters, not tokens, in non-configurable chunks of 128, and its trie is never purged and models no eviction.
production-stack’s kvaware and loadaware do not support chat completions. They tokenise an empty string.
The default tokeniser in llm-d’s EPP is a byte estimate, and with it the exact scorer degrades silently.
llm-d’s active-active mode is discouraged with approximate prefixes, and the reference guides fix one replica even with the exact one.
Dynamo’s overlap credit decay ships at zero, which can leave a freshly autoscaled replica with no traffic.
The SGLang router’s default values differ between the Rust and the Python CLI, by as much as four orders of magnitude.
vllm:gpu_prefix_cache_hit_rate no longer exists and vllm:gpu_cache_usage_perc is now called vllm:kv_cache_usage_perc. Old panels and autoscalers are measuring nothing.
The hit rate counts the reprefill after eviction as a hit. A high number can be reuse or it can be thrashing.
KV events do not publish the hits by default, only the additions, unless full mode is requested request by request.
The hashes travel truncated to 64 bits unless it is disabled with an environment variable.
The default block size is 16, and the attention backend can change it without anyone asking.
The Endpoint Picker left the Gateway API Inference Extension repository. Looking for it there today leads nowhere.

Closing

June’s error has a reading that goes beyond the erratum. It came from reading a product’s documentation and not its code, and from the fact that LiteLLM’s documentation uses the word “caching” in a pre-call check that does something else. The distance between what a name suggests and what a function does is, at this point in the track, the most repeated pattern of the whole series.

The architectural conclusion is that KV-aware routing is not a gateway function. It is not one today in LiteLLM and probably should not be: it requires keeping an index synchronised with each engine’s memory, replicating its hash function, consuming an event stream and sizing CPU per request per second. That belongs to the engine’s domain, not to that of identity and budget.

What remains is a two-part decision. The first is whether the traffic deserves it, and that is answered with two Prometheus counters and without installing anything. The second is which of the three routers fits the cluster you already have, and there the deciding variable is not the hit rate but whether you are willing to bring in Gateway API and the inference extension, or to pin vLLM 0.13.0, or to operate a Dynamo frontend.

And the honesty the material demands: the only independent measurement that specifically went looking for regimes where this makes things worse found them, and they are not marginal. Half to two thirds of reference capacity is a serious regression. That no project publishes that side of the curve does not mean it does not exist in theirs.

Sources

LiteLLM, Router - Load Balancing (estrategias, comprobaciones previas, afinidad): https://docs.litellm.ai/docs/routing.
LiteLLM, código: litellm/router.py, litellm/router_utils/prompt_caching_cache.py, litellm/router_utils/pre_call_checks/prompt_caching_deployment_check.py, litellm/router_utils/pre_call_checks/deployment_affinity_check.py, litellm/proxy/litellm_pre_call_utils.py, litellm/proxy/middleware/admission_control_middleware.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 28427, TTL de afinidad por caché de prompts fijado a cinco minutos: https://github.com/BerriAI/litellm/issues/28427.
vLLM, código: vllm/v1/core/kv_cache_utils.py, vllm/v1/core/block_pool.py, vllm/v1/core/kv_cache_manager.py, vllm/distributed/kv_events.py, vllm/config/kv_events.py, vllm/v1/metrics/loggers.py, vllm/v1/metrics/stats.py: https://github.com/vllm-project/vllm.
vLLM, KV offloading usage (niveles CPU, filesystem y objeto; compartición entre procesos): https://docs.vllm.ai/en/latest/features/kv_offloading_usage.html.
vLLM, notas de la versión 0.29.0 (semilla determinista, seguimiento de propiedad en eventos, flags de admisión): https://github.com/vllm-project/vllm/releases/tag/v0.29.0.
vLLM, RFC 16669, publicación de bloques KV y métricas: https://github.com/vllm-project/vllm/issues/16669.
vLLM production-stack, código: src/vllm_router/routers/routing_logic.py, src/vllm_router/prefix/hashtrie.py, src/vllm_router/parsers/parser.py: https://github.com/vllm-project/production-stack.
LMCache Lab, resultados de production-stack frente a AIBrix (parte interesada): https://blog.lmcache.ai/en/2025/03/06/open-source-llm-inference-cluster-performing-10x-faster-than-sota-oss-solution/.
llm-d Router, código y documentación de operaciones (pesos por defecto, dimensionamiento del EPP, aviso de activo-activo): https://github.com/llm-d/llm-d-router.
llm-d, guía de enrutado preciso por prefijo y resultados publicados (parte interesada): https://github.com/llm-d/llm-d.
llm-d Router, incidencia 1290, estado de prefijo no compartido entre réplicas del EPP: https://github.com/llm-d/llm-d-router/issues/1290.
Red Hat, Same 16 GPUs, twice the users (parte interesada): https://www.redhat.com/en/blog/same-16-gpus-twice-users-inference-aware-routing-llm-clusters.
Kubernetes SIG Network, Gateway API Inference Extension (traslado del EPP y del BBR): https://github.com/kubernetes-sigs/gateway-api-inference-extension.
NVIDIA Dynamo, diseño y ajuste del router KV: https://github.com/ai-dynamo/dynamo.
SGLang, sgl-model-gateway: https://github.com/sgl-project/sglang/tree/main/sgl-model-gateway.
AIBrix, enrutado por caché de prefijo: https://github.com/vllm-project/aibrix/blob/main/pkg/plugins/gateway/algorithms/prefix_cache_readme.md.
Ray Serve LLM, enrutado consciente de prefijo (API en alfa): https://docs.ray.io/en/latest/serve/llm/user-guides/prefix-aware-routing.html.
Huang Cheng (Meta), CacheRoute, agosto de 2026: https://arxiv.org/html/2608.19677.
Yuan et al., DualMap, febrero de 2026: https://arxiv.org/abs/2602.06502.
Ricci Toniolo et al., GORGO, junio de 2026: https://arxiv.org/html/2602.11688.
Yoshimura, Chiba y van de Beek, Accuracy Is Speed (EuroMLSys ‘26), abril de 2026: https://arxiv.org/html/2604.15732v1.

Serving reasoning models: the invisible scratchpad that decides your latency and your bill

Fri, 12 Jun 2026 05:30:00 +0200

TL;DR

A reasoning model generates, before the answer you see, a thinking block, hundreds or thousands of tokens enclosed between <think> and </think>, that the user does not read but that consumes the same compute, the same VRAM and the same bill as any other token. Serving them in production is not “serving a bigger model”: it is serving a model whose cost per request is variable and, by default, uncontrolled. Three levers change the equation: choosing well between think / non-think mode (the hybrid models of 2026 allow it within the same weights), setting a reasoning budget (thinking_token_budget in vLLM, s1-style budget forcing) and measuring the impact on the KV cache (a 30k-token reasoning chain can eat ~9 GB of KV in FP16). Managed well, you gain accuracy where it matters; managed badly, you pay up to 113× more energy for an answer that did not need that much thinking.

The analogy

Picture a candidate sitting a competitive exam. Before writing the good answer on the official sheet, they fill three sheets of scratch paper: they try approaches, correct themselves, discard. Those scratch sheets are not handed in, the board only reads the final answer, but the candidate has spent ink, time and half an hour of the exam on them.

A reasoning model does exactly that. The <think>…</think> block is the scratch paper: it reasons out loud, contradicts itself, backtracks, and finally emits the “clean” answer. The production problem is that you pay for the scratch paper too: every reasoning token takes a slot in the batch, grows the KV cache and adds to the bill just like an output token. And, like any nervous candidate, the model tends to overthink: it writes five sheets when one would have done.

The operational question in this post is not “does it reason well?”, but “how much scratch paper do I let it write, and when do I take the pen away?”.

What changes when the model “thinks”

In a normal model, a request is: prompt → output tokens. In a reasoning model the sequence has two stretches:

Reasoning block (<think> … </think>): the chain of thought (CoT). Long, variable, normally hidden from the end user.
Final answer: what the user sees.

vLLM models this explicitly. When started with a reasoning parser, the output carries a reasoning_content field separate from content:

vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
 --reasoning-parser deepseek_r1

The parser detects the thinking delimiters and separates both stretches in the API response (vLLM · Reasoning Outputs). This matters for serving for one practical reason: if your gateway or your UI does not separate reasoning_content from content, you will end up showing the scratch paper, or, worse, recording it in logs and traces without meaning to.

The 2026 landscape: from the model that “always reasons” to the switchable hybrid

The first generation (o1, DeepSeek-R1, QwQ-32B) were models that always reasoned. In 2026 the dominant pattern is the hybrid: a single set of weights that switches between think and non-think.

DeepSeek-V3.1 introduced a hybrid reasoning architecture in a single model: DeepSeek-V3.1-Think for complex problems and a faster direct mode for the trivial, switchable by the user (InfoQ). Important for serving: reasoning is off by default and is turned on with thinking=True in chat_template_kwargs.
Qwen3 brings Hybrid Thinking Modes and an explicit Thinking Budget to bound the reasoning compute; in its series thinking is on by default (like QwQ-32B).
gpt-oss shows a different reasoning structure from the R1/Qwen family, which in practice means that the parser and the delimiters are not universal: each family has its own.

The architectural consequence is clear: in 2026 you no longer decide “a model that reasons” vs “one that does not”, but when you let the same model reason. That turns reasoning into a routing and budgeting decision, not a model-catalogue one. It connects directly with the L7 inference router: the “cheap” route turns thinking off; the “expensive” one turns it on with a cap.

The economics: why the scratch paper drives up the bill

Here is the heart of the problem. Reasoning tokens are invisible to the user but real in compute and in billing. Providers charge them as output tokens; API providers bill those reasoning tokens at the output price, not for free (Test-Time Compute Quietly Changed the Economics of Inference). On-premise there is no external invoice, but there is a cost: they occupy batch slots, KV VRAM and watts.

The billable cost (or the equivalent cost in GPU·hours) of a request is:

$$\text{compute tokens} = \text{prompt} + \underbrace{\text{reasoning tokens}}_{\text{hidden, variable}} + \text{visible answer}$$

The middle term is the one that runs out of control. On hard tasks the reasoning/visible-answer ratio can be enormous, and empirical analysis has measured up to 113× the energy cost against standard inference on certain coding tasks (Towards Data Science · Inference Scaling).

And it does not always buy accuracy. The overthinking phenomenon is documented: more thinking does not guarantee a better answer, and beyond a certain point it makes things worse. There is work showing that shorter chains are up to 34.5 % more accurate than the longest one sampled for the same question (Don’t Overthink it), plus studies on the “mirage” of test-time scaling (Mirage of Test-Time Scaling) and on when more thinking hurts (When More Thinking Hurts). The first large-scale study of test-time scaling, more than 30 billion tokens generated with 8 open-source LLMs from 7B to 235B across 4 reasoning datasets, systematises where it scales and where it does not (The Art of Scaling Test-Time Compute).

The operational reading: reasoning is a cost you have to budget for, not a “more quality” switch you leave on full all the time.

Controlling the thinking budget

Three mechanisms, from coarsest to finest.

1. Switching think / non-think

The cheapest thing is not to reason when it is not needed. On hybrid models, turn thinking off for classification, extraction, formatting, trivial answers:

# DeepSeek-V3.1: reasoning OFF by default; turned on explicitly
chat_template_kwargs = {"thinking": True} # only on the "hard" route

This is 80 % of the saving for 20 % of the effort: route by difficulty and reserve thinking for what actually pays for it.

2. Setting a cap: `thinking_token_budget`

vLLM ships a sampling parameter thinking_token_budget that counts reasoning tokens and, on reaching the limit, forces the block to close (reasoning_end_str, typically </think>), making the model answer straight away. If it is not specified, there is no limit other than max_tokens (PR #37112, implementation note). Underneath, a ReasoningBudgetLogitsProcessor injects something like “Let me stop thinking and answer now.</think>” when the cap is reached.

# Define the delimiters of the reasoning block
vllm serve Qwen/Qwen3-32B \
 --reasoning-parser qwen3 \
 --reasoning-config '{"reasoning_start_str":"<think>","reasoning_end_str":"</think>"}'

// per request: cut the scratch paper off at 1024 tokens
{"messages":[...], "thinking_token_budget": 1024}

3. Budget forcing (state of the art)

The technique that popularised fine-grained control is budget forcing, from the s1 paper: to spend less, thinking is terminated by forcing the end delimiter; to spend more, the end is suppressed and the word “Wait” is appended, which pushes the model to review and often correct its own reasoning. With this, s1-32B (a Qwen2.5-32B-Instruct fine-tuned on just 1,000 examples) beat o1-preview by as much as 27 % on AIME24/MATH (s1: Simple Test-Time Scaling). The serving lesson: the budget is not only a cost cap, it is a bidirectional quality lever.

$$\text{accuracy} \approx f(\text{reasoning tokens}) \quad\text{with a maximum, not monotonic}$$

The curve rises, saturates and on many tasks falls. Your job is to operate it near the knee, not at the extreme.

The impact almost nobody measures: the KV cache

This is where reasoning really bites serving. Every token of scratch paper is one more token in the KV cache. A long reasoning chain inflates the working memory just like a long input context.

The KV size grows linearly with the sequence (reasoning included):

$$\text{KV bytes} = 2 \times L \times h_{kv} \times d_{head} \times s \times b$$

where $L$ is layers, $h_{kv}$ KV heads (few, with GQA), $d_{head}$ the per-head dimension, $s$ the sequence length and $b$ the bytes per element. The term reasoning blows up is $s$: a “normal” 500-token answer and one with 8,000 tokens of thinking do not take up the same space, not by a long way.

The real numbers are frightening: for a distilled reasoning model with standard attention such as DeepSeek-R1-Distill-Llama-70B, a single 30,000-token reasoning chain consumes ~9 GB of KV in FP16, which quantising the KV to FP8 reduces to ~4.5 GB, one of the highest-impact optimisations for chain-of-thought workloads (Spheron · KV Cache Optimization). Multiply that by concurrency and you understand why a cluster that served 200 simultaneous requests of a “normal” model drops to 40 with the same model reasoning.

Two state-of-the-art nuances worth knowing:

Prefix caching is not free here. The empirical study of reasoning-model serving finds that prefix caching clearly improves speed on models of 14B or more, but hurts on 7B models (cache management costs more than it saves) (Reasoning Language Model Inference Serving Unveiled). And since the bulk of the reasoning cost sits in the decode (generating tokens one by one), prefix caching, which only speeds up the prefill, barely touches the expensive phase. It connects with prefix cache: engineering the hit rate.
Reasoning-specific KV compression. Techniques such as R-KV have appeared, compressing the KV by exploiting the redundancy typical of chains of thought (the model repeats itself and rambles), recovering memory without touching quality (R-KV).

The most profitable and available lever today is still KV in FP8 (see FP8 end to end) combined with a reasoning budget that avoids 30k-token tails in the first place.

Reasoning + structured output: order matters

If you serve structured output (JSON with a schema) and reasoning at the same time, there is a subtle trap: the grammar must not be applied during the thinking block. The model needs free text to reason; forcing the schema inside the <think> breaks the reasoning. vLLM solves it by having the structured-output engine (xgrammar) use the reasoner’s end_token_id to skip the constraint while the block lasts and apply it only to the final answer (vLLM · Reasoning Outputs). If you build this by hand on another engine, remember: first let it think, then impose the form. It links with structured output.

Reference architecture

On the blog’s example cluster, a node of 4×H100 SXM (80 GB, NVLink), a sensible topology for serving reasoning without wrecking the SLO:

A cheap difficulty classifier up front (a small model or rules) that picks the route. Most of the traffic does not need to think.
Fast route (non-think): the same hybrid model with reasoning off. Low TTFT, predictable cost.
Reasoning route (think): reasoning on with thinking_token_budget by default (say 1–2k tokens) and the option to raise it per request for the genuinely hard cases.
KV in FP8 in the engine on the reasoning route, plus per-request reasoning-length metrics exported to your observability (instrumenting vLLM with OTel).

To prototype this routing outside the cluster, an RTX 5090 (Blackwell, 32 GB) serves a 7–14B reasoning model and lets you validate parsers, budgets and the reasoning_content/content separation before touching the H100s. Do not expect to serve a 32B in think mode at high concurrency on a consumer card: the reasoning KV eats the 32 GB in no time.

Sizing starts, as always, from an SLO; the difference is that now the “answer size” in your capacity planning has to count the scratch paper, not just the visible answer.

Operational pitfalls (and honest scepticism)

Leaking reasoning_content. If the gateway does not separate it, you end up showing or logging the scratch paper, which may contain wrong attempts, sensitive data or language you do not want in your UI. Always separate it at L7.
Serving reasoning “just in case”. Overthinking is real and sometimes lowers accuracy. Do not turn think on by default: turn it on per route, with a budget.
Forgetting the KV. The most common mistake: sizing concurrency from the length of the visible answer. The KV is dictated by the total sequence, reasoning included. Measure p50/p95 reasoning tokens per endpoint.
Prefix caching as a silver bullet. On small models it can make things worse, and in any case it does not touch the decode, which is where the reasoning cost goes.
Assuming universal delimiters. R1, Qwen3 and gpt-oss do not share a reasoning structure. Use the right --reasoning-parser per family and test streaming (the parser has to separate correctly token by token).
Believing more compute = more intelligence. The state of the art (s1, short-m@k, “When More Thinking Hurts”) points the other way in many cases: there is a knee, and operating beyond it is burning watts. Test-time compute is a tool with diminishing, and sometimes negative, returns, not a linear lever.

A note of caution for June 2026: the field of reasoning control (budget forcing, reasoning-specific KV compression, thinking early-exit) moves fast and almost all of it is very recent. The specific figures, the 113×, the 34.5 %, the 9 GB, depend on model, task and configuration; take them as an order of magnitude for design, not as constants. Measure on your own workload before committing to SLOs.

Closing

Serving a reasoning model means serving a model whose cost per request you decide with the thinking budget, not the catalogue. The chain goes: route by difficulty → turn think off where it adds nothing → set thinking_token_budget by default on the route that thinks → quantise the KV to FP8 → measure reasoning length as a first-class metric. Do that, and reasoning becomes a capability you pay for only when it buys accuracy. Do not, and your cluster will serve a fifth of the users while the model fills sheets of scratch paper nobody is going to read.

Sources

vLLM · Reasoning Outputs — https://docs.vllm.ai/en/stable/features/reasoning_outputs/
vLLM PR #37112 · reasoning_budget — https://github.com/vllm-project/vllm/pull/37112
Adding Reasoning Budget to vLLM — https://shuyo.wordpress.com/2026/03/26/adding-reasoning-budget-to-vllm-reasoning-token-limit/
s1: Simple Test-Time Scaling — https://arxiv.org/abs/2501.19393
The Art of Scaling Test-Time Compute — https://arxiv.org/abs/2512.02008
Don’t Overthink it (short-m@k) — https://arxiv.org/pdf/2505.17813
Mirage of Test-Time Scaling — https://arxiv.org/pdf/2506.04210
When More Thinking Hurts — https://arxiv.org/html/2604.10739v1
Reasoning Language Model Inference Serving Unveiled — https://arxiv.org/pdf/2510.18672
R-KV: Redundancy-aware KV Cache Compression — https://arxiv.org/pdf/2505.24133
Spheron · KV Cache Optimization Guide (2026) — https://www.spheron.network/blog/kv-cache-optimization-guide/
Test-Time Compute Quietly Changed the Economics of Inference — https://medium.com/@Elongated_musk/test-time-compute-quietly-changed-the-economics-of-inference-9d0b8d77641c
Inference Scaling (Test-Time Compute) · Towards Data Science — https://towardsdatascience.com/inference-scaling-test-time-compute-why-reasoning-models-raise-your-compute-bill/
DeepSeek-V3.1 Hybrid Reasoning · InfoQ — https://www.infoq.com/news/2025/09/deepseek-v31-hybrid/

Long context and KV offloading: when the notebook does not fit on the desk

Fri, 12 Jun 2026 05:00:00 +0200

TL;DR

Serving long context is not serving a smarter model, it is managing a notebook that does not fit on the desk. The context window is set by the model (and extended with RoPE scaling / YaRN), but the cost of serving it is set by the KV cache, which grows linearly with sequence length and, in production, explodes: a 300k-token contract on Llama 3 70B eats ~93 GB of KV, more than an entire H100, and a million tokens asks for ~125 GB. When the KV does not fit in HBM, there are only two ways out: recompute (extremely expensive, attention is quadratic) or offload the KV to a cheaper memory hierarchy: DRAM, NVMe, network. The 2026 OSS state of the art, LMCache, Mooncake (Kimi’s platform) and NVIDIA Dynamo/KVBM, turns that offload into a first-class KV layer, with reuse across requests, prefill/decode disaggregation and cache-aware routing. Reported results: 3×–10× less latency with LMCache and up to +525 % throughput in long-context scenarios with Mooncake. The price: every memory hop adds transfer latency, so offloading only pays off when what you save by not recomputing beats what it costs to move the bytes.

The analogy

A researcher works at a small desk. On it fit the papers they are pulling from right now, which is the GPU’s HBM, blazingly fast but tiny. When the case is long (a thousand-page file, a million-token context), the papers do not fit. There are three options:

Throw papers away and ask the archive for them again every time they are needed. That is recomputing the KV: correct, but agonisingly slow, because “asking the archive” in an LLM means running the whole prompt through attention again, and attention is quadratic.
Put a shelf next to the desk (the CPU’s DRAM) and, further out, a store room in the basement (NVMe) and a depot in another building (network/object). Moving papers between the desk and the shelf costs seconds, not hours. That is hierarchical KV offloading.
Have two researchers split the work: one reads and marks up the whole file (prefill), then passes their notes to the second, who only drafts (decode). That is the KVCache-centric disaggregated architecture.

This post is about options 2 and 3, the ones that make long context viable in production. Option 1 is what you pay by default if you do nothing.

Part 1 · Why long context is a memory problem

The KV cache grows with the sequence

For every token that goes in or out, the model stores its key and value vectors in each layer so it does not have to recompute them. The size is:

$$\text{KV bytes} = 2 \times L \times h_{kv} \times d_{head} \times s \times b$$

with $L$ layers, $h_{kv}$ KV heads (few, with GQA), $d_{head}$ the per-head dimension, $s$ the sequence length and $b$ the bytes per element. Everything is a model constant except $s$: the KV is linear in context length. Doubling the context doubles the KV. That is the basis of KV cache: the working memory of inference.

The numbers are frightening

At production scale, that linear term turns brutal:

A 300k-token contract on Llama 3 70B consumes ~93 GB of KV alone, more than the 80 GB of an entire H100 (DigitalOcean · Long-Context Inference Cost).
A 1M-token context needs ~125 GB of KV, which exceeds both an RTX 4090 (24 GB) and an 80 GB A100 (Introl · Long-Context LLM Infrastructure).
Even a 7B at 128k climbs to ~14 GB of KV (against ~6 GB at 4k).

The KV stops being an implementation detail and becomes the resource that dictates your concurrency.

And on top of that, attention is quadratic

The KV is linear, but the compute of attention is quadratic in length. When the prompt reaches 1M tokens, generating each token can take on the order of 1.765 seconds, with more than 96 % of the latency spent in attention (Introl). The aggregate effect is a throughput collapse of 10×–100× against short contexts. This is why long context is not “fixed” with more VRAM alone: there is a memory problem (KV) and a compute problem (attention) at the same time.

Extending the window ≠ serving it cheaply

A common confusion: “my model supports 1M of context” does not mean “I can serve 1M cheaply”. The window is extended with RoPE scaling techniques such as YaRN, the practical option for fine-tuning open source models to long context: it needs 10× fewer training tokens and 2.5× fewer steps than naive RoPE interpolation, and it took LLaMA-2 from 4k to 32k and 128k (YaRN/LongRoPE). But that solves the model attending to long context, not the KV fitting on your GPU. According to the 2026 round-ups, May 2026 marked the first generation of open million-token models (with families supporting 256k extensible to 1M via YaRN) (letsdatascience). Treat it as a trend, not a closed spec, and verify capabilities per concrete model.

Part 2 · The KV memory hierarchy

The core idea of offloading is old in systems: hierarchical caching. “Hot” KV (what is in use) lives in HBM; “warm” KV drops to DRAM; “cold” KV to NVMe; “shared across nodes” to network or object storage. Every hop multiplies capacity and divides bandwidth.

Working out when offloading pays off

Offloading is not magic: moving KV from DRAM/NVMe back to HBM costs time. The rule is simple. Offloading pays off when the cost of recomputing exceeds the cost of transferring:

$$t_{recompute}(s) \;>\; t_{transfer} = \frac{\text{KV bytes}}{BW_{enlace}}$$

Since $t_{recompute}$ grows with attention (quadratic) and $t_{transfer}$ grows only with the size of the KV (linear) divided by the link bandwidth, the longer the context, the more the balance tips in favour of offloading. That is why offloading is the lever for long context: it is exactly where recomputing becomes unbearable. The LMCache documentation says it without hedging: offloading only helps when the recomputation it avoids is larger than the overhead it introduces (LMCache benchmark).

Part 3 · KV offloading in practice (OSS)

The limit of out-of-the-box prefix caching

vLLM already caches prefixes in HBM (see prefix cache: hit rate engineering and PagedAttention). The problem: it only lives in HBM, which is small. When the useful KV exceeds HBM, it is evicted and, on the next request, has to be recomputed. There is no speedup even with prefix caching enabled, because the cache is no longer there (vLLM vs LMCache benchmark). Native prefix caching is necessary but insufficient for long context.

LMCache: the persistent KV layer

LMCache adds persistent storage backends to vLLM’s prefix cache (and also supports SGLang and NVIDIA Dynamo as engines). It pulls the KV out of HBM and shares it across engines and across queries, covering a three-level hierarchy: GPU HBM, CPU DRAM and NVMe SSD, with additional backends for Redis/Valkey, Mooncake, InfiniStore, S3 and NIXL/GDS (GitHub LMCache, arXiv 2510.09665). Because DRAM/NVMe hold far more KV than HBM, the hit ratio rises, and LMCache reports 3×–10× less latency combined with vLLM. Against vLLM’s basic CPU offload, LMCache loads the KV at chunk level with high-performance CUDA kernels, which cuts transfer overhead.

A minimal start, offloading to CPU:

# vLLM with LMCache as the KV layer (offload to DRAM)
pip install lmcache
LMCACHE_CONFIG_FILE=lmcache.yaml \
vllm serve meta-llama/Llama-3.1-70B-Instruct \
 --tensor-parallel-size 4 \
 --kv-transfer-config '{"kv_connector":"LMCacheConnector","kv_role":"kv_both"}'

# lmcache.yaml — hierarchy HBM -> DRAM -> NVMe
chunk_size: 256
local_cpu: true # warm KV in DRAM
max_local_cpu_size: 200 # GB
local_disk: "file:///nvme/lmcache" # cold KV on NVMe
max_local_disk_size: 2000 # GB

Offloading to CPU adds moderate overhead but retains a good part of the benefit against recomputing (LMCache docs). Mind the physical path: that HBM↔DRAM↔NVMe traffic goes over PCIe and across NUMA nodes. It is worth reading PCIe topology, GPUDirect and ACS and NUMA, hugepages and CPU isolation, because a badly wired offload can eat its own gain.

Combine it with FP8

Offloading reduces where the KV lives; quantization reduces how much it weighs. They are orthogonal and they add up: KV in FP8 halves the size (see FP8 end-to-end), which means half the bytes to transfer on every offload. In long context, FP8 + offload is the default combination.

Part 4 · KVCache-centric / disaggregated architecture

The next level is not just offloading KV, but redesigning serving around the KV.

Mooncake (Kimi’s platform)

Mooncake, the serving platform behind Kimi (Moonshot AI), is the reference KVCache-centric disaggregated architecture: it separates the prefill and decode clusters, and exploits the underused CPU, DRAM and SSD resources of the GPU cluster to build a distributed KV cache. Its core is a KVCache-centric scheduler that maximises throughput while respecting SLOs, with a prediction-based early rejection policy for overloaded scenarios (arXiv 2407.00079, USENIX FAST'25). The long-context numbers are striking: up to +525 % throughput in certain simulated scenarios while respecting the SLO, and in production Kimi handles 115 % and 107 % more requests on A800 and H800 clusters respectively against previous systems. The title of their FAST talk sums it up: “Trading More Storage for Less Computation”, exactly the rule from Part 2.

NVIDIA Dynamo and KVBM

NVIDIA Dynamo is the open source distributed serving framework with prefill/decode disaggregation, dynamic GPU scheduling and KV-aware routing: the PrefillRouter computes overlap scores between the incoming request and the KV blocks already cached, and routes to the GPUs that already have the relevant KV, avoiding recomputation (Dynamo · Disaggregated Serving, Dynamo · KV-aware routing). KV transfer between prefill and decode is handled by NIXL directly GPU-to-GPU over the best available transport (NVLink, InfiniBand). And its offload manager, KVBM (KV Block Manager), has a three-layer architecture (LLM runtime, logical block management, NIXL transport) that offloads cold KV to CPU RAM, NVMe or network storage to free up HBM (Dynamo · KV Cache Offloading). This connects with disaggregated serving: prefill and decode in specialised pods.

The llm-d project walks the same road, from vLLM’s prefix caching to KV-aware distributed scheduling (llm-d · KV-Cache Wins You Can See).

Part 5 · Reference architecture

On the blog’s example cluster, a node with 4×H100 SXM (80 GB, NVLink), local NVMe and plenty of DRAM, a sensible stack to serve long context without recomputing pointlessly:

KV in FP8 in the engine (vLLM), to start with half the bytes.
LMCache as the KV layer with a DRAM→NVMe hierarchy, so that long prefixes (a manual, a code base, a case file) are cached once and reused across sessions without recomputation.
KV-aware routing (Dynamo or llm-d) if you have several nodes: send the request to the GPU that already holds the prefix, not to any GPU at all.
Prefill/decode disaggregation once the prefill of enormous contexts starts stealing decode from the other requests. Prefilling 300k tokens monopolises the GPU and kills everyone’s latency.
Metrics for KV hit ratio, bytes transferred per level and prefill/decode ratio, exported to your observability stack (instrumenting vLLM with OTel). Without those three metrics, offloading is faith, not engineering.

To prototype the stack, validating the LMCache config, the backend formats, the hit/miss behaviour, an RTX 5090 (Blackwell, 32 GB) with NVMe is more than enough to serve a 7–14B and watch offloading work at contexts of 32–128k. Do not expect to serve 1M on a consumer card: the long-context problem is precisely that neither the KV nor the attention fit on small hardware. Real sizing starts from the SLO and from the context-length distribution of your traffic, not from the model’s maximum window, and that is where it links to capacity planning.

Part 6 · Operational pitfalls (and honest scepticism)

Believing that native prefix caching is enough. It only lives in HBM; in long context it is evicted and you recompute. You need a persistent layer (LMCache/KVBM) for the cache to survive.
Offloading without measuring the break-even. If your prefixes are not reused, offloading only adds transfer latency without saving recomputation. Offloading shines with shared prefixes (RAG over the same corpus, agents with the same system prompt, long sessions), not with one-off, never-repeated requests.
Forgetting that attention is still quadratic. Offloading fixes the memory of the KV, not the compute of attention. For real 1M tokens you also need efficient attention, sparse attention or retrieval approaches (RetrievalAttention, ShadowKV); offloading on its own does not bring down the 1.765 s/token.
Wiring the offload badly. HBM↔DRAM↔NVMe crosses PCIe and NUMA. An offload that crosses the wrong NUMA node or saturates a PCIe lane can be slower than recomputing. Measure the link’s real bandwidth, not the datasheet figure.
Treating the figures as constants. The 93 GB, the +525 %, the 3–10× all depend on model, length, hardware and reuse pattern. They are orders of magnitude for design; measure on your own load.
Trusting million-token model specs without verifying. The landscape of open 1M models is very recent (2026) and moves fast; the specific names and limits change between versions. Verify the window and the cost of serving it per model, do not take the headline on trust.

A note of caution for June 2026: the distributed KV layer (LMCache, Mooncake, KVBM, NIXL, llm-d) is setting right now, with APIs and backends that change release by release. It is exactly the kind of piece worth assembling behind an abstraction (vLLM’s kv_connector) rather than coupling to a single vendor.

Closing

Long context is a problem of memory and compute at the same time, and the KV cache is the bottleneck of both. The window is a gift from the model; serving it cheaply is systems engineering: quantise the KV (FP8), offload it by tiers (HBM→DRAM→NVMe→network) with a persistent layer, and reorganise serving around the KV (aware routing, separated prefill/decode) when scale demands it. The rule that governs it all is Mooncake’s: trade storage for computation. You store more KV in cheap places so you do not recompute expensive attention. Do it well and a 4×H100 node serves contexts that, by recomputing, would not even get started. Do it badly, or not at all, and every long request reads the whole case file from scratch again, while the GPU drowns in a quadratic attention you had already paid for once.

Sources

LMCache · GitHub — https://github.com/LMCache/LMCache
LMCache: An Efficient KV Cache Layer (arXiv 2510.09665) — https://arxiv.org/pdf/2510.09665
LMCache · Offload KV cache to CPU (docs) — https://docs.lmcache.ai/getting_started/quickstart/offload_kv_cache.html
vLLM Prefix Caching vs. LMCache: Benchmarking KV Reuse Tradeoffs — https://levelup.gitconnected.com/vllm-prefix-caching-vs-lmcache-benchmarking-kv-reuse-tradeoffs-944fbaf98b56
Mooncake: A KVCache-centric Disaggregated Architecture (arXiv 2407.00079) — https://arxiv.org/abs/2407.00079
Mooncake · USENIX FAST'25 (Trading More Storage for Less Computation) — https://www.usenix.org/conference/fast25/presentation/qin
Mooncake · GitHub — https://github.com/kvcache-ai/Mooncake/
NVIDIA Dynamo · Disaggregated Serving (docs) — https://docs.dynamo.nvidia.com/dynamo/design-docs/disaggregated-serving
NVIDIA Dynamo · KV-aware routing (docs) — https://docs.nvidia.com/dynamo/latest/user-guides/kv-cache-aware-routing
NVIDIA Dynamo · KV Cache Offloading / KVBM (docs) — https://docs.nvidia.com/dynamo/backends/v-llm/kv-cache-offloading
llm-d · KV-Cache Wins You Can See — https://llm-d.ai/blog/kvcache-wins-you-can-see
Long-Context Inference at Scale: The Hidden Infrastructure Cost · DigitalOcean — https://www.digitalocean.com/community/tutorials/long-context-inference-production-cost
Long-Context LLM Infrastructure · Introl — https://introl.com/blog/long-context-llm-infrastructure-million-token-windows-guide
YaRN / LongRoPE (arXiv 2402.13753) — https://arxiv.org/html/2402.13753v1

Multimodal on-premise: serving a VLM with vLLM (vision + language)

Thu, 11 Jun 2026 11:40:00 +0000

Third batch of an operational series on squeezing a generic on-premise LLM cluster of 4×H100 SXM. Its sibling pieces in this batch are FinOps and GPU multi-tenancy with LiteLLM, which puts a price on every token, and a visual token costs the same as a text one, and Speeding up cold start with Tensorizer, which matters twice as much here because a VLM loads two models (the vision encoder and the LLM). If the VLM ends up feeding a document assistant, that end-to-end setup with LibreChat and RAG is another story (in preparation).

TL;DR

A vision-language model (VLM) is not magic: it is a normal LLM with a sensory organ bolted on the front. Three pieces in series: a vision encoder (a ViT) that looks at the image, a projector or connector (typically an MLP) that translates the encoder’s output into the LLM’s embedding space, and the usual LLM, which receives those embeddings as if they were just more tokens. The consequence that governs the entire operational design is brutal in its simplicity: an image becomes visual tokens, and those tokens cost exactly the same as text tokens. A high-resolution document page is not “a couple of image tokens”: it is hundreds or thousands of tokens entering the context, inflating the prefill (compute-bound) and occupying KV cache (memory-bound) just as if you had pasted a thousand words. In Qwen2.5-VL the arithmetic is literal: the number of visual tokens is $H\times W / (14\times14\times4)$, so an $896\times896$ image is 1,024 tokens and an A4 page scanned at a decent resolution easily reaches 1,500–2,500 tokens for the image alone. This post opens up the anatomy, works through the token-cost maths and translates it into TTFT and VRAM on an 80 GB H100, explains vLLM’s multimodal support (--limit-mm-per-prompt, pixel budget via mm_processor_kwargs, chat template) and answers the question that really matters: classic OCR or VLM. The short answer: use Tesseract/PaddleOCR for the bulk of documents with clean text, and reserve the VLM, expensive in tokens and in GPU, for the subset that genuinely needs it: complex layout, nested tables, handwriting, stamps, charts. On the generic 4×H100 SXM cluster, with FP8 so it fits.

The analogy: the reader who sees the page versus the one who only hears the transcript

Picture two experts you ask to review a scanned contract. To the first you read the contract aloud, a blind transcript, word by word. It is fast, efficient, and for 90 % of the clauses it is enough: the text is the text. But there are things you cannot read out: that the signature at the end is handwritten and does not match the typed name; that there is a “PAID” stamp struck diagonally across the amount; that the table on page 3 has a merged cell that changes who each row applies to; that there is a pen annotation in the margin. The blind transcript loses all of that or flattens it into gibberish.

To the second expert you hand the whole page, to look at. They see the signature, they see the stamp, they understand the structure of the table because they perceive the lines and the cells, they read the marginal note. They grasp the document as a visual object with layout, not as a river of characters. They are strictly more capable.

Why not always use the second one, then? Because seeing costs more mental bandwidth. The first expert receives the contract as a flow of words: cheap, linear. The second receives it as an image that their brain has to parcel up, attend to region by region, and reconstruct. They process a great deal more information per page, most of which, in a clean text document, is redundant with what the transcript would already have given them. If you only need to read the clauses, hiring the expert-who-sees for every page is overpaying for capability you do not use.

That is the entire thesis of this post. The VLM is the expert who sees; OCR is the blind transcript; and “mental bandwidth” is tokens. The engineering consists of sending the expert-who-sees only the pages where seeing changes the answer, and resolving the rest with the cheap transcript.

Anatomy of a VLM: encoder, projector, LLM

A modern VLM of the Qwen-VL family (Qwen2.5-VL model card; Qwen3-VL Technical Report) is made up of three blocks in series:

Vision encoder (ViT). A Vision Transformer that receives the image, cuts it into patches and produces an embedding per patch (or per group of patches). In Qwen2.5-VL it is a dynamic-resolution ViT: it accepts images of arbitrary size, resizing them to multiples of 28 and splitting them into 14×14 pixel patches, with window attention to speed up the processing of large images. There is no “canonical size” everything is squashed to: a large image produces more patches than a small one.
Projector / connector. The ViT’s embedding space is not the LLM’s. The projector acts as the translator. In Qwen2.5-VL it is elegant and cheap: it groups every 2×2 block of adjacent patches (four ViT tokens), concatenates them and projects them with a two-layer MLP into a single fused token in the LLM’s dimension. That fusion factor of 4 is the reason for the $4$ that appears in the denominator of the token formula below: four ViT patches collapse into one visual token that the LLM sees.
The LLM. The usual language transformer. It receives a sequence of embeddings that is a mixture of text tokens (from your prompt) and visual tokens (from the image), all in the same space, and does what it knows how to do: attention over the complete sequence and autoregressive generation. To the LLM, a visual token and a text token are indistinguishable in cost: both go through the same QKV projections, both occupy an entry in the KV cache, both take part in the $O(C^2)$ attention.

The idea to internalise: the encoder and the projector are the sensory organ, but the inference cost lives in the LLM, and that cost is paid in tokens. The ViT adds a fixed prefill overhead (processing the image once), but the recurring bill, the LLM prefill and the KV cache throughout the whole generation, is set by the number of visual tokens the projector spits out. That is why the control lever is not “which encoder”, but how many pixels you let through.

The maths: what it costs to see a page

Let us put numbers on “an image costs many tokens”, with the real Qwen2.5-VL formula (Qwen team, Qwen2.5-VL blog):

$$N_{\text{tok}} = \frac{H \times W}{14 \times 14 \times 4} = \frac{H \times W}{784}$$

where $H$ and $W$ are height and width in pixels (rounded to the multiple of 28 the ViT imposes). The $14\times14$ is the patch size; the $\times4$ is the projector’s 2×2 fusion. A sanity check against the model card figure: an $896\times896$ image gives

$$N_{\text{tok}} = \frac{896 \times 896}{784} = \frac{802{,}816}{784} = 1{,}024 \text{ tokens.}$$

It checks out. Now an A4 page scanned at a resolution reasonable for reading small text, say $1120 \times 1568$ px (approximate height $\times$ width, already rounded to multiples of 28):

$$N_{\text{tok}} = \frac{1120 \times 1568}{784} = \frac{1{,}756{,}160}{784} \approx 2{,}240 \text{ tokens.}$$

A single page is ~2,240 visual tokens before you write a word of prompt. For context: that is roughly what 1,700 words of Spanish text would take up (at ~1.3 tokens/word). If that A4 page contains 600 words of actual text, the VLM is paying almost 4× in tokens to see it rather than read its transcript. If the page is a dense table or a handwritten document that OCR cannot read, that 4× is justified. If it is a paragraph of clean text, it is pure waste.

Qwen2.5-VL lets you control this with min_pixels / max_pixels: the number of tokens per image is dynamic and ranges from about 4 up to 16,384 by default, adjustable with those parameters (HF Qwen2.5-VL docs). Lowering max_pixels cuts tokens at the expense of resolution, and of the ability to read the fine print. It is exactly the lever of “how much mental bandwidth do I give the expert”.

From token cost to TTFT

Visual tokens are not free in latency. The prefill, processing the whole context before the first token, is compute-bound and its cost grows with the context length $C$ (linear in the projections, quadratic in the attention); this is worked through in detail in vLLM’s attention backend. Every visual token enters that $C$.

Let us model the TTFT as the time to process the prefill tokens at a given prefill throughput. Take a VLM of around 7–8B on an H100 with an illustrative prefill throughput of ~12,000 tok/s (an example figure; measure it, do not assume it, and it depends on how much the ViT forward pass weighs on top). For a request with an 80-token text prompt plus one page at 2,240 visual tokens:

$$\text{TTFT} \approx \frac{80 + 2240}{12{,}000 \text{ tok/s}} \approx \frac{2320}{12{,}000} \approx 0{.}19 \text{ s}$$

plus the fixed cost of the vision encoder’s forward pass over the ~9,000 ViT patches (another compute-bound slice of prefill). Now five pages in the same request (a short PDF):

$$\text{TTFT} \approx \frac{80 + 5 \times 2240}{12{,}000} = \frac{11{,}280}{12{,}000} \approx 0{.}94 \text{ s}$$

The TTFT multiplies almost fivefold, because the visual tokens completely dominate the context. And attention scales as $O(C^2)$: five pages are not “5× more expensive” in the attention component, but ~25× relative to one. The operational lesson: the number of images and their resolution are your TTFT budget, not a detail. Lowering max_pixels from 2,240 to, say, 1,100 tokens/page (lower resolution, enough for large text) cuts the prefill almost in half.

VRAM: the VLM’s weights and what is left for KV

On an 80 GB H100 SXM, let us budget a VLM of the Qwen-VL family. Take an 8B (LLM) plus the vision encoder (a ViT of a few hundred million parameters; call the whole thing ~8.5B effective parameters for weight purposes). In BF16 (2 bytes/parameter):

$$M_{\text{BF16}} \approx 8{.}5 \times 10^9 \times 2 \text{ B} \approx 17 \text{ GB.}$$

In FP8 (1 byte/parameter), the weights halve (Qwen3-VL-8B-Instruct-FP8, with reported quality almost identical to BF16):

$$M_{\text{FP8}} \approx 8{.}5 \times 10^9 \times 1 \text{ B} \approx 8{.}5 \text{ GB.}$$

For an 8B model this fits comfortably in BF16 on an H100; FP8 matters more for leaving VRAM free for KV cache, and here the KV is the problem, because visual tokens inflate it. The detail of end-to-end FP8 (weights and KV) is in FP8 end to end. What is relevant: one page is ~2,240 KV entries; serving several sessions that send long documents consumes KV at a rate a text-only case would never see. With a 32B VLM in FP8 (~32 GB of weights) on an 80 GB H100 you are left with ~45 GB for KV and activations, which with long multimodal contexts run out sooner than “text-only” intuition suggests. The rule: with a VLM, plan the VRAM counting the visual tokens in the KV, not just the weights.

Serving a VLM with vLLM

vLLM supports image multimodal input natively for the Qwen-VL family and many other VLMs (Multimodal Inputs, vLLM docs). The gears that matter:

--limit-mm-per-prompt. Limits how many multimodal items each prompt accepts, per modality. By default the limit is generous (999 per modality), but in production it is worth setting it low, --limit-mm-per-prompt '{"image": 2}', because every accepted image reserves token and KV budget. If you set a limit of 8 images and someone sends 8 A4 pages, that is ~18,000 visual tokens in a single request: it blows up max_model_len or the KV. The limit is a budget firebreak, not a cosmetic restriction. (A curious case: setting it to 0 for a modality lets you pass precomputed embeddings without loading the encoder module, saving VRAM, useful if you do the vision encoding outside vLLM.)

Pixel budget via mm_processor_kwargs. Here the token formula connects with the configuration. When serving Qwen2.5-VL you pass, for example, --mm-processor-kwargs '{"min_pixels": 12544, "max_pixels": 254016}' (vLLM Qwen2.5-VL discussion). With $254{,}016$ maximum pixels per image, the token ceiling per image is $254{,}016 / 784 \approx 324$ tokens, an aggressive trim that keeps the cost in check at the price of not being able to read very small print. This is the dial that really governs your token bill: raise it for dense documents, lower it for images where fine detail does not matter.

The chat template and the image placeholder. The model’s processor (loaded via AutoProcessor) inserts special tokens marking where the image goes in the sequence: in Qwen2.5-VL the sequence <|vision_start|><|image_pad|><|vision_end|>, where <|image_pad|> expands to the exact number of visual tokens the image produced. vLLM applies this template automatically; it is worth knowing there is a documented quirk, the image block tends to be placed before the user’s text in the sequence order, regardless of where you put it in the content (issue #15125, vllm-project/vllm), which rarely causes trouble but is worth bearing in mind if the image/text order is semantically important for your prompt.

Current models. As of June 2026, the reference open family is Qwen3-VL (dense 2B/4B/8B/32B variants and MoE 30B-A3B / 235B-A22B, interleaved contexts of up to 256K tokens combining text, image and video), with published FP8 checkpoints of almost identical quality to BF16 (Qwen3-VL Technical Report, arXiv 2511.21631; Qwen3-VL Usage Guide, vLLM recipes). Qwen2.5-VL remains perfectly serviceable and its token formula is the one we have used because it is public and clean; the cost figures are of the same order in both generations.

Classic OCR vs VLM: when to use which

Here is the decision that pays the bill. Parsing documents to feed them into a document ingestion pipeline has two paths, and choosing the split between them well is the difference between a healthy cluster and a choking one.

Classic OCR (Tesseract, PaddleOCR). Text-recognition engines that run on CPU, are cheap, fast and mature. Tesseract is the portable workhorse (edge, offline kiosks, embedded scanners). PaddleOCR is more capable on layout and Asian scripts, with its PP-Structure module for preserving structure, extracting tables and detecting key-value fields (PaddleOCR vs Tesseract, CodeSOTA 2026). On clean, well-scanned text, classic OCR is hard to beat on cost per page: zero GPU, millisecond latency, very high throughput. Its weakness is the “hard” document: complex multi-column layout, tables with merged cells, handwriting, stamps, charts, poor scan quality. There, pipeline-based OCR either fails or returns gibberish that destroys the structure.

VLM. It takes the image of the document and generates structured markdown or HTML in a single end-to-end pass, understanding the layout because it sees it. It shines exactly where classic OCR struggles: nested tables, reading charts, handwriting, understanding the spatial relationship between elements. The price is the one we have been calculating all through this post: expensive in tokens and in GPU. Worth noting a 2026 nuance: the frontier has moved. VLM models specialised in document parsing have appeared, compact (~1–3B), open source, scoring very high on benchmarks such as OmniDocBench (PaddleOCR-VL in the 94–96 point band; dots.ocr 3B at ~88 with full markdown including tables) (LLM-OCR vs Traditional OCR, Parsli 2026; Open-Source OCR Models, E2E 2025). In other words: for pure parsing, today you may not want a 32B generalist VLM, but a small specialised VLM-OCR. But it is still GPU and it still pays tokens per image: the cost calculation does not disappear, the model just gets cheaper.

The split rule. It is not “OCR or VLM” as a global choice, but per-document triage. Run everything through a cheap first filter: classic OCR already returns a confidence score; if it is high and the document has no tables or visual elements, the OCR transcript will do and you have spent CPU. If confidence is low, there are tables, there is handwriting or there are graphical elements, escalate that page to the VLM. In a typical corpus, 80–90 % of pages are resolved with cheap OCR and only the rest consumes GPU. This is what makes an on-premise VLM sustainable: you do not serve everything with it, you use it as the expensive specialist you send only the hard cases.

Applied to the generic 4×H100 cluster

How this fits into a cluster of 4×H100 SXM 80GB:

Where to place the VLM. A VLM deserves a whole H100 or a large slice, for a VLM-specific reason that text-only intuition does not anticipate: the token cost per image means not many concurrent sessions fit. If each request brings one or several pages, each consumes thousands of KV tokens; the KV cache runs out with few simultaneous sessions. Compared with a text-only chat service where dozens of short conversations fit per GPU, a document service saturates the KV with a handful of heavy requests. That is why it makes no sense to carve the GPU finely for the VLM with aggressive MIG (GPU sharing is covered in sharing a GPU: time-slicing, MPS and MIG): the bottleneck is not idle compute but the KV, and a small slice runs out of KV before it makes use of the compute.

FP8 so it fits and to free up KV. Serve the VLM in FP8: at equal reported quality, the weights take up half and you leave the maximum VRAM for the KV cache, which is the scarce resource with multimodal contexts. For a 32B in FP8 (~32 GB) on an H100, those ~45 GB free for KV are what determine how many concurrent pages you can handle.

The VLM only for the subset that needs it. The most important architectural piece: do not put the VLM on the critical path of every document. The triage from the previous section lives upstream; the bulk (clean text) is resolved with OCR/PaddleOCR on CPU, and in a cluster with 4×H100 there is probably spare CPU to run it in parallel with the GPUs, and only the hard subset reaches the VLM. This frees the H100s for what really needs them and keeps a reasonable cost per document. It connects directly with FinOps and GPU multi-tenancy with LiteLLM: when you put a price on every token, a visual token costs the same as a text one, so a page through the VLM can cost 4× its transcript, and that should show up in the chargeback of the team that decides to send everything to the VLM for convenience.

Double cold start. An operational detail that bites: a VLM loads two models into memory, the vision encoder and the LLM, so its cold start is heavier than that of an equivalent LLM. If you are going to do model swapping or on-demand scaling, fast loading matters twice as much; that is where speeding up cold start with Tensorizer comes in. If instead you serve several models on one GPU by rotating them, the swap+sleep pattern (serving several models on one GPU) has to account for the VLM taking up more room and loading more slowly.

Consumer GPU, for reference. If someone wants to prototype outside the cluster, an RTX 5090 (Blackwell, 32 GB) comfortably serves a 7–8B VLM in FP8 (~8.5 GB of weights) with KV to spare for a few pages per request; a 32B in FP8 (~32 GB) no longer fits with useful KV on that card and calls for the H100. The 5090 is good for validating the triage pipeline and the prompt; the real service belongs on the H100s.

What we have not covered

Video. Qwen3-VL handles video with dynamic FPS sampling; each frame is an image and the token cost multiplies by the number of sampled frames. One minute of video can be tens of thousands of tokens. It is the most expensive multimodal beast and deserves its own analysis.
Prefix caching of images. If the same image (a logo, a template) appears in many requests, can its KV be cached? It depends on the backend and on the stability of the placeholder; it connects with the attention backend.
Evaluation. How to measure that the VLM extracts the table correctly (it is not enough for it to “look right”): faithfulness against a set with structured ground truth.
Fine-tuning the VLM for a specific document domain (invoices, particular forms), which can put part of the domain’s “knowing how to see” into the weights and reduce what the prompt has to explain.

References

vLLM, Multimodal Inputs (limit_mm_per_prompt, precomputed embeddings): https://docs.vllm.ai/en/stable/features/multimodal_inputs/
vLLM, Vision Language Multi Image (mm_processor_kwargs, min/max pixels): https://docs.vllm.ai/en/latest/examples/offline_inference/vision_language_multi_image/
vLLM, Qwen3-VL Usage Guide (recipes): https://docs.vllm.ai/projects/recipes/en/latest/Qwen/Qwen3-VL.html
Qwen team, Qwen2.5-VL (blog, token formula, dynamic resolution): https://qwenlm.github.io/blog/qwen2.5-vl/
Qwen2.5-VL (Transformers docs, min/max pixels, token range): https://huggingface.co/docs/transformers/model_doc/qwen2_5_vl
Qwen3-VL Technical Report, arXiv 2511.21631: https://arxiv.org/abs/2511.21631
Qwen3-VL-8B-Instruct-FP8 (FP8 with quality almost identical to BF16): https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct-FP8
vLLM issue #15125 (image placeholder order): https://github.com/vllm-project/vllm/issues/15125
PaddleOCR vs Tesseract, CodeSOTA 2026: https://www.codesota.com/ocr/paddleocr-vs-tesseract
LLM-Based OCR vs Traditional OCR, Parsli 2026: https://parsli.co/blog/llm-ocr-vs-traditional-ocr
Open-Source OCR Models 2025, E2E Networks: https://www.e2enetworks.com/blog/complete-guide-open-source-ocr-models-2025

The pantry of pigeonholes: PagedAttention and the vLLM block manager

Mon, 08 Jun 2026 05:20:00 +0200

Part of the under the engine series. The scheduler post ended with a loose end: the scheduler has a second budget, the KV blocks, and when they run out it preempts. This post opens up that budget. It is the piece the KV cache post took for granted, what gets stored, in order to explain how it is managed in memory. And it is the one the FlashAttention post had been promising for months.

TL;DR

The KV cache grows a little with every token generated, and the problem was never its total size but the way it is reserved. The first servers asked, per request, for a contiguous chunk of HBM the size of the maximum possible context. Since almost no request reaches that maximum, the result was catastrophic: 60-80% of the HBM wasted on fragmentation. PagedAttention applies to the KV the oldest and most battle-tested idea in operating systems, paging: split the KV into fixed-size blocks (16 tokens by default), store them in non-contiguous HBM wherever there is room, and keep a block table that translates each sequence’s logical block into its physical one. Waste falls to ~4% (only the last block, half-filled). And since each block can be identified by the hash of its content, two requests that share a prefix point at the same physical block and share memory, with copy-on-write when one of them diverges: that is the engine of prefix caching. This post explains fragmentation with numbers, the block manager, the block table, COW, the block-size trade-off, the 10 knobs and the trap of confusing “fragmentation solved” with “zero waste”. On the generic 4×H100 SXM cluster.

Where you are: the pantry, underneath the scheduler

Go back to the kitchen of the previous post. The front-of-house manager assembles trays, but behind him there is a pantry where the ingredients each table accumulates over its meal are stored, its KV cache. The question in this post is how that pantry is organised.

The naive way: each table gets a whole contiguous shelf, sized for the greediest imaginable customer. The problem is obvious: a table that orders little leaves almost all of its shelf empty, but that shelf is already reserved and nobody else can use it. With many tables, the pantry fills up with half-empty shelves and no new table fits, even though the gaps added together leave plenty of room.

The PagedAttention way: the pantry is divided into small, identical pigeonholes. Each table is given the pigeonholes it needs, one at a time, wherever there is room, and they do not have to be next to each other. A map book records which physical pigeonholes each table has and in what order. When a table leaves, its pigeonholes go back to the pile. There are no half-empty shelves: the only waste is the last pigeonhole of each table, the half-filled one. That is, almost literally, the virtual memory of an operating system applied to the KV cache.

Why contiguous memory fragmented

Reserving contiguously and up front produces three distinct kinds of waste:

Reservation fragmentation. You set aside space for max_model_len (8192 tokens, say) even though the request will use 800. Reserved and never used.
Internal fragmentation. Within what you reserved, everything above what you actually use at any given moment.
External fragmentation. Gaps between contiguous reservations that are too small for a new request, even though added together they would be plenty.

The original vLLM paper measured that earlier systems wasted 60% to 80% of the KV memory through these three routes (Kwon et al., SOSP 2023). That is: on a GPU with room for 100 real requests, only 20-40 fitted. Paging attacks all three at once. It eliminates reservation waste (on-demand allocation) and external waste (blocks need not be contiguous), and leaves only a remainder of the internal kind: the last partial block.

The mechanism: blocks, block table and the kernel gather

The KV of a sequence is chopped into logical blocks of $b$ tokens (by default $b = 16$). Each logical block is mapped, via the block table, to a physical block somewhere in the HBM. The block table is the “map book”: a per-request list of which physical block corresponds to each logical one (vLLM implementation).

The key point is that the attention kernel knows how to read like that. Instead of assuming a contiguous KV tensor, the PagedAttention kernel receives the block table and does a gather: for each sequence it walks its physical blocks in logical order and reads K and V as if they were together. That is why PagedAttention is not just a data structure: it is a kernel that knows how to attend over paged memory. And that is why the attention backend and the block manager are tied together, the second deciding where the KV lives and the first knowing how to read it from there.

The block manager: the pantry librarian

The block manager (in V1, the KVCacheManager) is the one that keeps the map book. Its responsibilities:

Maintain a pool of free physical blocks (a queue of available blocks).
Allocate blocks to a sequence as it grows (a new block every $b$ tokens).
Free the blocks when the sequence finishes or is preempted.
Maintain the block tables (logical→physical) of each request.
Manage prefix caching: detect blocks with identical content and share them.
When free blocks run out, tell the scheduler to preempt (see the scheduler post).

When the block manager says “no blocks left”, the scheduler has to take somebody off the train. That is why the two budgets, tokens and blocks, are the two hands of the same engine.

Here is the elegant part. If two requests start with the same prefix, the same system prompt, the same context document, the first KV blocks of both are identical byte for byte. Why compute and store them twice?

vLLM gives each block a hash summarising its content (the tokens that formed it, plus the hash of the previous block, so that the hash captures position). It keeps a global table of blocks by hash. When a new request produces a block whose hash already exists, it allocates no new memory: it points its block table at the physical block that was already there (automatic prefix caching, vLLM).

Copy-on-write is the safeguard: while A and B share a block, neither can modify it. The moment one of the two needs to write something different into that block (because their sequences diverge, or in parallel sampling / beam search where several branches share a prefix), the block manager copies the block for that branch and only then writes (details, vLLM). It is the same COW that fork() uses in an OS: share until somebody writes.

The saving is direct: if 50 requests share a system prompt of 1,000 tokens, instead of 50 copies of that prefix’s KV there is one. How to maximise that saving in practice is the subject of the prefix cache hit rate post.

The maths that matter: how much KV, how many blocks

KV bytes per token. For a transformer block with $L$ layers, $h_{kv}$ KV heads (GQA), per-head dimension $d$ and $s$ bytes per element (2 in FP16):

$$\text{KV/token} = 2 \cdot L \cdot h_{kv} \cdot d \cdot s$$

For a Llama-70B ($L=80$, $h_{kv}=8$, $d=128$, FP16):

$$\text{KV/token} = 2 \cdot 80 \cdot 8 \cdot 128 \cdot 2 = 327680 \text{ bytes} \approx 320 \text{ KB}$$

A block of 16 tokens takes up $16 \times 320\,\text{KB} = 5.12$ MB.

How many requests fit. If after loading the weights ~120 GB of the node’s 320 are left for KV:

$$\text{KV tokens} = \frac{120 \cdot 10^9}{327680} \approx 366000 \text{ tokens} \approx 22900 \text{ blocks}$$

With average contexts of 4,000 tokens (250 blocks each), that is ~90 concurrent requests. That number, not max_num_seqs, is the real concurrency ceiling, and it is exactly the “block budget” of the scheduler.

The waste that remains. PagedAttention does not get to zero: each sequence wastes, on average, half a block (the last one, half-filled). With 16-token blocks and sequences of 4,000, that is $8 / 4000 = 0.2\%$ per sequence, and the famous aggregate “~4%” from the paper includes other overheads. The lesson: the waste does not disappear, it is bounded to the size of one block.

The block-size trade-off

The block_size (16 by default) is a trade-off, not a magic constant:

Block	Advantage	Drawback
Small (8)	less internal waste; finer-grained prefix sharing	more block table entries; more management and gather overhead
Large (32)	fewer metadata; more efficient gather	more waste in the last block; prefix caching shares at a coarser grain (fewer hits)

A large block shares worse: prefix caching can only reuse complete and identical blocks, so with 32-token blocks two prompts that agree on 20 tokens share nothing (they do not fill a common block), whereas with 8-token blocks they share two blocks. The default of 16 is the point vLLM found reasonable for most workloads; it is worth testing if your workload has short, heavily repeated prefixes.

The 10 knobs

#	Knob	What it controls	Cost if you overdo it
1	`block_size`	tokens per block	waste / overhead (see table)
2	`enable_prefix_caching`	sharing blocks by hash	almost none; usually on
3	`gpu_memory_utilization`	how many physical blocks there are	OOM if too high
4	`kv_cache_dtype` (FP8)	bytes per KV element	quality (measure, do not assume)
5	`swap_space`	blocks that fit on the host (SWAP)	PCIe traffic on preemption
6	`max_model_len`	maximum length per request	fewer requests if too high
7	eviction policy	who gets blocks taken away	prefix cache hit rate
8	sliding window	discarding old KV	quality on long contexts
9	TP / KV sharding	splitting the KV across GPUs	NVLink traffic
10	num_gpu_blocks (override)	forcing the block count	OOM or underuse

How it connects to the rest of the stack

With the scheduler. The “block budget” of the scheduler is administered by this block manager. When it says there are no blocks, the scheduler preempts (RECOMPUTE by default).

With the KV cache. The KV cache post explains what each token stores; this one, how it is placed in memory without fragmenting.

With prefix caching. COW and block hashes are the mechanism; hit rate engineering is how to squeeze it (prompt structure, prefix-aware routing).

With KV quantisation. Moving the KV to FP8 halves the bytes/token: the same node holds twice the tokens. It is the most direct lever on concurrency.

With the attention backend. The FlashAttention/FlashInfer kernel has to know how to attend over paged blocks; the block manager decides where they live, the kernel knows how to read them.

With disaggregated serving. Moving a request from a prefill pool to a decode pool in disaggregated serving is, at bottom, transferring its KV blocks between engines, over NVLink or over the network.

With multi-LoRA. In multi-LoRA serving, the base shares prefix KV between requests from different adapters as long as the prefix is identical.

Traps and things that are not what they seem

“PagedAttention eliminates waste.” It bounds it, it does not eliminate it. What remains is the last partial block per sequence (~half a block) plus the block table metadata. It is ~4% instead of 60-80%, but it is not zero. Sizing as if it were zero leaves you without a cushion.

“Bigger blocks always perform better.” The gather is somewhat more efficient, yes, but you lose sharing granularity: prefix caching shares worse and the waste in the last block grows. On workloads with many short repeated prefixes, small blocks can win.

“Prefix caching shares KV between users, that is a privacy problem.” It shares only blocks that are identical token by token (same system prompt, same document). It does not expose one user’s content to another: if the tokens do not match, there is no common block. What is worth watching is information leaking through timing side channels (a hit is faster than a miss), relevant only in very adversarial multi-tenant scenarios.

“FP8 in the KV is free: twice the concurrency.” It does double the tokens that fit, yes, but KV in FP8 degrades quality measurably on long contexts. It is a real lever, not a free lunch: quality has to be measured (FP8 end-to-end), not assumed.

“Going back to contiguous memory would be simpler and almost as good.” That is nostalgia for the contiguous tensor. The “simple” option reintroduces 60-80% fragmentation: on a GPU, that is the difference between 30 and 90 concurrent requests. The complexity of the block table pays for itself many times over.

SWAP versus RECOMPUTE when preempting. Configuring a lot of swap_space “so as not to lose KV” puts gigabyte transfers over PCIe on the critical path. In V1, RECOMPUTE is usually better; swap is for specific cases.

Conclusion

The bottleneck in serving an LLM was never only how much memory you have, but how you hand it out. The first servers treated the KV cache as a contiguous shelf per client and threw two thirds of the HBM in the bin without it showing up on any dashboard. PagedAttention stole from the operating system its best idea of fifty years ago, paging, and applied it at the exact place where it hurt: small pigeonholes, a map book, on-demand allocation and, as a bonus, the possibility that two requests starting the same way share the same pigeonholes until they stop resembling each other. The result is not magic: the waste is still there, but bounded to the size of one block instead of to the size of the worst imaginable case. And that difference, from 70% to 4%, is what turned a GPU serving thirty clients into one serving ninety, without touching the hardware. The pantry did not get bigger; it got better organised.

References

W. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023): https://arxiv.org/pdf/2309.06180.
vLLM, Automatic Prefix Caching (design, block hashing): https://docs.vllm.ai/en/v0.8.1/design/automatic_prefix_caching.html.
vLLM, Automatic Prefix Caching — Implementation (block table, COW): https://docs.vllm.ai/en/v0.6.1/automatic_prefix_caching/details.html.
H. Elshafie, Paged Attention from First Principles: A View Inside vLLM: https://hamzaelshafie.bearblog.dev/paged-attention-from-first-principles-a-view-inside-vllm/.
vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention (alternative, critical context): https://arxiv.org/pdf/2405.04437.

Disaggregated serving: prefill and decode in specialised pods

Fri, 22 May 2026 01:00:00 +0200

TL;DR

LLM inference has two phases with opposite profiles: prefill (processing the whole prompt in one go) is compute-bound, decode (generating token by token) is memory-bandwidth-bound. Running them on the same GPU forces a choice between two incompatible optimal hardware profiles, and leaves between 60 % and 80 % of peak capacity unused. The industry consolidated the pattern in 2026: disaggregated serving, separate pods for each phase, connected by a KV cache transfer channel (NIXL over UCX, RDMA, or NCCL as a fallback). DistServe demonstrated 7.4× more request rate at the same SLO; NVIDIA Dynamo 1.0 (GA at GTC 2026) takes the pattern to production at datacenter scale. Mixing heterogeneous hardware, H100 for prefill and commodity GPUs for decode, cuts up to 48 % off the cost per token. This article explains the why, the how, and the numbers that matter for a typical on-premise infrastructure.

You are here: Deploy

Disaggregated serving is an architectural decision in the Deploy stage of the six-stage LLMOps pipeline. It does not change the model, it does not change the data, it does not change the evals. It only changes how inference pods are spread over the GPU hardware. But that change moves aggregate throughput between 2× and 7×.

The analogy: the kitchen with two brigades

A serious industrial kitchen, any kitchen serving more than 50 covers a night, runs with two distinct brigades and two physically separate spaces.

The prep brigade starts at dawn. Its job is the mise en place: chopping, marinating, blanching, boiling stocks, preparing complex components. Equipment: good knives, big burners, convection ovens, 40-litre pots. It is capacity-intensive work and it is done in one batch. When it finishes, everything is left in labelled trays ready to use.

The service brigade comes in mid-afternoon. Its job is service: taking the trays from prep, heating portions, plating, running the pass. Equipment: salamanders, small griddles, thin spatulas, plenty of crockery. It is wrist work, rhythm work, not-failing-the-customer-who-has-the-plate-in-front-of-them work. Capacity per hour matters less than latency per plate.

If you make the same person do prep and service, both suffer. The cook is idle while doing mise en place in the middle of the afternoon. They have to stop and plate up when five orders land at once. Their working kit is designed for one or the other, not for both.

Serious kitchens solved this decades ago: separate brigades, separate spaces, separate equipment. The only thing that crosses between them is the trays of mise en place.

The trays are the KV cache. The separation is disaggregated serving. The handover from prep to service is the KV cache transfer, solved today with NIXL over RDMA. And the specialised pods are the two brigades with their optimal kit.

Quick recap: prefill and decode

A request to an LLM always goes through two phases:

Prefill. Take the complete prompt (say, 4,000 tokens) and process it in a single pass through all the layers of the model. The result is the KV cache of those 4,000 tokens (see the previous article on KV cache if you want to recall exactly what it holds). This step is massively parallel: all the tokens go through the attention matrices at once, which translates into huge, dense matrix multiplications. The GPU sits at 90-95 % compute utilisation. TTFT (time to first token) is determined by this phase.

Decode. Once the KV cache is ready, the model generates tokens one by one. Each new token is a pass through all the layers with a single query vector, reading the whole accumulated KV cache to compute attention. There is no parallelism between tokens (each one depends on the previous). What limits you here is not compute but bandwidth: every step has to read the model’s full weights from HBM. The GPU sits at 20-40 % compute utilisation, but at 90 % HBM utilisation. TBT (time between tokens) is determined by this phase.

Phase	Characteristic	Bottleneck	Key metric
Prefill	Massive parallel compute over N tokens at once	TFLOPS (compute)	TTFT
Decode	Streaming weights from HBM, 1 token at a time	HBM bandwidth	TBT (inter-token latency)

The asymmetry is structural: prefill burns compute and leaves memory half idle, decode does the opposite. A GPU designed to be excellent at both at once is a GPU designed to be badly used all the time.

Why putting them on the same GPU is a bad deal

Until 2023, the universal assumption was to run prefill and decode in the same inference process, on the same GPU. The engine scheduler (vLLM, TGI, Triton) decided on each cycle whether to prefill a new request or decode the ones already in flight. The intuition was that sharing hardware saves money.

The intuition is wrong. The problem has three faces:

Latency interference. When the engine decides to prefill a new request, it interrupts every decode in progress. That pushes up the TBT of the other requests. The user who was watching tokens flow smoothly down the screen notices a stall of several hundred milliseconds. This is known as prefill-decode interference and it degrades the experience visibly as concurrency rises.

Sub-optimal hardware for each phase. An H100 SXM has 989 TFLOPS BF16 of compute and 3.35 TB/s of HBM3. It is excellent for prefill, where compute is the limit. For decode, where the only thing that matters is bandwidth, those 989 TFLOPS are 60-70 % wasted. Conversely, a GPU with less compute but similar relative bandwidth (RTX 4090, L40S) would handle decode just as well for a fraction of the price.

Low aggregate utilisation. In real workloads with Llama 3 70B and 512-token outputs, around 80 % of the wall-clock is spent in decode. That means 80 % of your H100 cluster budget is doing memory reads, not calculations. It is like paying for a Ferrari to use it in a car park queue.

The idea: specialised pods, KV cache as the deliverable

Disaggregated serving breaks the inference cycle into two distinct services:

Prefill pod. Receives the prompt, runs the prefill, produces the KV cache. Hardware: high-compute GPUs (H100, H200, B200). Optimised for aggressive batching and throughput, not for individual latency: if 32 prompts arrive within 100 ms, it processes them together.

Decode pod. Receives the already-built KV cache, runs the token-by-token generation, streams to the client. Hardware: GPUs with good bandwidth but ideally cheaper per TFLOPS (RTX 4090, L40S, A100, even A30 depending on the case). Optimised for per-token latency (low TBT).

Between the two: a KV cache transfer over the network, which can be node-local (shared memory, NVLink), intra-rack (RDMA with InfiniBand or RoCE) or cross-rack (NIXL over UCX). The cost of this transfer scales linearly with context length, and it is the economic crux of the scheme.

The transfer protocol: the economics of movement

The KV cache transferred for a Llama 3 70B with 4K of context weighs roughly 2.6 GB (80 layers × 8 KV heads × 128 dim × 4,096 tokens × 2 (K and V) × 2 bytes in BF16). Moving 2.6 GB between two GPUs is not trivial:

Channel	Effective bandwidth	Time for 2.6 GB
Intra-node NVLink (NVSwitch)	~450 GB/s	~6 ms
Shared memory (same node, PCIe 5)	~60 GB/s	~45 ms
RDMA InfiniBand 400 Gbps	~50 GB/s	~55 ms
RDMA RoCE 200 Gbps	~25 GB/s	~105 ms
TCP/IP 10 GbE	~1 GB/s	~2.6 s

The immediate reading: above InfiniBand grade, the transfer is comfortable. Below it, it ruins the very TTFT we are trying to improve. Disaggregated serving is only viable with a decent interconnect. It is not a pattern for clusters built out of consumer Ethernet switches.

NVIDIA answered this with NIXL (NVIDIA Inference Transfer Library), released in mid-2025: a library that abstracts the transport (UCX, NCCL, direct RDMA verbs, shared memory) and picks the best available path automatically. vLLM has integrated it since late 2025 through the NixlConnector. It is now the de facto default for new deployments.

Real implementations in May 2026

The pattern’s path over two years:

2024 Jan · DistServe (HKU + UCSD): 7.4× requests at the same SLO
2024 May · SplitWise (Microsoft): variant with heterogeneous hardware
2024 Dec · experimental vLLM disagg (SharedStorage + PyNcclConnector)
2025 Mar · NIXL release (NVIDIA): unified transfer library
2025 Jul · stable vLLM NixlConnector
2025 Nov · SGLang, llm-d, MoonCake adopt the pattern
2026 Mar · NVIDIA Dynamo 1.0 GA (GTC 2026): production-ready at datacenter scale

As of today, the pattern is the default in any serious serving framework. The ones still monolithic are the small ones or the educational ones.

Three realistic options for an on-premise infrastructure:

vLLM disagg with NixlConnector. The most open route, requiring two sets of vLLM pods to be deployed (one with --kv-transfer-config '{"kv_role":"producer"}', another with "kv_role":"consumer") plus a proxy router. Enough for clusters of 4-16 GPUs.
SGLang with disagg. Conceptually equivalent, better performance on some MoE workloads.
NVIDIA Dynamo 1.0. The one taking over at datacenter scale. It covers routing, KV cache management, monitoring and scheduling in a single control plane. Heavier, but the reference solution if your cluster grows beyond 32 GPUs.

The numbers that matter

What disaggregation unlocks, in direct terms:

Metric	Aggregated (monolithic)	Disaggregated	Improvement
Goodput (req/s at the SLO)	baseline	1.4 – 2×	up to 2×
TTFT under high load	rises sharply from QPS 4	stable up to QPS 7+	~2×
Request rate at the same SLO (DistServe paper)	baseline	7.4×	7.4×
MoE throughput on Blackwell (Dynamo, GB300 NVL72)	baseline (Hopper)	up to 50×	depends on the model
Cost per token (heterogeneous H100 + commodity)	baseline (all H100)	-48 %	almost half

These numbers have to be read carefully: the most spectacular ones (7× and 50×) need specific hardware (Blackwell GB200/GB300 NVL72) and specific models (large MoE). The realistic range for a typical on-premise setup is 1.4-2× in goodput and -30 to -50 % in cost per token, depending on how heterogeneous the GPU mix is and how optimised the KV cache transfer is.

Heterogeneity: the radical version

The logical next step, proposed by SplitWise in 2024 and matured in 2025-2026 (Cronus, Tessera and others), is to mix GPU types: expensive high-compute GPUs for prefill, commodity GPUs with good bandwidth for decode.

Indicative cost (typical market prices in mid-2026):

H100 SXM: ~30-40 kUSD capex, ~3-4 USD/h amortised. Compute-heavy profile.
L40S: ~8-10 kUSD capex, ~1.5 USD/h. Intermediate profile, 864 GB/s of bandwidth.
RTX 4090: ~1.5 kUSD capex, ~0.30 USD/h. Modest compute profile but 1 TB/s of GDDR6X bandwidth, enough for decode of models up to ~30B parameters.

A realistic mixed cluster for serving an 8B model:

2× RTX 4090 (prefill batch) → ~3,000 USD capex, ~0.60 USD/h
4× RTX 4090 (decode pool) → ~6,000 USD capex, ~1.20 USD/h
TOTAL → ~9,000 USD capex, ~1.80 USD/h

Against the monolithic alternative with equivalent throughput:

2× H100 SXM (all in one) → ~70,000 USD capex, ~7 USD/h

The same throughput at a fraction of the capex and a quarter of the hourly cost, at the price of operational complexity: you now have two pools to coordinate, a transfer network to look after, and a scheduler that is not trivial.

For larger models (Llama 3 70B), the decode pool no longer fits on a single 4090 (the model does not fit in 24 GB, not even quantised to INT4 with headroom). There the sensible mix is H100 for prefill plus L40S or A100 80GB for decode, with a typical saving of 30-40 % over the all-H100 option.

Applied to typical on-premise hardware

Case 1 — One or two RTX 4090s: monolithic still wins

With a single GPU there is no disaggregation to speak of: the pattern needs at least two GPUs in separate pods. With two 4090s you can technically try it (one for prefill, one for decode with the KV cache transferred over PCIe 5 or basic RDMA), but the transfer overhead eats the gain for small models where prefill is already fast.

Recommendation: stay monolithic (traditional vLLM, properly configured with a quantised KV cache). The next justifiable level of complexity is a cluster with a fast interconnect.

Case 2 — 4×H100 SXM cluster (320 GB, NVLink): the sweet spot

The minimum realistic configuration for serious disaggregation, serving a 70B model in production:

2× H100 (TP=2) → 2 prefill pods
2× H100 (TP=2) → decode pods with several instances sharing TP
NIXL over NVLink → KV cache transfer <6 ms
Router (vLLM or Dynamo) → prompt distribution and streaming

Realistic expected result: goodput 1.6-1.9× relative to the same cluster running monolithic, with TTFT stable up to loads of QPS 7-8 (against the QPS 4 at which the monolithic setup starts to degrade).

If the heterogeneous mix is possible (adding 4-8 L40S to the cluster to make the decode pool), the cost per token drops a further 25 % to 35 %, while still serving the 70B model whole.

Position within the architecture

Disaggregated serving is a cross-cutting layer over almost everything discussed in previous articles. It touches:

The KV cache because it is the artefact transferred between pods. Without a solid grasp of how much the cache weighs and how it grows with context, you cannot size the transfer.
Continuous fine-tuning because multi-LoRA hot-swap keeps its semantics: each pod (prefill or decode) loads the adapters separately, and the router decides which adapter to apply in each phase.
Cluster topology: it changes the recommended HW, the networking required and the cost model.

If you are designing an inference infrastructure for 2026 from scratch, disaggregation stops being optional for any cluster beyond 4 GPUs of capacity. If you are modernising an existing one, it is the upgrade with the best return per euro invested, provided the networking between pods is decent (intra-node NVLink or intra-rack RDMA as a minimum).

What we have not covered (upcoming articles)

NIXL in detail: how it picks the optimal transport, how UCX is configured, what happens when RDMA fails and you have to fall back to TCP.
Routing scheduler: how the orchestrator decides which pod gets which request, dynamic batching, priority handling.
Multi-tenant disagg: KV cache isolation between tenants, per-adapter ACLs, multi-LoRA over specialised pods.
Disagg + prefix caching: how it combines with the KV cache reuse pattern when several prompts share a prefix (a common system prompt).
Disagg at the edge / local inference: viability on home hardware (4090 + Mac Studio, for example), where the transfer depends on Thunderbolt or residential Ethernet.

References

Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (OSDI 2024).
Patel et al., SplitWise: Efficient Generative LLM Inference Using Phase Splitting (ISCA 2024).
NVIDIA, NVIDIA Dynamo 1.0: Production-Ready Disaggregated Inference (GTC 2026, March): https://developer.nvidia.com/blog/nvidia-dynamo-1-production-ready/.
NVIDIA, NIXL: NVIDIA Inference Transfer Library — official documentation.
vLLM, Disaggregated Prefilling: https://docs.vllm.ai/en/stable/features/disagg_prefill/.
vLLM, NixlConnector Usage Guide: https://docs.vllm.ai/en/stable/features/nixl_connector_usage/.
Hao AI Lab, Disaggregated Inference: 18 Months Later (UCSD, 2025) — technical retrospective on the DistServe paper.

KV cache: the working memory that holds up LLM inference

Mon, 18 May 2026 10:00:00 +0200

TL;DR

The KV cache is the working memory a language model keeps during a conversation. Without it, every new token would force a recomputation of the whole conversation from the start, at a cost that is quadratic in the length of the text. With it, the cost is linear, but in exchange the cache lives in VRAM and grows with every token. In practice, it is not the model that limits how much context you can serve: it is the KV cache. On an RTX 4090 running Llama 3 8B, the model fits in 16 GB and barely leaves room for ~64 K tokens of cache in total, adding up all the simultaneous sessions. Understanding that number is the difference between promising a client “128 K of context” and actually delivering it.

You are here: Deploy

This post opens the series on LLM inference fundamentals. Within the six-stage LLMOps pipeline that articulates the whole system, the KV cache lives in the Deploy stage: it is the piece that dictates how much traffic fits in your inference engine and, therefore, how much platform you can offer on top of it.

The analogy: the speaker with amnesia

Imagine you are at a two-hour technical conference. Every time the speaker is about to say a new sentence, he mentally rewinds the whole talk from the beginning, reassembles the thread, and only then carries on. His next sentence requires recalling the previous one; the one after that, the two before it; an hour in, every new word costs him an hour of recap. A conference like that would be materially impossible.

Now imagine the same speaker with a notebook in which he jots down, as he goes, the two or three key ideas of each sentence: subject, object, link to what came before. Before each new sentence he glances at the notebook and continues. His next word only costs a glance at the notebook, not a rewind of the entire talk.

That notebook, in a transformer, is called the KV cache. Without it, conversational language models would be unworkable. With it, they are commercial products. But the notebook weighs something, and understanding how much, where and why is what separates an inference infrastructure that works from one that falls over at the third concurrent client.

The mechanism itself (in plain terms)

A transformer generates text one token at a time. To decide the next token, the model applies a mechanism called attention over all previous tokens: it asks “which parts of the earlier context are relevant for predicting what comes next?”.

Internally, each input token is projected into three vectors:

Q (Query): “what I am looking for”
K (Key): “what this token offers”
V (Value): “what information this token carries”

The attention of the current token against the context is computed by multiplying its Q against the K of all previous tokens, normalising with softmax, and weighting the corresponding V. The result is a contextualised representation of the current token.

Q·Kᵀ → softmax × V

representation of token N

Here is the key point: to predict token N, I only need the new Q (that of token N) and the K, V of all previous tokens. The K and V of tokens 1..N-1 have not changed since the previous iteration. Recomputing them would be throwing work away.

The KV cache is exactly that: the memory that holds the K and V of every token already processed, in every layer of the model, so they never have to be recomputed.

Why it exists: the quadratic cost without it

Generating a text of N tokens involves N steps. At step i, attention is computed over i previous tokens. Without a cache, at every step you recompute the K, V of the i-1 previous tokens plus those of the new one. The total count of attention computations grows as:

$$\sum_{i=1}^{N} i = \frac{N(N+1)}{2} \approx \frac{N^2}{2}$$

With a KV cache, you only process the new token at each step: cost linear in N.

Cumulative compute to generate N tokens (schematic scale — exact data is in the table)

tokens generated (N) attention operations

0 1K 2K 3K 4K

with KV cache O(N) — linear

without KV cache O(N²) — quadratic

The concrete numbers are devastating:

Tokens generated	Without KV cache (operations)	With KV cache	Ratio
128	8,256	128	64×
1,024	524,800	1,024	512×
4,096	8,390,656	4,096	2,048×
32,768	536,887,296	32,768	16,384×

At 32 K tokens, the cache saves you four orders of magnitude of compute. This is not an optimisation: it is what makes conversational inference possible.

The price: how heavy the backpack is

The KV cache is paid for in VRAM. The formula, per sequence, is:

KV_size = 2 · n_layers · n_kv_heads · head_dim · context_len · bytes_per_param
↑
K and V

Per token (dropping context_len), it is a constant of the model itself. Let us look at real numbers:

Model	n_layers	n_kv_heads	head_dim	Bytes/token (BF16)	GB at 8 K ctx	GB at 32 K	GB at 128 K
Llama 3 8B (hypothetical MHA)	32	32	128	524,288	4.00	16.00	64.00
Llama 3 8B (real GQA)	32	8	128	131,072	1.00	4.00	16.00
Llama 3 70B (GQA)	80	8	128	327,680	2.50	10.00	40.00
Qwen3 8B (GQA)	36	8	128	147,456	1.12	4.50	18.00
Mistral 7B (GQA)	32	8	128	131,072	1.00	4.00	16.00

Two immediate readings:

Without GQA there is no 128 K worth talking about. A Llama 3 8B with classic multi-head attention would need 64 GB of KV cache alone for a single sequence with 128 K tokens. In other words, it does not fit on any consumer GPU. That is why Meta, Mistral and the rest adopted Grouped Query Attention.
The KV cache can be larger than the model. Llama 3 8B in BF16 takes ~16 GB. With 128 K of context, its cache is another 16 GB. A single session ties with the model in VRAM.

0 10 20 30 40 GB

0 8K 32K 64K 128K

≈ VRAM free after loading an 8B on a 4090

Llama 3 8B Qwen3 8B Llama 3 70B

The dashed red line marks the realistic VRAM available on an RTX 4090 after loading the model. Any model whose curve crosses that line will not be able to serve that context without additional strategies: cache quantisation, offload, partitioning.

Inference is memory-bound, not compute-bound

There is a common misconception: thinking that “fast GPU = fast inference”. In the regime where inference services with a KV cache actually operate, what gets measured is memory bandwidth. Every new token requires reading the K and V of all previous tokens from HBM. The compute is modest; the data movement is massive.

That is why an H100 SXM (3.35 TB/s of HBM3) can be 2-3× faster than an A100 (1.55-2 TB/s) without the clock frequency or the core count fully explaining the difference. Bandwidth explains it.

And that is also why offers of “cheap GPUs with plenty of VRAM but slow HBM” (some variants with GDDR6 or LPDDR5) disappoint in inference with long contexts: they have room to store the cache but take forever to re-read it.

Tricks to make the notebook thinner

Three techniques, in chronological order, have progressively flattened the size of the KV cache:

Multi-Head Attention (MHA). The original transformer design (Vaswani et al., 2017). Every attention head has its own K and V. Expensive in cache but theoretically maximal in expressiveness. It is what models had until around 2023.

Multi-Query Attention (MQA). A single K and V shared by all heads. It reduces the cache by a factor of n_heads. It works reasonably well but degrades generation quality on some benchmarks.

Grouped Query Attention (GQA). The middle ground that won. Heads are grouped: in Llama 3 8B, 32 query heads share K, V in groups of 4, giving 8 KV groups. It cuts the cache 4× relative to MHA with almost identical quality. It has been the de facto standard since 2024.

Multi-Head Latent Attention (MLA). DeepSeek-V2/V3’s innovation: instead of storing K, V per head, it compresses the state into a smaller latent vector and projects out to K, V on the fly. The cache can come down to 70 bytes/token, two orders of magnitude less than GQA. It is the main reason DeepSeek-V3 (671 B parameters, 37 B active) is servable on affordable infrastructure.

Note: the MLA bar is illustrative, using typical values published by DeepSeek; the exact implementation depends on the latent size. What matters is the order of magnitude.

On top of this comes a fourth, orthogonal technique: quantising the cache to FP8, INT8 or even INT4. vLLM and TensorRT-LLM already support it in production. Going from BF16 (2 bytes) to FP8 (1 byte) halves the cache at a small cost in quality. Going to INT4 divides it by four, at a somewhat higher cost.

The next dragon: fragmentation

So far we have talked about the cache as if it were a contiguous block. In practice, an inference server handles dozens of simultaneous sessions, each with its own cache growing at a different rate. Naive allocation, reserving the maximum possible per session, wastes between 60 % and 80 % of the VRAM according to the original PagedAttention paper.

session A

session B

session C

session D

→ ~70 % of VRAM reserved and empty

 <rect x="0" y="22" width="30" height="20" class="used blk"/>
<rect x="30" y="22" width="30" height="20" class="used blk"/>
<rect x="60" y="22" width="30" height="20" class="used blk"/>
<rect x="90" y="22" width="30" height="20" class="used blk"/>
<rect x="120" y="22" width="30" height="20" class="used blk"/>
<rect x="150" y="22" width="30" height="20" class="used blk"/>
<rect x="180" y="22" width="30" height="20" class="free blk"/>
<rect x="210" y="22" width="30" height="20" class="free blk"/>
<rect x="0" y="44" width="30" height="20" class="used blk"/>
<rect x="30" y="44" width="30" height="20" class="used blk"/>
<rect x="60" y="44" width="30" height="20" class="used blk"/>
<rect x="90" y="44" width="30" height="20" class="free blk"/>
<rect x="120" y="44" width="30" height="20" class="free blk"/>
<rect x="150" y="44" width="30" height="20" class="free blk"/>
<rect x="180" y="44" width="30" height="20" class="free blk"/>
<rect x="210" y="44" width="30" height="20" class="free blk"/>
</g>

→ < 4 % waste (vLLM paper)

PagedAttention, the idea from Kwon et al. (2023) that gave rise to vLLM, solves this by borrowing a technique from operating systems: divide the VRAM into small blocks (typically 16 tokens) and keep a page table mapping logical to physical blocks per session. A session no longer reserves one huge contiguous block: it grows one block at a time, and the blocks can be scattered across the VRAM. The result is an effective occupancy of 90 % instead of 30 %, and therefore 2-4× more aggregate throughput on the same hardware.

PagedAttention deserves an article of its own. I am noting it down for the next one.

Applied to generic on-premise hardware

Let us come down to concrete cases.

Case 1 — RTX 4090 (24 GB, Ada Lovelace)

A typical configuration with Qwen3-8B BF16:

Model in BF16: ~16 GB
Activations + overhead: ~2 GB
VRAM available for KV cache: ~6 GB (with margin)

At 144 KB/token (Qwen3-8B GQA), that comes to ~43 K total tokens of cache distributed across all simultaneous sessions. In practice:

Concurrency	Maximum context per session
1	32,768
4	8,192
8	4,096
16	2,048

If you need to advertise “we support 32 K of context” with concurrency of 4 or more, you have to quantise the cache (FP8 brings it down to 72 KB/token, doubling capacity) or move up the model range (a 4B with GQA and a quantised cache would have room to spare).

Case 2 — Cluster of 4×H100 (320 GB total, NVLink)

With tensor parallel = 4 and Llama 3 70B BF16:

Model in BF16: ~140 GB (35 GB/GPU)
vLLM overhead per GPU: ~2 GB
VRAM free for KV per GPU: ~43 GB → ~172 GB aggregate

At 320 KB/token (Llama 3 70B GQA), that comes to ~537 K total tokens of cache. Ample margin for long contexts with high concurrency:

Concurrency	Maximum context per session
4	134,000
16	33,500
64	8,375

For DeepSeek-V3 671 B with MLA the economics change radically, because the cache is ~100× thinner. What limits you is no longer the cache but the VRAM of the model itself (quantised to FP8 it is ~335 GB, so it fits on 4×H100 with room for the KV cache).

Operational implications

Three observations we repeat in every consulting engagement:

First, the maximum context a model advertises is not the one you can serve on your hardware. Llama 3 8B “supports” 128 K, but on a 4090 with 4 simultaneous sessions your effective context is ~8 K. It is trivial to check before promising it to a client.

Second, quantising the KV cache is one of the optimisations with the best cost/benefit ratio in the ENS context. It does not touch the weights, it does not affect audit reproducibility, and it doubles capacity. vLLM supports it via --kv-cache-dtype fp8.

Third, if the SLAs dictate long contexts with many concurrent users, GQA is necessary but not sufficient. In the medium term you need to look at models with MLA or attention variants with compression.

What we have not covered (upcoming articles)

PagedAttention and its implementation in vLLM: blocks, page table, eviction.
Prefix caching: when several requests share the system prompt, there is no need to recompute the K, V of the common part.
Speculative decoding and its interaction with the cache.
Cache offloading: moving cold blocks to RAM or NVMe, a key technique for contexts above 1 M.

References

Vaswani et al., Attention Is All You Need (NeurIPS 2017) — foundational transformer paper.
Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (EMNLP 2023).
Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023) — original vLLM paper.
DeepSeek-AI, DeepSeek-V2 Technical Report (2024) — introduction of Multi-Head Latent Attention.
Official vLLM documentation: https://docs.vllm.ai/.
Llama 3 model card (Meta): GQA specs, n_layers, n_kv_heads.

Kv-Cache on lo0 — Tech Blog

Prefix routing: what LiteLLM does not do, who does, and how much it really improves

TL;DR

You are here: between the gateway and the engine

The analogy: the library with four counters

The correction: what I wrote in June

What LiteLLM does have: four affinities

prompt_caching: why it is no use for this

session_affinity: the one to use

The routers that do read the prompt, and why they are no good

What did land: admission control

What the engine exposes upwards

The ZMQ stream

The hash that has to be replicated

The hits are invisible by default

The metrics, and what broke

The last token

The ones that do do it

vLLM production-stack

llm-d

NVIDIA Dynamo

The rest of the field

Comparison table

How much it really improves

The five failure modes

The architecture that comes out of all of the above

When not to build this

Checklist

Traps and things that are not what they seem

Closing

See also

Sources

Serving reasoning models: the invisible scratchpad that decides your latency and your bill

TL;DR

The analogy

What changes when the model “thinks”

The 2026 landscape: from the model that “always reasons” to the switchable hybrid

The economics: why the scratch paper drives up the bill

Controlling the thinking budget

1. Switching think / non-think

2. Setting a cap: thinking_token_budget

3. Budget forcing (state of the art)

The impact almost nobody measures: the KV cache

Reasoning + structured output: order matters

Reference architecture

Operational pitfalls (and honest scepticism)

Closing

Sources

Long context and KV offloading: when the notebook does not fit on the desk

TL;DR

The analogy

Part 1 · Why long context is a memory problem

The KV cache grows with the sequence

The numbers are frightening

And on top of that, attention is quadratic

Extending the window ≠ serving it cheaply

Part 2 · The KV memory hierarchy

Working out when offloading pays off

Part 3 · KV offloading in practice (OSS)

The limit of out-of-the-box prefix caching

LMCache: the persistent KV layer

Combine it with FP8

Part 4 · KVCache-centric / disaggregated architecture

Mooncake (Kimi’s platform)

NVIDIA Dynamo and KVBM

Part 5 · Reference architecture

Part 6 · Operational pitfalls (and honest scepticism)

Closing

Sources

Multimodal on-premise: serving a VLM with vLLM (vision + language)

TL;DR

The analogy: the reader who sees the page versus the one who only hears the transcript

Anatomy of a VLM: encoder, projector, LLM

The maths: what it costs to see a page

From token cost to TTFT

VRAM: the VLM’s weights and what is left for KV

Serving a VLM with vLLM

Classic OCR vs VLM: when to use which

Applied to the generic 4×H100 cluster

What we have not covered

`prompt_caching`: why it is no use for this

`session_affinity`: the one to use

2. Setting a cap: `thinking_token_budget`