Dynamo on lo0 — Tech Blog

Prefix routing: what LiteLLM does not do, who does, and how much it really improves

Fri, 11 Sep 2026 00:30:00 +0200

Fifth article in the operational track of the control layer. The pair with Langfuse covered observability, the proxy’s day 2 covered availability, virtual keys covered governance and humans and agents covered the coexistence of two classes of traffic. Here a claim of mine from June gets taken apart. Everything that follows is verified against LiteLLM 1.102.0 (commit of 10 September 2026), vLLM 0.29.0 and the main branches of production-stack, llm-d and Dynamo from the second week of September 2026.

TL;DR

LiteLLM does not do prefix-aware routing, neither in the open version nor in the paid one. It is not a licensing question. A search across the whole tree for kv_aware, cache_aware, radix, prefix_cache and block_hash returns zero matches in the router. Nor is there any call from the router to /metrics or to any vLLM endpoint. In June I wrote the opposite in the inference router article and I was wrong.

What LiteLLM has is affinity, which is a different thing. Four pre-call checks pin a request to a replica by session identifier, by key hash, by previous response identifier or by exact hash of a marked prefix. None is active by default. Affinity distributes conversations; prefix routing distributes prefixes shared between different conversations, which is the case that multiplies throughput.

The only one that looks at the prompt is fragile in three ways. If the client does not send cache_control, the key comes out null and the check does nothing. If the cut point goes in the last turn, which is what the automatic injection does by default, the key changes on every turn and it never hits. And the pin’s TTL is 300 seconds hardcoded in two lines, while the provider offers one-hour caches.

The engine does not answer questions, it publishes events. There is no vLLM endpoint you can ask whether it has a prefix. What there is, is a ZMQ socket that emits stored and removed blocks in msgpack, switched off by default, with silent discarding when the consumer does not keep up. A router that wants to be exact has to replicate the engine’s hash in full: chained SHA256, deterministic seed, block granularity and the extra keys in their order.

And by default it does not publish the hits, only the additions. The index of a router that only listens drifts away from the engine’s real LRU without noticing. To see the hits you have to ask for it request by request with kv_cache_report_mode: full.

The three routers that do do it are very different from each other. The vLLM production-stack one indexes chunks of 128 raw characters in a trie that is never purged, models no eviction and dies on every restart. The llm-d one offers an approximate mode and an exact event-driven one, all its plugins are in alpha or beta, and its default tokeniser does not produce real tokens. NVIDIA Dynamo’s keeps an event-driven radix tree with a weighted cost function, and ships with the parameter that avoids starving a freshly scaled replica switched off.

The big improvements only appear in the best possible case. The two- and three-digit numbers are published by the projects themselves, with synthetic shared prefixes and a round-robin balancer as the reference. The most complete independent measurement, from Meta, obtains 2.3 times more throughput and also identifies load regimes where affinity leaves capacity at between half and two thirds of the reference. Another, on a multi-region deployment with 89 % reuse, measures single-digit improvements.

The correct architecture has two layers, not one. LiteLLM on top for identity, budget and audit, pointing at a single destination per model group. The KV-aware router underneath, inside the engine’s domain. Mixing it into a single piece is not an option that exists today.

You are here: between the gateway and the engine

This article occupies the gap between two layers that already have an article of their own. Above, the L7 switchboard and the gateway that has been operated since the first article in this track. Below, the fundamentals of the KV cache and hit rate engineering.

The gap is the piece that decides which replica each request goes to, knowing what each replica has in memory. It is the only function in the stack that cannot be resolved in the gateway or in the engine separately.

The analogy: the library with four counters

A library with four counters. Each librarian has in front of them a pile of the books they have consulted lately. When a reader comes in asking for the third volume of an encyclopaedia, if that volume is in the pile at counter two, serving them there takes seconds. At any other counter someone has to go down to the store room.

A normal balancer sends the reader to the counter with the shortest queue. It is the right decision if all the counters are interchangeable, and it is the wrong decision here, because they are not: what distinguishes them is not the queue, it is the pile.

Session affinity resolves one specific case of this. The returning reader is always sent to the same counter, because they will probably ask for the fourth volume of the same encyclopaedia. It works, and it does not resolve the general case: when a hundred different readers ask for the same third volume, session affinity spreads them across the four counters and four librarians go down to the store room for the same book.

Prefix routing is looking at what the reader asks for before assigning them a counter. It has two costs that the analogy makes visible straight away. The first is that somebody has to keep track of what is in each pile, and that bookkeeping takes work. The second is that if everybody asks for the third volume, counter two becomes the library’s only desk while the other three look on.

The correction: what I wrote in June

In the inference router article, on 2 June 2026, I closed the prefix routing section with this sentence: “vLLM Production Stack router implements it natively. NVIDIA Dynamo too. LiteLLM in its enterprise version has a beta.”

The first two hold up. The third is false and was no less false in June. While preparing the humans and agents article the first hint appeared, and full verification against 1.102.0 confirms it.

Case-insensitive search over litellm/, enterprise/ and gateway/, for the terms kv_aware, kv-aware, cache_aware, cache-aware, radix, prefix_cache, prefix-cache, kv_cache, block_hash, prefix_hash, prefix_score and longest_prefix: zero matches. Search for vllm inside litellm/router.py, litellm/router_strategy/ and litellm/router_utils/: zero. Search for num_requests_waiting, gpu_cache_usage or any scraping of the backends’ /metrics from the router: zero. LiteLLM’s router never asks the engine anything.

Search for premium_user in the files where those checks would live: a single result, and it is about something else. There is no paid version with this feature. The only routing piece that is behind a licence is per-tag budgets, in litellm/router_strategy/budget_limiter.py.

Where the error came from. LiteLLM’s documentation has an entry on dynamic TPM and RPM allocation marked as beta and enterprise, and a pre-call check called prompt_caching. Skim-read and added together, they give the impression of prefix routing in beta. Read in the code, prompt_caching is something else, and it deserves a section of its own because it is still recommended for this problem.

What LiteLLM does have: four affinities

The optional_pre_call_checks field accepts eight values. None is enabled by default: apply_default_settings passes an empty list. One of the eight, forward_client_headers_by_model_group, is declared in the type and in the interface schema and is consumed by no branch of the code. It is a dead value.

The four that pin a request to a replica are these.

Check	What it pins	Where it stores it	TTL
`session_affinity`	Session identifier	Redis with a Lua script, mirrored in memory	`deployment_affinity_ttl_seconds`, 3600 by default, refreshed
`deployment_affinity`	API key hash	Same	Same
`responses_api_deployment_check`	`previous_response_id`	Same	Same
`prompt_caching`	Exact hash of a marked prefix	`DualCache`	300 s, hardcoded

The other four are router_budget_limiting, encrypted_content_affinity, enforce_model_rate_limits and the dead value.

`prompt_caching`: why it is no use for this

The key is a SHA256 over the serialisation of the cacheable prefix, and “cacheable prefix” means everything up to and including the last block marked with cache_control: {"type": "ephemeral"}. Three consequences, checked by running the functions in isolation over the repository’s code.

If the request carries no cache_control, the extraction returns an empty list, the key comes out null and the check pins nothing. Automatic injection of cut points is only enabled with litellm.enable_anthropic_prompt_caching, which defaults to false, and only for Anthropic and Bedrock providers.

If the cut point is in the system prompt, the key does not change when turns are added and the pin works. This is the good case and it requires the client to mark it deliberately.

If the cut point is in the last message, the key changes on every turn and the pin never hits. And this is exactly the default case of the automatic injection, which places one point on the system and another at index -1. Since the second is the last, the cacheable prefix becomes the whole conversation.

On top of that there is the TTL. Two lines with ttl=300, one of them with the comment # store for 5 minutes. There is no parameter to change it. Issue 28427 has had it open for months with no response from the maintainers, and it points to the mismatch with the one-hour ephemeral caches that LiteLLM itself knows how to request via cache_control.ttl: "1h".

There is a nuance the issue does not capture and that makes diagnosis worse: in the default case the pin does not even last five minutes, because the key changes first.

`session_affinity`: the one to use

For conversations, this is the right piece and it is reasonably built. The session identifier is looked for in this exact order of headers:

x-litellm-trace-id
x-litellm-session-id
Any x-<vendor>-session-id, with the regular expression ^x-.+-session-id$ and a value of at least eight alphanumeric characters
Only if the User-Agent is Codex’s: session-id, session_id, thread-id, conversation_id
Plain x-session-id, for opencode

And if nothing matches, two fallbacks: a metadata.user_id in Anthropic format, and the W3C baggage with session.id=. The third point is the one that captures Claude Code with nothing configured.

Storage is a Lua script in Redis that does GET, SET NX EX and EXPIRE, so the first writer wins and the TTL is refreshed on every request that confirms the pin. It is an inactivity TTL, not a total session duration. Without Redis it degrades to a pod-local reservation, which is atomic but not shared.

A detail of the pipeline order that already came up in the previous article and still holds: the affinity checks run inside async_get_healthy_deployments before tag filtering. If the pin returns a single deployment and that deployment does not satisfy the request’s tag, the later filter removes it and the request fails. Affinity gets no vote on the tags.

The routers that do read the prompt, and why they are no good

LiteLLM has four pieces that read message content: AutoRouter with semantic matching, RequestComplexityRouter, AdaptiveRouter and QualityRouter. They all live in async_pre_routing_hook, which runs before looking for healthy deployments.

The reason they do not solve this problem is in what they choose. All four choose a model group, not a replica. They decide whether an easy question goes to the thirty-billion-parameter model or to the small one. None has access to the replica list or knows what is in each one’s memory.

What did land: admission control

A minor correction to the previous article, which placed this in the 1.101 branch as an upcoming feature. In 1.102.0 it is already there, in litellm/proxy/middleware/admission_control_middleware.py, with the three settings in general_settings: max_in_flight_requests_per_worker, max_queued_requests_per_worker and admission_queue_timeout_seconds, the last with a default value of 1.0 seconds.

It stays inactive as long as the first one is not declared. It exposes an authenticated /health/backlog and the three metrics litellm_admission_admitted_requests, litellm_admission_queued_requests and litellm_admission_rejected_requests_total, the last with a reason label. It is a per-process semaphore, not coordinated across pods.

What the engine exposes upwards

Before looking at the routers that do do this, you have to understand what raw material they work with. The natural question would be whether vLLM has an endpoint you can ask about a prefix. It does not.

The only cache router in the entry points is vllm/entrypoints/serve/dev/cache/api_router.py, and it only exposes destructive writes: POST /reset_prefix_cache, /reset_mm_cache and /reset_encoder_cache. All three are behind VLLM_SERVER_DEV_MODE=1 and do not exist in a normal deployment. There is no read.

What there is, is an event stream.

The ZMQ stream

It is enabled with --kv-events-config, and it is off by default. The configuration accepts a publisher (null or zmq), an endpoint (by default tcp://*:5557, which binds), an optional replay endpoint, the number of batches retained for that replay (10,000), and a high water mark limit of 100,000.

That limit is the one that matters operationally: above it, ZMQ discards events if the consumer does not keep up. Silently.

The transport is a PUB socket with msgpack serialisation and three-part frames: topic, eight-byte sequence number and payload. The sequence number lets a subscriber ask for replay from where it left off through the optional ROUTER socket. With data parallelism the port shifts by rank, so you have to subscribe to 5557+rank.

Three event types. BlockStored carries the hashes, the parent hash, the token identifiers, the block size, the LoRA adapter name, the extra keys, the medium (GPU, CPU or storage) and a session identifier. BlockRemoved carries only hashes. AllBlocksCleared carries no payload and is emitted when the prefix cache is reset.

They are published once per scheduler step, only if there are events. With the feature switched off the cost is zero. With it switched on, the full token identifiers of each block travel over the wire, so bandwidth scales with tokens per second and not with requests per second.

An example in vLLM’s own repository declares itself experimental on its eighth line.

The hash that has to be replicated

An exact router does not receive the engine’s state, it reconstructs it. For that it has to compute the same hashes, and the engine computes them like this:

# vllm/v1/core/kv_cache_utils.py
def hash_block_tokens(hash_function, parent_block_hash, curr_block_token_ids, extra_keys):
 if not parent_block_hash:
 parent_block_hash = NONE_HASH
 return hash_function((parent_block_hash, tuple(curr_block_token_ids), extra_keys))

It is chained: each hash identifies the complete prefix up to that boundary, not the isolated block. The default algorithm is SHA256, configurable with --prefix-caching-hash-algo among four values.

Five things a router has to get right for its hashes to match the engine’s:

The seed. Since 0.29, NONE_HASH is deterministic from a fixed seed, so two different processes produce the same hashes with nothing configured. It is a silent change of semantics from earlier versions, which required setting PYTHONHASHSEED. With the xxhash algorithms it is still needed.

The block size. By default 16 tokens, but the attention backend can override it if the user did not pass --block-size: 256 with sliding-window sparse attention, 64 with ROCm’s AITER, 64 or more with FlashAttention on XPU. A router must not assume 16, it has to read it from the event.

The hashing granularity, which is not necessarily the physical block size. There is --prefix-match-unit, which with several cache groups resolves to the greatest common divisor of the cacheable groups’ sizes.

The extra keys, in their order: LoRA adapter name (by name, not by identifier), multimodal identifiers with their relative offset inside the block, cache_salt only in block zero, and the hash of the prompt embeddings. Sampling parameters do not go in, which is correct: the prefix KV does not depend on the temperature.

And truncation. By default the hashes travel over the wire as 64-bit integers truncated from a SHA256, not as the full bytes. It can be disabled with VLLM_KV_EVENTS_USE_INT_BLOCK_HASHES=0. With the default value, a router has a theoretical collision risk that the engine does not have.

The hits are invisible by default

This is the detail that takes the most effort to discover and has the most consequences. The events are incremental: only what gets newly cached is announced. A reused block generates no event.

A router that only listens sees the additions and the removals, but does not see the hits. Its notion of what is recent and what is cold drifts away from the engine’s real LRU with nothing to warn it.

A full mode exists, and it is per request. sampling_params.extra_args["kv_cache_report_mode"] = "full" makes BlockStored events be emitted for blocks that were hits too. The function’s docstring says so plainly: it generates events so that external consumers, “for example a gateway”, know which blocks were reused. That is to say: for this to work, the router has to inject a parameter into every request it forwards.

The metrics, and what broke

For aggregate measurement two counters remain: vllm:prefix_cache_queries_total and vllm:prefix_cache_hits_total, both in tokens and not in requests, with model and engine labels. There are two equivalents for the external connector, which allow a hit in HBM to be distinguished from one brought in from CPU or NVMe.

And there are two removals that break existing configurations. vllm:gpu_prefix_cache_hit_rate no longer exists: zero appearances in the repository. It was deprecated in 0.8, hidden in 0.9 and removed in 0.10. And vllm:gpu_cache_usage_perc was renamed to vllm:kv_cache_usage_perc, which breaks Grafana panels and autoscalers that depended on the old name.

The hit rate has to be derived in PromQL with rate() over the two counters. The one that appears in the text log is something else: it uses a window of the last thousand requests.

And it has a bias you need to know about before taking decisions with it. When a request is evicted, its blocks return to the free queue keeping the hash, so the later reprefill finds them again and counts as a hit. A high hit rate can mean useful reuse between users or it can mean the engine is evicting and recomputing in a loop. The internal statistics separate the two things with a preempted field, which is not exposed as a Prometheus metric.

The last token

A small detail with an effect on long conversations: the maximum hit length is the request’s token count minus one. There is always something to recompute in order to get logits. Since the reservation requires alignment to the block size, in practice that can force recomputing the whole final block. With blocks of 16 it is noise. With a backend that prefers 256, it is up to 256 tokens recomputed per turn.

The ones that do do it

vLLM production-stack

Eight routing logics in a single --routing-logic selector: roundrobin, session, kvaware, loadaware, prefixaware, disaggregated_prefill, disaggregated_prefill_orchestrated and priority. Two of them, loadaware and priority, are not in the Helm chart’s enum and are only enabled through extraArgs.

The naming trap is that prefixaware and kvaware are radically different things.

prefixaware is a hash trie in the router’s process. It chunks the prompt string every 128 characters, not tokens, and hashes each chunk with 64-bit xxhash. Its own docstring says so: the chunk size is “in number of characters”. That 128 is hardcoded, there is no flag or Helm value to change it.

Four consequences come out of that. --prefix-min-match-length is measured in characters and is quantised to multiples of 128. A shared prefix that differs by one character inside the first chunk breaks the whole match, because there is no partial match inside a chunk. For chat, it concatenates the content of all the messages with newlines and without applying the chat template, so the indexed string is not what the engine tokenises. And the tie-break between tied replicas is random.choice, without looking at load.

The trie has no notion of saturation. A popular prefix concentrates all the traffic on the same replica indefinitely. The docstring assumes the simplified model and declares it: “we assume there is no prefix cache eviction”.

It has no pruning either. HashTrie only has insertion and longest prefix lookup; there is no deletion method. When a replica disappears, its URLs stay forever in every node’s sets, and filtering happens at query time. It works, and the trie grows monotonically: each distinct prompt adds its length divided by 128 permanent nodes, with no bound, no TTL and no LRU. The router’s memory limit in the chart is 1,000 MiB. By contrast, the round-robin router in the same file does bound its caches to 1,024 entries.

And it dies on every restart: it is in-process memory state, with no snapshot and no initial load.

kvaware and loadaware are better and have a different price. They do not index themselves: they ask the LMCache controller, which knows about real tokens. That requires lmcache==0.3.11 and vllm==0.13.0 pinned exactly, and the official router image drags in the whole of vLLM. The limitation to read before planning anything: they only look at request_json.get("prompt", ""), with an open TODO for chat completions. With /v1/chat/completions, both tokenise an empty string.

loadaware is the only one of the three that breaks affinity on saturation, with the formula score = relative_match - beta * relative_load and beta = 1.0 by default. The authors’ own reading: a replica at twice the average load loses the equivalent of a full cache hit. The guide suggests 0.25 to favour cache and 2.0 to favour balancing.

On published results, the documentation’s benchmarking file literally says that the measurement platform is coming soon. The project blog’s two posts claim between three and ten times lower response latency, and only one declares a partial configuration: Llama 3.1 70B on four nodes with tensor parallelism 2 over 80 GB A100s, inputs of 9K tokens and outputs of 10, against AIBrix 0.2.0 and a flat deployment. Charts with no labelled points, no absolute figures and no declaration of which routing logic was used in each.

llm-d

There has been a reorganisation here that invalidates a good part of the third-party documentation. The Endpoint Picker code left the Gateway API Inference Extension repository and was merged into llm-d’s, which has been renamed to llm-d Router. The Kubernetes project keeps the InferencePool API and a lightweight reference EPP that does nothing about prefixes. If you look for KV-aware routing in the extension’s repository, today it is not there.

The InferencePool is at v1 and is GA. The other two APIs moved to the llm-d.ai group at v1alpha2, and when they coexist with the old ones the EPP prefers the new ones and ignores the old.

Scoring is a weighted sum. The default weights, when no configuration is passed: queue 2, KV utilisation 2, prefix 3. Any scorer referenced without a weight counts as 1.0. There is no configuration through environment variables: the old ENABLE_*_SCORER no longer exist, everything goes in an EndpointPickerConfig in YAML.

The prefix scorer consumes an attribute published by a producer, and there are two producers.

The approximate one computes the hashes in the EPP itself without talking to the engine, and keeps a local LRU per pod and block pair that is populated after the decision. It is an approximation by construction. It has a hard floor: a configured block size below 64 tokens is raised to 64 at request time, because the index stores one entry per pod and block and going down to 16 would multiply memory by four.

The exact one builds the real index from the engine’s ZMQ events, with one subscriber per pod that is installed and torn down with the endpoint’s registration and deregistration. It has adapters for vLLM and SGLang, deduplicates evictions by reference counting, and the replay buffer that allows the index to be rebuilt after an EPP restart requires vLLM 0.26.0 or above.

The trap is in the tokeniser. The EPP tokenises through a plugin, and the default backend is estimate: byte packing with no tokeniser, roughly 4 bytes per token, with identifiers that do not correspond to real engine tokens. The exact scorer needs real tokens, and for that you have to explicitly configure the vllm backend, which does HTTP against the engine’s /v1/completions/render endpoints. If it is omitted, the auto-created estimate satisfies the dependency and the system degrades silently.

On maturity, the file itself declares it: all in-tree plugins are in alpha or beta, and they will be promoted to stable when the project approaches 1.0. An alpha plugin configured without the corresponding flag makes the EPP fail to start.

And on replicas, the warning is explicit: active-active mode should be avoided with approximate prefix routing, because the EPP replicas do not share state and each one only sees the traffic it has served itself, which significantly degrades the hit rate. The exact index is safe in high availability, and even so the reference guides fix a single replica because the accounting of in-flight requests is local to the process.

The cost is the best documented in the whole field, and it deserves a table. Simulator with Qwen3 8B, 100 pods, 100K input tokens:

Requests/s	Cap on tokens to match	Peak CPU	Peak memory
5.0	4,096	1.19 cores	0.26 GiB
5.0	100,000	3.82 cores	0.65 GiB
98.7	4,096	35.17 cores	2.46 GiB
98.8	100,000	46.50 cores	3.41 GiB

With outputs of 10K tokens at 50 requests per second, 32.53 cores and 12.54 GiB. The maintainers’ sizing rule is half a core to one core per request per second on large agentic loads. And idle consumption scales with the number of pods because of continuous scraping: with 100 pods, about 7.5 cores with no traffic.

NVIDIA Dynamo

A precise event-driven index, with a global radix tree in each frontend’s process. Since all the replicas consume the same event plane, there is no need to synchronise routers with each other for the prefix state.

The cost function is explicit and weighted. It discounts from the prefill the blocks that overlap in GPU, host and disk with different weights (1.0 for the overlap credit, 0.75 for host, 0.25 for disk), and adds the potential decode blocks and the active requests.

There is a parameter to look at before deploying: --router-kv-overlap-score-credit-decay is at 0 by default, that is, disabled. Its description says what it is for: preventing loaded, cache-rich replicas from winning again and again while the freshly autoscaled ones receive too little traffic. With the default value, a new pod has no cache, therefore never wins the scoring, therefore never generates cache.

The approximate mode exists (--no-router-kv-events, with a TTL of 120 seconds) and the documentation says it is not the recommended route in production.

Dynamo is the most honest project in its measurement methodology: its comparison guide recommends contrasting --router-mode random against kv over the public Mooncake trace from FAST'25, with a declared cache ratio of 59 %, and it generates the table locally instead of publishing a number.

The rest of the field

SGLang has renamed its router to sgl-model-gateway, version 0.3.2 within SGLang 0.5.19. Its cache_aware policy is an approximate radix tree over raw characters, with switching to shortest queue when the system is unbalanced according to two combined thresholds. One detail that can cost an afternoon: the Rust defaults and the Python CLI defaults do not match, and on the maximum tree size they differ by four orders of magnitude (10,000 nodes against 2^26).

AIBrix 0.7.0 offers a standard mode with a local hash table and a KV event synchronisation mode behind a flag and a build tag, with vLLM 0.7.0 or above.

Ray Serve LLM has PrefixCacheAffinityRouter with an explicit alpha API warning, a character-based prefix tree in a decoupled actor, and its imbalance threshold at infinity by default, that is, out of the box it never breaks affinity on load.

Envoy AI Gateway 1.1.0 implements none of this and delegates to InferencePool and the EPP. KServe the same, with its LLMInferenceService at v1alpha1. Mooncake Conductor is still an unimplemented proposal.

Comparison table

Project	Version and date	Index	Where it lives	Kubernetes	Declared maturity
production-stack `prefixaware`	main branch, 09-Sep-2026	Approximate, character trie, no eviction	Router process	Optional	WIP in the README
production-stack `kvaware`/`loadaware`	same	Exact via LMCache controller	LMCache controller	Optional	No chat completions
llm-d Router approximate	v0.10.0, 17-Aug-2026	Approximate, LRU per pod	EPP process	Yes	Beta, active-active discouraged
llm-d Router exact	same	Exact through ZMQ events	EPP process	Yes	Beta
NVIDIA Dynamo	v1.4.2, 27-Aug-2026	Exact, event-driven radix tree	Frontend process	No	GA with experimental subsystems
SGLang `sgl-model-gateway`	0.3.2, 03-Sep-2026	Approximate, character radix	Gateway process	No	Default policy
AIBrix	v0.7.0, 16-Jun-2026	Both	Gateway plugin	Yes	Synchronisation behind a flag
Ray Serve LLM	Ray 2.58	Approximate, characters	Ray actor	No	Declared alpha
Envoy AI Gateway	v1.1.0, 21-Aug-2026	None of its own	Delegates to EPP	Yes	Stable 1.x API

How much it really improves

Here two types of source have to be separated.

The projects’ own numbers. Red Hat published the llm-d case in May: Qwen3 32B, eight vLLM pods over 16 H100s with tensor parallelism 2, synthetic shared-prefix load with 150 groups of five prompts, 6,000 tokens of system prompt, 1,200 of question and 1,000 of output. Result: up to 109 % more throughput and up to 99 % less TTFT, around 200 concurrent users within SLO against around 20. The repository’s result files give more detail: peak throughput from 6,986 to 14,892 output tokens per second, p90 TTFT from 135.5 seconds to 0.26.

That same file publishes the regression, and that is why it is worth reading: p50 inter-token latency rises by 22.4 % with vLLM and by 45.4 % with SGLang. The trade-off is declared: routing by affinity concentrates more concurrent work on the pods with a warm cache.

It is the best possible case. A synthetic shared prefix and a reference that is a flat Kubernetes Service.

The independent measurements. There are three and they say different things.

CacheRoute, from Meta, August 2026, is the most complete. Llama 3.3 70B in fp8, 30 targets with tensor parallelism 2 over 60 H100s, a semi-synthetic trace from production telemetry with 128,824 business keys, and all the alternatives reimplemented in the same harness instead of comparing against a dumb balancer. It obtains 176 requests per second with a p99 SLO of 3.5 seconds against 76 for the best competitor, and a hit rate of 93.2 % against 72.0 %.

And it publishes two negative findings that nobody else publishes. There are load regimes where affinity reduces capacity to between half and two thirds of the reference, to the point that the authors make a mirrored-traffic trial mandatory before deploying. And growing the fleet can cool a prefix even though total cache capacity grows, because the time between revisits stretches beyond the eviction window. Scaling horizontally can make the hit rate worse.

GORGO, June 2026, measures on a multi-region deployment with a dataset where intra-user prefix reuse is 89.4 % and the average prompt is almost 18,000 tokens. An improvement of 6.9 to 15.5 % in p95 TTFT. With extremely high reuse, the improvement is single-digit, because network latency and queuing dominate.

LAAR, from IBM Research Tokyo, April 2026, is the uncomfortable counterpoint: in its comparison, session affinity routing was the worst of the baselines, and at 64K contexts a load-based routing gives lower absolute latency at the cost of fewer hits.

The joint reading fits in three conditions. The big improvements require high, concentrated prefix reuse, long prompts, and a weak reference. Take away any of the three and the numbers deflate.

The five failure modes

Hot spot from affinity. The busiest bucket inherits the load’s skew, and tail latency starts following the hottest target instead of the fleet average. Mitigations exist in the four main products and in three of the four they ship switched off: Dynamo’s credit decay at 0, Ray’s imbalance threshold at infinity, and production-stack’s prefixaware with none at all.

Starvation of the freshly scaled pod. A new pod has no cache, so it never wins the scoring, so it never warms up. Dynamo documents it as the reason for the decay parameter. I found no primary documentation from any of the projects about the specific interaction with KEDA.

Cost of the index. The llm-d numbers in the previous table are the best public reference that exists. Raising the cap on tokens to match from 16,384 to 400,000 can more than double the EPP’s CPU with little traffic.

Traffic with unique prefixes. Each project degrades to something different. SGLang routes to the smallest tree, which is not the same as the least loaded. Ray falls back to power of two choices. Dynamo has an explicit preset, --load-aware, to say “I want the load model without reuse”. And production-stack’s prefixaware keeps inserting into the trie, that is, keeps paying the cost while receiving nothing.

Sibling bursts. Parallel sampling, best of N, or an agent that opens five branches at once. With event-driven routing, no engine has yet emitted the block stored event, all the siblings score zero overlap, and the prefix is precomputed on every replica. Dynamo has a prediction TTL for this; the others do not.

The architecture that comes out of all of the above

Two layers, with the responsibilities separated.

On top, LiteLLM, with what it does do well: non-forgeable identity through the virtual key, budgets, spend logging, concurrency limits, fallback between providers and API unification. For the gateway, each model group has a single destination: the URL of the router below.

model_list:
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://kv-router.inferencia.svc:8000/v1
 api_key: os.environ/VLLM_KEY

router_settings:
 routing_strategy: simple-shuffle
 num_retries: 1
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600

general_settings:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

Below, the KV-aware router, inside the engine’s domain, talking to the vLLM replicas and consuming their events.

Four consequences of this separation that have to be accepted before building it.

Per-replica visibility in LiteLLM is lost. For the gateway there is a single deployment, so its cooldowns, its per-deployment counters and its strategies stop meaning anything. Observability of which replica served what goes downstairs.

Retries stack up. The client SDK retries, LiteLLM retries, and the router below may have its own failover. In the previous article 45 calls per turn came out with three layers; with four it is worse. Lowering num_retries in the gateway to 1, or to 0, is the sensible decision when there is a router below that already retries.

LiteLLM’s session affinity stops making sense for the KV, because it no longer chooses a replica. It is kept if it is used for something else, such as pinning a conversation to a specific provider.

And the choice of the router below conditions the rest of the cluster far more than the gateway’s does. llm-d requires Kubernetes 1.32 or above, Gateway API, the inference extension and a conformant Gateway. Dynamo does not require Kubernetes. production-stack has a Helm chart and works outside it. That decision is the expensive one, not the gateway’s.

When not to build this

Three cases where the correct answer is not to do it, at least not yet.

A single replica per model. With one replica, the engine’s prefix cache already does all the work and there is nothing to route. That is the case for a good part of the four H100 deployments with one large model and tensor parallelism.

Low or scattered prefix reuse. Before building anything, the measurement is rate(vllm:prefix_cache_hits_total[5m]) / rate(vllm:prefix_cache_queries_total[5m]) per replica, with a mental correction for the bias from reprefills after eviction. If it is already high with blind distribution, reuse is concentrated inside each conversation, and there LiteLLM’s session affinity costs one line of configuration and resolves the case.

Traffic where the network dominates. The GORGO result is the warning: with enormous prompts and a distributed deployment, the improvement is eaten up by queuing and network latency.

The case where it does pay off is specific: several replicas of the same model, a large system prompt shared between different users, or a RAG with a corpus of documents that repeat across requests. There the prefill saved is real and it is most of the work.

Checklist

Measure the current hit rate per replica with the two vLLM counters before touching anything, and check whether the engine is evicting with vllm:num_preemptions_total.
Check that prefix caching is switched on. It is no longer --enable-prefix-caching that switches it on: the default value is derived from the model and the log of the decision is at DEBUG level. What is actionable today is --no-enable-prefix-caching to switch it off.
Audit the prompts before routing: a timestamp or a session identifier at the start of the system prompt breaks the match from block zero and no routing layer fixes it.
If the KV event stream is enabled, size the bandwidth by tokens per second and watch discarding at the high water mark. With data parallelism, subscribe to one port per rank.
If an exact router is chosen, verify the vLLM version against the one the replay buffer requires.
If llm-d is chosen, explicitly configure the vllm tokenisation backend. The default estimate does not fail, it degrades.
If production-stack is chosen with kvaware or loadaware, check first whether the traffic goes through /v1/chat/completions, because today it does not work there.
Review the load-based affinity-breaking parameter, because in three of the four main projects it ships disabled.
Lower num_retries in LiteLLM when there is a router below that already does failover.
Trial with mirrored traffic before changing production. It is the explicit recommendation of the only independent measurement that went looking for bad regimes, and found them.

Traps and things that are not what they seem

LiteLLM has no prefix routing in any version. Neither the open one nor the paid one. I got it wrong in June.
prompt_caching does nothing if the client does not send cache_control, and with the default automatic injection it fails on every turn because it marks the last message.
That pin’s TTL is 300 hardcoded seconds, incompatible with the one-hour caches that LiteLLM itself knows how to request.
production-stack’s prefixaware indexes characters, not tokens, in non-configurable chunks of 128, and its trie is never purged and models no eviction.
production-stack’s kvaware and loadaware do not support chat completions. They tokenise an empty string.
The default tokeniser in llm-d’s EPP is a byte estimate, and with it the exact scorer degrades silently.
llm-d’s active-active mode is discouraged with approximate prefixes, and the reference guides fix one replica even with the exact one.
Dynamo’s overlap credit decay ships at zero, which can leave a freshly autoscaled replica with no traffic.
The SGLang router’s default values differ between the Rust and the Python CLI, by as much as four orders of magnitude.
vllm:gpu_prefix_cache_hit_rate no longer exists and vllm:gpu_cache_usage_perc is now called vllm:kv_cache_usage_perc. Old panels and autoscalers are measuring nothing.
The hit rate counts the reprefill after eviction as a hit. A high number can be reuse or it can be thrashing.
KV events do not publish the hits by default, only the additions, unless full mode is requested request by request.
The hashes travel truncated to 64 bits unless it is disabled with an environment variable.
The default block size is 16, and the attention backend can change it without anyone asking.
The Endpoint Picker left the Gateway API Inference Extension repository. Looking for it there today leads nowhere.

Closing

June’s error has a reading that goes beyond the erratum. It came from reading a product’s documentation and not its code, and from the fact that LiteLLM’s documentation uses the word “caching” in a pre-call check that does something else. The distance between what a name suggests and what a function does is, at this point in the track, the most repeated pattern of the whole series.

The architectural conclusion is that KV-aware routing is not a gateway function. It is not one today in LiteLLM and probably should not be: it requires keeping an index synchronised with each engine’s memory, replicating its hash function, consuming an event stream and sizing CPU per request per second. That belongs to the engine’s domain, not to that of identity and budget.

What remains is a two-part decision. The first is whether the traffic deserves it, and that is answered with two Prometheus counters and without installing anything. The second is which of the three routers fits the cluster you already have, and there the deciding variable is not the hit rate but whether you are willing to bring in Gateway API and the inference extension, or to pin vLLM 0.13.0, or to operate a Dynamo frontend.

And the honesty the material demands: the only independent measurement that specifically went looking for regimes where this makes things worse found them, and they are not marginal. Half to two thirds of reference capacity is a serious regression. That no project publishes that side of the curve does not mean it does not exist in theirs.

Sources

LiteLLM, Router - Load Balancing (estrategias, comprobaciones previas, afinidad): https://docs.litellm.ai/docs/routing.
LiteLLM, código: litellm/router.py, litellm/router_utils/prompt_caching_cache.py, litellm/router_utils/pre_call_checks/prompt_caching_deployment_check.py, litellm/router_utils/pre_call_checks/deployment_affinity_check.py, litellm/proxy/litellm_pre_call_utils.py, litellm/proxy/middleware/admission_control_middleware.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 28427, TTL de afinidad por caché de prompts fijado a cinco minutos: https://github.com/BerriAI/litellm/issues/28427.
vLLM, código: vllm/v1/core/kv_cache_utils.py, vllm/v1/core/block_pool.py, vllm/v1/core/kv_cache_manager.py, vllm/distributed/kv_events.py, vllm/config/kv_events.py, vllm/v1/metrics/loggers.py, vllm/v1/metrics/stats.py: https://github.com/vllm-project/vllm.
vLLM, KV offloading usage (niveles CPU, filesystem y objeto; compartición entre procesos): https://docs.vllm.ai/en/latest/features/kv_offloading_usage.html.
vLLM, notas de la versión 0.29.0 (semilla determinista, seguimiento de propiedad en eventos, flags de admisión): https://github.com/vllm-project/vllm/releases/tag/v0.29.0.
vLLM, RFC 16669, publicación de bloques KV y métricas: https://github.com/vllm-project/vllm/issues/16669.
vLLM production-stack, código: src/vllm_router/routers/routing_logic.py, src/vllm_router/prefix/hashtrie.py, src/vllm_router/parsers/parser.py: https://github.com/vllm-project/production-stack.
LMCache Lab, resultados de production-stack frente a AIBrix (parte interesada): https://blog.lmcache.ai/en/2025/03/06/open-source-llm-inference-cluster-performing-10x-faster-than-sota-oss-solution/.
llm-d Router, código y documentación de operaciones (pesos por defecto, dimensionamiento del EPP, aviso de activo-activo): https://github.com/llm-d/llm-d-router.
llm-d, guía de enrutado preciso por prefijo y resultados publicados (parte interesada): https://github.com/llm-d/llm-d.
llm-d Router, incidencia 1290, estado de prefijo no compartido entre réplicas del EPP: https://github.com/llm-d/llm-d-router/issues/1290.
Red Hat, Same 16 GPUs, twice the users (parte interesada): https://www.redhat.com/en/blog/same-16-gpus-twice-users-inference-aware-routing-llm-clusters.
Kubernetes SIG Network, Gateway API Inference Extension (traslado del EPP y del BBR): https://github.com/kubernetes-sigs/gateway-api-inference-extension.
NVIDIA Dynamo, diseño y ajuste del router KV: https://github.com/ai-dynamo/dynamo.
SGLang, sgl-model-gateway: https://github.com/sgl-project/sglang/tree/main/sgl-model-gateway.
AIBrix, enrutado por caché de prefijo: https://github.com/vllm-project/aibrix/blob/main/pkg/plugins/gateway/algorithms/prefix_cache_readme.md.
Ray Serve LLM, enrutado consciente de prefijo (API en alfa): https://docs.ray.io/en/latest/serve/llm/user-guides/prefix-aware-routing.html.
Huang Cheng (Meta), CacheRoute, agosto de 2026: https://arxiv.org/html/2608.19677.
Yuan et al., DualMap, febrero de 2026: https://arxiv.org/abs/2602.06502.
Ricci Toniolo et al., GORGO, junio de 2026: https://arxiv.org/html/2602.11688.
Yoshimura, Chiba y van de Beek, Accuracy Is Speed (EuroMLSys ‘26), abril de 2026: https://arxiv.org/html/2604.15732v1.

LLM serving engines compared on the Pareto frontier: vLLM, SGLang, TRT-LLM and Dynamo

Tue, 16 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter). Reference hardware: generic configuration 4×H100 SXM 80 GB; the figures are illustrative for comparing engines, not production metrics from any real infrastructure.

TL;DR

Four OSS engines dominate on-premise LLM serving in 2026: vLLM, SGLang, NVIDIA TensorRT-LLM and NVIDIA Dynamo. None is superior on every axis. In raw throughput with a shared prompt, SGLang beats vLLM by 29 % on Llama-3.3-70B FP8 over H100 (16,200 vs 12,500 tok/s). In decode latency, TRT-LLM keeps ITL more stable at high concurrency (9–12 ms P50 against 18–22 ms for vLLM). Dynamo is not an inference engine: it is the orchestration layer on top of the previous three for multi-node clusters, and it reports up to 7× additional throughput in disaggregated serving on Blackwell. The right choice depends on three variables: load pattern (batch/interactive), deployment scale (single node/multi-node) and operational constraints (OSS maturity, ease of operation). The Pareto decision table is at the end of the article.

Framing the article: what this B8 is and is not

This article (B8 of the benchmarking pillar) takes for granted the metrics from the introduction to benchmarking and the tool catalogue from the card-by-card piece. It applies those concepts to engine selection. The protocol from GuideLLM for SLO validation and AIPerf in depth detail how to run the harness; here the result is used. The fundamentals of continuous batching, disaggregated serving, the attention backend and quantisation are assumed known.

Card for each engine

vLLM

Attribute	Value
Reference version	v0.9.x (June 2026)
Licence	Apache 2.0
Maintainer / governance	vLLM Project (Linux Foundation AI), led by UC Berkeley + Red Hat
Supported hardware	NVIDIA (Ampere/Hopper/Blackwell), AMD ROCm, Intel Gaudi, TPU, CPU
Repository	github.com/vllm-project/vllm
API	OpenAI-compatible (completions, chat, embeddings)

Origin: PagedAttention (Kwon et al., 2023). The first implementation of continuous batching with a paged KV cache. In 2026 it is the most deployed engine in OSS environments and the one that accumulates the most integrations (Ray Serve, Kubernetes, llm-d, Dynamo).

SGLang

Attribute	Value
Reference version	v0.4.x (June 2026)
Licence	Apache 2.0
Maintainer / governance	LMSYS Org (Berkeley, CMU, UCSD)
Supported hardware	NVIDIA (Ampere/Hopper/Blackwell), AMD ROCm
Repository	github.com/sgl-project/sglang
API	OpenAI-compatible + SGLang runtime API

Origin: the SGLang paper, 2024 (Zheng et al.). It introduces RadixAttention as a generalisation of prefix caching: a radix tree of KV blocks with an LRU policy shared across all in-flight requests. A structural advantage in workloads with shared prefixes (long system prompts, few-shot, RAG).

NVIDIA TensorRT-LLM (TRT-LLM)

Attribute	Value
Reference version	v0.18.x (June 2026)
Licence	Apache 2.0
Maintainer / governance	NVIDIA (proprietary in practice, OSS in name)
Supported hardware	NVIDIA only: Ampere (A100), Hopper (H100/H200/GH200), Ada (L40/L40S), Blackwell (B200/GB200)
Repository	github.com/NVIDIA/TensorRT-LLM
API	Python + C++ runtime; integrates with Triton Inference Server

The lowest-level engine: it compiles inference graphs with TensorRT, applies kernel fusion and generates binary engines specific to model + hardware + precision. Maximum performance on NVIDIA, with no portability to other hardware. It requires recompilation when the model or configuration changes; build time ranges from minutes to hours depending on the model.

NVIDIA Dynamo

Attribute	Value
Reference version	v1.1.1 (May 2026)
Licence	Apache 2.0
Maintainer / governance	NVIDIA + OSS community (ai-dynamo org on GitHub)
Supported hardware	NVIDIA (through the underlying engine: vLLM, SGLang or TRT-LLM)
Repository	github.com/ai-dynamo/dynamo
Role	Orchestration layer, NOT an inference engine

Dynamo does not run inference directly: it is the scheduler and router that coordinates pools of workers (vLLM, SGLang, TRT-LLM) across multiple nodes. Implemented in Rust (performance) + Python (extensibility). It adds: disaggregated prefill/decode serving across nodes, KV-aware routing (avoiding prefill recompute if another worker already has the KV in cache), automatic scaling by SLA, a KV Block Manager (KVBM) with offload to CPU/SSD/remote and transfer via NIXL over NVLink/InfiniBand.

Honourable mentions: LMDeploy and HF TGI

LMDeploy (OpenMMLab/InternLM, Apache 2.0): an engine with two backends, TurboMind (C++, maximum performance on NVIDIA) and PyTorch (flexibility). KV cache blocking and persistent batching; 4-bit inference up to 2.4× faster than FP16 in its own benchmark. A narrower ecosystem; no AMD support.

HF TGI (Hugging Face, Apache 2.0): the reference engine for the HuggingFace ecosystem. TGI v3.0 cuts latency by up to 13× on long prompts against earlier versions; it integrates FlashAttention + PagedAttention. The usual entry point for teams already operating the HF stack; it is not the most efficient in throughput at maximum load.

Capability matrix

The table uses this notation: Yes = supported and stable, Beta = available but not recommended for production, No = not available or only with a manual workaround.

Capability	vLLM	SGLang	TRT-LLM	Dynamo (via backend)
Continuous batching	Yes	Yes	Yes	Yes (delegates to the engine)
PagedAttention	Yes	Yes	Yes	Yes
RadixAttention / prefix caching	Yes (chunked prefill + prefix)	Yes (native RadixAttention)	Yes (prefix caching)	Yes + KV-aware routing
Speculative decoding	Yes (n-gram, EAGLE, DFlash, suffix)	Yes (draft+verify)	Yes (EAGLE-3, MTP)	Yes
FP8 quantisation	Yes	Yes	Yes	Yes
MXFP4/NVFP4 quantisation	Yes	Beta	Yes (Blackwell)	Yes
INT4 AWQ quantisation	Yes	Yes	Yes	Yes
GPTQ quantisation	Yes	Yes	Not native	Yes (via vLLM)
INT8 quantisation (SmoothQuant)	Yes	Yes	Yes	Yes
Structured output	Yes	Yes (compressed FSM)	Beta	Yes
Dynamic multi-LoRA	Yes	Yes	Beta	Yes
Disaggregated prefill/decode	Beta (experimental)	Yes	Yes	Yes (first class)
Chunked prefill	Yes	Yes	Yes	Yes
Multimodal support (VLM)	Yes	Yes	Yes	Yes
Multi-node tensor parallelism	Yes	Yes	Yes	Yes (+ pipeline across nodes)
OpenAI API compatible	Yes	Yes	Yes (via Triton)	Yes
Non-NVIDIA hardware	Yes (AMD/Intel/TPU)	Yes (AMD)	No	No

Sources for the matrix: docs.vllm.ai, docs.sglang.io, nvidia.github.io/TensorRT-LLM, docs.nvidia.com/dynamo, github.com/ai-dynamo/dynamo (feature matrix, June 2026).

Why they are not compared lightly: the variables that break a comparison

Before the table of figures, the four decisions that invalidate any cross-engine benchmark:

1. A different harness. A benchmark run with each engine’s native micro-bench (vllm bench serve vs SGLang bench) measures the client, not the engine. The discrepancy can be up to 7.2× at high concurrency. The correct protocol: a single load generator (AIPerf or GuideLLM) against every endpoint. The engine is the variable, the tool is the constant.

2. Different ISL/OSL. Input Sequence Length and Output Sequence Length determine the prefill/decode ratio and therefore which engine is favoured. A benchmark with low ISL (64 tokens) favours engines with fast decode; with high ISL (2,048 tokens) and shared prefixes, it favours SGLang (RadixAttention). Fixing ISL and OSL is mandatory for the comparison to be valid.

3. Different precision. FP16 and FP8 are not comparable: FP8 gives between 1.5× and 2× more throughput on Hopper. Comparing vLLM FP16 with TRT-LLM FP8 says nothing about the engine.

4. Not reporting P99. Maximum throughput is reached at the point where TTFT P99 already violates the SLO. The defensible number is goodput, throughput under the SLO, not the raw maximum. The end-to-end latency formula:

$$\text{latency}_{e2e} \approx \text{TTFT} + (N_{\text{out}} - 1) \times \text{TPOT}$$

where $N_{\text{out}}$ is the number of output tokens, makes it clear that a high TTFT with low TPOT has the same effect as the inverse case only for one specific $N_{\text{out}}$: both must be reported separately.

The complete fair-comparison protocol is in the tools article.

Performance on the Pareto frontier: illustrative figures

Reference configuration for every row: 4×H100 SXM 80 GB NVLink, model Llama-3.1-70B-Instruct FP8, harness AIPerf (formerly genai-perf, the multi-process successor), dataset ShareGPT (realistic length distribution), mean ISL ~512 tok, mean OSL ~256 tok, SLO TTFT P99 < 500 ms. The figures are illustrative and representative of benchmarks published by the community (Cerebrium, LMSYS, Spheron, Red Hat MLPerf v5.1); they are not measurements from any real infrastructure. Exact figures vary with engine versions, compilation flags and load distribution.

Throughput vs latency: Pareto table

Engine	Config	Throughput (tok/s)	TTFT P50 (ms)	TTFT P99 (ms)	ITL P50 (ms)	Goodput (@SLO)
vLLM v0.9	FP8, chunked prefill	12,500	160	420	18	~96 %
SGLang v0.4	FP8, RadixAttention	16,200	140	390	21	~97 %
TRT-LLM v0.18	FP8, compiled engine	14,800	190	480	10	~93 %
vLLM + Dynamo	FP8, disagg. P/D 2+2	18,500	120	310	19	~98 %
SGLang + Dynamo	FP8, disagg. P/D 2+2	21,000	110	280	22	~99 %
TRT-LLM + Dynamo	FP8, disagg. P/D 2+2	22,500	130	350	11	~97 %

Reading notes:

“Disagg. P/D 2+2” = 2 prefill GPUs + 2 decode GPUs (the same 4×H100 budget).
Goodput falls against raw throughput when the latency tail exceeds the SLO of 500 ms P99. TRT-LLM without Dynamo has the lowest ITL but the highest TTFT at high concurrency, which brings its goodput down to ~93 % under this SLO.
Dynamo adds ~30–40 % of effective throughput over 4 GPUs thanks to disaggregated serving and KV-aware routing, in the scenario with reusable prefixes (ShareGPT).
In workloads without shared prefixes (short ISL, unique prompts), Dynamo’s advantage shrinks to ~10–15 % over the base engine.

Saturation point (concurrency sweep, vLLM FP8, 4×H100)

Concurrency	TTFT P99 (ms)	ITL P50 (ms)	Throughput (tok/s)	Goodput (tok/s)
8	210	14	5,200	5,200
16	370	18	9,800	9,800
32	420	18	12,500	12,200
48	680	26	13,800	7,400
64	1,200	41	14,100	2,100

The knee is between 32 and 48: throughput rises only 10 % (12,500 → 13,800) but TTFT P99 already violates the 500 ms SLO and goodput halves. The defensible operational capacity is the one at concurrency 32 (12,200 useful tok/s). Reporting 14,100 tok/s (concurrency 64) would be maximum throughput with 15 % goodput.

Fair-comparison methodology: the harness as the constant

The protocol that makes numbers comparable across different engines. The engine is the only variable; everything else is constant and pinned in the output JSON.

Variables that are fixed (constants of the experiment)

Variable	Value fixed in the reference experiment
Load tool	AIPerf v2.x (multi-process)
Model	Llama-3.1-70B-Instruct
Precision	FP8
Hardware	4×H100 SXM 80 GB NVLink
Dataset	ShareGPT (realistic ISL/OSL distribution)
SLO	TTFT P99 < 500 ms, ITL P50 < 30 ms
Warm-up	200 requests discarded before measuring
Sweep	concurrencies 1, 4, 8, 16, 32, 48, 64
Primary metric	goodput (useful tok/s under SLO)

Harness steps

Deploy the engine with the exact config (pinned flags, fixed version).
Warm-up: 200 requests; discard the results.
Concurrency sweep: aiperf profile --concurrency-range 1:64:step with AIPerf.
Extend the sweep past the knee (until goodput falls below 50 %).
Collect the JSON: TTFT P50/P99, ITL P50, raw throughput, goodput.
Change only the engine; repeat steps 1–5.
Compare the goodput column across rows.

The sweep with GuideLLM is equivalent for SLO validation; AIPerf is preferable for engine comparisons because its automatic estimatedCapacity normalises the saturation point. Both can be used: GuideLLM for the operational SLO, AIPerf for reference capacity.

Reproducible output format

{
 "harness": "aiperf", "harness_version": "2.x",
 "model": "Llama-3.1-70B-Instruct", "precision": "FP8",
 "hardware": "4xH100_SXM_80GB_NVLink",
 "dataset": "sharegpt",
 "slo": {"ttft_p99_ms": 500, "itl_p50_ms": 30},
 "motor": "vllm", "motor_version": "0.9.x",
 "concurrency_sweep": [8, 16, 32, 48, 64],
 "results_at_knee": {
 "concurrency": 32,
 "ttft_p50_ms": 210, "ttft_p99_ms": 420,
 "itl_p50_ms": 18, "throughput_tok_s": 12500,
 "goodput_tok_s": 12200
 }
}

This versioned JSON is the auditable datum: anyone can reproduce the figure with the same tool, version, model, hardware and load.

State of the art 2026: relative maturity

Dimension	vLLM	SGLang	TRT-LLM	Dynamo
Production maturity	High (3+ years)	Medium-high (2 years)	High (NVIDIA)	Medium (1.0 GA Mar 2026)
Release cadence	Weekly	Weekly	Monthly	Fortnightly
Integration ecosystem	Very broad (Ray, k8s, llm-d, Dynamo, NIM)	Broad (Dynamo, k8s)	NVIDIA-centric (Triton, NIM)	NVIDIA-centric
Ease of operation	High (pip install, OpenAI API)	High	Medium (requires compiling engines)	Medium-low (multi-node, etcd/NATS)
Non-NVIDIA hardware	Yes (AMD, Intel, TPU)	Yes (AMD)	No	No
Disaggregated serving	Beta	Stable	Stable	First class
Dynamic multi-LoRA	Stable	Stable	Beta	Stable (via vLLM/SGLang)
Structured output	Stable	Stable (FSM)	Beta	Stable (via the engine)
OSS community support	Very high (>30 K GitHub stars)	High (~20 K stars)	High (~10 K stars)	Growing (6.8 K stars)

Pareto decision table: when to choose each engine

No prose; the decision as a table of criteria and outcome.

Selection criterion	Recommended engine	Quantitative justification
Single node, quick prototype, team with no LLM serving experience	vLLM	pip install + OpenAI API in minutes; the broadest community support
Workload with long shared prefixes (RAG, few-shot, system prompts > 512 tok)	SGLang	RadixAttention: up to 6.4× more throughput vs a baseline without prefix caching
Maximum throughput on NVIDIA, no portability, team with TRT experience	TRT-LLM	Fused kernels + native FP8: ITL 9–12 ms vs 18–22 ms for vLLM on the same hardware
Multi-node (> 8 GPUs), mixed prefill-intensive/decode-intensive traffic	Dynamo + SGLang	Disaggregated P/D: up to 7× additional throughput on Blackwell; 2× TTFT with KV-aware routing
AMD ROCm or Intel Gaudi hardware	vLLM	The only SOTA engine with stable non-NVIDIA support
Structured output + high concurrency	SGLang	Compressed FSM: guided decode with no appreciable overhead
Stable dynamic multi-LoRA in production	vLLM or SGLang	TRT-LLM multi-LoRA still in beta (Jun 2026)
Hugging Face ecosystem, models < 13B	HF TGI v3	Direct HF Hub integration; competitive latency on small models
Maximum energy efficiency (J/token) on NVIDIA	TRT-LLM	Lower-overhead kernels → fewer J/token at iso-throughput

Quantisation: effect on throughput and quality

The four main options in order of decreasing speed and increasing quality:

Format	Relative throughput (H100)	Quality degradation (perplexity)	Engines with stable support
NVFP4 / MXFP4	~2.2× vs FP16	1–3 % on MMLU	TRT-LLM (Blackwell), vLLM (experimental)
FP8 (W8A8)	~1.7× vs FP16	< 1 % on MMLU	vLLM, SGLang, TRT-LLM
INT4 AWQ	~1.5× vs FP16	1–2 % on MMLU	vLLM, SGLang, TRT-LLM
GPTQ (INT4)	~1.4× vs FP16	1–3 % on MMLU	vLLM, SGLang
FP16 (baseline)	1.0×	0 %	All

FP8 is the de facto standard on Hopper for production: 1.7× throughput with degradation below 1 %. The article on quantisation in depth develops the quality/speed trade-off by format and model architecture.

Disaggregated serving: when the overhead is worth it

Separating prefill/decode into independent pools introduces transfer latency for KV tensors between nodes (NIXL over NVLink or InfiniBand). The overhead pays off when:

Condition	Effect of disaggregated serving
Mean ISL > 1,024 tokens	Prefill dominates; separate pools stop it blocking decode
Prefill/decode ratio > 3:1 in time	The decode pool sits idle waiting for prefill
Mixed traffic (prefill burst + decode queue)	Independent scaling of each pool by SLA
< 512 tokens ISL, unique prompts	Transfer overhead exceeds the benefit; an aggregated engine is enough

Dynamo quantifies the break-even point: disaggregated serving pays off when the transfer time for KV tensors (a function of ISL and NVLink/IB bandwidth) is lower than the time the decode worker would spend waiting for the prefill worker in aggregated mode. On 4×H100 with an ISL of 512 tok, break-even occurs at a prefill/decode ratio > 2.5:1. The article on disaggregated serving develops the full formula.

MLPerf Inference v5.1: cross-vendor reference figures

MLPerf Inference v5.1 (September 2025) is the standard for cross-vendor comparability. Server scenario (Llama-3.1-70B, tokens/s):

Submitter	Hardware	Scenario	Throughput (tok/s)
Red Hat (vLLM)	1×H100 80 GB	Server	5,103
Red Hat (vLLM)	1×L40S	Server	1,207
NVIDIA (TRT-LLM)	8×H100 SXM	Server	~52,000 (estimated)

The MLPerf figures are not directly comparable with the internal benchmarks above (a different harness, MLPerf’s Server scenario with a Poisson distribution and fixed SLOs). They are useful for comparing vendors under the same rules; not for sizing a specific case. For that, run your own sweep with AIPerf/GuideLLM against the target endpoint.

Variables the benchmark does not capture

Omitted variable	Effect in production
Network latency (gateway, load balancer)	Can add 20–100 ms to the TTFT measured on the server
Engine compilation time (TRT-LLM)	Hours per model×precision×hardware; penalises cold starts and CI
Weight loading time on the GPU	Relevant in autoscaling; ModelExpress (Dynamo) cuts it 7×
Operational cost of the system (Dynamo vs a simple engine)	Dynamo requires etcd/NATS, a Rust runtime, a more specialised team
Response quality	Performance does not imply quality; measure with lm-evaluation-harness (article B7)
Energy (J/token)	Does not come out of a throughput benchmark; requires DCGM running alongside

Sources

vLLM · documentación oficial — https://docs.vllm.ai/en/latest/
vLLM · blog de rendimiento (v0.6.0, 2,7× throughput) — https://blog.vllm.ai/2024/09/05/perf-update.html
vLLM · disaggregated prefill (experimental) — https://docs.vllm.ai/en/latest/features/disagg_prefill/
SGLang · documentación oficial — https://docs.sglang.io/
SGLang · paper RadixAttention (LMSYS, 2024) — https://arxiv.org/pdf/2312.07104
SGLang · blog LMSYS sobre Llama-3 serving — https://www.lmsys.org/blog/2024-07-25-sglang-llama3/
NVIDIA TensorRT-LLM · GitHub — https://github.com/NVIDIA/TensorRT-LLM
NVIDIA TensorRT-LLM · overview y release notes — https://nvidia.github.io/TensorRT-LLM/overview.html
NVIDIA TensorRT-LLM · quantización — https://nvidia.github.io/TensorRT-LLM/features/quantization.html
NVIDIA Dynamo · GitHub (ai-dynamo/dynamo) — https://github.com/ai-dynamo/dynamo
NVIDIA Dynamo · documentación oficial — https://docs.nvidia.com/dynamo/latest
NVIDIA Dynamo · disaggregated serving — https://docs.dynamo.nvidia.com/dynamo/design-docs/disaggregated-serving
Cerebrium · benchmark vLLM vs SGLang vs TRT-LLM (Llama 3.1-70B) — https://cerebrium.ai/blog/benchmarking-vllm-sglang-tensorrt-for-llama-3-1-api
Spheron · vLLM vs TRT-LLM vs SGLang H100 benchmarks 2026 — https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/
MLCommons · MLPerf Inference v5.1 results — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
Red Hat · MLPerf Inference v5.1 con vLLM — https://www.redhat.com/en/blog/efficient-and-reproducible-llm-inference-red-hat-mlperf-inference-v51-results
arXiv 2605.24217 · sesgo sistemático en benchmarks de inferencia LLM — https://arxiv.org/html/2605.24217
IETF Draft · LLM Benchmarking Methodology (enero 2026) — https://www.ietf.org/archive/id/draft-gaikwad-llm-benchmarking-methodology-00.html

Long context and KV offloading: when the notebook does not fit on the desk

Fri, 12 Jun 2026 05:00:00 +0200

TL;DR

Serving long context is not serving a smarter model, it is managing a notebook that does not fit on the desk. The context window is set by the model (and extended with RoPE scaling / YaRN), but the cost of serving it is set by the KV cache, which grows linearly with sequence length and, in production, explodes: a 300k-token contract on Llama 3 70B eats ~93 GB of KV, more than an entire H100, and a million tokens asks for ~125 GB. When the KV does not fit in HBM, there are only two ways out: recompute (extremely expensive, attention is quadratic) or offload the KV to a cheaper memory hierarchy: DRAM, NVMe, network. The 2026 OSS state of the art, LMCache, Mooncake (Kimi’s platform) and NVIDIA Dynamo/KVBM, turns that offload into a first-class KV layer, with reuse across requests, prefill/decode disaggregation and cache-aware routing. Reported results: 3×–10× less latency with LMCache and up to +525 % throughput in long-context scenarios with Mooncake. The price: every memory hop adds transfer latency, so offloading only pays off when what you save by not recomputing beats what it costs to move the bytes.

The analogy

A researcher works at a small desk. On it fit the papers they are pulling from right now, which is the GPU’s HBM, blazingly fast but tiny. When the case is long (a thousand-page file, a million-token context), the papers do not fit. There are three options:

Throw papers away and ask the archive for them again every time they are needed. That is recomputing the KV: correct, but agonisingly slow, because “asking the archive” in an LLM means running the whole prompt through attention again, and attention is quadratic.
Put a shelf next to the desk (the CPU’s DRAM) and, further out, a store room in the basement (NVMe) and a depot in another building (network/object). Moving papers between the desk and the shelf costs seconds, not hours. That is hierarchical KV offloading.
Have two researchers split the work: one reads and marks up the whole file (prefill), then passes their notes to the second, who only drafts (decode). That is the KVCache-centric disaggregated architecture.

This post is about options 2 and 3, the ones that make long context viable in production. Option 1 is what you pay by default if you do nothing.

Part 1 · Why long context is a memory problem

The KV cache grows with the sequence

For every token that goes in or out, the model stores its key and value vectors in each layer so it does not have to recompute them. The size is:

$$\text{KV bytes} = 2 \times L \times h_{kv} \times d_{head} \times s \times b$$

with $L$ layers, $h_{kv}$ KV heads (few, with GQA), $d_{head}$ the per-head dimension, $s$ the sequence length and $b$ the bytes per element. Everything is a model constant except $s$: the KV is linear in context length. Doubling the context doubles the KV. That is the basis of KV cache: the working memory of inference.

The numbers are frightening

At production scale, that linear term turns brutal:

A 300k-token contract on Llama 3 70B consumes ~93 GB of KV alone, more than the 80 GB of an entire H100 (DigitalOcean · Long-Context Inference Cost).
A 1M-token context needs ~125 GB of KV, which exceeds both an RTX 4090 (24 GB) and an 80 GB A100 (Introl · Long-Context LLM Infrastructure).
Even a 7B at 128k climbs to ~14 GB of KV (against ~6 GB at 4k).

The KV stops being an implementation detail and becomes the resource that dictates your concurrency.

And on top of that, attention is quadratic

The KV is linear, but the compute of attention is quadratic in length. When the prompt reaches 1M tokens, generating each token can take on the order of 1.765 seconds, with more than 96 % of the latency spent in attention (Introl). The aggregate effect is a throughput collapse of 10×–100× against short contexts. This is why long context is not “fixed” with more VRAM alone: there is a memory problem (KV) and a compute problem (attention) at the same time.

Extending the window ≠ serving it cheaply

A common confusion: “my model supports 1M of context” does not mean “I can serve 1M cheaply”. The window is extended with RoPE scaling techniques such as YaRN, the practical option for fine-tuning open source models to long context: it needs 10× fewer training tokens and 2.5× fewer steps than naive RoPE interpolation, and it took LLaMA-2 from 4k to 32k and 128k (YaRN/LongRoPE). But that solves the model attending to long context, not the KV fitting on your GPU. According to the 2026 round-ups, May 2026 marked the first generation of open million-token models (with families supporting 256k extensible to 1M via YaRN) (letsdatascience). Treat it as a trend, not a closed spec, and verify capabilities per concrete model.

Part 2 · The KV memory hierarchy

The core idea of offloading is old in systems: hierarchical caching. “Hot” KV (what is in use) lives in HBM; “warm” KV drops to DRAM; “cold” KV to NVMe; “shared across nodes” to network or object storage. Every hop multiplies capacity and divides bandwidth.

Working out when offloading pays off

Offloading is not magic: moving KV from DRAM/NVMe back to HBM costs time. The rule is simple. Offloading pays off when the cost of recomputing exceeds the cost of transferring:

$$t_{recompute}(s) \;>\; t_{transfer} = \frac{\text{KV bytes}}{BW_{enlace}}$$

Since $t_{recompute}$ grows with attention (quadratic) and $t_{transfer}$ grows only with the size of the KV (linear) divided by the link bandwidth, the longer the context, the more the balance tips in favour of offloading. That is why offloading is the lever for long context: it is exactly where recomputing becomes unbearable. The LMCache documentation says it without hedging: offloading only helps when the recomputation it avoids is larger than the overhead it introduces (LMCache benchmark).

Part 3 · KV offloading in practice (OSS)

The limit of out-of-the-box prefix caching

vLLM already caches prefixes in HBM (see prefix cache: hit rate engineering and PagedAttention). The problem: it only lives in HBM, which is small. When the useful KV exceeds HBM, it is evicted and, on the next request, has to be recomputed. There is no speedup even with prefix caching enabled, because the cache is no longer there (vLLM vs LMCache benchmark). Native prefix caching is necessary but insufficient for long context.

LMCache: the persistent KV layer

LMCache adds persistent storage backends to vLLM’s prefix cache (and also supports SGLang and NVIDIA Dynamo as engines). It pulls the KV out of HBM and shares it across engines and across queries, covering a three-level hierarchy: GPU HBM, CPU DRAM and NVMe SSD, with additional backends for Redis/Valkey, Mooncake, InfiniStore, S3 and NIXL/GDS (GitHub LMCache, arXiv 2510.09665). Because DRAM/NVMe hold far more KV than HBM, the hit ratio rises, and LMCache reports 3×–10× less latency combined with vLLM. Against vLLM’s basic CPU offload, LMCache loads the KV at chunk level with high-performance CUDA kernels, which cuts transfer overhead.

A minimal start, offloading to CPU:

# vLLM with LMCache as the KV layer (offload to DRAM)
pip install lmcache
LMCACHE_CONFIG_FILE=lmcache.yaml \
vllm serve meta-llama/Llama-3.1-70B-Instruct \
 --tensor-parallel-size 4 \
 --kv-transfer-config '{"kv_connector":"LMCacheConnector","kv_role":"kv_both"}'

# lmcache.yaml — hierarchy HBM -> DRAM -> NVMe
chunk_size: 256
local_cpu: true # warm KV in DRAM
max_local_cpu_size: 200 # GB
local_disk: "file:///nvme/lmcache" # cold KV on NVMe
max_local_disk_size: 2000 # GB

Offloading to CPU adds moderate overhead but retains a good part of the benefit against recomputing (LMCache docs). Mind the physical path: that HBM↔DRAM↔NVMe traffic goes over PCIe and across NUMA nodes. It is worth reading PCIe topology, GPUDirect and ACS and NUMA, hugepages and CPU isolation, because a badly wired offload can eat its own gain.

Combine it with FP8

Offloading reduces where the KV lives; quantization reduces how much it weighs. They are orthogonal and they add up: KV in FP8 halves the size (see FP8 end-to-end), which means half the bytes to transfer on every offload. In long context, FP8 + offload is the default combination.

Part 4 · KVCache-centric / disaggregated architecture

The next level is not just offloading KV, but redesigning serving around the KV.

Mooncake (Kimi’s platform)

Mooncake, the serving platform behind Kimi (Moonshot AI), is the reference KVCache-centric disaggregated architecture: it separates the prefill and decode clusters, and exploits the underused CPU, DRAM and SSD resources of the GPU cluster to build a distributed KV cache. Its core is a KVCache-centric scheduler that maximises throughput while respecting SLOs, with a prediction-based early rejection policy for overloaded scenarios (arXiv 2407.00079, USENIX FAST'25). The long-context numbers are striking: up to +525 % throughput in certain simulated scenarios while respecting the SLO, and in production Kimi handles 115 % and 107 % more requests on A800 and H800 clusters respectively against previous systems. The title of their FAST talk sums it up: “Trading More Storage for Less Computation”, exactly the rule from Part 2.

NVIDIA Dynamo and KVBM

NVIDIA Dynamo is the open source distributed serving framework with prefill/decode disaggregation, dynamic GPU scheduling and KV-aware routing: the PrefillRouter computes overlap scores between the incoming request and the KV blocks already cached, and routes to the GPUs that already have the relevant KV, avoiding recomputation (Dynamo · Disaggregated Serving, Dynamo · KV-aware routing). KV transfer between prefill and decode is handled by NIXL directly GPU-to-GPU over the best available transport (NVLink, InfiniBand). And its offload manager, KVBM (KV Block Manager), has a three-layer architecture (LLM runtime, logical block management, NIXL transport) that offloads cold KV to CPU RAM, NVMe or network storage to free up HBM (Dynamo · KV Cache Offloading). This connects with disaggregated serving: prefill and decode in specialised pods.

The llm-d project walks the same road, from vLLM’s prefix caching to KV-aware distributed scheduling (llm-d · KV-Cache Wins You Can See).

Part 5 · Reference architecture

On the blog’s example cluster, a node with 4×H100 SXM (80 GB, NVLink), local NVMe and plenty of DRAM, a sensible stack to serve long context without recomputing pointlessly:

KV in FP8 in the engine (vLLM), to start with half the bytes.
LMCache as the KV layer with a DRAM→NVMe hierarchy, so that long prefixes (a manual, a code base, a case file) are cached once and reused across sessions without recomputation.
KV-aware routing (Dynamo or llm-d) if you have several nodes: send the request to the GPU that already holds the prefix, not to any GPU at all.
Prefill/decode disaggregation once the prefill of enormous contexts starts stealing decode from the other requests. Prefilling 300k tokens monopolises the GPU and kills everyone’s latency.
Metrics for KV hit ratio, bytes transferred per level and prefill/decode ratio, exported to your observability stack (instrumenting vLLM with OTel). Without those three metrics, offloading is faith, not engineering.

To prototype the stack, validating the LMCache config, the backend formats, the hit/miss behaviour, an RTX 5090 (Blackwell, 32 GB) with NVMe is more than enough to serve a 7–14B and watch offloading work at contexts of 32–128k. Do not expect to serve 1M on a consumer card: the long-context problem is precisely that neither the KV nor the attention fit on small hardware. Real sizing starts from the SLO and from the context-length distribution of your traffic, not from the model’s maximum window, and that is where it links to capacity planning.

Part 6 · Operational pitfalls (and honest scepticism)

Believing that native prefix caching is enough. It only lives in HBM; in long context it is evicted and you recompute. You need a persistent layer (LMCache/KVBM) for the cache to survive.
Offloading without measuring the break-even. If your prefixes are not reused, offloading only adds transfer latency without saving recomputation. Offloading shines with shared prefixes (RAG over the same corpus, agents with the same system prompt, long sessions), not with one-off, never-repeated requests.
Forgetting that attention is still quadratic. Offloading fixes the memory of the KV, not the compute of attention. For real 1M tokens you also need efficient attention, sparse attention or retrieval approaches (RetrievalAttention, ShadowKV); offloading on its own does not bring down the 1.765 s/token.
Wiring the offload badly. HBM↔DRAM↔NVMe crosses PCIe and NUMA. An offload that crosses the wrong NUMA node or saturates a PCIe lane can be slower than recomputing. Measure the link’s real bandwidth, not the datasheet figure.
Treating the figures as constants. The 93 GB, the +525 %, the 3–10× all depend on model, length, hardware and reuse pattern. They are orders of magnitude for design; measure on your own load.
Trusting million-token model specs without verifying. The landscape of open 1M models is very recent (2026) and moves fast; the specific names and limits change between versions. Verify the window and the cost of serving it per model, do not take the headline on trust.

A note of caution for June 2026: the distributed KV layer (LMCache, Mooncake, KVBM, NIXL, llm-d) is setting right now, with APIs and backends that change release by release. It is exactly the kind of piece worth assembling behind an abstraction (vLLM’s kv_connector) rather than coupling to a single vendor.

Closing

Long context is a problem of memory and compute at the same time, and the KV cache is the bottleneck of both. The window is a gift from the model; serving it cheaply is systems engineering: quantise the KV (FP8), offload it by tiers (HBM→DRAM→NVMe→network) with a persistent layer, and reorganise serving around the KV (aware routing, separated prefill/decode) when scale demands it. The rule that governs it all is Mooncake’s: trade storage for computation. You store more KV in cheap places so you do not recompute expensive attention. Do it well and a 4×H100 node serves contexts that, by recomputing, would not even get started. Do it badly, or not at all, and every long request reads the whole case file from scratch again, while the GPU drowns in a quadratic attention you had already paid for once.

Sources

LMCache · GitHub — https://github.com/LMCache/LMCache
LMCache: An Efficient KV Cache Layer (arXiv 2510.09665) — https://arxiv.org/pdf/2510.09665
LMCache · Offload KV cache to CPU (docs) — https://docs.lmcache.ai/getting_started/quickstart/offload_kv_cache.html
vLLM Prefix Caching vs. LMCache: Benchmarking KV Reuse Tradeoffs — https://levelup.gitconnected.com/vllm-prefix-caching-vs-lmcache-benchmarking-kv-reuse-tradeoffs-944fbaf98b56
Mooncake: A KVCache-centric Disaggregated Architecture (arXiv 2407.00079) — https://arxiv.org/abs/2407.00079
Mooncake · USENIX FAST'25 (Trading More Storage for Less Computation) — https://www.usenix.org/conference/fast25/presentation/qin
Mooncake · GitHub — https://github.com/kvcache-ai/Mooncake/
NVIDIA Dynamo · Disaggregated Serving (docs) — https://docs.dynamo.nvidia.com/dynamo/design-docs/disaggregated-serving
NVIDIA Dynamo · KV-aware routing (docs) — https://docs.nvidia.com/dynamo/latest/user-guides/kv-cache-aware-routing
NVIDIA Dynamo · KV Cache Offloading / KVBM (docs) — https://docs.nvidia.com/dynamo/backends/v-llm/kv-cache-offloading
llm-d · KV-Cache Wins You Can See — https://llm-d.ai/blog/kvcache-wins-you-can-see
Long-Context Inference at Scale: The Hidden Infrastructure Cost · DigitalOcean — https://www.digitalocean.com/community/tutorials/long-context-inference-production-cost
Long-Context LLM Infrastructure · Introl — https://introl.com/blog/long-context-llm-infrastructure-million-token-windows-guide
YaRN / LongRoPE (arXiv 2402.13753) — https://arxiv.org/html/2402.13753v1

The LLM inference router: the L7 switchboard we called a LoadBalancer in the canary post

Tue, 02 Jun 2026 03:00:00 +0200

This post is the natural continuation of Canary, blue-green and shadow for LLM models. There the promotion mechanics dumped all the traffic-splitting complexity into a box we called “LoadBalancer”. The description was operational, good enough to understand the choreography, but structurally vague: what actually does that splitting is an L7 inference router with LLM awareness, a piece of the stack in its own right (layer 1 of the seven layers) that deserves its own post.

TL;DR

In the previous post on canary we called LoadBalancer the piece that splits traffic between the stable v1 and candidate v2 pools. The description was enough to understand the flow, but technically it was blurry: neither an L4 LoadBalancer (kube-proxy, MetalLB, IPVS) nor a generic L7 HTTP LoadBalancer (NGINX or HAProxy with no extension) knows what a model is, what a version is, how many tokens a request costs, what prefix the prompt has or which replica has which KV cache hot. The right piece is an LLM inference router: an L7 proxy with explicit knowledge of the domain. It combines four functions: model catalogue (resolving model=llama-70b@v2 → service.namespace:port), traffic splitting (applying the canary weight with a deterministic hash or deliberate stickiness for A/B), cross-cutting policy (OIDC auth, per-tenant rate limit and quota, pre-prompt PII redaction, lightweight inline guardrails, gen_ai.* tracing propagation) and failover/degradation (if v2 falls over, redirect to v1; if the whole cluster is saturated, return 503 with Retry-After instead of queueing forever). The non-obvious piece that justifies its technical existence beyond the operational one is prefix-aware routing: the router decides which fleet replica each request goes to based on the prompt prefix, so that a RAG system with the same system prompt plus the same block of retrieved documents systematically hits the prefix cache (RadixAttention in SGLang, PrefixCaching in vLLM, KV reuse in TensorRT-LLM) of the same replica, multiplying the hit rate from 5–15 % (blind round-robin) to 60–85 % (prefix affinity). The concrete pieces in May 2026 are LiteLLM Proxy (the simplest option, OpenAI-compatible, declarative YAML catalogue), vLLM Production Stack router (specific to vLLM fleets, aware of the KV cache and the prefix), Envoy AI Gateway (LLM-aware Envoy filters, integrable with Istio), Kong AI Gateway (enterprise alternative with a plugin ecosystem), KGateway (CNCF, still incubating) and the NVIDIA Dynamo router (production-grade, aware of disaggregated prefill/decode serving). In the seven-layer stack it lives in layer 1 (gateway); in the five maturity levels it appears from level 3 onwards; in the seven-phase deployment cycle it is the last piece that F6 closes. This post includes a minimal manifest applicable to a generic 4×H100 SXM cluster.

You are here: DEPLOY (layer 1 of the stack)

The background: what the canary post called a “LoadBalancer”

In Canary, blue-green and shadow for LLM models we described the flow like this: “the LoadBalancer progressively splits traffic following a schedule: 1 % → 5 % → 25 % → 100 %”. It was a correct operational description, and the reader understood the choreography without needing more. But technically it left unnamed a piece that deserves explicit treatment, because neither of the two usual senses of “LoadBalancer” does what that paragraph assumed:

An L4 LoadBalancer (kube-proxy with iptables/IPVS, MetalLB, F5 BIG-IP in TCP mode) splits IP packets without looking inside the payload. It does not know which model is being requested, nor which version, nor how many tokens it carries, nor whether the client has quota. It cannot apply the canary weight “for model X version 2”: to it, every packet heading for the vllm-llama70b VIP is indistinguishable.
A generic L7 HTTP LoadBalancer (NGINX or HAProxy in HTTP mode with no extension, a ClusterIP Service with multiple backends) does split by URL and can route by header, but it does not understand the OpenAI-compatible body of the request. It does not know that {"model": "llama-70b", "messages": [...]} carries the routing key in the model field; it does not count tokens; it does not apply policies over LLM structures; it does not do prefix-aware routing because that requires parsing messages and hashing the common prefix.

The piece the canary post assumed was doing this work is an L7 inference router with LLM awareness. A layer in its own right, with its own configuration, its own CI/CD, its own metrics and its own pitfalls. This post names it and takes it apart.

The analogy: the switchboard and triage of a hospital with multiple specialities

A large hospital receives patients who arrive at A&E through different doors and who need different specialities: trauma, cardiology, paediatrics, oncology. There are three possible models of “front door”.

A single door with no triage. All patients wait in the same room and are sent in arrival order to the first free doctor, whatever their speciality. It works in a village surgery with a single general practitioner. When there are 200 patients a day and 12 specialities, it falls into dysfunction fast: the cardiologist treats sprains, the paediatrician treats heart attacks, specialised resources are wasted. This is the equivalent of the L4 LoadBalancer, which distributes bodies without understanding what they bring.

A door with a receptionist who asks about the symptom. Someone at the front desk asks “what’s wrong?” and directs the patient to the right corridor. The cardiologist sees only cardiology, the paediatrician only children. Better, but the receptionist is slow, does not gauge urgency and does not know the state of the rooms: they may send you to the cardiologist in corridor A when the one in B is free. This is the equivalent of a generic L7 HTTP proxy with path-based routing, which distributes by category but with no information about internal state.

Professional triage with full awareness. A trained triage nurse who knows the catalogue of specialities, knows which bay is busy and which is free, remembers the recurring patient whose file is already open in the system (and sends them to the same doctor for continuity), applies cross-cutting policy (checks insurance cover, records allergens, redirects to paediatric A&E if the patient is a minor) and, if the cardiology room goes down because the electrocardiograph has broken, redirects to the hospital on the other side of the city. This is the piece a large hospital needs. In LLM terms it is called an inference router.

The analogy holds down to the last detail, including the “file already open” one: the patient who returns to the same doctor is exactly the client whose prompt shares a prefix with the one from five minutes ago. If the router sends it to the same replica, that replica still has the KV cache hot and the request hits the prefix cache. If it sends it to a different replica because that one was “next in round-robin”, the KV cache has to be rebuilt from scratch and TTFT doubles. The triage nurse knows this. The blind LoadBalancer does not.

The four functions of the inference router

Function 1 — Model catalogue

The router keeps a declarative catalogue mapping model identity to a concrete deployment:

models:
 - name: "llama-70b" # stable alias
 version: "v2" # canary version
 weight: 5 # 5% of the traffic
 endpoint: "vllm-llama70b-v2.inference.svc.cluster.local:8000"
 capabilities: [chat, tool_use]
 lifecycle: canary
 - name: "llama-70b"
 version: "v1"
 weight: 95
 endpoint: "vllm-llama70b-v1.inference.svc.cluster.local:8000"
 capabilities: [chat, tool_use]
 lifecycle: stable
 - name: "embedding-multilingual"
 version: "v1"
 weight: 100
 endpoint: "tei-bge-m3.inference.svc.cluster.local:8080"
 capabilities: [embeddings]
 lifecycle: stable

The client sends {"model": "llama-70b", "messages": [...]} without knowing there are two replica pools behind it. The router resolves. If tomorrow you migrate from vLLM to SGLang for a specific version, the client never notices; you change the endpoint in the catalogue and that is it.

What this decoupling buys you is the freedom to move topology without breaking clients. What it costs is keeping the naming convention disciplined (llama-70b is always the stable alias; llama-70b@v2 is the specific version for canary). Without that discipline, the aliases get dirty with llama-70b-prod-fixed-real-final-v3 and the catalogue stops being navigable within a few weeks.

Function 2 — Traffic splitting

The splits from the canary post (1 % → 5 % → 25 % → 100 %) materialise here, not in the inference engine. The router computes a deterministic hash of the request_id (or of the user_id, if you want stickiness) and maps it to the weight range in the catalogue. For a weight of [v1: 95, v2: 5], 5 % of the hash space falls in v2 and 95 % in v1.

Three design decisions that matter:

Hash by random request_id = independent sampling. Each request is an independent observation of the v1 vs v2 distribution. It is the right setting for statistically comparable canaries.
Hash by user_id = per-user stickiness. The same client always sees the same pool. Useful for A/B testing with persisted conversational memory, but it breaks the statistical comparability of the canary because user populations are not symmetric, a pitfall explained in the previous post.
Hash by tenant_id = hard partitioning. Tenant A goes to v1, tenant B to v2. This is the pattern for clients with different SLAs, or for validating v2 in an internal tenant before exposing it to external clients.

Function 3 — Cross-cutting policy

Once, across all models, the router applies:

Auth: OIDC with JWT tokens validated against Keycloak / Authentik. Authorization: Bearer ... headers translated into tenant_id and roles.
Rate limit: token bucket per tenant (X req/min) or per model (Y req/min for llama-70b because it is expensive).
Quota: monthly quota of tokens consumed per tenant. The router counts gen_ai.usage.input_tokens + gen_ai.usage.output_tokens and rejects with 429 Quota exceeded when it runs out.
Pre-prompt PII redaction: Presidio or Llama Guard inline before the prompt touches the model. What the model does not see is not trained on, not logged, not leaked.
Lightweight inline guardrails: PromptGuard 2, Llama Guard 4, Granite Guardian, the ones covered in Guardrails and safety in LLMs, run in the router because their latency (30–150 ms) fits within the TTFT budget.
gen_ai.* tracing propagation: the router starts the parent span, propagates traceparent to the engine and emits the gen_ai.system, gen_ai.request.model and gen_ai.request.version attributes that OTel GenAI tracing consumes.
Semantic cache: for prompts that repeat exactly or with high semantic similarity (embedding cosine > 0.97 against the previous cache), it returns the cached answer without touching the engine. Typical saving in RAG with frequently asked questions: 20–40 % of requests.

Function 4 — Failover and degradation

The router knows the health state of each endpoint (active health probes every 5–15 s, recent TTFT latency) and decides:

If v2 returns persistent 5xx or does not respond, the circuit breaker opens: the router redirects the traffic that was going to v2 towards v1 until the probes go green again. This is automatic canary rollback in its simplest form.
If the whole cluster is saturated (all replicas reporting num_requests_waiting > N for T seconds), the router returns 503 Service Unavailable with Retry-After: 30 instead of queueing forever. It is better to tell the client “come back in 30 seconds” than to keep them waiting four minutes and then time out.
If there is multi-region or multi-cluster, cross-cluster failover via DNS or L7: the primary region goes down and the secondary router takes over.

The non-obvious piece: prefix-aware routing

This is the function a conventional LoadBalancer cannot do and the one that justifies an LLM-specific router beyond the four generic ones.

The KV cache in vLLM, SGLang and TensorRT-LLM can reuse common prefixes across requests, see KV cache. Specifically:

vLLM with --enable-prefix-caching: it detects that the current request shares a prefix (of a length that is a multiple of the block size, 16 tokens by default) with a previous request whose pages are still in HBM, and reuses those pages instead of reprocessing them.
SGLang with RadixAttention: it structures the cache as a radix tree indexed by tokens; each request hits the common path of the tree and only computes the tail.
TensorRT-LLM: a similar feature, called KV cache reuse.

The prefix cache hit rate is the key metric: every token that hits is a token that is not processed in prefill, reducing TTFT in direct proportion. For a typical RAG system, with a 400-token system prompt plus 2,000 tokens of retrieved documents plus a 50-token user question, the common prefix (system_prompt + docs) is 2,400 of the 2,450 total tokens. If the cache hits, prefill only processes 50 tokens instead of 2,450: TTFT falls to roughly a twentieth.

But the cache lives per replica, not globally. If two requests with the same 2,400-token prefix land on different replicas, both do the full prefill: the first one’s cache is no use to the second. The second pays the full cost.

With blind round-robin (any conventional LB), requests are spread uniformly across N replicas. For a cluster of 4 replicas and 1,000 requests with the same system_prompt + docs, each replica receives ~250 requests, but all 4 do their own “first prefill” and the following 249 benefit within their replica. The global hit rate is decent but not optimal. For traffic with many different system prompts and little intra-prefix repetition, the hit rate hovers around 5–15 %.

With prefix-aware routing, the router computes a hash of the prompt prefix (the first N tokens, or the system_prompt declared in messages[0]) and keeps an affinity table of hash → replica. All requests with the same prefix land on the same replica. The first pays the full prefill; the following 999 hit the cache. Global hit rate: 60–85 %.

The cost of implementing it: the router has to parse the request body (not just the HTTP header), apply a lightweight tokeniser or a byte-based hash, and maintain an LRU/consistent-hash affinity table that rebalances when a replica comes in or out. That is server work, not generic proxy work. vLLM Production Stack router implements it natively. NVIDIA Dynamo does too. LiteLLM has a beta in its enterprise version. Envoy AI Gateway is adding it as an experimental filter.

Correction (September 2026). The previous sentence about LiteLLM is false: prefix-aware routing does not exist in any version, open or paid, verified against the code of 1.102.0. The detail, what LiteLLM does have, and the real state of the field are in Prefix routing: what LiteLLM does not do.

The operational difference for a production RAG system: with prefix-aware routing, the same cluster serves 2–4× more requests without adding GPUs, simply because prefill disappears in most cases.

Token-aware load balancing

The second non-obvious piece. Classic round-robin splits by number of requests; but a 50-token prompt and an 8,000-token one cost radically different amounts (a factor of ~160× in prefill). Splitting equally by count severely unbalances the real load.

Token-aware load balancing adds up the expected prefill tokens (prompt length) and expected decode tokens (the client’s max_tokens) per active replica, and sends the new request to the replica with the lowest accumulated load. It is what both vLLM Production Stack and NVIDIA Dynamo implement as the default strategy when it is enabled.

The metric feeding the calculation is, once again, vllm:num_requests_running and vllm:gpu_cache_usage_perc, see GPU observability for LLM inference, ideally complemented with an estimator of the incoming prompt’s tokens. Mature routers use tiktoken or the model’s real tokeniser to count prompt tokens before choosing a replica.

Comparison of concrete pieces (May 2026)

Piece	LLM awareness	Prefix-aware	Token-aware LB	Multi-model	Semantic cache	Plug & play
LiteLLM Proxy	High	Beta (enterprise)	Yes	Excellent	Yes (Redis)	Very high
vLLM Production Stack router	vLLM-specific	Yes, native	Yes	vLLM only	No (external)	Medium
NVIDIA Dynamo router	High + disagg-aware	Yes	Yes	vLLM/SGLang/TRT-LLM	No (external)	Low
Envoy AI Gateway	Medium (filters)	Experimental	Yes	Yes	Via filter	Medium
Kong AI Gateway	Medium (plugins)	No	Yes	Yes	Yes (plugin)	Medium
KGateway	Medium	Roadmap	Yes	Yes	Roadmap	Low (CNCF incubating)
NGINX + custom Lua	Manual	No	Manual	Manual	No	Low (build it yourself)

LiteLLM Proxy is the default option to start with. OpenAI-compatible, simple YAML, supports the commercial providers plus any self-hosted OpenAI-compatible backend. The OSS version covers the four basic functions and semantic cache; prefix-aware routing and the enterprise version add advanced multi-tenancy.

vLLM Production Stack router is the right option if the fleet is 100 % vLLM. Aware of the KV cache, the prefix and the LoRA loaded per replica. It integrates better with native vLLM metrics.

NVIDIA Dynamo router is the most complete production-grade option, especially if you operate disaggregated serving (separate prefill workers and decode workers). It requires an NVIDIA-aligned stack.

Envoy AI Gateway and Kong AI Gateway are the options if the organisation already has Envoy or Kong as its corporate gateway and wants to extend it with LLM awareness without introducing another new piece.

Minimal manifest: LiteLLM Proxy on a generic cluster

apiVersion: v1
kind: ConfigMap
metadata: { name: litellm-config, namespace: inference }
data:
 config.yaml: |
 model_list:
 - model_name: llama-70b
 litellm_params:
 model: openai/llama-70b
 api_base: http://vllm-llama70b-v1.inference.svc:8000/v1
 weight: 95
 model_info:
 version: v1
 lifecycle: stable
 - model_name: llama-70b
 litellm_params:
 model: openai/llama-70b
 api_base: http://vllm-llama70b-v2.inference.svc:8000/v1
 weight: 5
 model_info:
 version: v2
 lifecycle: canary
 - model_name: embedding-multilingual
 litellm_params:
 model: openai/bge-m3
 api_base: http://tei-bge-m3.inference.svc:8080
 router_settings:
 routing_strategy: least-busy # basic token-aware
 num_retries: 1
 timeout: 60
 general_settings:
 master_key: "os.environ/LITELLM_MASTER_KEY"
 database_url: "os.environ/DATABASE_URL"
 litellm_settings:
 cache: true
 cache_params:
 type: redis
 host: redis.inference.svc
 port: 6379
 similarity_threshold: 0.97
 success_callback: ["langfuse"]
 failure_callback: ["langfuse"]
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: litellm-router, namespace: inference }
spec:
 replicas: 3
 selector: { matchLabels: { app: litellm } }
 template:
 metadata: { labels: { app: litellm } }
 spec:
 containers:
 - name: litellm
 image: ghcr.io/berriai/litellm:v1.55.0
 args: ["--config=/config/config.yaml", "--port=4000", "--num_workers=4"]
 ports: [{ containerPort: 4000, name: http }, { containerPort: 4000, name: metrics }]
 env:
 - { name: LITELLM_MASTER_KEY, valueFrom: { secretKeyRef: { name: litellm-secret, key: master_key } } }
 - { name: DATABASE_URL, valueFrom: { secretKeyRef: { name: litellm-secret, key: db_url } } }
 - { name: LANGFUSE_PUBLIC_KEY, valueFrom: { secretKeyRef: { name: langfuse-keys, key: public } } }
 - { name: LANGFUSE_SECRET_KEY, valueFrom: { secretKeyRef: { name: langfuse-keys, key: secret } } }
 volumeMounts: [{ name: config, mountPath: /config }]
 readinessProbe: { httpGet: { path: /health, port: 4000 } }
 volumes: [{ name: config, configMap: { name: litellm-config } }]
---
apiVersion: v1
kind: Service
metadata: { name: litellm-router, namespace: inference }
spec:
 selector: { app: litellm }
 ports: [{ name: http, port: 80, targetPort: 4000 }]
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata: { name: litellm-metrics, namespace: inference }
spec:
 selector: { matchLabels: { app: litellm } }
 podMetricsEndpoints:
 - port: metrics
 path: /metrics
 interval: 15s

The end client points at litellm-router.inference.svc:80/v1/chat/completions, sets model=llama-70b, and the router decides on each request whether it goes to v1 (95 %) or v2 (5 %), applies the rate limit, looks in the semantic cache, propagates tracing to Langfuse, and translates from OpenAI-compatible to the OpenAI-compatible API of the destination vLLM. Three router replicas for HA and so the gateway itself can scale horizontally with KEDA if needed, see LLM autoscaling on Kubernetes.

Four operational pitfalls

Pitfall 1 — the router becomes a SPoF if it is not replicated. Three or more replicas of the router itself, behind a LoadBalancer Service (this one genuinely L4) with healthchecks. A single router replica means every configuration deploy closes the whole service for a few seconds.

Pitfall 2 — the router’s latency adds to the model’s. Each function adds milliseconds: body parsing (5–10 ms), JWT auth (2–5 ms), rate limit (1–2 ms), PII redaction with Presidio (20–80 ms), inline guardrails with Llama Guard (50–150 ms), prefix hash (5–10 ms), token counting with a tokeniser (10–30 ms). In total, 100–300 ms of overhead before touching the engine. If the model’s TTFT is 400 ms and the router’s is 200 ms, the client sees 600 ms, so it is worth measuring what each function costs and disabling the non-critical ones in the low-latency path.

Pitfall 3 — the catalogue drifts from the real state of the cluster. The router believes vllm-llama70b-v2 exists because it is in its YAML, but the deployment was withdrawn three days ago and nobody updated the config. The router returns 502 on 5 % of the traffic. Fix: validate the catalogue against kubectl get svc in CI; no endpoint in the catalogue may point at a non-existent Service. Or better: the router discovers the available endpoints dynamically via a label selector (app=vllm,model=llama-70b) and applies the catalogue weights over the ones that are alive.

Pitfall 4 — semantic cache with an outdated embedding. The semantic cache compares the new prompt’s embedding against the embeddings of cached prompts. If you update the embedding model (see RAG corpus curation), the distances are computed in a different space and the cache stops working correctly (false hits or false misses). Policy: the cache is invalidated when the embedding model changes; generations are never mixed.

Fit within the stack and maturity

In the seven-layer stack, the router is layer 1: the front door that precedes the inference engine (layer 2), the KV cache + PagedAttention (layer 3) and the rest. It is the only piece that sees all the traffic from the outside; any policy not applied here gets duplicated N times in the engines.

In the five maturity levels, the router appears from level 3 (MANAGED) onwards: without OIDC + RBAC + cert-manager + default-deny NetworkPolicy, the router has nobody to authenticate and nobody to apply quotas to; before level 3, what you should build is a minimal proxy with no pretension of a catalogue. Platforms that try to have a polished router at level 1 end up with a large YAML nobody maintains.

In the seven deployment phases, the router is what closes F6: the last atomic step that puts the cluster into production. Without a router, F6 does not finish, because the catalogue, the quotas, the canaries and the failovers are a necessary condition for opening production traffic.

Applied to typical on-premise hardware

For a generic cluster of 4 nodes × 4×H100 SXM 80 GB, the inference router consumes modest resources: 3 replicas of the router pod (2 CPU cores, 4 GiB of memory each) are enough to sustain thousands of RPS because its work is light (parsing, hashing, routing, no inference). The router lives on non-GPU nodes of the cluster (control plane nodes or general workload nodes), and never consumes nvidia.com/gpu.

Traffic volume that a LiteLLM with 3 replicas and 4 workers each will sustain: 2,000–5,000 RPS routing to a vLLM backend, with 80–150 ms of overhead on the full path (auth + rate limit + cache check + propagation). If more is needed, scaling the router with KEDA on litellm:requests_per_second is trivial.

For larger clusters (16+ GPU nodes), consider the vLLM Production Stack router or the NVIDIA Dynamo router, which are more complex but squeeze out the prefix-aware routing and token-aware LB that LiteLLM OSS does not cover. For multi-region clusters, Envoy AI Gateway with Istio Service Mesh is the standard choice.

What we have not covered (upcoming articles)

Deep comparison of LiteLLM vs vLLM PStack vs Dynamo, with prefix-aware benchmarks on a real on-premise cluster.
Semantic cache with Redis Stack + RedisVL: hit rate, false positives, TTL policy.
Multi-region routing: how the router decides between the DC1 and DC2 clusters based on latency, health and load.
AI Gateway specific features: token-bucket cost-based rate limiting (penalising long prompts), guardrails policy engine in the router.
Migration path: how to introduce a router into a cluster that already has clients pointing straight at the vLLM service, with no downtime.

References

LiteLLM project — litellm.ai (documentación de Proxy, routing strategies, semantic cache).
vLLM Production Stack — github.com/vllm-project/production-stack (router con prefix-aware nativo).
NVIDIA Dynamo — developer.nvidia.com/blog/nvidia-dynamo-1-production-ready/ (router production-grade con disaggregated-aware).
Envoy AI Gateway — gateway.envoyproxy.io/docs/tasks/ai-gateway/ (proyecto en gestación dentro de Envoy).
Kong AI Gateway — konghq.com/products/kong-ai-gateway (proxy enterprise con plugin LLM).
KGateway — kgateway.dev (alternativa CNCF en gestación).
Zheng et al. — SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS 2024) — RadixAttention y prefix caching.
vLLM project — Automatic Prefix Caching (docs.vllm.ai/en/latest/features/automatic_prefix_caching.html).
Patel et al. — SplitWise: Efficient Generative LLM Inference Using Phase Splitting (ISCA 2024) — base teórica del routing prefill/decode aware.

LLM inference operators on Kubernetes: OME, vLLM Production Stack, NVIDIA Dynamo and llm-d

Mon, 18 May 2026 17:00:00 +0200

TL;DR

Serving an LLM in production is not running a binary: it is coordinating a model (tens of gigabytes that take minutes to load), a runtime (vLLM, SGLang, TensorRT-LLM with a hundred flags), heterogeneous GPUs (NVLink, MIG, PCIe), prefill and decode that live better apart, a KV cache that wants offloading to colder tiers, intelligent routing that exploits prefix caching, and autoscaling that reacts to metrics other than CPU%. A flat Kubernetes Deployment only covers the first 20% of that. The other 80% is covered by LLM inference operators, of which four matter in 2026: OME (LMSYS, July 2025, multi-engine with a focus on SGLang), vLLM Production Stack (a curated Helm chart from vLLM itself with LMCache for tiered KV), NVIDIA Dynamo (the official successor to Triton, multi-engine, with its own Grove scheduler) and llm-d (a CNCF donation of March 2026 by Red Hat + Google + IBM + CoreWeave + NVIDIA, built on vLLM, focused on distributed scale). Behind all four sits KServe, the CNCF parent operator that normalised the InferenceService concept and that several of them lean on. This article walks through the full hierarchy, gives a decision map and teaches you not to get lost when someone drops seven acronyms in the first meeting.

This article closes the LLM inference series. The previous ones were KV cache: the working memory that holds up LLM inference, vLLM on Kubernetes: the LLM inference piece that actually scales and PagedAttention from the inside and the state of the art of the KV cache in 2026. There we explained what happens inside one inference process. Here we explain how many inference processes are coordinated through Kubernetes.

The analogy: from `init.d` to systemd to operators

Anyone with 20 years of sysadmin behind them will recognise the pattern. Decades ago, starting a service on Linux was a shell script in /etc/init.d/: start, stop, status, reloaded by hand. When services got more complex, with dependencies between them, monitoring, restart on failure, per-user slots, it became obvious that a script was not enough. Along came systemd, which turned “a service” into a declarative unit with dependencies, resources, restart policy, sockets and timers. The script did not disappear; it moved up one level of abstraction.

Kubernetes made the same move for distributed services. A Deployment declares “I want N replicas of this container”; a Service declares “these replicas are exposed like this”; an Ingress declares “this HTTP traffic comes in here”. The controller translates the declaration into real state and keeps the system converged.

Serving LLMs in 2024 was the equivalent of /etc/init.d/: every team wrote its own Deployment/Service/HPA with custom scripts for model loading, session draining and GPU handling. We covered it in the vLLM on Kubernetes article: it can be done, and it does work, but it is repetitive, fragile and nobody is extracting the right abstractions. Serving LLMs in 2026 has gone through the same transition as services: the equivalent of systemd has appeared, the inference operators, which normalise the abstractions and let the engineer declare what matters: “this model, with this runtime, this scalable, with this routing policy”.

There are four relevant operators in 2026 and a fifth common ancestor. Let us take them in order.

Why an operator, and not just a Deployment

Listing what an inference operator adds over a flat Deployment is the best way to understand what problem it solves:

The model as a first-class citizen. In a Deployment, the model is “the thing you download in an initContainer and mount as a volume”. In an operator, the model is a CustomResource with metadata (origin, fingerprint, licence, GPU requirements). It can be shared between InferenceServices, versioned, replicated to multiple nodes. It is the difference between “a file” and “a managed artifact”.

The runtime as a first-class citizen. The same goes for the runtime (vLLM/SGLang/TRT-LLM): it is not “a Docker image with flags”; it is a ServingRuntime that declares which args it accepts, which metrics it exports, which deployment types it supports (single-node, multi-node TP, PD-disag). Changing runtime is changing a reference, not rewriting every manifest.

Declarative composition. An InferenceService (the core CRD of KServe and its descendants) references a model and a runtime, declares the scaling policy, wires up observability and configures routing. The controller composes all the pieces: Deployment(s), Service, HPA, possibly a LeaderWorkerSet, a KEDA ScaledObject, a Gateway API HTTPRoute. You declare intent; the operator emits the 8 derived resources.

Operational prefill–decode disaggregation. As we saw in the PagedAttention article, separating prefill and decode into distinct pools can give 7× goodput. Modelling that with flat Deployments is feasible, but it requires coordinating two sets of pods, a transport to move the KV cache and conditional routing. An operator models it as a single InferenceService with two sub-pools.

Autoscaling on LLM metrics. The standard HPA does not understand vllm:num_requests_waiting. An operator integrates KEDA or Prometheus Adapter automatically and exposes the right metrics as knobs on the CRD.

Multi-tenancy. Multiple models on the same cluster, with quotas, priorities and fairness. One Deployment per model scaling independently is fine up to the fifth model; beyond that, coordinating GPUs between tenants becomes operationally hostile.

Model lifecycle. Weights on a shared PVC, warm-up of the first pod, rolling updates with maxUnavailable: 0, draining of active sessions, integrated observability. Things that with a flat Deployment have to be reinvented by every team.

If your workload is one model, one node, up to three replicas, a flat Deployment is enough and an operator is overkill. If your workload is two or more models, serious scaling, disaggregation or multi-tenancy, an operator stops being optional.

KServe: the common ancestor

Before the four new ones, we have to mention KServe, the parent operator from which all the rest conceptually descend. It was born as KFServing inside the Kubeflow project in 2019, was renamed KServe when it became independent in 2021, and in 2025 was accepted into the CNCF as a project incubating towards graduation.

KServe’s conceptual contribution is the InferenceService CRD, which has become the field’s common vocabulary: a declarative K8s object that joins a model (origin + metadata) with a predictor (runtime + resources) and produces a ready HTTP service. Under the hood, the controller emits Deployments, Services, HorizontalPodAutoscalers, Knative Services if you go serverless, Istio VirtualServices if you do traffic splitting.

KServe was designed in a pre-LLM era: its first use cases were traditional scikit-learn, TensorFlow and PyTorch models served as simple REST APIs. That gives it strengths (it is mature, with 6 years in production at Bloomberg, JPMorgan and others) and weaknesses (it was not designed to manage multi-node tensor parallel, prefill–decode disaggregation, or the patterns specific to LLMs).

The way the ecosystem has reacted is elegant: the new LLM operators inherit from or take inspiration from InferenceService but extend the API with LLM-specific primitives. OME is the clearest example: it uses the name InferenceService and the idea of “model + runtime → service”, but adds BaseModel, a ServingRuntime with LLM-aware flags, and deployment modes (PD-disag, multi-node) that KServe does not cover natively.

OME (Open Model Engine)

OME was published by the LMSYS team in July 2025 (announced on their blog). It is an operator that understands SGLang deeply (that is its first-class runtime) but also supports vLLM, TensorRT-LLM and Triton.

The CRD hierarchy

OME models the domain with four main CRDs:

BaseModel and ClusterBaseModel: the model itself. It defines origin (Hugging Face, S3, URL), fingerprint and metadata. The Cluster* version is global; BaseModel is namespaced. It lets multiple InferenceService objects reference the same model without duplicating the download.
FineTunedWeight: LoRA adapters or finetuned weights served on top of a BaseModel. Critical for multi-tenant setups where each customer has their own finetune.
ServingRuntime and ClusterServingRuntime: the runtime (vLLM, SGLang, and so on) with its configuration. It declares which args it accepts, which metrics it exports, which deployment modes it supports.
InferenceService: the central declarative piece, joining BaseModel + ServingRuntime + infrastructure.

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
 name: llama3-70b-prod
 namespace: inference
spec:
 model:
 name: meta-llama-3-70b-instruct  # reference to a BaseModel
 runtime:
 name: sglang-h100  # reference to a ServingRuntime
 deploymentMode: PrefillDecodeDisaggregated  # standard | PD | MultiNode | Serverless
 prefill:
 minReplicas: 2
 maxReplicas: 8
 resources:
 requests:
 nvidia.com/gpu: 4
 decode:
 minReplicas: 4
 maxReplicas: 16
 resources:
 requests:
 nvidia.com/gpu: 1
 router:
 type: cache-aware  # SGLang router with cache awareness
 autoscaling:
 metricSource: keda
 metrics:
 - type: prometheus
 metricName: vllm_requests_waiting
 threshold: "10"

That is what the operator takes as input. The output is roughly 8 derived resources that would be horrible to declare by hand: two LeaderWorkerSets (one per prefill/decode pool), two Services, a Deployment for the router, KEDA ScaledObjects for each pool, a Gateway API HTTPRoute, and a PriorityClass that hooks into Kueue for gang scheduling.

The four deployment modes

OME materialises the InferenceService differently depending on deploymentMode:

Standard: one Deployment with N replicas; the classic. For small or single-GPU models.
PrefillDecodeDisaggregated: two coordinated pools; the SGLang router routes between them.
MultiNode: tensor parallel across multiple nodes via LeaderWorkerSet, with NCCL/InfiniBand. For models above 70B where a single node is not enough.
Serverless: Knative-style scale-to-zero. For sporadic workloads where the cost of keeping GPUs powered up does not pay off. The trade-off: the first request pays the model’s cold start cost (minutes).

Integration with the K8s ecosystem

OME does not invent primitives where they already exist. It leans on:

Kueue for gang scheduling: all the pods of a tensor parallel group have to start at once or not at all; Kueue guarantees it.
LeaderWorkerSet (LWS) for multi-node: workers join the leader’s Ray cluster, with an atomic lifecycle (one going down restarts the group).
KEDA for autoscaling on LLM-specific Prometheus metrics (queue depth, GPU cache usage, TTFT p95).
Gateway API and its Inference Extension for advanced routing (model-aware, prefix-aware, weighted canary).

The consequence: OME feels “idiomatically Kubernetes”. It does not introduce new concepts where they are not needed; it uses standard primitives and concentrates on what is specific to the LLM domain.

When to choose it

OME is the natural choice if SGLang is your main runtime and/or if you come from the KServe ecosystem and want an idiomatic evolution of it. It is mature but relatively young (a year old at the time of writing); expect rough edges on advanced features.

vLLM Production Stack

vLLM Production Stack is the official project from vLLM itself for production on Kubernetes. Its philosophy is the opposite of OME’s: instead of an operator with new CRDs, it is a curated Helm chart that deploys a coherent set of pieces.

The three pieces

The stack has three components:

Serving engines: vLLM pods, configured with the flags we have been seeing throughout the series (--enable-prefix-caching, --kv-cache-dtype fp8, and so on). The Helm chart lets you declare them as a list; it deploys the underlying Deployments and Services.
Request router: a proxy in front of the engines that decides which one each request goes to. It supports several policies:
- Round-robin: trivial, for a baseline.
- Session-based: pins each session to one replica to keep its KV cache.
- Prefix-aware: detects shared prefixes between requests and routes them to the replica that already has them cached.
- KV-aware: sees each replica’s gpu_cache_usage_perc and avoids the saturated ones.
- Disaggregated-prefill with native LMCache: separates prefill and decode, with LMCache as the transport for the KV cache between the two.
Observability stack: Prometheus + Grafana with ready-made dashboards. It measures TTFT, TBT (Time-Between-Tokens), throughput, queue depth and GPU memory.

LMCache and tiered KV

One of the most interesting pieces the stack brings is LMCache, which adds a KV cache with multiple tiers: GPU HBM as L1, CPU RAM as L2, local disk as L3, and optionally remote storage as L4. When a KV cache block does not fit in HBM, instead of evicting and recomputing it, LMCache drops it to a lower tier. For workloads with shared prefixes and multi-turn conversations, the saving is huge.

LMCache integrates as a sidecar to the engines and as part of the transport in disaggregated-prefill. Production Stack ships it enabled by default in its Helm chart.

A typical manifest (values.yaml)

servingEngineSpec:
 modelSpec:
 - name: llama3-8b
 repository: vllm/vllm-openai
 tag: v0.6.3
 modelURL: meta-llama/Meta-Llama-3-8B-Instruct
 replicaCount: 3
 requestCPU: 4
 requestMemory: 16Gi
 requestGPU: 1
 vllmConfig:
 enablePrefixCaching: true
 kvCacheDtype: fp8
 maxModelLen: 32768
 enableChunkedPrefill: true

routerSpec:
 routingLogic: prefix-aware  # round-robin | session | prefix-aware | kv-aware
 sessionKey: x-user-id  # when routingLogic=session

cacheserverSpec:
 enabled: true # LMCache for tiered KV
 storageBackends:
 - cpu
 - disk  # offload to local disk

observabilitySpec:
 prometheus:
 enabled: true
 grafana:
 enabled: true
 dashboards:
 - vllm-engine-metrics
 - lmcache-metrics

This is declarative but these are not CRDs: they are Helm chart values. The difference with OME is not semantic (both start from declaration) but operational: with Helm, changes go through helm upgrade; with CRDs, they go through kubectl apply. For teams already living in GitOps with Argo CD or Flux, both approaches integrate cleanly, but the flows are different.

When to choose it

If vLLM is your only runtime and you want the closest thing to “the happy path the project recommends”, this one. It is the productionised version maintained by the same people who write the engine. The downsides: it ties you to vLLM (it is not generic) and it does not solve some advanced cases such as multi-tenancy with strict quotas or gang scheduling, where OME or full-fledged operators are better.

NVIDIA Dynamo

NVIDIA Dynamo is the official successor to Triton Inference Server, announced at GTC 2025 and merged into the brand as Dynamo-Triton in March of that year. Triton had spent years as the most-used inference engine in “serious” NVIDIA infrastructures; Dynamo is what NVIDIA believes the new generation needs.

What it is exactly

Dynamo is a distributed inference framework, not exactly a Kubernetes operator. It has its own runtime (it can run engines), a scheduler (Grove), intelligent routing, multi-tier KV cache management and disaggregation. It supports SGLang, TensorRT-LLM and vLLM as engines, but the engines are executed by Dynamo, not the other way round: the model is “Dynamo manages, the engine executes”.

On Kubernetes, Dynamo is deployed via its own operator plus CRDs, normalised with the K8s integration NVIDIA formalised at the end of 2025 (covered by this InfoQ piece). The CRDs are product-specific: they define a DynamoCluster, a topology of prefill/decode workers and a routing policy.

The four contributions

Dynamo is sold on four pillars, with figures reported by NVIDIA:

Disaggregated serving built in, with its own scheduler.
Smart routing based on cache state: if a worker already has most of a prompt cached, the request goes there.
Multi-tier KV cache: analogous to LMCache, with HBM/RAM/SSD/NVMe.
Autoscaling integrated with Dynamo’s scheduler.

The marketing figure: up to 30× more throughput than legacy Triton on the same hardware. With all the caution a vendor benchmark deserves.

Grove: its own scheduler

A controversial decision in Dynamo is not to rely 100% on the Kubernetes scheduler and instead bring its own scheduler, called Grove, that understands GPU topologies. Grove decides which worker runs on which physical GPU, which interconnects (NVLink/InfiniBand) are relevant, and how to distribute tensor parallel across nodes. That gives it more control than the standard kube-scheduler.

Operationally: if your cluster is “pure Kubernetes” with kube-scheduler and heterogeneous workloads (not only LLMs), Grove adds one more component to operate. If your cluster is dedicated to LLM inference and there is already a team dedicated to running it, Grove gives you more levers.

When to choose it

Dynamo makes sense if:

Your infrastructure is NVIDIA-heavy (Hopper, Blackwell, GB200) and you want to exploit the latest in TensorRT-LLM with the familiar Triton integration, modernised.
You were already a Triton user for legacy inference (vision, recommendation) and want to keep the ecosystem.
You have a dedicated SRE team for inference and Grove’s extra operational complexity is not a problem.

It is the vendor-specific option of the quartet. In exchange it gives you NVIDIA’s support and first-rate integration with their hardware. If your organisation is already fighting NVIDIA for GPUs, they may even call you offering help with Dynamo.

llm-d

llm-d is the youngest and the most “political” of the four. In March 2026, at KubeCon Europe Amsterdam, Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA announced the joint donation of the project to the CNCF as a Sandbox project, with backing from AMD, Cisco, Hugging Face, Intel, Lambda, Mistral AI, UC Berkeley and the University of Chicago. An explicit coalition for vendor neutrality.

Philosophy

llm-d positions itself as the vendor-neutral “Kubernetes blueprint” for distributed inference. It is not a runtime; it is a system that sits on top of vLLM (the default engine) and orchestrates the control plane.

The primitives the project puts on the table:

Intelligent routing with prefix-cache awareness and load-aware balancing.
Tiered KV cache with offload to CPU and disk for multi-turn.
Prefill/decode disaggregation over fast interconnects.
Wide expert parallelism for serving very large Mixture-of-Experts (MoE) models, a critical pattern popularised by DeepSeek-V3 and Mixtral, where the experts live on different GPUs and tokens have to be routed to the right expert.

Numbers

The v0.5 release validates ~3.1k tok/s per decode GPU on B200, and up to 50k output tok/s on a 16×16 B200 prefill/decode topology. The most interesting benchmark: an order-of-magnitude reduction in TTFT against a round-robin baseline. In other words, intelligent routing is worth what they say it is.

CNCF and the future

Donating to the CNCF as a Sandbox project means neutral governance: no vendor is in charge. For an organisation wary of being tied to a single supplier, llm-d is probably the safest medium-term bet. The price: like any Sandbox project, it is not yet “boring” in the way vLLM is. There is API churn, features that move around, documentation that lags the code.

When to choose it

llm-d makes sense if:

You want multi-vendor portability without ties to NVIDIA, Red Hat or Google.
Your workload includes large MoE models (DeepSeek-V3, Mixtral 8x22B, Llama 4 Behemoth if the size is confirmed), where wide expert parallelism is decisive.
Your organisation is already comfortable with CNCF Sandbox (projects under active evolution, not yet a stable 1.0).
You want to bet on the project that will probably be the de facto standard in 2-3 years.

The common ancestor is still there: KServe

It is worth reconnecting before the comparison: KServe is still alive and heavily used in organisations that serve both LLMs and traditional models (scikit-learn, XGBoost, PyTorch CV). Its InferenceService is generic enough to serve any model, including vLLM or SGLang as a ServingRuntime. What it does not do well is the LLM-specific part: disaggregation, multi-node tensor parallel, routing with KV cache awareness. If your organisation already has KServe in production for other models, adding an LLM-specific operator alongside it (OME, vLLM Stack or llm-d) is reasonable. Fighting it all out from pure KServe is not.

Decision map

Dimension	OME	vLLM Prod Stack	NVIDIA Dynamo	llm-d
Philosophy	Classic K8s-idiomatic operator	Curated Helm chart	Framework with its own scheduler	Vendor-neutral CNCF blueprint
Own CRDs	Yes (BaseModel, ServingRuntime, InferenceService…)	No (Helm values)	Yes (DynamoCluster)	Yes (KServe-derived + extensions)
Primary runtime	SGLang (first class), also vLLM/TRT-LLM/Triton	vLLM only	TensorRT-LLM (first class), also SGLang/vLLM	vLLM (first class)
PD-disaggregation	Yes, declarative	Yes, with LMCache	Yes, own scheduler	Yes, native
Multi-node TP	Yes, via LWS	Limited	Yes, via Grove	Yes, via LWS and MoE EP
Multi-model in a cluster	Yes, mature multi-tenant	Yes (list of models in values)	Yes	Yes
Multi-LoRA	Yes, first class (FineTunedWeight CRD)	Limited	Yes	On the roadmap
Tiered KV cache	Via LMCache (external integration)	Native LMCache	Own multi-tier	Yes, native
Intelligent routing	Cache-aware via the SGLang router	Prefix-aware / KV-aware / session-based	Own smart routing	Prefix-cache + load-aware
GPU scheduler	kube-scheduler + Kueue	kube-scheduler	Grove (its own)	kube-scheduler + Kueue
Hardware	NVIDIA, AMD ROCm, Intel	NVIDIA, AMD ROCm	NVIDIA only (with emphasis)	NVIDIA, AMD, Intel — neutral
Maturity (mid-2026)	Young, evolving	Stable	Stable, vendor-driven	CNCF Sandbox, evolving fast
Governance	LMSYS (academic-industrial)	vLLM project (academic)	NVIDIA (vendor)	CNCF (neutral)
Learning curve	Medium (4 new CRDs)	Low (familiar Helm values)	Medium-high (Grove + own CRDs)	Medium (similar to extended KServe)

When to choose each one

Choose OME if:

SGLang is your main engine.
You need multi-LoRA serving in production.
The hierarchical abstraction (BaseModel → ServingRuntime → InferenceService) fits you and you come from, or live alongside, KServe.
You have an appetite for a young and very active project.

Choose vLLM Production Stack if:

vLLM is your only engine and you want to align with what the project recommends.
Your team already lives in Helm and does not want to learn new CRDs.
LMCache + advanced routing inside a single Helm chart is exactly what you need.
Your scale is medium (tens of replicas), not extreme.

Choose NVIDIA Dynamo if:

Your infrastructure is NVIDIA-heavy and you want the most optimised path for Hopper/Blackwell.
You already ran Triton for legacy inference and the transition is natural.
You accept vendor lock-in in exchange for direct NVIDIA support.
Your organisation has a dedicated SRE team for inference.

Choose llm-d if:

You want to bet on the future CNCF standard, neutral between vendors.
Your workload includes large MoE models with wide expert parallelism.
You operate multi-cloud or multi-hardware and portability is valuable.
You accept the immaturity of a Sandbox project in exchange for the bet on the future.

Choose pure KServe if:

You already serve non-LLM models and want to unify; LLMs are a minority of your workload.
You need the most conservative and mature use case.
You accept that advanced LLM features (disaggregation, MoE EP, smart routing) are yours to add with external pieces.

Concrete scenarios

Scenario A — Small startup, 1-2 models, 1-3 GPU nodes. You probably do not need an operator. Deployment + Service + HPA with KEDA metrics, as in the vLLM on Kubernetes article. When you grow to 5+ models, reassess.

Scenario B — Mid-sized company, 5-15 models, internal multi-tenant. vLLM Production Stack or OME are the reasonable options. Production Stack if vLLM is all you are going to use; OME if you want runtime flexibility and idiomatic CRDs.

Scenario C — Internal corporate platform or an external service for end customers. llm-d or Dynamo. llm-d if you value vendor neutrality; Dynamo if you live on NVIDIA infrastructure and want the path they recommend.

Scenario D — Mixed cluster of LLMs plus traditional models. KServe as the base, with an LLM operator alongside (OME is the most natural given its conceptual kinship).

Common traps

“I’ll start with pure KServe because it’s mature”. For mid-sized LLMs and up, pure KServe leaves a lot of optimisation on the table. The sensible route is KServe as the base if you live alongside other models, but with an LLM-specific operator next to it.

“I’ll wire everything by hand to understand it”. Reasonable in a PoC, suicidal in production. There are 8 derived resources per model. Multiply by 10 models. You are writing 80 YAMLs and maintaining them. Use an operator.

“I’ll pick the one I like best and pivot if I’m wrong”. Pivoting between operators is not free: although the InferenceService abstraction is converging, the details (how LoRA is modelled, how routing is configured, how metrics are exposed) vary. Migrating from OME to Dynamo is a project of weeks, not days.

“I’ll go with Dynamo because it’s NVIDIA’s and therefore better”. Only if your organisation is already aligned with its operational philosophy (its own scheduler, acceptable vendor lock-in). For many cases, vLLM Production Stack or llm-d give 95% of the value with less friction.

“Helm chart vs operator is a technical decision”. It is a cultural and operational decision. If your team ships via Argo CD with Helm values in Git, Production Stack fits without friction. If your team lives in direct kubectl apply -f and the idea of operators feels natural, OME or llm-d.

What we have not covered

Mooncake: the KV cache system shared between instances that Kimi/Moonshot runs in production across hundreds of millions of queries. It is a primitive (not a complete operator), but it integrates as a cache tier with several of the above.
Ray Serve LLM: Anyscale’s offering, on Kubernetes through KubeRay. More tied to the Ray ecosystem than to native K8s CRDs. Useful if Ray is already part of your infrastructure.
Fireworks AI, Modular MAX: commercial platforms with similar primitives, but hosted. They are not K8s operators; they are competitors at another layer.
Gateway API Inference Extension: the sigwg proposal to extend Gateway API with LLM primitives (model-aware routing, sticky sessions, fairness). In 2026 it is in alpha; the operators above are already starting to support it. When it matures, routing will stop being each operator’s problem and become part of the Kubernetes standard.
A generic inference observability stack: Prometheus + Grafana is standardising around the vllm:* metrics we covered in the vLLM article. There is an OpenTelemetry effort for LLMs (gen-ai semantic conventions) that is probably the next link in the chain.

Closing the series

This series of four articles has walked through LLM inference in production from the bottom up:

KV cache: the working memory that holds up LLM inference — why each token consumes VRAM, and how much.
vLLM on Kubernetes: the LLM inference piece that actually scales — how a model is served in production with a serious Deployment.
PagedAttention from the inside and the state of the art of the KV cache in 2026 — what happens inside the engine at block level, and what has arrived since.
This one — how many models are orchestrated in a cluster.

If you have got this far, you have the vocabulary and the map to sit in a meeting where five people throw acronyms around and place each one correctly. And, more importantly, to start making reasoned decisions about where to begin.

References

Operators and projects covered:

OME — Open Model Engine (GitHub) — LMSYS operator for LLM serving with SGLang/vLLM/TRT-LLM/Triton.
Introducing OME (LMSYS Blog, jul 2025) — announcement and architecture.
vLLM Production Stack (GitHub) — official vLLM Helm chart for K8s.
vLLM Production Stack docs — installation and configuration.
LMCache (GitHub) — KV cache with tiers.
NVIDIA Dynamo — successor to Triton.
NVIDIA Dynamo Addresses Multi-Node LLM Inference Challenges (InfoQ, dic 2025) — K8s integration.
llm-d (GitHub) — CNCF Sandbox project.
IBM, Red Hat, and Google donated llm-d to CNCF (The New Stack) — KubeCon EU 2026 announcement.
Red Hat bets big on Kubernetes inference with llm-d (SiliconANGLE, mar 2026) — coverage of the announcement.

Ancestors and primitives:

KServe (site) and KServe joins CNCF (The New Stack).
Kueue — gang scheduling.
LeaderWorkerSet — coordinated workloads such as multi-pod tensor parallel.
KEDA — autoscaling on external metrics.
Gateway API — the successor to Ingress.

Analysis and perspectives:

Building Efficient LLM Inference with the Cloud Native Quartet: KServe, vLLM, llm-d, and WG Serving (Jimmy Song) — an integrating view.
Complete Guide to llm-d CNCF Sandbox (DEV Community) — an operational walkthrough.
Previous articles on this blog: KV cache, vLLM on Kubernetes, PagedAttention deep dive.

Dynamo on lo0 — Tech Blog

Prefix routing: what LiteLLM does not do, who does, and how much it really improves

TL;DR

You are here: between the gateway and the engine

The analogy: the library with four counters

The correction: what I wrote in June

What LiteLLM does have: four affinities

prompt_caching: why it is no use for this

session_affinity: the one to use

The routers that do read the prompt, and why they are no good

What did land: admission control

What the engine exposes upwards

The ZMQ stream

The hash that has to be replicated

The hits are invisible by default

The metrics, and what broke

The last token

The ones that do do it

vLLM production-stack

llm-d

NVIDIA Dynamo

The rest of the field

Comparison table

How much it really improves

The five failure modes

The architecture that comes out of all of the above

When not to build this

Checklist

Traps and things that are not what they seem

Closing

See also

Sources

LLM serving engines compared on the Pareto frontier: vLLM, SGLang, TRT-LLM and Dynamo

TL;DR

Framing the article: what this B8 is and is not

Card for each engine

vLLM

SGLang

NVIDIA TensorRT-LLM (TRT-LLM)

NVIDIA Dynamo

Honourable mentions: LMDeploy and HF TGI

Capability matrix

Why they are not compared lightly: the variables that break a comparison

Performance on the Pareto frontier: illustrative figures

Throughput vs latency: Pareto table

Saturation point (concurrency sweep, vLLM FP8, 4×H100)

Fair-comparison methodology: the harness as the constant

Variables that are fixed (constants of the experiment)

Harness steps

Reproducible output format

State of the art 2026: relative maturity

Pareto decision table: when to choose each engine

Quantisation: effect on throughput and quality

Disaggregated serving: when the overhead is worth it

MLPerf Inference v5.1: cross-vendor reference figures

Variables the benchmark does not capture

Sources

Long context and KV offloading: when the notebook does not fit on the desk

TL;DR

The analogy

Part 1 · Why long context is a memory problem

The KV cache grows with the sequence

The numbers are frightening

And on top of that, attention is quadratic

Extending the window ≠ serving it cheaply

Part 2 · The KV memory hierarchy

Working out when offloading pays off

Part 3 · KV offloading in practice (OSS)

The limit of out-of-the-box prefix caching

LMCache: the persistent KV layer

Combine it with FP8

Part 4 · KVCache-centric / disaggregated architecture

Mooncake (Kimi’s platform)

NVIDIA Dynamo and KVBM

Part 5 · Reference architecture

Part 6 · Operational pitfalls (and honest scepticism)

Closing

Sources

The LLM inference router: the L7 switchboard we called a LoadBalancer in the canary post

TL;DR

`prompt_caching`: why it is no use for this

`session_affinity`: the one to use

The analogy: from `init.d` to systemd to operators