Vllm on lo0 — Tech Blog

Sizing for agents: 275 input tokens for every output token, and why the bottleneck is still decode

Sat, 12 Sep 2026 06:00:00 +0200

Seventh article in the operational track of the control layer. The fourth dealt with the coexistence of human traffic and agentic traffic on a fixed fleet. This one deals with the prior question: how big the fleet has to be, and how big the gateway in front of it. Verified against LiteLLM 1.102.0 (commit of 11 September 2026) and vLLM 0.29.0, with load data published between May and September 2026.

TL;DR

Until this year, sizing for agents meant extrapolating a chat profile and adding margin. In 2026 the first production-scale characterisations were published and the profile turned out to be something else. These are the seven conclusions.

The median prompt of an agentic call is 68,000 tokens, of which 63,000 arrive cached. The measurement covers 13.5 million sessions, 3.2 million users, 760.5 million model calls and 44.9 trillion prompt tokens in one week of June 2026. The median input-to-output ratio exceeds 275 to 1. Any calculation done on a profile of 800 prompt tokens and 250 output tokens describes a different system.

The load is prefill-dominant when counting tokens and decode-dominant when counting time. The genuinely new tokens that have to be prefilled on each turn are only between 1.5 and 7.3 times the output, not 275. With the prefix cache working, the split of execution time runs from 91 % to 98.6 % in decode. You size by memory bandwidth and by KV cache capacity, not by prefill FLOPs.

The sizing risk is not the average, it is the collapse of the prefix cache. The median hit rate within a turn is 98 %, it falls to 55 % at the boundary between turns and to 8 % when the client changes model. With dynamically retrieved context, prefix reuse drops to 1 %. One miss turns 142,000 prefix tokens into real prefill. Session affinity per replica stops being an optimisation and becomes a capacity requirement.

The intra-turn concurrency measured in production is 1.15 at the median and 1.4 at the 90th percentile. 93 % of tool batches contain a single invocation. The image of the agent opening twenty calls in parallel does not appear in the data. The real multiplier comes from the duty cycle and from the fact that 87 % of calls are started by the agent, not by the person.

The gateway is sized per worker, and each worker is a universe. One vCPU and 4 GiB per worker, with requests equal to limits. The limit counters, the caches and the cooldowns live in the worker’s memory unless there is Redis, so a limit of 100 requests per minute with four workers and three replicas allows 1,200. The Prisma pool is also per process: the number of connections open against Postgres is workers times connection_limit.

The CPU cost that grows with agentic load is the token counter. It serialises the entire tool catalogue on every request and tokenises it. The project itself measured 46, 53 and 100 ms for prompts of 50K, 75K and 100K tokens, and the operation passes through a concurrency limiter of four per worker. With agents sending long catalogues and six-figure contexts, that limit is the ceiling before the CPU is.

Autoscaling by memory does not work on this piece and the startup probe takes five minutes. The official chart deliberately leaves memory without a target because Prisma’s query engine never returns the maximum reached. That leaves CPU at 60 %, with a startupProbe of 300 seconds that makes any new replica arrive late. For agents there are two better metrics, and they are in /metrics.

You are here: the arithmetic that comes before buying

Classic capacity planning starts from a chat SLO and crosses two budgets, the VRAM one and the time one. That method is still correct. What changes with agents are the inputs: the prompt length, the number of calls per unit of human work, the split between prefill and decode, and the very definition of an SLO. This article replaces those inputs with measured figures and drags the result through to two numbers: gateway replicas and GPUs.

Out of scope is the splitting of traffic once it is built, which was already covered by the coexistence of humans and agents and prefix routing.

The analogy: the repair shop and the front desk

An appliance repair shop sizes its staff by counting faults, not phone calls. The difference matters because the person who brings in the washing machine occupies the front desk for three minutes, while the technician who takes it apart goes back to consult the machine’s data sheet fourteen times over the afternoon. If the shop manager sizes by visits to the front desk, they end up short of technicians and overstaffed at reception.

The inference platform has the same asymmetry. The person sends one turn. The agent sends fifteen calls for that turn, and each one drags along the complete file of everything done so far. The front desk is the gateway and the technicians are the GPUs, and the usual mistake is to size both with the same counter.

There is a second detail the analogy captures well. The technician does not read the whole data sheet each time: they have it open on the bench. That is the prefix cache. The day somebody moves them to a different bench between consultations, they have to fetch it from the archive again, and the work multiplies without the number of faults having increased.

Part 1. The real load profile

For two years, any capacity calculation for agents was done on assumptions. In 2026 three pieces of work appeared that measure the matter, and it is worth reading them together because they measure different scaffolds and their figures are not averageable.

The three sources

The characterisation at scale was published by a research team working on GitHub Copilot production telemetry (arXiv 2608.00101, 30 July 2026): 13.5 million sessions, 3.2 million users, 95.1 million user turns, 760.5 million model calls and 44.9 trillion prompt tokens, in one week of June 2026. It is the largest sample published to date.

The characterisation by benchmark suites comes from an architecture team (arXiv 2605.26297, 25 May 2026), which instruments five of them (ADE-Bench, DABStep, GAIA, SWE-bench Pro and Terminal-Bench 2.0) and measures the internal split of time.

The production measurement on a different scaffold was published by the vLLM project itself together with AgentX (vLLM blog, 8 September 2026).

The three disagree on turns per session: 15 calls at Copilot’s median, 43 turns in AgentX, 62.4 in SWE-bench Pro with Qwen. It is not a contradiction, they are scaffolds with different loops. The sizing lesson is that the number of calls per unit of human work depends on the client that connects, and that is why you have to measure it at home before fixing the size.

The figures you can actually use

Quantity	Measured value	Source
Median prompt per call	68K tokens, 63K of them cached	2608.00101
Input:output ratio (median)	above 275:1	2608.00101
Median session	3 user turns, 15 calls, 4.2 min	2608.00101
Mean session	6.1 turns, 40.6 calls, 62.6 min	2608.00101
Calls per turn (median)	4.5	2608.00101
Calls started by the agent	87 %	2608.00101
Range across user profiles	50× (23K to 1.1M tokens per turn)	2608.00101
Median input context	142K tokens; output 444 tokens per turn	AgentX
Accumulated context in SWE-bench Pro	mean 68.7 to 80.1K, maximum 146 to 166K	2605.26297
Reasoning tokens over the output	29.0 % to 67.6 %	2605.26297

Two operational observations come out of that and appear in no sizing guide.

The first: agents that fail accumulate up to 1.8 times more average context than those that finish well (2605.26297). The cost does not correlate with success, it correlates with difficulty and with getting stuck. A sizing done on the task that goes well falls short.

The second: the range across user profiles is 50 to 1. A user who only converses spends 23,000 tokens per turn; one who leaves the agent in a long loop spends 1.1 million. Segmenting by usage profile pays off more than any difficulty-based routing, because the variance is in the people, not in the requests.

What to measure before calculating

Three of your own figures, taken from a week of real traffic, are worth more than any table in this article:

Model calls per hour of human work, by user profile.
Prompt length distribution (median and 95th percentile), not the mean.
Aggregate prefix cache hit rate, with the time series, not the single number.

All three come from what the gateway already stores. In LiteLLM_SpendLogs you have the tokens per request and the key and team identifiers; in proxy_server_request you have the tools block of each call; and in vLLM’s /metrics you have vllm:prefix_cache_queries and vllm:prefix_cache_hits.

Part 2. Prefill, decode and the cache trap

Here is the finding that corrects almost everyone’s intuition, including mine when I started this article.

Counting tokens, agentic load is overwhelmingly input: 275 to 1. The apparently obvious conclusion would be to size by prefill capacity, that is by FLOPs. It is false.

What has to be prefilled on each turn is not the whole prompt, but what was added since the previous turn. That addition, measured, sits between 1.5 and 7.3 times the output length (2605.26297). The rest is served by the prefix cache. And the real split of the model’s execution time comes out like this: between 91.0 % and 98.6 % in decode, between 1.4 % and 9.0 % in prefill (2605.26297).

The hit rate that sustains that split is high and has been measured in three places: theoretical from 87.9 % to 99.3 % and empirical from 84.6 % to 99.5 % (2605.26297); above 96 % in production (AgentX); and a median of 98 % within a turn in Copilot (2608.00101).

Where it falls over

The same Copilot work measures the degradation, and it is the most actionable figure in the article:

98 % median hit rate within a turn. The detail matters: the first call of a turn hits around 45 %, the second rises to 86 % and from the third onwards it stabilises between 92 % and 94 %.
55 % at the boundary between turns, with idle gaps of 4.1 minutes at the median.
8 % when the client changes model.

And there is a fourth, worse regime: with dynamically retrieved context, that is when the scaffold injects search or RAG results that change on every iteration, prefix reuse falls to 1 % or less (arXiv 2608.15127, 15 August 2026).

A cache miss with a context of 142,000 tokens turns a prefill of a few thousand tokens into a six-figure one. The work multiplies by a factor of between 20 and 60 without the number of requests having changed. That is the difference between a fleet that meets its targets and one that seizes up mid-afternoon.

Two sizing rules come from that:

Without session affinity towards the same replica, the calculated sizing is worthless. LiteLLM’s concrete mechanism is session_affinity, which is not active by default and which was already covered in the prefix routing article. Without it, the calculation has to be redone assuming a prefill regime, which is between 20 and 60 times more expensive.
Changing model halfway through a session costs the entire cache. Any difficulty-based routing policy that moves a live conversation from one model group to another pays an 8 % hit rate on the next call. If you want to route by difficulty, do it at the start of the session.

Co-batching, which is the other effect nobody measures

Mixing a short request into the same batch as long-context requests degrades the short one’s inter-token latency. The measured figure: one long request in the batch raises time per token by 38.3 %; two raise it by 79.7 % (2608.15127). It is the technical reason why separating human and agent pools improves the human experience more than the capacity split suggests, and it is also the reason not to raise --max-num-seqs above what the KV budget allows.

Part 3. The KV budget, with the numbers of the agentic case

The formula does not change with respect to classic capacity planning, but the inputs do, and the result is surprising:

$$\text{KV}_{\text{bytes/token}} = 2 \times n_{\text{layers}} \times n_{\text{kv heads}} \times d_{\text{head}} \times \text{bytes}_{\text{elem}}$$

For a 70B-class geometry (80 layers, 8 KV heads with grouped attention, head dimension 128, 2-byte elements) that gives 0.3125 MiB per token. With the median agentic context of 142,000 tokens:

$$142{,}000 \times 0.3125\ \text{MiB} \approx 43.3\ \text{GiB per sequence}$$

For a 30B-class geometry (48 layers, everything else the same) that gives 0.1875 MiB per token, that is around 26.0 GiB per sequence.

It is worth pausing on what that means. An 80 GB H100 with the weights of a 70B model in FP8 already taking up space does not have 43 GiB free for a single sequence. The conclusion is harsh and has to be written down: serving real 70B-class agentic context demands tensor parallelism of at least 2 and a quantised KV cache. And the number of simultaneous sequences that fit, which is what sets --max-num-seqs, comes from dividing the available KV by that figure, not from a chat heuristic.

$$\text{sequences} \approx \frac{\text{VRAM}_{\text{KV available}}}{\text{KV}_{\text{bytes/token}} \times \text{max context}}$$

With FP8 in the KV, the divisor halves. It is the cheapest capacity lever there is in this load profile, and its cost in quality is validated with evals, not assumed.

Part 4. From people to concurrent requests

This is where most calculations overshoot, because it is assumed that an agent equals many simultaneous requests. The data says otherwise.

The concurrency of calls within a turn, measured in production, is 1.15 at the median and 1.4 at the 90th percentile, and 93 % of tool batches contain a single invocation (2608.00101). Intra-turn parallelism is small. Subagents appear in 44 % of AgentX sessions, with a median of 4 deployments among those that use them, so there is a long tail, but it is not the usual regime.

What does multiply is the duty cycle. The mean session lasts 62.6 minutes and contains 40.6 calls. At 444 output tokens and an interactivity of around 60 tokens per second, each call occupies around 7.4 seconds of decode, which gives between 5 and 7 active minutes out of those 62.6.

$$C = D \times \rho \times A$$

where $D$ is the people working, $\rho$ the duty cycle and $A$ the amplification. With $\rho \approx 0.10$ to $0.12$ derived from the previous figures, twenty developers give between 2 and 2.4 concurrent sequences in the steady state. It is the peak, not the average, that has to be sized for.

The amplification

Three factors published separately:

Tool failures: 9 % of turns trigger them, with a compute amplification of 4 times (2608.00101).
Subagents: 44 % of sessions, median of 4 deployments (AgentX).
Internal variance: the same task from the same agent varies by up to 30 times in cost within a single trace (2608.15127).

The product of the first two gives $(1 + 0.09 \times 3) \times (1 + 0.44 \times 3) \approx 2.9$. Nobody publishes that product as a single figure, so it is flagged as my own derivation, but an amplification factor of 3 is the reasonable design provision and it fits the measured variance.

For the twenty developers in the example: between 6 and 7 concurrent design sequences, with 142K contexts. At 43.3 GiB of KV per sequence in the 70B class, that is around 300 GiB of KV cache, that is four H100s dedicated to KV alone with the weights on top. With KV in FP8, half. With a 30B-class model, a third.

That calculation, done with the chat profile of 800 prompt tokens, would have given a single GPU. That is the distance between the two regimes.

Part 5. The gateway, which also has to be sized

The proxy consumes no GPU, but it has its own arithmetic and it is the part that produces the most surprises in production, because almost all of its state is per process.

A worker is a universe

The default value of --num_workers is 1 (litellm/constants.py:57). The usual startup is uvicorn directly with uvloop (proxy_cli.py:1477), and there are alternatives with gunicorn, hypercorn and granian.

What lives in each worker’s memory, and therefore multiplies:

The in-memory cache, limited to 200 elements with a TTL of 600 seconds (litellm/caching/in_memory_cache.py:27). With many deployments, the cooldowns and the counters compete for that same space and get evicted before they expire.
The key authentication cache, with a TTL of 60 seconds (proxy_server.py:1650), adjustable with general_settings.user_api_key_cache_ttl.
The limit and budget counters when there is no Redis (parallel_request_limiter_v3.py, local_only=True paths).
The router cooldowns, with DEFAULT_COOLDOWN_TIME_SECONDS = 5 (constants.py:75).
A periodic task scheduler of its own per worker (proxy_server.py:9674), coordinated between pods with a lock in Redis.

The practical consequence: a limit of 100 requests per minute with four workers and three replicas allows up to 1,200 without Redis. With Redis, the v3 limiter registers seven Lua scripts at startup (parallel_request_limiter_v3.py:608) and the limits become global.

One detail of the v3 limiter that matters with agentic load: LITELLM_TPM_TOKEN_RESERVATION_ENABLED comes enabled by default and reserves estimated tokens before the call, which adds a round trip to Redis per request. And there is an asyncio.Lock per instance that serialises the atomic batch checks (:665), documented by the code itself as a 99th-percentile risk under contention.

If Redis goes down, requests do not fail: there is a circuit breaker with a threshold of 5 failures and a recovery of 60 seconds (constants.py:456), and the proxy degrades to local memory. What is lost silently is the globality of the limits.

Postgres

The pool is also per process. The default values are database_connection_pool_limit = 10 and database_connection_pool_timeout = 60 (proxy_cli.py:62), injected as parameters of the Prisma URL. A pod with four workers opens forty connections. Three pods, one hundred and twenty. It is the fast route to exhausting Postgres’s max_connections without having served any traffic, and the reason the official chart offers PgBouncer inside the pod itself with maxDbConnections: 20 and maxClientConn: 1000.

On the request path there is only the authentication, which is cached for 60 seconds, and the rate limiting. Spend is deferred. The SpendLogs queue is capped at 64 MB and on overflow it discards the oldest elements with an error in the log (utils.py:6704). The write interval is proxy_batch_write_at, with a default value of 10 seconds plus a random spread of up to 5 (constants.py:1719, proxy_server.py:9704); the production guide recommends raising it to 60.

With a slow Postgres, the queue grows to those 64 MB and then loses billing data without returning any error to the client. It is the silent failure mode you have to alert on.

The token counter, which is the agentic trap

This is the piece that turns a correct sizing into an insufficient one, and it appears in no guide.

Before admitting a request, the proxy counts tokens. With chat load the cost is negligible. With agentic load it is not, for two reasons that add up.

The first is the prompt length. The project itself measured the admission counting cost at 46, 53 and 100 ms for prompts of 50K, 75K and 100K tokens (benchmarks, consulted on 12 September 2026), and published that with a counter written in Rust those figures fall to 4.9, 6.8 and 10.2 ms.

The second is the tool catalogue. The function that prepares the definitions serialises the entire catalogue to a pseudo-TypeScript and tokenises it (litellm_core_utils/token_counter.py:934), plus nine fixed tokens. The cost is linear in the size of the catalogue and it is paid on every request. That is exactly the pattern of an agent connected to an MCP gateway: a long catalogue, resent on every turn. The token bill of that catalogue was already covered in the MCP gateway article; here its other face appears, which is proxy CPU.

And on top of that there is an explicit ceiling. The counting runs in a thread with a capacity limiter whose default value is 4 per worker (token_counter.py:340, constants.py:407). With 100K prompts, four concurrent counts of 100 ms mean a theoretical ceiling of around 40 admissions per second per worker before the limiter starts queueing, and that is without having called the model yet.

What is cached is the tokeniser, with an LRU of 64 entries (utils.py:2206), not the result of the count.

Three levers, in order of return: raise TOKEN_COUNTER_MAX_CONCURRENT_COUNTS if CPU is spare, reduce the tool catalogue exposed per key, and watch the gateway overhead figure in /metrics to know whether this is happening.

Streaming and admission

There is no connection limit by default. --limit_concurrency exists but is optional, and it returns 503. The built-in admission control is only activated if general_settings.max_in_flight_requests_per_worker is defined (middleware/admission_control_middleware.py:285), with max_queued_requests_per_worker and admission_queue_timeout_seconds at 1.0 by default, rejecting with 503 and a retry-after: 1 header.

For agents, that header matters: an agentic client that retries without respecting it multiplies the load exactly when the gateway is saying it cannot take any more. The combination of gateway retries was already calculated by the fourth article and it reached 45 calls per turn in the worst case.

The default request timeout is 6,000 seconds (constants.py:512). The production guide recommends 600. With long agentic tasks, leaving it at 6,000 means a hung connection occupies a slot for an hour and a half.

Resources and autoscaling

The official chart recommends 1 vCPU and 4 GiB per worker, with requests equal to limits (helm/litellm-helm/values.yaml:235), and the production documentation suggests one worker per pod scaling horizontally. The project’s own high-performance benchmark, by contrast, uses four workers per pod with PgBouncer in front.

The HPA comes with targetCPUUtilizationPercentage: 60. And there is a comment in the chart worth reading twice: memory is deliberately left without a target because the resident consumption of Prisma’s query engine is a high-water mark that is never returned, so that an HPA on memory would add replicas and never remove them (values.yaml:260).

The other problem is startup. The startupProbe has failureThreshold: 30 with a period of 10 seconds, that is up to 300 seconds before the pod is considered ready (values.yaml:109). An HPA that reacts at 60 % CPU and takes five minutes to add capacity arrives late to any agentic burst.

The alternative, optional in the chart, is to scale by Prometheus metrics: targetRequestsPerSecond and targetTokensPerSecond over rate(litellm_proxy_total_requests_metric_total[1m]) and rate(litellm_total_tokens_metric_total[1m]). The chart itself warns that the token metric lags with long streaming responses, because tokens are accounted for on completion. For agents, that is almost always.

Part 6. What to watch: the metrics that work and the ones that broke

At the gateway

Metric	What for
`litellm_in_flight_requests`	Real saturation per worker. It is the scaling metric
`litellm_overhead_latency_metric`	Proxy cost per request; it rises with token counting
`litellm_overhead_with_guardrails_latency_metric`	The same including guardrails
`litellm_request_queue_time_seconds`	Arrival through to pre-call, includes authentication and ASGI queueing
`litellm_deployment_cooled_down`	Replicas out of the pool
`litellm_service_latency`	Redis and database latency, per service

The /health/backlog endpoint returns in_flight_requests, admitted_requests, queued_requests and rejected_requests (health_endpoints/_health_endpoints.py:1871).

What it does not expose: the depth of the SpendLogs queue or its bytes, the Prisma connections in use against the limit, nor the depth of the logging worker’s queue. All three are real failure modes and have to be instrumented from outside.

At the engine

Names verified in the vLLM 0.29.0 metrics documentation:

vllm:num_requests_running, vllm:num_requests_waiting
vllm:time_to_first_token_seconds, vllm:e2e_request_latency_seconds, vllm:request_queue_time_seconds, vllm:inter_token_latency_seconds
vllm:prefix_cache_queries, vllm:prefix_cache_hits, vllm:kv_cache_usage_perc

And the ones that break dashboards on upgrade:

vllm:time_in_queue_requests is duplicated by vllm:request_queue_time_seconds.
The direct prefix cache hit rate indicator disappeared. It is now computed as rate(vllm:prefix_cache_hits) / rate(vllm:prefix_cache_queries).
vllm:num_requests_swapped and vllm:cpu_cache_usage_perc are deprecated along with CPU swapping.
vllm:gpu_cache_usage_perc became vllm:kv_cache_usage_perc, which also breaks any HPA that uses it.

The SLO worth chasing

A 95th percentile of time to first token describes nobody’s experience when the unit of work is 43 turns. Coding tasks frequently exceed ten minutes and research tasks reach hours (2608.15127).

Two indicators serve better:

90th percentile of full-task latency, measured per session, not per request. It comes from grouping LiteLLM_SpendLogs by session identifier.
90th percentile of interactivity, in tokens per second per user. It is the one AgentX uses, with values between 58.3 and 74.2 in its runs.

To autoscale the engine, the official production-stack example uses vllm:num_requests_waiting with a threshold of 5, polling every 15 seconds and a cooldown of 360. With agentic load it is worth adding vllm:kv_cache_usage_perc as a second trigger, because with 26 to 43 GiB of KV per sequence the memory pressure arrives before the queue does.

Part 7. The engine levers, ordered by return

Verified against vLLM 0.29.0, released on 9 September 2026.

--long-prefill-token-threshold 512. The AgentX measurement attributes to it +93 % in tokens per GPU per second, with a 2.3-times improvement at the 90th percentile, by attacking the head-of-line blocking that a long prefill produces in front of short requests. It is the highest-return lever documented for this profile.

KV cache in FP8. It halves the memory budget per sequence, which is the limiting resource. Validate with evals.

--max-num-queued-reqs and --max-num-queued-tokens. New in 0.29. They are admission control in the engine, and they are the right mechanism to protect the SLO instead of letting the queue grow without limit.

max_num_batched_tokens downwards. High values improve time to first token; low values improve inter-token latency. In a regime where 91 % of the time is decode, inter-token latency is what is perceived. Chunked prefill is enabled by default whenever possible in the V1 engine.

Session affinity at the gateway. It is not an engine setting, but it is what holds up everything above. Without it, that 98 % hit rate turns into random distribution across replicas.

Two version warnings: Model Runner V2 becomes the default in 0.29 and V1 is deprecated with removal planned for 0.32; and starting with python -m vllm.entrypoints.openai.api_server is deprecated.

The full calculation, from start to finish

Case: twenty developers, a coding scaffold, a 30B-class model in the fleet, maximum context of 128K, KV in FP8.

1. Steady-state concurrency. $C = 20 \times 0.11 \times 3 \approx 6.6$ sequences.

2. KV budget per sequence. 0.1875 MiB per token in BF16, half in FP8, that is around 0.094 MiB. At 128K of context: around 12.0 GiB per sequence. For 6.6 sequences with margin, rounding to 8: 96 GiB of KV.

3. GPUs. The weights of a 30B in FP8 take up around 30 GB. On an 80 GB H100 with gpu_memory_utilization of 0.90 there are around 72 GB usable per card. With tensor parallelism of 2, the weights are split: around 15 GB per GPU, so around 57 GB of KV per GPU and around 114 GB in the replica. That covers the 96 GiB calculated, and TP=2 also adds memory bandwidth, which is the resource that dominates decode. Two H100s, one replica, TP=2. It is worth seeing the contrast: on a single H100 the full weights leave around 42 GB of KV, that is three and a half sequences, less than half of what is needed. With a second replica for availability and maintenance windows: four GPUs.

4. Requests per second at the gateway. 20 people, a mean session of 40.6 calls in 62.6 minutes, duty cycle included: of the order of 0.2 to 0.3 requests per second on average, with peaks of 2 to 3.

5. Gateway workers. The ceiling is not set by throughput but by token counting: four concurrent counts per worker, at 100 ms with long prompts. With peaks of 3 requests per second and six-figure prompts, two replicas of one worker each cover the case with margin for the rolling update, at 1 vCPU and 4 GiB each. Postgres will see 20 connections.

6. What needs switching on. Session affinity, admission control at the gateway and at the engine, --long-prefill-token-threshold, and Redis so that the limits are global.

The same calculation with the chat profile would have given one GPU and one worker. The distance is not in the safety margin, it is in the inputs.

A reference `config.yaml`

model_list:
 - model_name: agentes-30b
 litellm_params:
 model: hosted_vllm/qwen3-coder-30b
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 rpm: 600
 model_info:
 input_cost_per_token: 0.00000018
 output_cost_per_token: 0.00000072

router_settings:
 routing_strategy: simple-shuffle
 # without this, the capacity calculation in this article is worthless
 optional_pre_call_checks: ["session_affinity"]
 num_retries: 2
 allowed_fails: 3
 cooldown_time: 30

general_settings:
 # admission control: below this the queue grows without limit
 max_in_flight_requests_per_worker: 24
 max_queued_requests_per_worker: 48
 admission_queue_timeout_seconds: 1.0
 # the spend queue discards at 64 MB; shorter interval, less backlog
 proxy_batch_write_at: 60
 enable_drain_endpoint: true
 drain_endpoint_token: os.environ/DRAIN_TOKEN
 database_connection_pool_limit: 10

litellm_settings:
 # 6000 s by default: a hung connection holds a slot for an hour and a half
 request_timeout: 600
 callbacks: ["langfuse_otel", "prometheus"]
 cache: true
 cache_params:
 type: redis
 host: os.environ/REDIS_HOST
 namespace: litellm

Environment variables that go with it:

NUM_WORKERS=1
TOKEN_COUNTER_MAX_CONCURRENT_COUNTS=8 # default 4; raise only if CPU is spare
LITELLM_TPM_TOKEN_RESERVATION_ENABLED=true
PROXY_CONFIG_RELOAD_INTERVAL_SECONDS=30

And on the engine side:

vllm serve Qwen/Qwen3-Coder-30B \
 --tensor-parallel-size 2 \
 --max-model-len 131072 \
 --kv-cache-dtype fp8 \
 --long-prefill-token-threshold 512 \
 --max-num-queued-reqs 64 \
 --gpu-memory-utilization 0.90

Checklist

Measure three figures of your own over a week: calls per hour of human work and per profile, prompt length distribution, and the time series of prefix cache hit rate.
Calculate the KV per sequence with the model’s real geometry, taken from its config.json, not from a table.
Set --max-num-seqs from that budget, not from a chat heuristic.
Switch on session affinity before accepting the calculated number of GPUs.
Switch on admission control in both layers and check that the client respects retry-after.
Lower request_timeout from 6,000 to 600.
Raise proxy_batch_write_at to 60 and alert on the growth of the spend queue from outside, because there is no metric.
Leave the HPA on CPU at 60 %, never on memory, and assume the 300 seconds of the startupProbe when planning the margin.
Watch litellm_in_flight_requests and litellm_overhead_latency_metric; if the overhead rises with the size of the tool catalogue, the culprit is the token counter.
Go over the dashboards for the vLLM metric renames before upgrading.
Check that no routing policy moves a live session between model groups.
Redo the calculation when a new user profile arrives: the range across profiles is 50 to 1.

Traps and things that are not what they seem

The ratio of 275 to 1 does not mean the system is prefill-bound. The new tokens per turn are between 1.5 and 7.3 times the output. The time is between 91 % and 98.6 % in decode.

The aggregate prefix cache hit rate hides the problem. An aggregate 90 % may be 98 % intra-turn and 55 % at boundaries. You have to look at the series, not the number, and segment by request type.

vLLM’s hit rate counts the reprefill after an eviction as a hit. The counter exists internally and is not exposed, so the published number is optimistic under a memory-pressure regime.

Agents that fail cost more than those that succeed, up to 1.8 times more context. Sizing on the task that goes well leaves the calculation short.

The database pool is per process, not per pod. Four workers are forty connections.

The HPA on memory does not work with this piece, and not because of a configuration mistake: Prisma’s query engine does not return the memory.

The 300-second startupProbe turns any autoscaling into late reaction. For agentic peaks, it is better to over-provision than to trust the scaling.

Raising --max-num-seqs to “make use of the GPU” degrades perceived latency: two long requests in the batch raise time per token by 79.7 %.

The token counter has a limiter of four per worker and it appears in no sizing guide. With large tool catalogues and long contexts it is the first ceiling you hit.

The figures in this article come from three different scaffolds and are not averageable. Fifteen, forty-three and sixty-two turns per session are three systems, not three measurements of the same one.

Closing

The arithmetic of sizing has not changed. What has changed are the inputs, and now there are public measurements at scale instead of intuitions. Three figures sum up the change of regime: 68,000 tokens of median prompt, 87 % of calls started by the agent rather than by the person, and a time split that is decode for more than 90 % as long as the prefix cache hits.

From that comes an order of work that is not the usual one. First, session affinity, because without it the rest of the calculation describes a different system. Second, the KV budget with the model’s real geometry, which is what sets how many sequences fit. Third, the gateway, where the ceiling is not throughput but a token counter with four threads per worker.

And a warning about your own figures. The measured range across user profiles is 50 to 1. Any platform that does not segment its measurement by profile is averaging two different populations and producing a number that describes neither of them.

Sources

Agentic Coding in the Wild: Characterizing GitHub Copilot at Production Scale, arXiv 2608.00101, 30 de julio de 2026.
Agentic AI Workload Characteristics, arXiv 2605.26297, 25 de mayo de 2026.
From LLM Inference to Agentic Workloads, arXiv 2608.15127, 15 de agosto de 2026.
vLLM x AgentX: serving agentic workloads, 8 de septiembre de 2026.
vLLM metrics design y optimization guide, consultados el 12 de septiembre de 2026.
LiteLLM benchmarks y production setup, consultados el 12 de septiembre de 2026.
Código de LiteLLM 1.102.0, commit 9071ca50 del 11 de septiembre de 2026.
Autoescalado con KEDA en vLLM production-stack, consultado el 12 de septiembre de 2026.

Prefix routing: what LiteLLM does not do, who does, and how much it really improves

Fri, 11 Sep 2026 00:30:00 +0200

Fifth article in the operational track of the control layer. The pair with Langfuse covered observability, the proxy’s day 2 covered availability, virtual keys covered governance and humans and agents covered the coexistence of two classes of traffic. Here a claim of mine from June gets taken apart. Everything that follows is verified against LiteLLM 1.102.0 (commit of 10 September 2026), vLLM 0.29.0 and the main branches of production-stack, llm-d and Dynamo from the second week of September 2026.

TL;DR

LiteLLM does not do prefix-aware routing, neither in the open version nor in the paid one. It is not a licensing question. A search across the whole tree for kv_aware, cache_aware, radix, prefix_cache and block_hash returns zero matches in the router. Nor is there any call from the router to /metrics or to any vLLM endpoint. In June I wrote the opposite in the inference router article and I was wrong.

What LiteLLM has is affinity, which is a different thing. Four pre-call checks pin a request to a replica by session identifier, by key hash, by previous response identifier or by exact hash of a marked prefix. None is active by default. Affinity distributes conversations; prefix routing distributes prefixes shared between different conversations, which is the case that multiplies throughput.

The only one that looks at the prompt is fragile in three ways. If the client does not send cache_control, the key comes out null and the check does nothing. If the cut point goes in the last turn, which is what the automatic injection does by default, the key changes on every turn and it never hits. And the pin’s TTL is 300 seconds hardcoded in two lines, while the provider offers one-hour caches.

The engine does not answer questions, it publishes events. There is no vLLM endpoint you can ask whether it has a prefix. What there is, is a ZMQ socket that emits stored and removed blocks in msgpack, switched off by default, with silent discarding when the consumer does not keep up. A router that wants to be exact has to replicate the engine’s hash in full: chained SHA256, deterministic seed, block granularity and the extra keys in their order.

And by default it does not publish the hits, only the additions. The index of a router that only listens drifts away from the engine’s real LRU without noticing. To see the hits you have to ask for it request by request with kv_cache_report_mode: full.

The three routers that do do it are very different from each other. The vLLM production-stack one indexes chunks of 128 raw characters in a trie that is never purged, models no eviction and dies on every restart. The llm-d one offers an approximate mode and an exact event-driven one, all its plugins are in alpha or beta, and its default tokeniser does not produce real tokens. NVIDIA Dynamo’s keeps an event-driven radix tree with a weighted cost function, and ships with the parameter that avoids starving a freshly scaled replica switched off.

The big improvements only appear in the best possible case. The two- and three-digit numbers are published by the projects themselves, with synthetic shared prefixes and a round-robin balancer as the reference. The most complete independent measurement, from Meta, obtains 2.3 times more throughput and also identifies load regimes where affinity leaves capacity at between half and two thirds of the reference. Another, on a multi-region deployment with 89 % reuse, measures single-digit improvements.

The correct architecture has two layers, not one. LiteLLM on top for identity, budget and audit, pointing at a single destination per model group. The KV-aware router underneath, inside the engine’s domain. Mixing it into a single piece is not an option that exists today.

You are here: between the gateway and the engine

This article occupies the gap between two layers that already have an article of their own. Above, the L7 switchboard and the gateway that has been operated since the first article in this track. Below, the fundamentals of the KV cache and hit rate engineering.

The gap is the piece that decides which replica each request goes to, knowing what each replica has in memory. It is the only function in the stack that cannot be resolved in the gateway or in the engine separately.

The analogy: the library with four counters

A library with four counters. Each librarian has in front of them a pile of the books they have consulted lately. When a reader comes in asking for the third volume of an encyclopaedia, if that volume is in the pile at counter two, serving them there takes seconds. At any other counter someone has to go down to the store room.

A normal balancer sends the reader to the counter with the shortest queue. It is the right decision if all the counters are interchangeable, and it is the wrong decision here, because they are not: what distinguishes them is not the queue, it is the pile.

Session affinity resolves one specific case of this. The returning reader is always sent to the same counter, because they will probably ask for the fourth volume of the same encyclopaedia. It works, and it does not resolve the general case: when a hundred different readers ask for the same third volume, session affinity spreads them across the four counters and four librarians go down to the store room for the same book.

Prefix routing is looking at what the reader asks for before assigning them a counter. It has two costs that the analogy makes visible straight away. The first is that somebody has to keep track of what is in each pile, and that bookkeeping takes work. The second is that if everybody asks for the third volume, counter two becomes the library’s only desk while the other three look on.

The correction: what I wrote in June

In the inference router article, on 2 June 2026, I closed the prefix routing section with this sentence: “vLLM Production Stack router implements it natively. NVIDIA Dynamo too. LiteLLM in its enterprise version has a beta.”

The first two hold up. The third is false and was no less false in June. While preparing the humans and agents article the first hint appeared, and full verification against 1.102.0 confirms it.

Case-insensitive search over litellm/, enterprise/ and gateway/, for the terms kv_aware, kv-aware, cache_aware, cache-aware, radix, prefix_cache, prefix-cache, kv_cache, block_hash, prefix_hash, prefix_score and longest_prefix: zero matches. Search for vllm inside litellm/router.py, litellm/router_strategy/ and litellm/router_utils/: zero. Search for num_requests_waiting, gpu_cache_usage or any scraping of the backends’ /metrics from the router: zero. LiteLLM’s router never asks the engine anything.

Search for premium_user in the files where those checks would live: a single result, and it is about something else. There is no paid version with this feature. The only routing piece that is behind a licence is per-tag budgets, in litellm/router_strategy/budget_limiter.py.

Where the error came from. LiteLLM’s documentation has an entry on dynamic TPM and RPM allocation marked as beta and enterprise, and a pre-call check called prompt_caching. Skim-read and added together, they give the impression of prefix routing in beta. Read in the code, prompt_caching is something else, and it deserves a section of its own because it is still recommended for this problem.

What LiteLLM does have: four affinities

The optional_pre_call_checks field accepts eight values. None is enabled by default: apply_default_settings passes an empty list. One of the eight, forward_client_headers_by_model_group, is declared in the type and in the interface schema and is consumed by no branch of the code. It is a dead value.

The four that pin a request to a replica are these.

Check	What it pins	Where it stores it	TTL
`session_affinity`	Session identifier	Redis with a Lua script, mirrored in memory	`deployment_affinity_ttl_seconds`, 3600 by default, refreshed
`deployment_affinity`	API key hash	Same	Same
`responses_api_deployment_check`	`previous_response_id`	Same	Same
`prompt_caching`	Exact hash of a marked prefix	`DualCache`	300 s, hardcoded

The other four are router_budget_limiting, encrypted_content_affinity, enforce_model_rate_limits and the dead value.

`prompt_caching`: why it is no use for this

The key is a SHA256 over the serialisation of the cacheable prefix, and “cacheable prefix” means everything up to and including the last block marked with cache_control: {"type": "ephemeral"}. Three consequences, checked by running the functions in isolation over the repository’s code.

If the request carries no cache_control, the extraction returns an empty list, the key comes out null and the check pins nothing. Automatic injection of cut points is only enabled with litellm.enable_anthropic_prompt_caching, which defaults to false, and only for Anthropic and Bedrock providers.

If the cut point is in the system prompt, the key does not change when turns are added and the pin works. This is the good case and it requires the client to mark it deliberately.

If the cut point is in the last message, the key changes on every turn and the pin never hits. And this is exactly the default case of the automatic injection, which places one point on the system and another at index -1. Since the second is the last, the cacheable prefix becomes the whole conversation.

On top of that there is the TTL. Two lines with ttl=300, one of them with the comment # store for 5 minutes. There is no parameter to change it. Issue 28427 has had it open for months with no response from the maintainers, and it points to the mismatch with the one-hour ephemeral caches that LiteLLM itself knows how to request via cache_control.ttl: "1h".

There is a nuance the issue does not capture and that makes diagnosis worse: in the default case the pin does not even last five minutes, because the key changes first.

`session_affinity`: the one to use

For conversations, this is the right piece and it is reasonably built. The session identifier is looked for in this exact order of headers:

x-litellm-trace-id
x-litellm-session-id
Any x-<vendor>-session-id, with the regular expression ^x-.+-session-id$ and a value of at least eight alphanumeric characters
Only if the User-Agent is Codex’s: session-id, session_id, thread-id, conversation_id
Plain x-session-id, for opencode

And if nothing matches, two fallbacks: a metadata.user_id in Anthropic format, and the W3C baggage with session.id=. The third point is the one that captures Claude Code with nothing configured.

Storage is a Lua script in Redis that does GET, SET NX EX and EXPIRE, so the first writer wins and the TTL is refreshed on every request that confirms the pin. It is an inactivity TTL, not a total session duration. Without Redis it degrades to a pod-local reservation, which is atomic but not shared.

A detail of the pipeline order that already came up in the previous article and still holds: the affinity checks run inside async_get_healthy_deployments before tag filtering. If the pin returns a single deployment and that deployment does not satisfy the request’s tag, the later filter removes it and the request fails. Affinity gets no vote on the tags.

The routers that do read the prompt, and why they are no good

LiteLLM has four pieces that read message content: AutoRouter with semantic matching, RequestComplexityRouter, AdaptiveRouter and QualityRouter. They all live in async_pre_routing_hook, which runs before looking for healthy deployments.

The reason they do not solve this problem is in what they choose. All four choose a model group, not a replica. They decide whether an easy question goes to the thirty-billion-parameter model or to the small one. None has access to the replica list or knows what is in each one’s memory.

What did land: admission control

A minor correction to the previous article, which placed this in the 1.101 branch as an upcoming feature. In 1.102.0 it is already there, in litellm/proxy/middleware/admission_control_middleware.py, with the three settings in general_settings: max_in_flight_requests_per_worker, max_queued_requests_per_worker and admission_queue_timeout_seconds, the last with a default value of 1.0 seconds.

It stays inactive as long as the first one is not declared. It exposes an authenticated /health/backlog and the three metrics litellm_admission_admitted_requests, litellm_admission_queued_requests and litellm_admission_rejected_requests_total, the last with a reason label. It is a per-process semaphore, not coordinated across pods.

What the engine exposes upwards

Before looking at the routers that do do this, you have to understand what raw material they work with. The natural question would be whether vLLM has an endpoint you can ask about a prefix. It does not.

The only cache router in the entry points is vllm/entrypoints/serve/dev/cache/api_router.py, and it only exposes destructive writes: POST /reset_prefix_cache, /reset_mm_cache and /reset_encoder_cache. All three are behind VLLM_SERVER_DEV_MODE=1 and do not exist in a normal deployment. There is no read.

What there is, is an event stream.

The ZMQ stream

It is enabled with --kv-events-config, and it is off by default. The configuration accepts a publisher (null or zmq), an endpoint (by default tcp://*:5557, which binds), an optional replay endpoint, the number of batches retained for that replay (10,000), and a high water mark limit of 100,000.

That limit is the one that matters operationally: above it, ZMQ discards events if the consumer does not keep up. Silently.

The transport is a PUB socket with msgpack serialisation and three-part frames: topic, eight-byte sequence number and payload. The sequence number lets a subscriber ask for replay from where it left off through the optional ROUTER socket. With data parallelism the port shifts by rank, so you have to subscribe to 5557+rank.

Three event types. BlockStored carries the hashes, the parent hash, the token identifiers, the block size, the LoRA adapter name, the extra keys, the medium (GPU, CPU or storage) and a session identifier. BlockRemoved carries only hashes. AllBlocksCleared carries no payload and is emitted when the prefix cache is reset.

They are published once per scheduler step, only if there are events. With the feature switched off the cost is zero. With it switched on, the full token identifiers of each block travel over the wire, so bandwidth scales with tokens per second and not with requests per second.

An example in vLLM’s own repository declares itself experimental on its eighth line.

The hash that has to be replicated

An exact router does not receive the engine’s state, it reconstructs it. For that it has to compute the same hashes, and the engine computes them like this:

# vllm/v1/core/kv_cache_utils.py
def hash_block_tokens(hash_function, parent_block_hash, curr_block_token_ids, extra_keys):
 if not parent_block_hash:
 parent_block_hash = NONE_HASH
 return hash_function((parent_block_hash, tuple(curr_block_token_ids), extra_keys))

It is chained: each hash identifies the complete prefix up to that boundary, not the isolated block. The default algorithm is SHA256, configurable with --prefix-caching-hash-algo among four values.

Five things a router has to get right for its hashes to match the engine’s:

The seed. Since 0.29, NONE_HASH is deterministic from a fixed seed, so two different processes produce the same hashes with nothing configured. It is a silent change of semantics from earlier versions, which required setting PYTHONHASHSEED. With the xxhash algorithms it is still needed.

The block size. By default 16 tokens, but the attention backend can override it if the user did not pass --block-size: 256 with sliding-window sparse attention, 64 with ROCm’s AITER, 64 or more with FlashAttention on XPU. A router must not assume 16, it has to read it from the event.

The hashing granularity, which is not necessarily the physical block size. There is --prefix-match-unit, which with several cache groups resolves to the greatest common divisor of the cacheable groups’ sizes.

The extra keys, in their order: LoRA adapter name (by name, not by identifier), multimodal identifiers with their relative offset inside the block, cache_salt only in block zero, and the hash of the prompt embeddings. Sampling parameters do not go in, which is correct: the prefix KV does not depend on the temperature.

And truncation. By default the hashes travel over the wire as 64-bit integers truncated from a SHA256, not as the full bytes. It can be disabled with VLLM_KV_EVENTS_USE_INT_BLOCK_HASHES=0. With the default value, a router has a theoretical collision risk that the engine does not have.

The hits are invisible by default

This is the detail that takes the most effort to discover and has the most consequences. The events are incremental: only what gets newly cached is announced. A reused block generates no event.

A router that only listens sees the additions and the removals, but does not see the hits. Its notion of what is recent and what is cold drifts away from the engine’s real LRU with nothing to warn it.

A full mode exists, and it is per request. sampling_params.extra_args["kv_cache_report_mode"] = "full" makes BlockStored events be emitted for blocks that were hits too. The function’s docstring says so plainly: it generates events so that external consumers, “for example a gateway”, know which blocks were reused. That is to say: for this to work, the router has to inject a parameter into every request it forwards.

The metrics, and what broke

For aggregate measurement two counters remain: vllm:prefix_cache_queries_total and vllm:prefix_cache_hits_total, both in tokens and not in requests, with model and engine labels. There are two equivalents for the external connector, which allow a hit in HBM to be distinguished from one brought in from CPU or NVMe.

And there are two removals that break existing configurations. vllm:gpu_prefix_cache_hit_rate no longer exists: zero appearances in the repository. It was deprecated in 0.8, hidden in 0.9 and removed in 0.10. And vllm:gpu_cache_usage_perc was renamed to vllm:kv_cache_usage_perc, which breaks Grafana panels and autoscalers that depended on the old name.

The hit rate has to be derived in PromQL with rate() over the two counters. The one that appears in the text log is something else: it uses a window of the last thousand requests.

And it has a bias you need to know about before taking decisions with it. When a request is evicted, its blocks return to the free queue keeping the hash, so the later reprefill finds them again and counts as a hit. A high hit rate can mean useful reuse between users or it can mean the engine is evicting and recomputing in a loop. The internal statistics separate the two things with a preempted field, which is not exposed as a Prometheus metric.

The last token

A small detail with an effect on long conversations: the maximum hit length is the request’s token count minus one. There is always something to recompute in order to get logits. Since the reservation requires alignment to the block size, in practice that can force recomputing the whole final block. With blocks of 16 it is noise. With a backend that prefers 256, it is up to 256 tokens recomputed per turn.

The ones that do do it

vLLM production-stack

Eight routing logics in a single --routing-logic selector: roundrobin, session, kvaware, loadaware, prefixaware, disaggregated_prefill, disaggregated_prefill_orchestrated and priority. Two of them, loadaware and priority, are not in the Helm chart’s enum and are only enabled through extraArgs.

The naming trap is that prefixaware and kvaware are radically different things.

prefixaware is a hash trie in the router’s process. It chunks the prompt string every 128 characters, not tokens, and hashes each chunk with 64-bit xxhash. Its own docstring says so: the chunk size is “in number of characters”. That 128 is hardcoded, there is no flag or Helm value to change it.

Four consequences come out of that. --prefix-min-match-length is measured in characters and is quantised to multiples of 128. A shared prefix that differs by one character inside the first chunk breaks the whole match, because there is no partial match inside a chunk. For chat, it concatenates the content of all the messages with newlines and without applying the chat template, so the indexed string is not what the engine tokenises. And the tie-break between tied replicas is random.choice, without looking at load.

The trie has no notion of saturation. A popular prefix concentrates all the traffic on the same replica indefinitely. The docstring assumes the simplified model and declares it: “we assume there is no prefix cache eviction”.

It has no pruning either. HashTrie only has insertion and longest prefix lookup; there is no deletion method. When a replica disappears, its URLs stay forever in every node’s sets, and filtering happens at query time. It works, and the trie grows monotonically: each distinct prompt adds its length divided by 128 permanent nodes, with no bound, no TTL and no LRU. The router’s memory limit in the chart is 1,000 MiB. By contrast, the round-robin router in the same file does bound its caches to 1,024 entries.

And it dies on every restart: it is in-process memory state, with no snapshot and no initial load.

kvaware and loadaware are better and have a different price. They do not index themselves: they ask the LMCache controller, which knows about real tokens. That requires lmcache==0.3.11 and vllm==0.13.0 pinned exactly, and the official router image drags in the whole of vLLM. The limitation to read before planning anything: they only look at request_json.get("prompt", ""), with an open TODO for chat completions. With /v1/chat/completions, both tokenise an empty string.

loadaware is the only one of the three that breaks affinity on saturation, with the formula score = relative_match - beta * relative_load and beta = 1.0 by default. The authors’ own reading: a replica at twice the average load loses the equivalent of a full cache hit. The guide suggests 0.25 to favour cache and 2.0 to favour balancing.

On published results, the documentation’s benchmarking file literally says that the measurement platform is coming soon. The project blog’s two posts claim between three and ten times lower response latency, and only one declares a partial configuration: Llama 3.1 70B on four nodes with tensor parallelism 2 over 80 GB A100s, inputs of 9K tokens and outputs of 10, against AIBrix 0.2.0 and a flat deployment. Charts with no labelled points, no absolute figures and no declaration of which routing logic was used in each.

llm-d

There has been a reorganisation here that invalidates a good part of the third-party documentation. The Endpoint Picker code left the Gateway API Inference Extension repository and was merged into llm-d’s, which has been renamed to llm-d Router. The Kubernetes project keeps the InferencePool API and a lightweight reference EPP that does nothing about prefixes. If you look for KV-aware routing in the extension’s repository, today it is not there.

The InferencePool is at v1 and is GA. The other two APIs moved to the llm-d.ai group at v1alpha2, and when they coexist with the old ones the EPP prefers the new ones and ignores the old.

Scoring is a weighted sum. The default weights, when no configuration is passed: queue 2, KV utilisation 2, prefix 3. Any scorer referenced without a weight counts as 1.0. There is no configuration through environment variables: the old ENABLE_*_SCORER no longer exist, everything goes in an EndpointPickerConfig in YAML.

The prefix scorer consumes an attribute published by a producer, and there are two producers.

The approximate one computes the hashes in the EPP itself without talking to the engine, and keeps a local LRU per pod and block pair that is populated after the decision. It is an approximation by construction. It has a hard floor: a configured block size below 64 tokens is raised to 64 at request time, because the index stores one entry per pod and block and going down to 16 would multiply memory by four.

The exact one builds the real index from the engine’s ZMQ events, with one subscriber per pod that is installed and torn down with the endpoint’s registration and deregistration. It has adapters for vLLM and SGLang, deduplicates evictions by reference counting, and the replay buffer that allows the index to be rebuilt after an EPP restart requires vLLM 0.26.0 or above.

The trap is in the tokeniser. The EPP tokenises through a plugin, and the default backend is estimate: byte packing with no tokeniser, roughly 4 bytes per token, with identifiers that do not correspond to real engine tokens. The exact scorer needs real tokens, and for that you have to explicitly configure the vllm backend, which does HTTP against the engine’s /v1/completions/render endpoints. If it is omitted, the auto-created estimate satisfies the dependency and the system degrades silently.

On maturity, the file itself declares it: all in-tree plugins are in alpha or beta, and they will be promoted to stable when the project approaches 1.0. An alpha plugin configured without the corresponding flag makes the EPP fail to start.

And on replicas, the warning is explicit: active-active mode should be avoided with approximate prefix routing, because the EPP replicas do not share state and each one only sees the traffic it has served itself, which significantly degrades the hit rate. The exact index is safe in high availability, and even so the reference guides fix a single replica because the accounting of in-flight requests is local to the process.

The cost is the best documented in the whole field, and it deserves a table. Simulator with Qwen3 8B, 100 pods, 100K input tokens:

Requests/s	Cap on tokens to match	Peak CPU	Peak memory
5.0	4,096	1.19 cores	0.26 GiB
5.0	100,000	3.82 cores	0.65 GiB
98.7	4,096	35.17 cores	2.46 GiB
98.8	100,000	46.50 cores	3.41 GiB

With outputs of 10K tokens at 50 requests per second, 32.53 cores and 12.54 GiB. The maintainers’ sizing rule is half a core to one core per request per second on large agentic loads. And idle consumption scales with the number of pods because of continuous scraping: with 100 pods, about 7.5 cores with no traffic.

NVIDIA Dynamo

A precise event-driven index, with a global radix tree in each frontend’s process. Since all the replicas consume the same event plane, there is no need to synchronise routers with each other for the prefix state.

The cost function is explicit and weighted. It discounts from the prefill the blocks that overlap in GPU, host and disk with different weights (1.0 for the overlap credit, 0.75 for host, 0.25 for disk), and adds the potential decode blocks and the active requests.

There is a parameter to look at before deploying: --router-kv-overlap-score-credit-decay is at 0 by default, that is, disabled. Its description says what it is for: preventing loaded, cache-rich replicas from winning again and again while the freshly autoscaled ones receive too little traffic. With the default value, a new pod has no cache, therefore never wins the scoring, therefore never generates cache.

The approximate mode exists (--no-router-kv-events, with a TTL of 120 seconds) and the documentation says it is not the recommended route in production.

Dynamo is the most honest project in its measurement methodology: its comparison guide recommends contrasting --router-mode random against kv over the public Mooncake trace from FAST'25, with a declared cache ratio of 59 %, and it generates the table locally instead of publishing a number.

The rest of the field

SGLang has renamed its router to sgl-model-gateway, version 0.3.2 within SGLang 0.5.19. Its cache_aware policy is an approximate radix tree over raw characters, with switching to shortest queue when the system is unbalanced according to two combined thresholds. One detail that can cost an afternoon: the Rust defaults and the Python CLI defaults do not match, and on the maximum tree size they differ by four orders of magnitude (10,000 nodes against 2^26).

AIBrix 0.7.0 offers a standard mode with a local hash table and a KV event synchronisation mode behind a flag and a build tag, with vLLM 0.7.0 or above.

Ray Serve LLM has PrefixCacheAffinityRouter with an explicit alpha API warning, a character-based prefix tree in a decoupled actor, and its imbalance threshold at infinity by default, that is, out of the box it never breaks affinity on load.

Envoy AI Gateway 1.1.0 implements none of this and delegates to InferencePool and the EPP. KServe the same, with its LLMInferenceService at v1alpha1. Mooncake Conductor is still an unimplemented proposal.

Comparison table

Project	Version and date	Index	Where it lives	Kubernetes	Declared maturity
production-stack `prefixaware`	main branch, 09-Sep-2026	Approximate, character trie, no eviction	Router process	Optional	WIP in the README
production-stack `kvaware`/`loadaware`	same	Exact via LMCache controller	LMCache controller	Optional	No chat completions
llm-d Router approximate	v0.10.0, 17-Aug-2026	Approximate, LRU per pod	EPP process	Yes	Beta, active-active discouraged
llm-d Router exact	same	Exact through ZMQ events	EPP process	Yes	Beta
NVIDIA Dynamo	v1.4.2, 27-Aug-2026	Exact, event-driven radix tree	Frontend process	No	GA with experimental subsystems
SGLang `sgl-model-gateway`	0.3.2, 03-Sep-2026	Approximate, character radix	Gateway process	No	Default policy
AIBrix	v0.7.0, 16-Jun-2026	Both	Gateway plugin	Yes	Synchronisation behind a flag
Ray Serve LLM	Ray 2.58	Approximate, characters	Ray actor	No	Declared alpha
Envoy AI Gateway	v1.1.0, 21-Aug-2026	None of its own	Delegates to EPP	Yes	Stable 1.x API

How much it really improves

Here two types of source have to be separated.

The projects’ own numbers. Red Hat published the llm-d case in May: Qwen3 32B, eight vLLM pods over 16 H100s with tensor parallelism 2, synthetic shared-prefix load with 150 groups of five prompts, 6,000 tokens of system prompt, 1,200 of question and 1,000 of output. Result: up to 109 % more throughput and up to 99 % less TTFT, around 200 concurrent users within SLO against around 20. The repository’s result files give more detail: peak throughput from 6,986 to 14,892 output tokens per second, p90 TTFT from 135.5 seconds to 0.26.

That same file publishes the regression, and that is why it is worth reading: p50 inter-token latency rises by 22.4 % with vLLM and by 45.4 % with SGLang. The trade-off is declared: routing by affinity concentrates more concurrent work on the pods with a warm cache.

It is the best possible case. A synthetic shared prefix and a reference that is a flat Kubernetes Service.

The independent measurements. There are three and they say different things.

CacheRoute, from Meta, August 2026, is the most complete. Llama 3.3 70B in fp8, 30 targets with tensor parallelism 2 over 60 H100s, a semi-synthetic trace from production telemetry with 128,824 business keys, and all the alternatives reimplemented in the same harness instead of comparing against a dumb balancer. It obtains 176 requests per second with a p99 SLO of 3.5 seconds against 76 for the best competitor, and a hit rate of 93.2 % against 72.0 %.

And it publishes two negative findings that nobody else publishes. There are load regimes where affinity reduces capacity to between half and two thirds of the reference, to the point that the authors make a mirrored-traffic trial mandatory before deploying. And growing the fleet can cool a prefix even though total cache capacity grows, because the time between revisits stretches beyond the eviction window. Scaling horizontally can make the hit rate worse.

GORGO, June 2026, measures on a multi-region deployment with a dataset where intra-user prefix reuse is 89.4 % and the average prompt is almost 18,000 tokens. An improvement of 6.9 to 15.5 % in p95 TTFT. With extremely high reuse, the improvement is single-digit, because network latency and queuing dominate.

LAAR, from IBM Research Tokyo, April 2026, is the uncomfortable counterpoint: in its comparison, session affinity routing was the worst of the baselines, and at 64K contexts a load-based routing gives lower absolute latency at the cost of fewer hits.

The joint reading fits in three conditions. The big improvements require high, concentrated prefix reuse, long prompts, and a weak reference. Take away any of the three and the numbers deflate.

The five failure modes

Hot spot from affinity. The busiest bucket inherits the load’s skew, and tail latency starts following the hottest target instead of the fleet average. Mitigations exist in the four main products and in three of the four they ship switched off: Dynamo’s credit decay at 0, Ray’s imbalance threshold at infinity, and production-stack’s prefixaware with none at all.

Starvation of the freshly scaled pod. A new pod has no cache, so it never wins the scoring, so it never warms up. Dynamo documents it as the reason for the decay parameter. I found no primary documentation from any of the projects about the specific interaction with KEDA.

Cost of the index. The llm-d numbers in the previous table are the best public reference that exists. Raising the cap on tokens to match from 16,384 to 400,000 can more than double the EPP’s CPU with little traffic.

Traffic with unique prefixes. Each project degrades to something different. SGLang routes to the smallest tree, which is not the same as the least loaded. Ray falls back to power of two choices. Dynamo has an explicit preset, --load-aware, to say “I want the load model without reuse”. And production-stack’s prefixaware keeps inserting into the trie, that is, keeps paying the cost while receiving nothing.

Sibling bursts. Parallel sampling, best of N, or an agent that opens five branches at once. With event-driven routing, no engine has yet emitted the block stored event, all the siblings score zero overlap, and the prefix is precomputed on every replica. Dynamo has a prediction TTL for this; the others do not.

The architecture that comes out of all of the above

Two layers, with the responsibilities separated.

On top, LiteLLM, with what it does do well: non-forgeable identity through the virtual key, budgets, spend logging, concurrency limits, fallback between providers and API unification. For the gateway, each model group has a single destination: the URL of the router below.

model_list:
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://kv-router.inferencia.svc:8000/v1
 api_key: os.environ/VLLM_KEY

router_settings:
 routing_strategy: simple-shuffle
 num_retries: 1
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600

general_settings:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

Below, the KV-aware router, inside the engine’s domain, talking to the vLLM replicas and consuming their events.

Four consequences of this separation that have to be accepted before building it.

Per-replica visibility in LiteLLM is lost. For the gateway there is a single deployment, so its cooldowns, its per-deployment counters and its strategies stop meaning anything. Observability of which replica served what goes downstairs.

Retries stack up. The client SDK retries, LiteLLM retries, and the router below may have its own failover. In the previous article 45 calls per turn came out with three layers; with four it is worse. Lowering num_retries in the gateway to 1, or to 0, is the sensible decision when there is a router below that already retries.

LiteLLM’s session affinity stops making sense for the KV, because it no longer chooses a replica. It is kept if it is used for something else, such as pinning a conversation to a specific provider.

And the choice of the router below conditions the rest of the cluster far more than the gateway’s does. llm-d requires Kubernetes 1.32 or above, Gateway API, the inference extension and a conformant Gateway. Dynamo does not require Kubernetes. production-stack has a Helm chart and works outside it. That decision is the expensive one, not the gateway’s.

When not to build this

Three cases where the correct answer is not to do it, at least not yet.

A single replica per model. With one replica, the engine’s prefix cache already does all the work and there is nothing to route. That is the case for a good part of the four H100 deployments with one large model and tensor parallelism.

Low or scattered prefix reuse. Before building anything, the measurement is rate(vllm:prefix_cache_hits_total[5m]) / rate(vllm:prefix_cache_queries_total[5m]) per replica, with a mental correction for the bias from reprefills after eviction. If it is already high with blind distribution, reuse is concentrated inside each conversation, and there LiteLLM’s session affinity costs one line of configuration and resolves the case.

Traffic where the network dominates. The GORGO result is the warning: with enormous prompts and a distributed deployment, the improvement is eaten up by queuing and network latency.

The case where it does pay off is specific: several replicas of the same model, a large system prompt shared between different users, or a RAG with a corpus of documents that repeat across requests. There the prefill saved is real and it is most of the work.

Checklist

Measure the current hit rate per replica with the two vLLM counters before touching anything, and check whether the engine is evicting with vllm:num_preemptions_total.
Check that prefix caching is switched on. It is no longer --enable-prefix-caching that switches it on: the default value is derived from the model and the log of the decision is at DEBUG level. What is actionable today is --no-enable-prefix-caching to switch it off.
Audit the prompts before routing: a timestamp or a session identifier at the start of the system prompt breaks the match from block zero and no routing layer fixes it.
If the KV event stream is enabled, size the bandwidth by tokens per second and watch discarding at the high water mark. With data parallelism, subscribe to one port per rank.
If an exact router is chosen, verify the vLLM version against the one the replay buffer requires.
If llm-d is chosen, explicitly configure the vllm tokenisation backend. The default estimate does not fail, it degrades.
If production-stack is chosen with kvaware or loadaware, check first whether the traffic goes through /v1/chat/completions, because today it does not work there.
Review the load-based affinity-breaking parameter, because in three of the four main projects it ships disabled.
Lower num_retries in LiteLLM when there is a router below that already does failover.
Trial with mirrored traffic before changing production. It is the explicit recommendation of the only independent measurement that went looking for bad regimes, and found them.

Traps and things that are not what they seem

LiteLLM has no prefix routing in any version. Neither the open one nor the paid one. I got it wrong in June.
prompt_caching does nothing if the client does not send cache_control, and with the default automatic injection it fails on every turn because it marks the last message.
That pin’s TTL is 300 hardcoded seconds, incompatible with the one-hour caches that LiteLLM itself knows how to request.
production-stack’s prefixaware indexes characters, not tokens, in non-configurable chunks of 128, and its trie is never purged and models no eviction.
production-stack’s kvaware and loadaware do not support chat completions. They tokenise an empty string.
The default tokeniser in llm-d’s EPP is a byte estimate, and with it the exact scorer degrades silently.
llm-d’s active-active mode is discouraged with approximate prefixes, and the reference guides fix one replica even with the exact one.
Dynamo’s overlap credit decay ships at zero, which can leave a freshly autoscaled replica with no traffic.
The SGLang router’s default values differ between the Rust and the Python CLI, by as much as four orders of magnitude.
vllm:gpu_prefix_cache_hit_rate no longer exists and vllm:gpu_cache_usage_perc is now called vllm:kv_cache_usage_perc. Old panels and autoscalers are measuring nothing.
The hit rate counts the reprefill after eviction as a hit. A high number can be reuse or it can be thrashing.
KV events do not publish the hits by default, only the additions, unless full mode is requested request by request.
The hashes travel truncated to 64 bits unless it is disabled with an environment variable.
The default block size is 16, and the attention backend can change it without anyone asking.
The Endpoint Picker left the Gateway API Inference Extension repository. Looking for it there today leads nowhere.

Closing

June’s error has a reading that goes beyond the erratum. It came from reading a product’s documentation and not its code, and from the fact that LiteLLM’s documentation uses the word “caching” in a pre-call check that does something else. The distance between what a name suggests and what a function does is, at this point in the track, the most repeated pattern of the whole series.

The architectural conclusion is that KV-aware routing is not a gateway function. It is not one today in LiteLLM and probably should not be: it requires keeping an index synchronised with each engine’s memory, replicating its hash function, consuming an event stream and sizing CPU per request per second. That belongs to the engine’s domain, not to that of identity and budget.

What remains is a two-part decision. The first is whether the traffic deserves it, and that is answered with two Prometheus counters and without installing anything. The second is which of the three routers fits the cluster you already have, and there the deciding variable is not the hit rate but whether you are willing to bring in Gateway API and the inference extension, or to pin vLLM 0.13.0, or to operate a Dynamo frontend.

And the honesty the material demands: the only independent measurement that specifically went looking for regimes where this makes things worse found them, and they are not marginal. Half to two thirds of reference capacity is a serious regression. That no project publishes that side of the curve does not mean it does not exist in theirs.

Sources

LiteLLM, Router - Load Balancing (estrategias, comprobaciones previas, afinidad): https://docs.litellm.ai/docs/routing.
LiteLLM, código: litellm/router.py, litellm/router_utils/prompt_caching_cache.py, litellm/router_utils/pre_call_checks/prompt_caching_deployment_check.py, litellm/router_utils/pre_call_checks/deployment_affinity_check.py, litellm/proxy/litellm_pre_call_utils.py, litellm/proxy/middleware/admission_control_middleware.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 28427, TTL de afinidad por caché de prompts fijado a cinco minutos: https://github.com/BerriAI/litellm/issues/28427.
vLLM, código: vllm/v1/core/kv_cache_utils.py, vllm/v1/core/block_pool.py, vllm/v1/core/kv_cache_manager.py, vllm/distributed/kv_events.py, vllm/config/kv_events.py, vllm/v1/metrics/loggers.py, vllm/v1/metrics/stats.py: https://github.com/vllm-project/vllm.
vLLM, KV offloading usage (niveles CPU, filesystem y objeto; compartición entre procesos): https://docs.vllm.ai/en/latest/features/kv_offloading_usage.html.
vLLM, notas de la versión 0.29.0 (semilla determinista, seguimiento de propiedad en eventos, flags de admisión): https://github.com/vllm-project/vllm/releases/tag/v0.29.0.
vLLM, RFC 16669, publicación de bloques KV y métricas: https://github.com/vllm-project/vllm/issues/16669.
vLLM production-stack, código: src/vllm_router/routers/routing_logic.py, src/vllm_router/prefix/hashtrie.py, src/vllm_router/parsers/parser.py: https://github.com/vllm-project/production-stack.
LMCache Lab, resultados de production-stack frente a AIBrix (parte interesada): https://blog.lmcache.ai/en/2025/03/06/open-source-llm-inference-cluster-performing-10x-faster-than-sota-oss-solution/.
llm-d Router, código y documentación de operaciones (pesos por defecto, dimensionamiento del EPP, aviso de activo-activo): https://github.com/llm-d/llm-d-router.
llm-d, guía de enrutado preciso por prefijo y resultados publicados (parte interesada): https://github.com/llm-d/llm-d.
llm-d Router, incidencia 1290, estado de prefijo no compartido entre réplicas del EPP: https://github.com/llm-d/llm-d-router/issues/1290.
Red Hat, Same 16 GPUs, twice the users (parte interesada): https://www.redhat.com/en/blog/same-16-gpus-twice-users-inference-aware-routing-llm-clusters.
Kubernetes SIG Network, Gateway API Inference Extension (traslado del EPP y del BBR): https://github.com/kubernetes-sigs/gateway-api-inference-extension.
NVIDIA Dynamo, diseño y ajuste del router KV: https://github.com/ai-dynamo/dynamo.
SGLang, sgl-model-gateway: https://github.com/sgl-project/sglang/tree/main/sgl-model-gateway.
AIBrix, enrutado por caché de prefijo: https://github.com/vllm-project/aibrix/blob/main/pkg/plugins/gateway/algorithms/prefix_cache_readme.md.
Ray Serve LLM, enrutado consciente de prefijo (API en alfa): https://docs.ray.io/en/latest/serve/llm/user-guides/prefix-aware-routing.html.
Huang Cheng (Meta), CacheRoute, agosto de 2026: https://arxiv.org/html/2608.19677.
Yuan et al., DualMap, febrero de 2026: https://arxiv.org/abs/2602.06502.
Ricci Toniolo et al., GORGO, junio de 2026: https://arxiv.org/html/2602.11688.
Yoshimura, Chiba y van de Beek, Accuracy Is Speed (EuroMLSys ‘26), abril de 2026: https://arxiv.org/html/2604.15732v1.

Humans and agents on the same gateway: why `priority` does not prioritise, the 429 that brings down the whole pool, and the two ways to separate traffic

Thu, 10 Sep 2026 08:00:00 +0200

Fourth article in the operational track of the control layer. The pair with Langfuse covered observability, the proxy’s day 2 covered availability and virtual keys covered governance. Here comes the new tenant: agents, which speak the same protocol as humans and behave in a different way. Everything that follows is verified against LiteLLM 1.100.1 and vLLM 0.29.0, both from the first ten days of September 2026.

TL;DR

The day a team connects an agent to the corporate gateway, the platform changes regime without anyone touching a YAML. These are the seven conclusions.

LiteLLM’s priority parameter prioritises nothing under normal conditions. It is still marked as beta and its queue is only consulted when all the deployments in the group are in cooldown. With free capacity, the only thing it produces is a response header. On top of that, the success branch does not pop the item off the queue, so the list grows indefinitely; and on /chat/completions the value 0, which is the one used by all the documentation examples, is falsy in Python and disables queuing.

Separating traffic is done with two pools, not with priorities. The non-forgeable way is model_group_alias at key or team level: the client always asks for qwen-30b and the credential decides whether that resolves to qwen-30b-interactivo or to qwen-30b-agentes. The way that does not touch clients is tag_regex against the User-Agent, with a pattern like ^User-Agent: claude-code\/, which serves to classify load and not to control access, because that header is set by the client.

The useful admission control arrives in the 1.101 branch, with max_in_flight_requests_per_worker and a per-process queue that returns 503 with retry-after. It rejects before authenticating, so what it discards does not appear attributed to any key.

max_parallel_requests is only applied at virtual key level. The team, user, end customer and organisation endpoints accept the field and store it in the database, and nobody reads it. global_max_parallel_requests is a no-op unless the old limiter is reactivated, and general_settings.max_parallel_requests is read by no code path.

vLLM with --scheduling-policy priority does not push ahead anyone who is already generating. The only thing that changes is who it evicts when KV cache runs short. If the agents have filled max_num_seqs, a chat with the highest priority waits just the same. And the two knobs that in 2025 allowed short prompts to get ahead of long ones, max_num_partial_prefills and max_long_partial_prefills, disappeared from the code in 0.27.

The retry product reaches 45 calls per agent turn. Three from the client SDK times three from the router times up to five fallback groups. The proxy sets max_retries=0 on the SDK it uses itself, but it cannot touch the client’s. The timeout is per attempt, not per request, and LiteLLM’s Timeout comes out as a 408, which is retryable.

A single 429 takes a deployment out of the pool for five seconds, without going through the allowed_fails threshold. With an agentic burst against a fleet of four replicas, the whole pool enters cooldown and the gateway returns a 429 of its own to everybody, humans included.

You are here: the gateway layer with two classes of traffic

In the seven-layer stack this still lives in the gateway layer, with one difference compared to the previous three articles. There the variable was load: more requests, more teams, more budget to share out. Here the variable is the shape of the load.

A gateway sized for three hundred engineers using a chat works. The same gateway, with thirty of those engineers running coding agents, serves a fraction of the human traffic with the previous latency and the rest degrades. The GPUs stay just as busy, the utilisation metrics stay good, and the support team gets complaints from people waiting nine seconds for a paragraph.

The analogy: the registry desk and the courier with two hundred files

A registry office serves people in order of arrival. Citizens come in one at a time, present a piece of paper, and leave. The system is fair and the queue moves.

One Tuesday a courier turns up with a trolley of two hundred files. He joins the queue like anyone else, because rules are rules, and when his turn comes he occupies the desk for forty minutes. Behind him are twelve people with one piece of paper each. Nobody has broken any rule. The courier has the same right as everyone else and uses the same procedure. What fails is that the office was designed assuming a certain size of transaction and another one three orders of magnitude bigger has turned up.

The real solutions for an office are three, and all three have an exact equivalent in the platform. Open a desk for bulk deliveries, which is separating pools. Put up a capacity sign and send people back later, which is admission control with its 503. And say that files are handed in during the afternoon, which is the part that on an inference platform can almost never be applied, because the agent does not negotiate schedules.

What does not work in the office is giving the citizen a priority number when the courier is already at the desk. That number does not interrupt the transaction in progress. It is, literally, what priority does in the two layers of the platform, and it is worth keeping in mind while reading the next section.

The two kinds of traffic, measured

Before configuring anything, you have to be able to state the difference in numbers, because every later adjustment comes out of it.

Trait	Human chat	Agentic loop
Input per request	hundreds of tokens	tens of thousands, and it grows with each turn
Output per request	hundreds of tokens	from two tokens (a tool call) to several thousand
Concurrency per user	1	3 to 20 requests in flight
The metric that hurts	time to first token	total turn throughput
Tolerance for waiting	seconds	minutes
Cancelled requests	rare	routine, by the agent’s own decision
Retries	the browser’s	the SDK’s, plus the loop’s
Witness when it fails	the user	nobody

Two rows deserve comment because they contradict intuition.

The first is the metric row. An agent does not perceive time to first token, so on a shared engine it is the natural candidate to give up that metric. The operational conclusion is that the agent pool can be configured for aggressive throughput, with large batches and deep queues, while the interactive pool is configured for the opposite.

The second is the cancellations row. An agent that launches three speculative continuations and discards two generates two aborted streams per turn. At the gateway, that touches spend accounting, the release of concurrency counters and the open connection to the engine. LiteLLM 1.100 handles it reasonably well, and it did not six months ago.

What does not work: `priority` in LiteLLM

The first reaction of any operator is to look for a priority field. LiteLLM has one, documents it on its Request Prioritization page, and that page carries the beta label and the sentence saying it is for testing. The label is correct and has to be taken literally, because behind it there are four things.

The queue is only consulted when the whole group is in cooldown. The scheduler’s poll() method returns true if there is any healthy deployment, without touching the queue. With available capacity, which is the normal situation, the request goes straight through and priority orders nothing. The only thing that changes is that the response carries the x-litellm-request-prioritization-used header.

The queue does not empty. In that success branch there is no heappop and no call to the cleanup, which only happens in the expiry branch. The list under the key scheduler:queue:{model} grows without bound, is serialised whole into Redis on every insertion, and there is no maximum size and no discard policy. When the group does enter cooldown, the head of the heap is occupied by fossil entries from requests that finished hours ago, and new ones compare their identifier against that head, never match, and wait until the timeout runs out.

On /chat/completions, the highest priority disables the feature. The code reads the value with a kwargs.get("priority") or self.default_priority, and in Python 0 or None is None. Since in this scheduler the lowest number is the highest priority, priority: 0, which is the value in the documentation’s own examples, goes down the path without queuing. On /completions the check is done with is not None and it works; the two routes do not behave the same.

And default_priority in router_settings breaks the request. If the client does not send a priority but the global setting is in place, the value is truthy, the scheduling branch is entered and a function is invoked that requires the positional argument nobody has injected. The setting has no associated test and does not appear in the router_settings reference.

There is also a /queue/chat/completions, marked as experimental and hidden from the OpenAPI schema, which reads data["priority"] with no default value: a request without that field raises an exception that comes out as a 400 authentication error, with a message that has no relation to the cause.

There is a priority feature that is solid, and it is a different thing. The v3 dynamic limiter shares out token and request per minute quota between classes with weights, through litellm.priority_reservation with values like {"premium": 0.75, "standard": 0.25}. It shares out capacity, it does not order a queue: below the saturation threshold it lends free capacity and above it applies the weights. It never makes anyone wait, either it passes or it returns a 429. It requires Postgres and an enterprise licence, and it is still marked as beta.

What does work: two pools

Effective separation is done earlier, by deciding which set of replicas each class of traffic goes to. LiteLLM offers two mechanisms with very different security properties.

By credential: `model_group_alias` at key or team level

Router settings are resolved in the order key, team, global. Among those that accept that override is model_group_alias, and there lies the piece:

model_list:
 - model_name: qwen-30b-interactivo
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 model_info:
 id: chat-01
 max_input_tokens: 32768
 - model_name: qwen-30b-agentes
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 model_info:
 id: agentes-01
 max_input_tokens: 131072

And on the agent team’s key:

curl -X POST "$PROXY/key/generate" \
 -H "Authorization: Bearer $MASTER_KEY" \
 -d '{
 "team_id": "plataforma-agentes",
 "models": ["qwen-30b"],
 "router_settings": {
 "model_group_alias": {"qwen-30b": "qwen-30b-agentes"}
 }
 }'

The client asks for qwen-30b and knows nothing about any of it. The resolution is tied to the credential, so the caller cannot get around it by changing a header. The same list allows routing_strategy, fallbacks, context_window_fallbacks, retry_policy, cooldown_time and allowed_fails to be set per key or team, which gives different reliability policies for each class of traffic on the same proxy instance.

It should not be confused with access groups (model_info.access_groups), which are permissions and not routing: they decide whether a key can invoke a model name, not which replica the request goes to.

By `User-Agent`: `tag_regex`

When separate credentials cannot be issued, because the agentic client uses the same personal key as that person’s chat, LiteLLM classifies by regular expression against the request headers:

model_list:
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 tag_regex: ["^User-Agent: claude-code\\/", "^User-Agent: .*codex"]
 model_info: {id: agentes-01}
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 tags: ["default"]
 model_info: {id: chat-01}

router_settings:
 enable_tag_filtering: true
 tag_filtering_match_any: true

The proxy composes strings of the form User-Agent: <value> and tests them with re.search. Exact tag matching takes precedence over the regex. The documentation itself carries a warning that has to be respected: the User-Agent is written by the client, so this is traffic classification and not a security boundary. As load sharing it is enough, because an agent that lies about its identity to sneak into the interactive pool is a governance problem, not a routing one.

The behaviour when no deployment carries the requested tag has a nuance that comes as a surprise in production. If the tag is unknown to the group, the request falls through to the pool marked with tags: ["default"]. If the tag exists in the group but no healthy deployment carries it at that moment, the request fails with the tag configuration error, even though a default pool is available. To get deterministic fallthrough to the default pool you have to set model_info.allow_fail_open: true.

A different strategy per pool with `routing_groups`

router_settings:
 routing_strategy: simple-shuffle
 routing_groups:
 - group_name: interactivo
 models: [qwen-30b-interactivo]
 routing_strategy: simple-shuffle
 - group_name: agentes
 models: [qwen-30b-agentes]
 routing_strategy: simple-shuffle

The recommendation to leave simple-shuffle on both is not laziness, and sustaining it requires looking at what the alternatives do on a homogeneous vLLM fleet.

least-busy keeps a complete dictionary under a single key and updates it with non-atomic reads and writes, without using INCR, so with several proxy replicas increments are lost. It has no TTL, so a request that never completes leaves the counter raised forever and that replica is permanently excluded. Counters can go negative. And when the minimum corresponds to an identifier that is no longer in the healthy list, it falls back to random.choice without warning. The pattern of requests that do not complete is exactly that of an agent that cancels.

latency-based-routing measures seconds per output token, not wall-clock latency. A chat turn of twenty tokens looks extremely slow by that metric and an agent generation of two thousand tokens looks extremely fast, so mixing both kinds of traffic in the same group poisons the signal. It also seeds new deployments with zero latency, and with the buffer defaulting to zero, a freshly started replica takes all the traffic until its first success callback.

usage-based-routing-v2 uses minute windows that reset all at once, and it accounts for tokens when the request completes, so long generations in flight are invisible to the router. With long outputs, it systematically underestimates the real load.

cost-based-routing sorts by the sum of the input and output unit prices and takes the first, with no random tie-break. With replicas of the same model at the same price, all the traffic always goes to the same place. And a model that is not in the price map gets a default cost of 5.0 per token, a figure designed to deprioritise it.

Admission control: the new piece in the 1.101 branch

When the agent pool saturates, what you want is to reject fast and with a code the client understands, not to accumulate requests inside the proxy until memory runs out. That arrives with per-worker admission control, added in the 1.101 branch, in release candidate as this is written.

general_settings:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

It is a semaphore with a per-uvicorn-process queue, without Redis. When the queue fills or the wait expires it returns 503 with retry-after: 1 and a body with "type": "overloaded_error". It is observed through /health/backlog, which exposes requests in flight, admitted, queued and rejected, and through the Prometheus litellm_admission_* metrics, whose rejection counter distinguishes queue_full from queue_timeout.

Two properties have to go into the runbook. The middleware rejects before authenticating, so what is discarded is not attributed to any key in the spend records, and the real number of requests rejected per team cannot be reconstructed from the spend table. And the limits are read on the first request: changing them requires a restart.

Concurrency per key, and the three settings that do nothing

max_parallel_requests is the right brake for a specific agent, and its configuration surface is full of dead ends.

The only level where it is applied is the virtual key. The v3 limiter, active by default since 1.94, has descriptors for user, team, team member, end customer, organisation, model per key, tag and agent, and all of them carry only request and token counters per window. The only reference to max_parallel_requests is the one in the key descriptor. The old limiter says so in a comment: supporting it for model, user and team is pending.

At the same time, /team/new, /user/new, /customer/new and the organisation endpoints accept the field, validate it and write it to Postgres, and the documentation shows it in their examples. Nobody reads it afterwards. A team with max_parallel_requests: 20 in its row has no concurrency limit at all.

global_max_parallel_requests is injected into the request metadata and is only consulted by the old limiter, which is disabled unless LEGACY_MULTI_INSTANCE_RATE_LIMITING=true is set. The server settings documentation still describes it as a global limit coordinated through Redis. And general_settings.max_parallel_requests does not appear in any execution path: its only two appearances are the copy from the database and the type schema of the interface.

What does work, in detail:

curl -X POST "$PROXY/key/generate" -H "Authorization: Bearer $MASTER_KEY" \
 -d '{"team_id":"plataforma-agentes","max_parallel_requests":8,"rpm_limit":600}'

Counting is done with a sorted set in Redis scored by Redis’s own clock, with Lua scripts to acquire and release. Each slot expires after 3,600 seconds, which is at once the maximum request duration the meter can track and the time a leaked slot takes to heal by itself. Release is hooked to success, to failure and to client disconnection during streaming, which was the classic leak with clients that cancel.

One live leak remains and it affects exactly the case in this article: every tool call through the MCP gateway acquires a slot and does not release it. The MCP route invokes the pre-call hook, which acquires, and none of the release functions is called from that module. With an agent that makes twenty tool calls per turn and a key with eight slots, the 429 arrives on the first turn and lasts an hour.

The order of evaluation matters for interpreting 429s. First the window counters, requests and tokens per minute, are checked, and only if they pass is the concurrency slot acquired. The code comment explains why: the other way round, every rate limit rejection would leave an orphan slot. In practice, with a low rpm you always see the requests type 429, and the max_parallel_requests one only appears when the relationship between concurrency and duration makes it binding first. The message distinguishes the two cases with the Limit type: field and the rate_limit_type header.

A minor detail with consequences on the client: the response to a concurrency rejection carries retry-after and reset_at computed from the window size, sixty seconds by default, when a concurrency meter has no window. A well-behaved client that respects that header will wait a minute when it might have had a free slot in two seconds.

Session affinity: the KV cache that gets thrown away

An agent turn resends the whole conversation context, system prompt and tools included. If that request lands on a different replica from the previous turn’s, vLLM’s prefix cache does not have the prefix and the prefill is recomputed in full. With twenty turns and four replicas balanced blindly, the long prefill is paid for fifteen times over.

LiteLLM has two mechanisms for this and they are not equally good.

The prompt_caching check computes a SHA-256 of the cacheable prefix, defined as everything up to the last block with cache_control, and pins the deployment with that hash. It has two problems. The TTL is hardcoded to 300 seconds in two places in the code, with no way to configure it, something which is reported and open; with one-hour caches, affinity is lost after five minutes of inactivity. And the key is an exact hash, so a client that moves the cut point forward on every turn, which is exactly what agents do to extend the cached prefix, generates a new key every time.

The session_affinity check is the one to use:

router_settings:
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600

It pins the deployment with an atomic Lua script in Redis, with a configurable TTL that defaults to one hour and that is refreshed on every request, so it bounds the idle time between turns and not the duration of the conversation. If the pinned deployment enters cooldown, the request goes out through the normal strategy and keeps the pin so it can come back when it heals.

The session identifier comes from a precedence chain that starts with x-litellm-trace-id, continues with x-litellm-session-id and then accepts any header of the form x-<something>-session-id whose value looks like an identifier. That includes the session header Claude Code already sends, with nothing configured on the client. Whatever matches populates both the session identifier and the trace identifier, which links this to the correlation from the first article in the track.

One dangerous interaction when combining it with tags: affinity is evaluated before tag-based routing. If the pin reduces the list to a deployment that does not satisfy the request’s tag, the request fails instead of falling through to the default pool, unless allow_fail_open has been enabled. Retagging a pool with live sessions breaks them.

And a clarification that saves a pointless search: LiteLLM does not do prefix routing. There is no prefix tree and nothing that directs the request to the vLLM replica that already has that prefix warm. What there is, is affinity by session or by key. For the other thing you need a specific router underneath the gateway, and that architecture has its own article pending.

The engine: what vLLM can and cannot do

Everything above shares out requests between engines. Inside one engine, the question is what happens when a twenty-token request and a sixty-thousand-token one live side by side.

Priority exists and does less than it looks like. --scheduling-policy priority turns the waiting queue into a heap ordered by priority, arrival time and identifier, with the lowest number first. The request can carry the priority field in the body, and there is also the X-Vllm-Priority header, which takes precedence over the body and is not documented anywhere.

What priority does not do is push ahead of whoever is already generating. In the running queue loop, when KV cache runs short, the policy only changes who is chosen as the victim of eviction: with priority the one with the worst priority and arrival time is evicted, and with fcfs the last in the queue. In the waiting queue loop, if there are no blocks available the iteration is cut short, without evicting anyone. And the max_num_seqs cap is a hard cut. If the agents have filled that cap, a chat with maximum priority waits just the same. There has been an open issue since April and an unmerged pull request about this.

That nuance about the victim is not negligible, because with fcfs the victims are the most recently admitted, that is, the chats that have just come in. Switching the priority policy on already improves the situation even if it pushes nobody ahead.

Eviction is always by recomputation. The swap-to-host-memory mode and the enumeration that controlled it disappeared from the code; all that is left is freeing blocks and returning the request to the queue with the computed token counter at zero. An evicted request loses all its prefill, and the prefix cache recovers part of it, but the perceived time to first token restarts.

Two admission settings help and do not appear in the narrative documentation. watermark reserves a fraction of free KV blocks on admission, with the declared purpose of avoiding repeated eviction, and it ships disabled. And scheduler_reserve_full_isl, active by default, checks that the full input length fits in the cache before admitting, instead of looking only at the first chunk.

Chunked prefill is active by default and prioritises decode. The scheduler walks the whole running request loop before touching the waiting queue, so a long agent prefill does not freeze a chat’s generation in progress. That protects the inter-token latency of what has already been admitted, and it does not protect the time to first token of what is waiting.

Here there is a breaking change that forces a review of the manifests. max_num_partial_prefills and max_long_partial_prefills were removed in 0.27. They were precisely the mechanism that allowed short prompts to get ahead of long ones. A deployment with those flags in its args does not start. long_prefill_token_threshold still exists, and its meaning changed: today it is a per-request chunk size cap, not a classification of long requests.

The partial replacement arrives in 0.29 with max_num_queued_tokens and max_num_queued_reqs, admission valves that return 503 when the pending prefill work exceeds the target. The code itself frames it as a quality of service mechanism for time to first token, with the formula of setting it to the latency target multiplied by prefill throughput. They are blind to the class of traffic: they reject a chat just the same as an agent.

And the default value of --max-num-batched-tokens depends on the hardware and on the start-up mode. With vllm serve on H100 or H200 it is 8,192, on cards of 160 GiB or more it is 16,384, and on the rest 2,048. The 2,048 that appears in the scheduler configuration only applies to tests.

The metrics for diagnosing coexistence, with the exact names from 0.29:

Metric	What for
`vllm:num_requests_waiting`	queue depth, the autoscaling signal
`vllm:num_requests_waiting_by_reason`	separates waiting for capacity from waiting in a blocked state
`vllm:num_preemptions_total`	evictions, the signal that the two classes are treading on each other
`vllm:kv_cache_usage_perc`	cache pressure, the cause of the evictions
`vllm:request_queue_time_seconds`	what a chat waits with the agents inside
`vllm:prefix_cache_hits_total` and `_queries_total`	measured in tokens, not in requests

A warning for anyone with old alerts. The log warning about evicted sequences that appears in the optimisation documentation belongs to the previous version of the engine and no longer exists in the code. Preemptions are only visible as a counter. An alert based on searching for that string in the logs has not fired for months, and not because there are no evictions.

On isolation between classes inside one engine, the short answer is that it does not exist. There are two queues and neither is per class of traffic. The conclusion, which is the one that holds up this article’s architecture: effective isolation means separate fleets, and on top of them the gateway deciding who goes to which.

Retries: the product nobody calculates

A human who gets an error sees it and decides. An agentic loop retries, and beneath it there are two more layers that also retry.

Layer	Default
Client SDK	2 retries, 3 attempts
LiteLLM router	2 retries, 3 attempts
Fallbacks	up to 5 groups

The product is about 45 calls to the engine for one agent turn, before counting the loop’s own retry. LiteLLM does what it can: when the call comes from the router, it forces max_retries=0 on the SDK it itself uses against the engine. What it cannot touch is the client’s SDK. The only layer that breaks the product is that one, and it is broken by setting max_retries=0 on the client and delegating the retry to the gateway.

The retryable errors are 408, 409, 429 and anything 500 or above. And there lies the problem with long reasoning: litellm.Timeout is a subclass of OpenAI’s timeout exception and comes out with code 408, which is retryable, so the most expensive request of all, the one that expired after ten minutes of generating, is launched again in full. Twice by the router, and as many times again by the client.

On timeouts you have to be precise because the code comment is: the configured timeout is per attempt, not per request. A request_timeout: 600 with two retries and two fallback groups gives a worst case of almost two hours before the client receives anything. The default value is still 6,000 seconds, as was already covered in the day 2 article.

The retry storm has two specific causes and both are default settings.

The first: a single 429 takes the deployment out of the pool, without going through the allowed_fails threshold. The cooldown logic has an explicit branch which, on a 429 in a group with more than one deployment, returns immediate cooldown. With an agentic burst against four replicas, all four leave the pool at once and the gateway answers everybody with its own 429 of “no deployments available”. The default cooldown is five seconds, which in a sustained burst gets renewed.

The second: backoff is zero while the pool looks healthy. The function that computes the wait returns 0 if the healthy deployment list is not empty. Exponential backoff with jitter, capped at eight seconds, only kicks in when there is nothing healthy left. Under partial degradation, the proxy hammers away without waiting and only slows down once everything has already fallen over.

On top of that, the engine’s Retry-After header is only respected if its value is between 1 and 60 seconds. A longer limit window is ignored and replaced by the proxy’s own backoff.

The settings that cut the problem off:

router_settings:
 num_retries: 1
 retry_after: 2 # real floor; the default 0 is hammering
 cooldown_time: 30
 allowed_fails_policy:
 RateLimitErrorAllowedFails: 3 # stops a 429 taking the deployment out
 retry_policy:
 TimeoutErrorRetries: 0 # do not relaunch the expensive request
 AuthenticationErrorRetries: 0
litellm_settings:
 request_timeout: 900 # per attempt
 drop_params: false

That last setting deserves an explanation, because it is a silent way of breaking an agent. With drop_params: true, LiteLLM removes from the body any parameter the destination provider does not support, and in that branch it writes no warning at all. The candidates for disappearing are tools, tool_choice, response_format and parallel_tool_calls. An agent that has tools taken away receives free text where it expected a function call, and interprets it as the model having decided not to use tools.

Worse still is prompt-based emulation. For a non-OpenAI-compatible provider that does not support tools, LiteLLM does not fail: it turns the tools into text inside the prompt, forces JSON format, discards tool_choice, and on the way switches on a process-global variable that makes any other tool-less provider in the same process go to emulation instead of erroring.

With that in mind, the warning about fallbacks is clear: LiteLLM does not check capabilities before falling back. The function that knows whether a model supports function calling exists and is only used in an informational endpoint. The pre-call checks filter by context window, request limit and region, and nothing else. A fallback list that goes from a model with tools to one without them degrades the agent silently. There are guards for provider-scoped resources, files and batches, so the absence of the capability guard is an omission and not a design oversight.

A month’s budget in twenty minutes

A runaway agent consumes in minutes what a team consumes in weeks, and the operational question is how long the gateway takes to find out.

The answer has changed, and the old mental model, the one where spend consolidates every minute and until then there is no brake, is no longer correct. Since 1.84 there are two new pieces. An optimistic reservation before the call, which estimates the maximum cost of the request and reserves it atomically in the counter before heading out to the engine, reconciling it with the real cost on completion. And a counter increment after the call that is explicitly awaited, with the comment that this way the counter is up to date before the next request goes through authentication. Budget checks read that counter, with Redis first and seeding from the database when cold.

Batch writing to Postgres still exists, every 10 to 15 seconds, and it affects persistence and what is shown in the interface, not the cut-off. The production documentation’s recommendation to raise that interval to 60 does not relax the control, and it does delay by a minute what /key/info shows, something which frequently generates operational panic. And the budget rescheduler’s ten minutes do not consolidate spend: they fire the job that resets expired budgets.

The overspend window then comes down to the cost of the requests in flight, provided there is Redis. Without Redis, the counters are per pod, and with N replicas you can spend up to N times the budget. That is the configuration failure that matters and it is the same Redis that the day 2 article declared non-optional for other reasons.

Two settings that do not appear in the public documentation, only in the data model descriptions, and that decide behaviour at the worst moment:

general_settings:
 fail_closed_budget_enforcement: true # false by default: if the counter fails, it lets things through
 # disable_budget_reservation: do not set it to true

With the default value, if the counter backend does not respond, the proxy fails open and carries on serving. For an agent in a loop that is exactly the opposite of what you want.

Time for the honest warning: this route is still unstable. There are open issues in both directions, one where the key and user budget limit stops being applied while spend is recorded, and another where a 429 is returned for stale spend in the reservation counters while the key query shows spend below the limit. Project budgets are not on the atomic reservation route and concurrency overshoots them. The practical conclusion is not to trust the hard cut-off to the budget alone.

On the manual cut-off, there is an asymmetry to know about before you need it. /key/block invalidates the local cache, the Redis one, and notifies all workers by publish, so with Redis the effect is almost immediate. /customer/block does the same. /team/block only writes to the database, without invalidating or notifying, so it takes however long the management object cache TTL takes, sixty seconds by default. To cut a team off dead you have to block its keys one by one, or lower user_api_key_cache_ttl. And /user/block no longer exists, even though the docstring of the function that replaced it still shows that route in its example.

On alerts, two numbers. The default thresholds warn at 85 % and 95 % of the budget. And deduplication of those alerts is 24 hours per event and identifier, so crossing 85 % generates one warning and then silence for a day. For agentic traffic, where 85 % and 100 % can be twenty minutes apart, the second warning arrives when it is no longer any use.

Attribution: who the user is when the caller is an agent

With agents, the question of who the spend is charged to has two answers and both are needed: the agent that executes and the human on whose behalf it acts.

The chain that extracts the end customer identifier tries the standard headers first, then the configurable header mapping, and only then the user field of the body, the litellm_metadata.user of the Anthropic format, the metadata.user_id and the safety_identifier. The last four are body fields and are therefore written by the client. The code itself warns about it: for untrusted callers the identifier has to be set from headers or from a middleware on the server, to avoid impersonation.

Out of that comes the correct pattern, which is the inverse of the one people put together out of inertia: the virtual key identifies the agent and is the non-forgeable part, and the header or the user field identify the human and serve for attribution, not for control. An agent that lies about the human behind it is still bounded by the budget and the limits of its key.

The spend table stores user, end_user, team_id, organization_id, request_tags, session_id and agent_id, with daily aggregation per agent, so the detail is available once it is sent. The x-litellm-agent-id header is what populates that last field.

A note about the MCP gateway, which on a platform with agents ends up being the other entry door. Tool calls do appear in the spend records, with call types of their own and a metadata block that stores tool name, arguments, result and server. What is missing is token counting: the cost is 0.0 by default and is only populated if the administrator configures a price per query, per server or per tool. It is the same pattern of silent zero cost that already came up with unpriced models in the first article in the track.

And the permission setting to review on day one: access control for MCP servers is open by default. If no level of the hierarchy defines a list, the request reaches every configured server. general_settings.require_key_mcp_access_defined: true inverts that inheritance, and it is not the default value.

A reference `config.yaml`

All of the above, together, on a fleet split into two pools:

model_list:
 - model_name: qwen-30b-interactivo
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 input_cost_per_token: 0.00000018 # without this, silent zero cost
 output_cost_per_token: 0.00000072
 model_info:
 id: chat-01
 max_input_tokens: 32768 # without this there is no context filter

 - model_name: qwen-30b-agentes
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 input_cost_per_token: 0.00000018
 output_cost_per_token: 0.00000072
 tag_regex: ["^User-Agent: claude-code\\/"]
 model_info:
 id: agentes-01
 max_input_tokens: 131072
 allow_fail_open: true

router_settings:
 routing_strategy: simple-shuffle
 enable_pre_call_checks: true
 enable_tag_filtering: true
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600
 num_retries: 1
 retry_after: 2
 cooldown_time: 30
 allowed_fails_policy:
 RateLimitErrorAllowedFails: 3
 retry_policy:
 TimeoutErrorRetries: 0
 AuthenticationErrorRetries: 0

general_settings:
 fail_closed_budget_enforcement: true
 require_key_mcp_access_defined: true
 user_api_key_cache_ttl: 10
 proxy_batch_write_at: 10
 # 1.101 branch onwards:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

litellm_settings:
 request_timeout: 900 # per attempt
 drop_params: false
 cache: true # Redis, not optional

And the start-up of the two engines, with the deliberate asymmetry:

# interactive pool: protect the first token
vllm serve Qwen/Qwen3-30B \
 --scheduling-policy priority \
 --max-num-seqs 64 \
 --max-model-len 32768 \
 --max-num-batched-tokens 4096

# agent pool: throughput and deep queues
vllm serve Qwen/Qwen3-30B \
 --max-num-seqs 256 \
 --max-model-len 131072 \
 --max-num-batched-tokens 8192

Checklist

Issue separate keys for agents, with model_group_alias at key or team level. It is the only separation the client cannot evade.
Add tag_regex over the User-Agent as a safety net for agents that use personal credentials, with a default pool and allow_fail_open: true.
Leave simple-shuffle on both pools. The other four strategies have verifiable defects on homogeneous replicas.
Enable session_affinity with a one-hour TTL. Do not use prompt_caching for this while its TTL stays hardcoded.
Declare max_input_tokens on every deployment. Without that value, the context window filter is not applied and it fails silently for models that are not in the price map.
Declare per-token prices on every self-hosted model, or the agents’ spend will be zero and their budget will never run out.
max_parallel_requests on the agent’s key, never on its team. On the team it is stored and not applied.
Lower num_retries to 1, put a floor on retry_after, raise cooldown_time and disable retry on timeout. On the client, max_retries=0.
fail_closed_budget_enforcement: true and mandatory Redis. Without Redis, N replicas are N times the budget.
Start the interactive engine with --scheduling-policy priority even though it pushes nobody ahead: it changes who it evicts.
Alert on vllm:num_preemptions_total and vllm:num_requests_waiting_by_reason, not on log strings that are no longer emitted.
Prepare the cut-off: block by key, never by team, and verify the real propagation time in your own deployment.

Traps and things that are not what they seem

priority: 0 on /chat/completions disables prioritisation. It is the value in the documentation examples and it is falsy in Python.
max_parallel_requests on a team is stored and not applied. The field exists in the API, in the database and in the documentation; the limiter only reads it from the key descriptor.
global_max_parallel_requests is a no-op with the default limiter, and the server settings documentation still describes it as active.
Every MCP call leaks a concurrency slot, which only heals when its hour expires. With agents calling tools in a loop, the 429 appears early and lasts a long time.
The TTL of prompt cache affinity is 300 hardcoded seconds. For agent conversations, session affinity is the right option.
Affinity is evaluated before tag-based routing. A pin to a deployment that stops satisfying the tag makes the request fail instead of falling through to the default pool.
LiteLLM’s timeout is per attempt. The worst case is the timeout multiplied by attempts and by fallback groups.
A 429 takes the deployment out of the pool without going through allowed_fails, and backoff is zero while anything healthy remains.
The engine’s Retry-After is ignored if it asks for more than 60 seconds.
drop_params: true removes tools and response_format without leaving a warning, and the fallback does not check whether the destination model supports function calling.
/team/block does not invalidate the cache. It takes the full TTL; /key/block does propagate by publish in Redis.
Access to MCP servers is open by default if no level defines a list.
max_num_partial_prefills and max_long_partial_prefills no longer exist in vLLM. A manifest with those flags does not start from 0.27 onwards.
vLLM’s priority does not push ahead of running requests, and with max_num_seqs full it pushes nobody ahead.
The eviction log warning that appears in the vLLM documentation is no longer emitted. Only the Prometheus counter is left.

Closing

The coexistence of humans and agents on the same platform is not solved with a priority field, and that is the conclusion that takes the most effort to accept because the field exists and looks made for this. Neither the gateway nor the engine has a usable notion of class of service today: one orders a queue that is almost never consulted, the other picks who to evict. What they do have is the ability to direct each request to a different place, and that is where the win is.

The architecture that comes out of all of the above fits in one sentence. Two vLLM fleets with opposite parameters, a credential that decides which one each request goes to, session affinity so as not to throw away the prefill work, concurrency limits on the agent’s key and not on its team, retries trimmed back on the three layers where they multiply, and budgets that fail closed.

The rest is measurement. If vllm:num_preemptions_total rises in the interactive pool, the two classes are treading on each other despite the separation. If chat queue time grows while total throughput holds, the partition is badly sized. And if the agents’ spend comes out as zero, the problem is not coexistence but undeclared prices, which is where this track started.

Sources

LiteLLM, [BETA] Request Prioritization: https://docs.litellm.ai/docs/scheduler.
LiteLLM, Router - Load Balancing (estrategias, afinidad de sesión, comprobaciones previas): https://docs.litellm.ai/docs/routing.
LiteLLM, Tag Based Routing (incluye tag_regex y el ejemplo con clientes agénticos): https://docs.litellm.ai/docs/proxy/tag_routing.
LiteLLM, Router Settings for Keys and Teams (resolución clave, equipo, global): https://docs.litellm.ai/docs/proxy/keys_teams_router_settings.
LiteLLM, Server Tuning (control de admisión por worker): https://docs.litellm.ai/docs/proxy/server_tuning.
LiteLLM, Dynamic TPM/RPM Allocation (reserva de prioridad, beta y enterprise): https://docs.litellm.ai/docs/proxy/dynamic_rate_limit.
LiteLLM, Budgets, Rate Limits: https://docs.litellm.ai/docs/proxy/users.
LiteLLM, Production Best Practices: https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Reliability (fallbacks, ventana de contexto, enfriamientos): https://docs.litellm.ai/docs/proxy/reliability.
LiteLLM, código: litellm/scheduler.py, litellm/router_strategy/tag_based_routing.py, litellm/router_utils/pre_call_checks/deployment_affinity_check.py, litellm/proxy/hooks/parallel_request_limiter_v3.py, litellm/proxy/middleware/admission_control_middleware.py, litellm/router_utils/cooldown_handlers.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 28427, TTL de la afinidad por caché de prompts fijado a cinco minutos: https://github.com/BerriAI/litellm/issues/28427.
LiteLLM, incidencia 34534, fuga de slot de concurrencia en cada llamada MCP: https://github.com/BerriAI/litellm/issues/34534.
LiteLLM, incidencia 26672, presupuesto de clave y usuario no aplicado: https://github.com/BerriAI/litellm/issues/26672.
LiteLLM, incidencia 872, comprobación previa de soporte de llamada a funciones, cerrada sin implementar: https://github.com/BerriAI/litellm/issues/872.
vLLM, Optimization and Tuning (prefill troceado, prioridad de decodificación): https://docs.vllm.ai/en/v0.29.0/configuration/optimization.html.
vLLM, Metrics design (nombres, convención _total, autoescalado como problema abierto): https://docs.vllm.ai/en/v0.29.0/design/metrics.html.
vLLM, Data Parallel Deployment (caché KV independiente por réplica): https://docs.vllm.ai/en/v0.29.0/serving/data_parallel_deployment.html.
vLLM, código: vllm/v1/core/sched/scheduler.py, vllm/v1/core/sched/request_queue.py, vllm/config/scheduler.py, vllm/engine/arg_utils.py, vllm/v1/metrics/loggers.py: https://github.com/vllm-project/vllm.
vLLM, incidencia 40004, la prioridad no desaloja peticiones en ejecución: https://github.com/vllm-project/vllm/issues/40004.

LiteLLM on day 2: one worker per pod, six thousand seconds of timeout and the other five things to change before opening the traffic

Tue, 08 Sep 2026 07:00:00 +0200

Second article in the operational track of the control layer. The operational pair with Langfuse covered the seam between the gateway and observability; here everything stays inside the gateway. The prior decisions, which gateway and why, are in choosing the OSS gateway and in the L7 inference router.

TL;DR

LiteLLM Proxy starts from a twenty-line config.yaml and serves traffic that same afternoon. The operational work consists of correcting seven defaults designed for a scenario other than an in-house inference factory.

One uvicorn worker per pod. The project’s production guide asks for --num_workers 1 on Kubernetes and horizontal scaling, with 1 vCPU and 4 GiB per worker set at the same time as requests and as limits. The 4 GiB are a floor, not a target: the Prisma query engine marks a resident memory high-water mark that grows up to the largest statement it has executed and that glibc does not return to the system.

Autoscale on CPU, never on memory. A direct consequence of the above. Memory goes up and never down, so an HPA on memory scales and never scales back. The recommended target is targetCPUUtilizationPercentage: 60.

The Postgres connection arithmetic. database_connection_pool_limit is 10, and the number of connections is instances times workers times the pool. The Helm charts ship maxReplicas: 100, on the order of a thousand connections, far above what a stock Postgres accepts.

Redis stops being optional from around a thousand requests per second or ten instances. Without it, each instance applies its limits on its own, caches are not shared, and spend updates against the same rows produce locks and exhaust the connections with FATAL: sorry, too many clients already.

Three health endpoints and only two are usable as probes. /health/liveliness and /health/readiness require no authentication and answer fast. Bare /health demands a key and fires a real request against every model in the catalogue, with its cost in tokens.

The resilience numbers are 2, 3 and 5: two retries, three allowed failures and five seconds of cooldown, the last one despite the function’s own documentation saying the default is 1.

request_timeout is 6,000 seconds. A hundred minutes holding a connection against an engine that does not answer.

And a warning about the performance figures: the project publishes 8 ms of p95 overhead on one benchmark and 257.7 ms of p99 on another. Both are theirs, and the difference is in the shape of the load.

You are here: the gateway layer, the day after

In the seven-layer stack, this article lives entirely in the gateway layer. The difference with the selection article is the one that separates buying from maintaining: there the question was which piece to put in front of the fleet, and here it is what happens when that piece has been serving for six months, the team has tripled the traffic and someone has added an agent that fires twenty calls per interaction.

The gateway has an uncomfortable property worth keeping in mind: it sits on the critical path of every request and contributes not a single token. All it does is coordination, and any coordination that fails turns into an outage of the whole inference service, even with the GPUs perfectly healthy.

The analogy: the switchboard and its notebook

A hospital switchboard does three jobs at once. It puts calls through, which is the urgent part. It writes down in a notebook who called, to which extension and for how long, which is what sustains the billing. And it consults a list of extensions to know where to put each call through.

The three jobs compete. If the operator stops to write each line in the notebook before putting the next call through, the waiting queue grows. If there are three operators and each keeps her own notebook, the totals do not add up. And if the list of extensions is consulted in a filing cabinet on another floor, every call costs the trip.

The three tensions have an exact equivalent in LiteLLM. The notebook is the SpendLogs in Postgres, which is why the transaction buffer in Redis exists. The three operators with three notebooks are the pod replicas applying limits independently. And the filing cabinet on another floor is usage-based routing, which adds a Redis lookup inside the request path.

The process model: one worker per pod

The production documentation is explicit: on Kubernetes, --num_workers 1 and scale with replicas. On a virtual machine with no orchestrator, NUM_WORKERS equal to the number of vCPUs.

The reason is not ideological. An additional worker inside the same pod shares the container’s memory limit with the others, multiplies the database connections by the same factor, and registers a copy of every periodic job. With separate replicas, each one has its own resource budget, its own connection quota and a place where the scheduler can put it.

The recommended sizing is 1 vCPU and 4 GiB per worker, and the part that gets overlooked is that these go as requests and as limits at the same time. Setting less than 4 GiB produces an OOM restart loop that shows up late, once some large request has grown the query engine’s footprint.

resources:
 requests: { cpu: "1", memory: "4Gi" }
 limits: { cpu: "1", memory: "4Gi" }

That footprint needs to be described precisely, because it determines the autoscaling policy. The Prisma query engine keeps a resident memory that behaves as a high-water mark: it grows up to the largest statement the process has executed and does not come back down, because glibc does not return that freed memory to the system. A pod that served a spike yesterday still shows the footprint of that spike today.

Two rules come out of this. The first: autoscale on CPU and never on memory. An HPA on memory sees a metric that only goes up, scales at the peak and never scales back. The recommended target is targetCPUUtilizationPercentage: 60, lower than the 80 the charts ship by default, because the startup probe allows up to 300 seconds before passing the first readiness check and there needs to be headroom for the new pod to be ready before the saturated one goes down.

The second: bound the process lifetime. --max_requests_before_restart 10000 recycles the worker before the high-water mark matters.

Postgres: the arithmetic that blows up when you scale

database_connection_pool_limit is 10 by default. The total number of connections the deployment opens is replicas times workers times pool, and it has to be compared against the Postgres max_connections.

The formula the documentation gives is the inverse, and it is the one to apply when sizing:

database_connection_pool_limit = MAX_DB_CONNECTIONS / (instances × workers)

The detail that hurts in production is in the Helm charts, which ship autoscaling.maxReplicas and keda.maxReplicas at 100. With the default pool, that is on the order of a thousand connections at full scale-out, far above what an untouched Postgres accepts. And the moment the HPA reaches those replicas is exactly the moment of highest load, so connection exhaustion arrives when it can least be afforded.

There are two more valves on the same path. Provider errors are written to the database by default, and under a sustained engine failure that inflates the spend table without contributing anything that is not already in the metrics. They are switched off like this:

general_settings:
 disable_error_logs: true # stops writing provider errors
 proxy_batch_write_at: 60 # batches the spend writes

disable_spend_logs: true is the radical version, which removes the per-request detail from the interface and leaves the cost in Prometheus and in the tracing backend. Before going that far you have to decide whether that table is the system’s audit record, which is the discussion in the previous article: if it is, it cannot be switched off, and the way out is the Redis buffer.

Redis: when it stops being optional

Without Redis, LiteLLM works. Each instance keeps its own in-memory cache and applies its own counters. The consequences are three, and all of them show up as you grow.

Requests-per-minute limits are applied per instance, so a limit of 100 with five replicas is in practice a limit of 500. Cache hits are local, so the same repeated question hits once in five. And there is no leader election, so periodic jobs run in every process at once.

The threshold the documentation gives for enabling the transaction buffer is a thousand requests per second or ten instances. Below that, each instance updates the key, user and team rows directly; above it, they all write to the same rows, locks appear and Postgres starts rejecting with FATAL: sorry, too many clients already.

general_settings:
 use_redis_transaction_buffer: true

The metrics to watch when that is enabled are litellm_in_memory_spend_update_queue_size, litellm_redis_spend_update_queue_size and litellm_pod_lock_manager_size. If the first grows without the second coming down, the flush to Postgres is not keeping up.

A note about the routing strategy, which interacts with this. simple-shuffle consults nothing external. Usage-based routing does, and it adds a round trip to Redis inside the request path. On a homogeneous vLLM fleet behind the same model, the improvement it brings rarely pays for that latency; on a heterogeneous fleet, the conversation is different and the router article covered it.

Background jobs register per worker

This detail appears in no getting-started guide and produces considerable bewilderment when it is discovered. The proxy’s periodic jobs register per uvicorn worker, not per pod. With --num_workers 4 and ten replicas that is forty copies of every job, all of them running with no coordination if there is no Redis for leader election.

The variable that separates them is LITELLM_JOB_ROLE: value serving on the pods that serve traffic and a separate replica with value worker for the jobs. With the one-worker-per-pod recommendation the problem shrinks, but it does not go away while there are several replicas.

Probes: three endpoints and only two are usable

The three endpoints exist and do different things:

Endpoint	Authentication	What it does	What it is for
`/health/liveliness`	No	Returns `I'm alive!`, or 503 during shutdown	`livenessProbe`
`/health/readiness`	No	State of the process and the database. 503 if the configured Postgres does not answer	`readinessProbe`
`/health`	Yes	Fires a real request against every model in the catalogue	Manual diagnosis, never a probe

The last row deserves insisting on. /health spends tokens on every call, one per configured model. Set as a probe, with the Kubernetes default interval and a fleet of eight models, that is thousands of inference requests a day that serve nobody.

There are more endpoints useful for diagnosis: /health/readiness/details, /health/services?service=langfuse to check one specific callback, /health/history and /health/latest for the history, and /health/backlog, which returns the in-flight requests and shows up in the latency section. /health/drain exists but returns 404 unless it is enabled with enable_drain_endpoint: true, and it is protected with X-Drain-Token.

The periodic model checks are configured separately and not in the probe:

general_settings:
 background_health_checks: true
 health_check_interval: 300 # seconds
 background_health_check_model_groups: ["llama-70b", "qwen-30b"]

And one setting with consequences in deployments with no Internet egress: allow_requests_on_db_unavailable: true allows serving traffic with the database down, but leaves /health/readiness always returning 200. With that set, the readiness probe stops detecting the very failure it was designed to detect, and budget and model errors keep blocking just the same.

Retries, cooldowns and fallbacks: the real numbers

The defaults, read from the project’s code and not from the documentation:

Constant	Value	What it governs
`DEFAULT_MAX_RETRIES`	2	Retries before moving on to fallbacks
`DEFAULT_ALLOWED_FAILS`	3	Failures before cooling down a deployment
`DEFAULT_COOLDOWN_TIME_SECONDS`	5	Seconds a deployment stays out
`SINGLE_DEPLOYMENT_TRAFFIC_FAILURE_THRESHOLD`	1000	Minimum requests before applying the cooldown logic with a single deployment
`DEFAULT_REQUEST_TIMEOUT_SECONDS`	6000.0	Request timeout

Besides the failure counter, there is a proportional criterion: a deployment enters cooldown if 50% of its requests fail in any given minute.

On the cooldown there is a discrepancy worth knowing about before debugging blind. The docstring of the router function says cooldown_time defaults to 1, and the code uses the constant of 5. The code wins.

The fallbacks are three separate lists and are applied after the retries are exhausted:

router_settings:
 num_retries: 2
 timeout: 120
 fallbacks: [{"llama-70b": ["qwen-30b"]}]
 context_window_fallbacks: [{"llama-70b": ["llama-70b-128k"]}]
 content_policy_fallbacks: [{"llama-70b": ["llama-70b-sin-guardrail"]}]

A note on the behaviour when everything is in cooldown: if no deployment is left available in the group, LiteLLM falls back onto a specific model_info.id, skipping the cooldown check. It is a deliberate degradation, serve something rather than return 503, and it is worth knowing it exists because it masks the real state of the fleet.

And a capability lost along the way: since version 1.85.0, the mock_testing_fallbacks parameter and its two siblings are stripped from requests arriving through the proxy and have no effect. Fallbacks can no longer be tested against the proxy with a doctored request; it has to be done against litellm.Router directly, in a test. Anyone who had a smoke test built on that has it broken without warning.

The six thousand second timeout

It deserves its own section because it is the setting with the worst damage-to-effort ratio in the whole system. request_timeout is 6,000 seconds. A hundred minutes. An inference engine that stops answering without closing the connection holds a worker for that long, and with one worker per pod that is an entire pod out of service per trapped request.

general_settings:
 request_timeout: 600

Ten minutes is still generous for a long generation, and it bounds the damage.

Overhead: the project’s two figures

LiteLLM publishes two measurements of its own overhead that differ by a factor of thirty, and both are legitimate because they measure different loads.

The benchmarks page, with Locust, a thousand users with think time and a fake endpoint, gives for four instances an x-litellm-overhead-duration-ms of median 2 ms, p95 8 ms and p99 13 ms at 1,170 requests per second. With two instances, median 12 ms, p95 29 ms and p99 43 ms. The page itself warns that those figures correspond to around 130 in-flight requests, and that a closed-loop client with no think time keeps a thousand in flight and sees on the order of eight times that latency at the same rate, by Little’s law.

The AIGatewayBench benchmark, from July 2026, measures the Python proxy at 257.7 ms of added p99 and 329.5 MB of peak memory, with no callbacks, no spend tracking and no persistence. The Rust variant, in beta, gives 0.7 ms.

The reading for an in-house platform: the first figure is the best case with human clients, and the second is the order of magnitude to plan for agentic load, which arrives in a closed loop against fast responses. There is one reported case that resembles this scenario closely: an in-house OpenAI-compatible engine served around 16 requests per second directly and around 9 through LiteLLM, with the degradation growing with concurrency, on a 4 vCPU, 8 GB pod.

The latency gap the metrics cannot see

LiteLLM’s timers start when its handler starts. The time the request spends queued in the uvicorn event loop, before getting there, appears in none of its metrics. The example in its own troubleshooting guide is a case in which LiteLLM records 10 seconds and the user experiences 20.

It is detected by comparing two sources: GET /health/backlog or the litellm_in_flight_requests gauge against the response time measured by the load balancer in front. If they diverge, the gap is in the input queue and the answer is more replicas, not more tuning.

Two headers are always on and serve to watch it without instrumenting anything: x-litellm-overhead-duration-ms and x-litellm-callback-duration-ms. For the second, the documentation sets the suspicion threshold at 100 ms, above which the diagnosis is that the payloads are too large.

And the first cause of latency that guide mentions is none of the above: LITELLM_LOG=DEBUG serialises the payload with json.dumps(indent=4) synchronously, and with payloads over 2 MB that alone can cost between 2 and 5 seconds per request. A log level set to debug a problem and forgotten there produces exactly the problem it was meant to debug.

The failure mode to put in the runbook

There is an open issue describing a cascade with a characteristic shape. Under sustained 429s from the engine upstream, the pods stopped answering readiness probes during startup, Kubernetes killed them on failed probes, and the restart put more pressure on an upstream that was already saturated. Peaks of 500 requests per second, average of 60, between two and five replicas with 1.3 CPU and 4 GB.

The shape of the failure is what matters more than the specific case: the engine’s saturation turns into a restart loop of the gateway, and from the outside it looks like the problem is the gateway. The defences are the ones already in this article, applied together: startupProbe with a wide margin so readiness does not compete with startup, request_timeout bounded so trapped requests get released, cooldowns that take the deployment returning 429 out of rotation, and CPU autoscaling at 60% so the capacity exists before it is needed. It fits with what was covered in incident response runbooks.

Upgrading without cutting traffic

The deployment is a normal Deployment with RollingUpdate, maxUnavailable: 0 and maxSurge: 1, and there is no further mystery to it except on one point: schema migrations. LiteLLM uses Prisma and applies migrations at startup. With several replicas coming up at once after a version change, the migration must be run by a single process, which is another reason for the dedicated replica with LITELLM_JOB_ROLE: worker.

Before a version bump, two checks that pay for themselves: read the release notes looking for changes in the metadata emitted to the callbacks, because a change like that breaks saved dashboards without touching the service, and verify that the new configuration starts with --detailed_debug on a separate pod before applying it to the fleet.

Go-live checklist

--num_workers 1 and scaling by replicas.
requests equal to limits, 1 vCPU and 4 GiB per worker.
HPA on CPU at 60%. Never on memory.
database_connection_pool_limit computed from max_connections, and the chart’s maxReplicas lowered to something the database can take.
Redis as soon as there is more than one replica, and use_redis_transaction_buffer from ten instances or a thousand requests per second.
livenessProbe on /health/liveliness, readinessProbe on /health/readiness, and /health out of the probes.
startupProbe with a margin of up to 300 seconds.
request_timeout: 600.
LITELLM_LOG out of DEBUG, and disable_error_logs: true.
A dashboard with x-litellm-overhead-duration-ms, x-litellm-callback-duration-ms, litellm_in_flight_requests and the response time of the load balancer in front.

Traps and things that are not what they look like

The HPA scales up and never back down. It is measuring memory, which in this process only goes up.

The pods die in a loop right after deploying. Less than 4 GiB of limit. The footprint appears with the first large request, not at startup.

Postgres starts rejecting connections at the peak. The chart has maxReplicas: 100 and the pool defaults to 10.

The per-minute limits are enforced badly. There is no Redis, and each replica keeps its own count.

The health check consumes tokens. Someone set /health as the readiness probe.

The readiness probe returns 200 with the database down. allow_requests_on_db_unavailable is set.

A deployment keeps receiving traffic after failing. The cooldown is 5 seconds, not minutes, and with a single deployment in the group you have to pass a thousand requests for the logic to apply.

The fallback test no longer tests anything. The simulation parameters are stripped from requests arriving through the proxy since 1.85.0.

The p99 is ten times worse than the published benchmark. The benchmark has think time and the real load is agentic.

Closing

None of these settings is complicated and all of them are one line. What unites them is that their defaults describe a small deployment, with one instance, a human client on the other side and a commercial provider taking care of capacity. An in-house inference factory is the opposite on all four axes: several replicas, automated clients, and an engine whose capacity is finite and known.

Of the ten points in the checklist, three decide almost everything in an on-premise deployment: the process model with its sizing, the database connection arithmetic, and the request timeout. The other seven are the ones that avoid the Saturday phone call.

Sources

LiteLLM, Production best practices (topología, workers, memoria, pool de conexiones, LITELLM_JOB_ROLE, buffer de transacciones en Redis, request_timeout): https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Health checks (los tres endpoints, comprobaciones de fondo, enable_drain_endpoint): https://docs.litellm.ai/docs/proxy/health.
LiteLLM, Reliability (fallbacks, reintentos, deprecación de los parámetros de simulación en la 1.85.0): https://docs.litellm.ai/docs/proxy/reliability.
LiteLLM, Benchmarks y Load test advanced (las cifras de sobrecarga por número de instancias y la advertencia de la ley de Little): https://docs.litellm.ai/docs/benchmarks · https://docs.litellm.ai/docs/load_test_advanced.
LiteLLM, Rust AI Gateway benchmarks (los 257,7 ms de p99 del proxy en Python): https://docs.litellm.ai/blog/rust-ai-gateway-benchmarks.
LiteLLM, Latency overhead (el hueco de la cola de uvicorn, las cabeceras de diagnóstico, el coste de LITELLM_LOG=DEBUG): https://docs.litellm.ai/docs/troubleshoot/latency_overhead.
LiteLLM, DB info (disable_spend_logs, disable_error_logs y qué se pierde con cada uno): https://docs.litellm.ai/docs/proxy/db_info.
LiteLLM, litellm/constants.py (los valores por defecto de reintentos, fallos permitidos, enfriamiento y timeout): https://github.com/BerriAI/litellm/blob/main/litellm/constants.py.
LiteLLM, litellm/router.py (la discrepancia entre el docstring de cooldown_time y la constante que usa el código): https://github.com/BerriAI/litellm/blob/main/litellm/router.py.
LiteLLM, issue #15526 — cascada de 429 sostenidos, sondas de readiness fallidas y bucle de reinicios: https://github.com/BerriAI/litellm/issues/15526.
LiteLLM, issue #21046 — degradación de throughput contra un motor propio compatible con OpenAI: https://github.com/BerriAI/litellm/issues/21046.

LiteLLM and Langfuse: the operational pair, and the four places where the trace is lost between the gateway and the dashboard

Mon, 07 Sep 2026 18:00:00 +0200

This post opens the operational track of the control layer. The previous articles left the pieces chosen and assembled: choosing the OSS gateway and the L7 inference router explain why LiteLLM ends up in front of the fleet, and Langfuse from the inside takes apart the backend that receives the traces. Here we deal with the seam between the two, which is where the work of the next three years lives.

TL;DR

The LiteLLM plus Langfuse pair is configured with a list of callbacks and two API keys, and from there four matters decide whether the dashboard is good for operating or only for showing in a demo.

There are three integration routes and they are not equivalent. The native langfuse callback is tied to the Python SDK v2 (langfuse>=2.59.7,<3.0 in LiteLLM’s pyproject.toml), which writes against the legacy ingestion that Langfuse Cloud retires on 16 November 2026. The langfuse_otel preset exports over OTLP and is the path recommended in the current documentation of both projects. The OTel v2 route, behind the LITELLM_OTEL_V2=true flag, changes span names, leaves the success status at UNSET and, above all, its Langfuse mapper does not read the request metadata: session_id, trace_user_id, tags and trace_id stop arriving.

The cost of a self-hosted model comes out as zero unless a price is registered. get_model_info() raises an exception for a model that is not in the pricing map, completion_cost() propagates it, the logger catches it, stores it in a debugging field and returns None, and the payload turns it into 0.0. There is no warning at the default log level. A team’s budget is never exhausted because its spend is always zero.

Client-gateway-engine correlation does not come out of the box. LiteLLM does continue the client’s trace if a traceparent arrives, and it also extracts from it the 32-hex trace-id to use as litellm_trace_id, which makes the Langfuse identifier equal to the W3C one. Upstream the story changes: forward_traceparent_to_llm_provider is False by default, the function that applies it returns early if no otel logger is loaded, and when it does act it forwards the client’s original header, so the vLLM span ends up a sibling of LiteLLM’s rather than a child. Add to that the fact that vLLM exports OTLP over gRPC by default and Langfuse does not accept gRPC.

Between the request and the dashboard there are four bounded queues that drop events. LiteLLM’s logging thread pool (100 threads, 10,000 pending tasks, dropping once full), the asynchronous LoggingWorker (50,000, and once full it aggressively flushes half), OTel’s BatchSpanProcessor (2,048 spans by default) and Langfuse’s ClickhouseWriter, which drops rows after exhausting the retries with no dead-letter queue. None of the four applies back pressure on the request, which is exactly what you want, and none of them guarantees delivery, which is exactly what an auditor asks about.

The rest of the article develops the four points with configuration applicable to a generic 4×H100 cluster with vLLM, the overhead figures the project itself publishes, and the honest limit of what this data sustains before ENS or ISO/IEC 42001.

You are here: the seam between DEPLOY and OBSERVE

In the seven-layer stack, LiteLLM lives in the gateway layer and Langfuse in the observability one. The separation is clean on the diagram and false in operation: the gateway is the only point in the system that sees the complete request before chopping it up and the complete response after reassembling it, so it is also the only reasonable place from which to emit the trace. The inference engine sees tokens and latencies; it does not see who is asking or how much that team has spent this month.

That turns LiteLLM into a telemetry producer as well as a traffic distributor, and moves onto the gateway a responsibility that was not in the original design: if the callback fails, the request is served all the same and nobody notices that the dashboard has been incomplete for two days.

The analogy: the toll booth that issues the ticket

A motorway toll booth does two independent jobs. It raises the barrier, which is its visible function, and it issues a ticket with number plate, time, section and amount, which is the function the concession lives off. The two share a booth and do not share a destination: the barrier responds in a hundred milliseconds because there is a car waiting, and the ticket travels to a processing centre over a slower channel that batches thousands of records before sending them.

The design is well thought out as long as its consequence is understood. If the processing centre goes down, cars keep going through. If the ticket queue fills up, tickets are thrown away, traffic is not stopped. The concession accepts losing records rather than causing a jam, and charges on a statistical sample it knows to be imperfect.

LiteLLM and Langfuse work the same way. The four queues in the corresponding section are the slow channel, and all of them prefer to drop rather than block. Whoever builds the pair inheriting the complete analogy gets the operation right. Whoever thinks the ticket is an accounting entry ends up explaining to an auditor why traces from a Tuesday afternoon are missing.

Three routes, and one with an expiry date

As of September 2026 three paths coexist for getting a request from LiteLLM to Langfuse. LiteLLM’s documentation already labels the first as legacy.

Route	How it is enabled	Transport	Status
Native `langfuse` callback	`success_callback: ["langfuse"]`	Langfuse Python SDK v2, legacy ingestion	Maintained for compatibility. Tied to `langfuse<3.0`
`langfuse_otel` preset (v1)	`callbacks: ["langfuse_otel"]`	OTLP/HTTP against `/api/public/otel`	Recommended by both projects
OTel v2 route	`callbacks: ["langfuse_otel"]` + `LITELLM_OTEL_V2=true`	OTLP/HTTP, new mapper	Opt-in. Changes semantics and loses metadata

The date that orders the decision is 16 November 2026. Langfuse’s compatibility matrix marks the legacy batch ingestion of the Python SDK v2 as retired on that date for Cloud, at which point trace ingestion through that route stops working. On self-hosted the endpoint survives, but a v4 server in events_only mode returns 400 for every event type except score-create and sdk-log.

Since LiteLLM’s native callback is pinned to langfuse>=2.59.7,<3.0 in its own pyproject.toml, and its code calls APIs that only exist in SDK v2 (self.Langfuse.trace(...), trace.generation(...)), there is no way to move up an SDK without changing route. Anyone who today has success_callback: ["langfuse"] against Langfuse Cloud has a migration with a date on it. Anyone who has it against a self-hosted Langfuse has one with some slack, because v3 receives security patches until the end of January 2027, but they have one all the same.

One detail of the OTLP route worth checking in older deployments: the x-langfuse-ingestion-version: 4 header is the one that enables real-time ingestion. Without it the traces arrive, but with a delay that Langfuse’s documentation puts at around ten minutes (the compatibility page says fifteen, and that discrepancy remains unresolved in their own documents). LiteLLM started sending it in version 1.95.0, of August 2026. A proxy older than that version against a Langfuse v4 produces the classic symptom: the dashboard works, but it is always behind and nobody knows why.

The configuration that works today

For a generic 4×H100 cluster with vLLM behind it and Langfuse self-hosted in the same Kubernetes, the OTLP route in its v1 variant is the one that gives the most functionality with the fewest surprises.

model_list:
 - model_name: llama-70b
 litellm_params:
 model: hosted_vllm/meta-llama/Llama-3.3-70B-Instruct
 api_base: http://vllm-llama70b.inference.svc.cluster.local:8000/v1
 api_key: os.environ/VLLM_API_KEY
 # Price derived from the cluster's €/GPU-hour. Without this, spend = 0.0
 input_cost_per_token: 0.00000031
 output_cost_per_token: 0.00000089

litellm_settings:
 callbacks: ["langfuse_otel"]
 turn_off_message_logging: false # a conscious decision, see the PII section
 langfuse_default_tags: ["model_group", "user_api_key_team_alias", "cache_hit"]

general_settings:
 disable_error_logs: true # provider errors inflate SpendLogs
 proxy_batch_write_at: 60

The credentials travel through the environment. LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_HOST are enough: the preset builds the Authorization: Basic base64(public:secret) header and adds the ingestion version one, so there is no need to touch OTEL_EXPORTER_OTLP_HEADERS by hand. The preset’s default host points to the US region of Langfuse Cloud, a relevant detail for anyone building a platform with data sovereignty requirements: if the variable is not defined, the telemetry leaves Spain without anyone having decided it.

For real multi-tenancy, LiteLLM supports per-team Langfuse credentials (POST /team/<id>/callback with callback_vars) and per-key ones (metadata.logging[] when generating the key), and the key’s wins over the team’s without mixing. There is a limitation that breaks the architecture of anyone who planned to separate tenants across different Langfuse instances: under OTel v2, a per-key or per-team langfuse_host does not move that tenant to another host, because the exporter fixes the destination at startup and the only thing that varies per tenant are the headers.

And one operational warning with years of history behind it: MAX_LANGFUSE_INITIALIZED_CLIENTS is 50 by default because each Langfuse client starts a thread. The comment in LiteLLM’s code is explicit about the incident that motivated it, with the CPU at 100% from repeated initialisations. A deployment with per-team credentials and many teams hits that ceiling.

What is lost when moving to OTel v2

The LITELLM_OTEL_V2=true flag brings a step up in quality in the proxy’s general instrumentation, and at the same time a concrete regression on the path to Langfuse that the documentation acknowledges without underlining it.

The Langfuse mapper of the v2 route emits langfuse.observation.type, langfuse.observation.model.name, the observation identifier, the model parameters, input and output, usage_details and cost_details, plus langfuse.trace.metadata.team_id and team_alias. It does not emit session.id, nor user.id, nor langfuse.trace.tags, nor langfuse.trace.id, nor the generation name, nor parent_observation_id. LiteLLM’s documents put it this way: those attributes are set by the preset from the request and the response, not from metadata supplied by the client.

The practical consequence is that an application that today groups conversations by sending metadata: {"session_id": "...", "trace_user_id": "..."} in the request body stops grouping them when the flag is enabled, with no error and no warning. The traces keep arriving, loose.

The other v2 changes are semantic and affect alerts already written. The root span goes from being called Received Proxy Server Request to being named after the route (POST /v1/chat/completions), the inference span goes from litellm_request to {operation} {model}, the guardrail spans stop hanging off the inference span and hang off the root instead, and the success status stays at UNSET instead of OK. Any rule that fires on status == OK stops firing. The way back is to remove the flag and restart.

Cost: why the trace says zero

This is the part that takes the most time to diagnose and the least to fix.

LiteLLM computes the cost of a request from a per-model pricing map. For commercial provider models the map comes filled in. For hosted_vllm/whatever, which is all there is in an in-house inference factory, the map is empty, and the failure path is as follows:

litellm.get_model_info() raises an exception with the message This model isn't mapped yet.
completion_cost() and response_cost_calculator() propagate the exception.
Logging._response_cost_calculator() catches it, writes a StandardLoggingModelCostFailureDebugInformation into response_cost_failure_debug_information, logs it at debug level and returns None.
The standard logging payload does llm_response_cost = raw_response_cost or 0.0.

The system delivers a spend of zero, with no visible error at the default log level, and writes it into the spend column of LiteLLM_SpendLogs. The per-key, per-team and per-organisation budgets are evaluated against that zero and are never exhausted. The Langfuse dashboard shows a total cost of zero for everything self-hosted, which is precisely the part you wanted to measure.

Registering the price

The pricing keys are accepted both in litellm_params and in model_info, and those in litellm_params win when they are in both places. The complete set has 91 fields, but for a deployment with vLLM two are enough, and there are three things to know about them.

The first is that the price is per individual token, in USD. There are no per-thousand variants: searching for per_1k or per_1000 in the pricing model returns nothing. A three-orders-of-magnitude error here is easy to make and hard to see, because it produces plausible figures.

The second is that input_cost_per_request exists in the schema and is not applied in chat calls: a model registered with that field returns a cost of 0.0. To charge per request it has to be converted into a cost per token.

The third is that input_cost_per_second needs the response time and is computed as price times response_time_ms / 1000, so outside the proxy’s request path it silently returns zero.

The number that goes into those two keys comes from the article on FinOps and multi-tenancy with LiteLLM and the one on cost per token and per request: amortisation plus energy divided by tokens served. The example arithmetic, to set the order of magnitude: a cluster whose fully loaded cost is 9.60 USD per hour sustaining 3,000 output tokens per second produces 10.8 million tokens an hour, that is 0.00000089 USD per output token. That is the value that goes into output_cost_per_token. If the cluster’s average utilisation is 30 % and not 100 %, the divisor changes and the price triples, which is the same lever that article develops.

Before sending traffic, the /spend/calculate endpoint accepts the same parameters as completion_cost and returns the computed cost. It is the cheap way to check that the price resolves.

How that number reaches Langfuse, and who wins

The two integration routes write the cost into different fields. The native callback sends it as usage.total_cost of the v2 schema, and duplicates it in the trace metadata as litellm_response_cost. The OTLP preset emits langfuse.observation.cost_details as a JSON string of the form {"total": cost}, and langfuse.observation.usage_details with input, output and total.

The rule on the Langfuse side is documented and is the right one for this case: when there is ingested cost and inferred cost, the ingested one takes priority. Langfuse only computes from its own pricing table when the emitter sends no cost, and changes to a model definition apply to new generations, with no retroactive recalculation. A self-hosted model therefore needs no pricing definition in Langfuse: it is enough for LiteLLM to send cost_details.

One detail remains that explains a puzzling symptom. Langfuse stores the ingested cost and the computed one in separate columns, totalCost and calculatedTotalCost, and the session header totals add up only the first. A mixed deployment where part of the traffic brings cost and part lets Langfuse compute it shows sessions at zero while the individual generations show cost. There is an open issue about it.

The tokens of a streaming response

Almost all of an assistant’s traffic is streamed, and there the OpenAI usage object only appears if the client asks for stream_options.include_usage. LiteLLM solves this by injecting the option on its own: on the asynchronous completion routes with stream: true, if the caller did not set it and all the candidate deployments support it, it adds it and marks _litellm_strip_stream_usage so the usage block is used in the cost calculation and not returned to the client. The general_settings.always_include_stream_usage flag forces or disables that behaviour, and is worth setting to True when capability detection does not recognise the vLLM deployment.

When the usage is missing anyway, LiteLLM reassembles the response with stream_chunk_builder() and counts tokens with the tokeniser. It is an approximation, and it produces the second path towards a cost of zero. The two paths to zero are told apart by looking at total_tokens in LiteLLM_SpendLogs: zero tokens with a correct price is a usage problem, correct tokens with zero cost is a price problem.

Correlation: from the client to the engine

The operational question that justifies the whole setup is being able to take a user complaint, find their request and see where the time went. That requires the trace identifier to survive three hops.

What does work

If the client sends traceparent, LiteLLM’s spans hang off the existing trace. In addition, and independently of the OTel subsystem, litellm_pre_call_utils.py extracts by regular expression the 32-character hexadecimal trace-id and assigns it to data["litellm_trace_id"] as a last resort, and takes the session_id from the W3C baggage header. That makes the Langfuse trace identifier equal to the W3C trace identifier, which is the most robust correlation key available: with it, an identifier copied from the application log opens the trace in Langfuse with no translation.

For the routes that are not OTel v2, moreover, headers prefixed with langfuse_ inject per-request metadata. langfuse_trace_id, langfuse_trace_user_id and langfuse_trace_metadata are documented, but the code strips the prefix and writes the rest into the metadata, so any body key works as a header. Two warnings about this: header values arrive as strings and are not interpreted as JSON, and Nginx drops headers with underscores by default (underscores_in_headers off), which produces the symptom that the metadata works locally and disappears behind the ingress.

What does not work without effort

Towards the engine, propagation is off. The flag is litellm.forward_traceparent_to_llm_provider, it is False, and its implementation has three behaviours to know about before enabling it:

The function returns immediately if open_telemetry_logger is None. A deployment with callbacks: ["langfuse_otel"] and without the generic otel callback propagates nothing, whatever the flag is set to.
When it does propagate, it copies the client’s traceparent header as is, instead of injecting the context of LiteLLM’s active span. The vLLM span ends up hanging off the client’s span-id, a sibling of the gateway call’s span. The trace is one, the hierarchy is wrong.
If the client sends no traceparent, none is created.

Add to that the fact that LiteLLM’s general policy is not to forward client headers to the provider, and that the forward_client_headers_to_llm_api allowlist covers headers prefixed with x-, among which traceparent is not. This is the only route.

The vLLM side, and why the Collector stops being optional

vLLM enables traces with --otlp-traces-endpoint, and --collect-detailed-traces with model, worker or all adds forward and execute times, with a performance impact its own documentation warns about. The attributes it emits are in vllm/tracing/utils.py and are the most useful part of the setup: gen_ai.latency.time_in_queue, time_to_first_token, time_in_scheduler, time_in_model_prefill, time_in_model_decode, time_in_model_forward, time_in_model_execute, time_in_model_inference and e2e, plus the token counters. That breakdown is what turns a “it’s slow” into “the time goes in the queue, not in decode”, which is the conversation you want to have during an incident. It fits with what was already covered in instrumenting vLLM with OTel and with the metrics in DCGM and vLLM.

There are three constraints that decide the architecture:

vLLM exports over gRPC unless OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf is set, and its gRPC exporter is built with insecure=True hard-coded. Langfuse, for its part, does not support gRPC. And vLLM emits the old convention names (gen_ai.usage.prompt_tokens, gen_ai.usage.completion_tokens), renamed in the specification to input_tokens and output_tokens, plus a gen_ai.latency.* family that was never standard and that the file itself documents as its own attributes until they are normalised.

An OpenTelemetry Collector in between solves all three at once: it receives gRPC, exports HTTP with the Basic header towards /api/public/otel/v1/traces, and renames attributes with a transform. It is also the only place where tail sampling can be done, which is the next point. And add the warning from the llm-d community, which holds for any deployment with Envoy or sidecars in between: OTEL_SERVICE_NAME has to be set per engine, because vLLM sends no service name and without it every span arrives as unknown_service.

The four serial queues

A request’s telemetry crosses four bounded buffers before it is queryable. All four drop once full and none blocks the request.

Queue	Capacity	What it does once full
LiteLLM logging thread pool	100 threads, 10,000 pending tasks	Drops the new ones with a rate-limited warning
Asynchronous `LoggingWorker`	50,000 events, concurrency 100	Aggressively flushes 50 % of the queue; `enqueue()` documents that it never blocks
OTel’s `BatchSpanProcessor`	2,048 spans (`OTEL_BSP_MAX_QUEUE_SIZE`), batch 512, sent every 5 s	Drops spans silently
Langfuse’s `ClickhouseWriter`	Bounded retries	Drops rows after exhausting retries, with no dead-letter queue

The first is documented in LiteLLM’s own code with a sentence that saves half an hour of discussion: logging is best-effort and, once the cap is reached, new entries are dropped with a rate-limited warning instead of being queued forever. The third is a barely visible consequence: LiteLLM builds the BatchSpanProcessor without adjusting max_queue_size, so the OTel SDK defaults govern, and no LiteLLM documentation mentions the OTEL_BSP_* variables that change them. The Langfuse SDK does the same on its own account: it sets batch size and interval, and leaves the queue size at the default.

On top of this comes a fifth loss point that is not a queue: LiteLLM does not call flush() per request on the native route, so a SIGKILL on the pod takes the last batch with it. On Kubernetes, this is an argument for a generous terminationGracePeriodSeconds and a preStop that gives some slack.

None of this is a defect. It is the right design decision for a piece that sits on the critical path of inference. What changes is what can be asserted with the resulting data.

What cannot be promised to the auditor

A dashboard built on this pipeline answers statistical questions well: what is the p95 of time to first token for the model group, which team consumes the most, what proportion of requests fails, how cost per request evolves. That is what it is designed for, and that is what it is good for.

It does not answer completeness questions well, which are the ones an ENS auditor or an ISO/IEC 42001 management system auditor asks. Faced with “show me that I have a record of every interaction with the AI system during March”, the honest answer is that the traces are a best-effort record with four documented points of silent dropping, with no dead-letter queue on the last hop and no durability guarantee on any of them.

The clean separation is the one already proposed in technical controls ENS, 42001 and the AI Act: the regulatory audit record is a different artefact, with a different write path, durability guarantees and retention of its own. In the pair this article deals with, the LiteLLM_SpendLogs table in Postgres is a much better candidate than Langfuse for that function, because it is a transactional write against a relational database, and because it already contains request_id, api_key in hashed form, team_id, organization_id, end_user, model, tokens, startTime, endTime, status and requester_ip_address. Writing the audit event from the same place and in the same transaction as the spend is the sensible route; delegating it to the tracing pipeline is not.

Here an operational tension appears that has to be resolved deliberately. LiteLLM’s performance recommendation for production includes disable_error_logs: true and, in very loaded deployments, disable_spend_logs: true, because provider errors inflate the table and concurrent writes against the same rows produce locks and exhaust the Postgres connections. Disabling spend_logs to gain performance and at the same time leaning on that table for the audit record are incompatible decisions. Above a thousand requests per second or more than ten instances, the way out is use_redis_transaction_buffer: true, which batches the spend updates in Redis before pushing them down to Postgres, watching the litellm_in_memory_spend_update_queue_size and litellm_redis_spend_update_queue_size metrics.

PII: switching off the content before it leaves

An LLM trace contains, by definition, the user’s prompt. In an on-premise deployment for a regulated customer, that content may be exactly what cannot leave the application’s domain, not even towards the observability backend in the same cluster.

The controls on the proxy side, from most global to most fine-grained:

litellm_settings.turn_off_message_logging: true cuts messages and responses in every callback, and keeps the metadata and the spend. The redaction replaces the content with the REDACTED_BY_LITELLM sentinel in messages, choices, reasoning_content, streaming deltas, tool call arguments and in the standard logging object.
The x-litellm-enable-message-redaction: true header enables redaction per request, and LiteLLM-Disable-Message-Redaction: true disables it. The second is checked first and wins over everything else, including the global configuration, which makes it a control to block at the ingress if the policy is that content is never logged.
redact_user_api_key_info: true removes the hashed token, the user_id and the team_id from the log.
Per request and per callback, metadata: {"mask_input": true, "mask_output": true}.
The body parameter "no-log": true disables logging of that call. It is a body parameter, not a header, and it can be disabled globally with global_disable_no_log_param: true so a client cannot make itself invisible.

On the OTel v2 route the policy is inverted towards the safe side: content is off by default and is opened with OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, with values no_content, span_only, event_only and span_and_event. The documentation underlines that the gate is applied centrally and that a user’s request cannot force their prompt towards the backend while capture is disabled. It is also the stance the OpenTelemetry specification recommends, whose preferred pattern for production is to store the content in external storage and leave only the reference in the span.

One specific version to check in the inventory: below LiteLLM 1.98.0, with per-team Langfuse credentials, the team’s own credentials ended up inside its traces. That version changed the origin of the metadata emitted to the StandardLoggingPayload to cut it off. The same change removed a couple of dozen metadata fields (model_group, deployment, queue_time_seconds, attempted_retries and others), so it breaks saved dashboards and alerts that filter on them. It is an upgrade with work attached, and it has to be done anyway. The same applies to the general hardening of the stack, covered in hardening and secrets of the sovereign stack.

The price in latency

LiteLLM publishes two overhead figures that differ by a factor of thirty, and both are theirs.

The benchmarks page, with Locust, a thousand users with think time and a fake OpenAI endpoint, gives for four instances an x-litellm-overhead-duration-ms of median 2 ms, p95 8 ms and p99 13 ms, at 1,170 requests per second. For two instances, median 12 ms, p95 29 ms and p99 43 ms. The page itself warns that those figures hold with around 130 in-flight requests, and that a closed-loop client with no think time keeps a thousand in flight and sees on the order of eight times that latency at the same rate, by Little’s law.

The AIGatewayBench benchmark, of July 2026, measures the Python proxy at 257.7 ms of added p99 with 329.5 MB of peak memory, against 0.7 ms for the Rust variant in beta. That measurement has no callbacks, no spend tracking and no persistence.

The operational reading is that the first figure is the best case and the second the order of magnitude to plan for agentic load, which is the one that arrives in a closed loop against fast responses. To watch it there are two always-on headers: x-litellm-overhead-duration-ms and x-litellm-callback-duration-ms, and the documentation sets the useful threshold of the second at 100 ms, above which the diagnosis is that the payloads are too large.

Three production settings that appear in LiteLLM’s documentation and that are wrong by default:

request_timeout is 6,000 seconds. A hundred minutes holding a connection against a hung upstream. The recommendation is to lower it to 600.

LITELLM_LOG=DEBUG serialises the payload with json.dumps(indent=4) synchronously, and the documentation puts the cost of that serialisation at 2 to 5 seconds with payloads over 2 MB. It is the first cause of latency their troubleshooting guide mentions.

Background jobs register per uvicorn worker, not per pod. With --num_workers 4 and ten replicas that is forty copies of every periodic job. The LITELLM_JOB_ROLE variable separates the pods that serve traffic from the replica that runs jobs.

And one failure mode with an open issue that deserves to be in the runbook: under sustained 429s from the upstream, pods have been reported unable to answer readiness probes during startup, with the ensuing restart loop. It fits with what was covered in incident response runbooks.

Start-up checklist

Ten checks before signing off the installation of the pair:

callbacks: ["langfuse_otel"] instead of success_callback: ["langfuse"], unless you depend on prompt management, which the OTLP preset does not support.
LANGFUSE_HOST defined explicitly. The preset’s default value points to the US region.
LiteLLM at 1.98.0 or higher, because of the team credential leak into the traces, and at 1.95.0 or higher because of the v4 ingestion header.
A price registered for every self-hosted model, per token and in USD, verified with /spend/calculate before opening traffic.
A test request with a known traceparent, checking that the same identifier appears in Langfuse.
An OpenTelemetry Collector between vLLM and Langfuse, with OTEL_SERVICE_NAME per engine and renaming of the old token attributes.
request_timeout: 600 and LITELLM_LOG out of DEBUG.
Tail sampling in the Collector, with a latency and status code policy, to keep 100 % of what is slow and what failed.
A written decision about prompt content, and blocking of the LiteLLM-Disable-Message-Redaction header at the ingress if the policy is not to log.
A dashboard with x-litellm-overhead-duration-ms, x-litellm-callback-duration-ms and Langfuse’s ingestion queue depth (langfuse.queue.ingestion.depth), which is the autoscaling signal for its workers.

Traps and things that are not what they look like

The trace arrives, but ten minutes late. The x-langfuse-ingestion-version: 4 header is missing, which LiteLLM sends from 1.95.0. It is not a ClickHouse capacity problem.

The metadata works in development and disappears in production. Nginx drops headers with underscores by default, and all of this integration’s have one.

Session cost comes out as zero and generation cost does not. Langfuse adds up only the ingested cost in the session totals, not the one it computes itself.

Enabling OTel v2 breaks nothing visible and ungroups the conversations. The v2 mapper does not read session_id or trace_user_id from the request.

A team’s budget is never exhausted. Its spend is zero because the model has no price, and the calculation failure is only logged at debug level.

The alerts stop firing after the migration to v2. The success status becomes UNSET; no rule on OK is satisfied.

Forwarding the traceparent is enabled and it does not reach the engine. The generic otel callback is missing, without which the function that applies it returns before looking at the flag.

All the engines’ spans appear under unknown_service. vLLM emits no service name; OTEL_SERVICE_NAME has to be set per deployment.

Closing

The gateway and the tracing backend are the two pieces of the inference stack that take the least time to install and the longest to operate well. The installation fits in a ConfigMap. The operation consists of four decisions no getting-started guide raises: which integration route to take knowing that one expires in November, where the price per token of a model nobody invoices comes from, what to do with the traceparent at each hop, and what to tell an auditor about data the system drops by design when things get tight.

The answer to the fourth is the one that orders the other three. This pipeline is operational instrumentation, with statistical precision and best-effort guarantees, and separating it from day one from the regulatory audit record avoids the awkward conversation two years from now. With that separation made, the LiteLLM and Langfuse pair is the pair of pieces that turns a fleet of GPUs into a platform with an owner, a known cost and explainable latency.

Sources

LiteLLM, Langfuse integration (rutas de integración, metadata aceptada, cabeceras langfuse_*): https://docs.litellm.ai/docs/observability/langfuse_integration.
LiteLLM, OpenTelemetry v2 y Migración a OTel v2 (preset, captura de contenido, cambios de semántica): https://docs.litellm.ai/docs/observability/opentelemetry_v2 · https://docs.litellm.ai/docs/observability/opentelemetry_v2_migration.
LiteLLM, Logging y Config settings (redacción, turn_off_message_logging, variables de entorno): https://docs.litellm.ai/docs/proxy/logging · https://docs.litellm.ai/docs/proxy/config_settings.
LiteLLM, Custom pricing y Spend tracking (claves de precio, SpendLogs, endpoints de gasto): https://docs.litellm.ai/docs/proxy/custom_pricing · https://docs.litellm.ai/docs/proxy/cost_tracking.
LiteLLM, Production best practices (topología, request_timeout, buffer de transacciones en Redis, LITELLM_JOB_ROLE): https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Benchmarks y Latency overhead (cifras de sobrecarga y cabeceras de diagnóstico): https://docs.litellm.ai/docs/benchmarks · https://docs.litellm.ai/docs/troubleshoot/latency_overhead.
LiteLLM, Rust AI Gateway benchmarks (la cifra de 257,7 ms de p99 del proxy Python): https://docs.litellm.ai/blog/rust-ai-gateway-benchmarks.
LiteLLM, notas de versión v1.95.0, v1.98.0 y v1.100.0: https://docs.litellm.ai/release_notes/v1.95.0/v1-95-0 · https://docs.litellm.ai/release_notes/v1.98.0/v1-98-0 · https://docs.litellm.ai/release_notes/v1.100.0/v1-100-0.
Langfuse, OpenTelemetry (endpoint, autenticación, tabla de mapeo de atributos, ausencia de gRPC): https://langfuse.com/integrations/native/opentelemetry.
Langfuse, Compatibility y Migrate v3 to v4 (la fecha del 16 de noviembre de 2026 y los modos de escritura): https://langfuse.com/docs/compatibility · https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4.
Langfuse, Token and cost tracking (prioridad del coste ingerido sobre el inferido): https://langfuse.com/docs/observability/features/token-and-cost-tracking.
Langfuse, Scaling y Sampling (dimensionado, señal de autoescalado, muestreo de cabecera): https://langfuse.com/self-hosting/configuration/scaling · https://langfuse.com/docs/observability/features/sampling.
Langfuse, issue #13468 — descarte de filas en ClickhouseWriter sin cola de mensajes muertos: https://github.com/langfuse/langfuse/issues/13468. Issue #15109 — coste de sesión a cero: https://github.com/langfuse/langfuse/issues/15109. Issue #5173 — Nginx y cabeceras con guión bajo: https://github.com/langfuse/langfuse/issues/5173.
vLLM, vllm/tracing/utils.py y vllm/tracing/otel.py en la etiqueta v0.28.0 (atributos emitidos, protocolo por defecto, insecure=True): https://github.com/vllm-project/vllm/blob/v0.28.0/vllm/tracing/utils.py.
Langfuse, vLLM integration (la advertencia de que vLLM solo exporta contadores y latencias): https://langfuse.com/integrations/model-providers/vllm.
LiteLLM, litellm/constants.py y litellm/litellm_core_utils/logging_worker.py (capacidades de las colas y política de descarte): https://github.com/BerriAI/litellm/blob/main/litellm/constants.py.
OpenTelemetry, Semantic Conventions for Generative AI y el repositorio semantic-conventions-genai (estado de desarrollo, renombrados, captura de contenido): https://github.com/open-telemetry/semantic-conventions-genai.
OpenTelemetry Collector, tailsamplingprocessor (políticas de muestreo por cola y la restricción de instancia única por traza): https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md.
llm-d, End-to-end and fine-grained tracing in llm-d (OTEL_SERVICE_NAME y los spans huérfanos tras Envoy): https://llm-d.ai/blog/end-to-end-and-fine-grained-tracing-in-llm-d.

Model chain of trust (1/4): KServe and the Open Inference Protocol, the control plane your serving is missing

Sun, 26 Jul 2026 20:00:00 +0200

This blog has devoted a good part of its 145 articles to what happens inside the inference process and another good part to what happens around it. What is missing is a thread that cuts across all of that and answers a question that takes thirty seconds to ask in an audit and does not get answered in three weeks in production: when the endpoint returns a token, where did that model come from, who signed it, and who proves that the binary running on the GPU is the one that was approved?

This series walks that chain backwards from the only visible point, the endpoint: the control plane and the API contract (this article), the registry and the distribution of the bytes (2/4), the signing and provenance that let you trust them (3/4), and the identity and isolation of whoever runs them (4/4). All of it with projects under the governance of CNCF, LF AI & Data or the Linux Foundation: a chain of trust that depends on a proprietary product is no better than a support contract.

TL;DR

A vLLM Deployment serves tokens perfectly well. What it does not give you is a contract: no model versioning, no rollout by percentage, no scale-to-zero, and no common surface for the thirty non-LLM artifacts of an inference factory.
KServe provides the vocabulary: InferenceService, ServingRuntime/ClusterServingRuntime, storage initializer and storageUri. It has been CNCF incubating since 29 September 2025, after seven years as KFServing and a spell in LF AI & Data. The cadence has accelerated: v0.18.0 on 29 April 2026 and v0.19.0 on 14 June 2026.
The LLM branch is recent and different: the LLMInferenceService CRD (serving.kserve.io/v1alpha1) arrived in v0.16 in November 2025, on top of llm-d, with spec.prefill, spec.worker, spec.parallelism and spec.router.
The Open Inference Protocol (V2) is the contract: /v2/health/*, /v2/models/<m>, /v2/models/<m>/infer, in REST and gRPC, implemented by Triton, OpenVINO Model Server, Seldon MLServer and AMD Inference Server. It is tensor-centric, and the LLM world standardised de facto on the OpenAI API: KServe serves both, with the OpenAI ones under the openai/ prefix.
The honest position: the portability of an LLM today comes from the OpenAI API, not from V2. Accepting the dual track is correct; pretending there is a single standard is not.
KServe is overkill if you serve one model on one node. It starts to pay off from the fifth heterogeneous artifact onwards, and it competes on a different axis from the LLM operators: contract and lifecycle versus engine and topology.

The analogy: the station master and the track gauge

An inference factory is like a railway station with several lines.

The station master does not drive any train. His job is the service timetable: which formation leaves from which platform, how many units are coupled each hour, when an engine is pulled out for maintenance and, when new rolling stock arrives, how to divert ten per cent of the passengers to the test train before committing the whole service. That is the control plane. KServe does not generate a single token, but it decides what runs, with what configuration, with how many replicas and with what traffic split. The trains are the engines (vLLM, SGLang, TensorRT-LLM, Triton, TEI), and a competent station master does not marry a manufacturer.

And then there is the track gauge: as long as everybody builds to the same one, any locomotive rolls on any line and changing supplier is a commercial decision, not a civil engineering project. That gauge is the Open Inference Protocol, the contract that makes a client written against Triton work against OpenVINO Model Server without touching a line.

The analogy has a twist that needs no explanation in Spain: the agreed gauge is not the gauge the new rolling stock runs on. The conventional network was built to Iberian gauge and the high-speed one to standard gauge, and the practical solution was not to convert the network but to install gauge changers at the boundary points. The same thing has happened in inference: V2 is the agreed gauge and the OpenAI API is the gauge everything generative runs on. What sustains interoperability is the changer, not a single standard. We will come back to it.

A `Deployment` serves tokens, but offers no contract

In vLLM on Kubernetes we built a service with Deployment, Service, HPA and an initContainer that pulls the weights down. It works, and for one model on one node it is the right answer. What it does not give you, and cannot give you without reinventing it every time:

A stable API contract. The endpoint is “whatever the image you put there exposes”: moving from vLLM to TensorRT-LLM changes routes, error format and metric names, and the clients find out in production.

Versioning with semantics. The model is a path inside a volume; knowing which version a pod is serving requires kubectl exec. There is no declarative field that says “this service serves artifact X” and can be audited from outside.

Progressive model rollout. A RollingUpdate replaces pods; it does not split traffic between two versions by percentage. For that you need an L7 gateway in front, like the one in canary, blue-green and shadow, or a control plane that emits it.

Scale-to-zero. The HPA does not go below one replica: for a model queried twelve times a day, that is an 80 GB GPU tied up twenty-four hours.

And, above all, a common surface for what is not an LLM. A real factory does not serve three LLMs: it serves two or three LLMs, four embedding models, a couple of rerankers, ticket classifiers, a PII detector, an OCR, industrial vision and three scikit-learn artifacts that nobody remembers training. Fifteen or thirty artifacts, each with its own runtime and its own way of saying “I am ready”. The cost is not in serving the first one: it is in making sure the thirtieth does not cost thirty times what the first one did.

KServe: from KFServing to a CNCF incubating project

KServe was born in 2019 as KFServing, inside Kubeflow, in a collaboration between Google, IBM, Bloomberg, NVIDIA and Seldon. In February 2022 it was donated to the LF AI & Data Foundation; in September 2022 it split from Kubeflow and was renamed. The relevant jump: on 29 September 2025 the CNCF accepted it directly at incubating level, without going through sandbox, with a public announcement on 11 November. The figures it declared then, more than 300 contributors, 19 maintainers, more than 30 adopting organisations (Bloomberg, Red Hat, Cloudera, Nutanix, SAP, NVIDIA), come from an interested party, but the list is verifiable and the 625 contributing organisations on the CNCF dashboard support the thesis of governance not captured by a vendor.

On cadence you have to be precise, because the cliché of “KServe is slow” no longer describes reality:

Version	Date	What it brought
v0.15.0	31 March 2025	`LocalModelCache`, multi-node inference, OpenAI-compatible embeddings API, Gateway API in raw mode, KEDA integration
v0.16.0	November 2025	`LLMInferenceService`; rename of `RawDeployment` to `Standard` and of `Serverless` to `KNative`; progressive rollout in raw mode
v0.18.0	29 April 2026	WVA + KEDA/HPA autoscaling for LLMISvc, `/v1/responses` route (OpenAI Responses API), `restricted` Pod Security Standards by default on LLMISvc, namespace-scoped `ModelCache`
v0.19.0	14 June 2026	Routing by model name, static LoRA adapters, balancing across heterogeneous GPUs, dual REST/gRPC routing in Standard mode, automatic migration to `InferencePool` v1

Two major releases in under two months, with the bulk of the changelog in the LLM branch. The project is not slow: it has pivoted from classic predictive serving to generative serving. The consequences run in both directions.

The negative one is called ModelMesh. It was the high-density mode (hundreds of small models sharing processes, with dynamic load and unload) and it was, on paper, the natural answer to the problem of thirty heterogeneous artifacts. Its last release was v0.12.0 on 9 July 2023, and the kserve/modelmesh-serving repository was archived read-only on 14 April 2026. Red Hat published specific guidance for migrating to Standard mode. If somebody proposes ModelMesh in a design today, it is a sign that the documentation they read is three years old.

Anatomy of the control plane

`InferenceService`: predictor, transformer, explainer

The core CRD is InferenceService (serving.kserve.io/v1beta1), and its contribution, the one that became the common vocabulary of the field, is to break an inference service down into three components each with its own contract:

predictor: mandatory. It declares modelFormat, optionally runtime, the storageUri the weights come from and the protocolVersion it exposes.
transformer: optional. Pre- and post-processing. That it is a separate component, rather than code inside the server, is what allows it to be scaled separately (it is usually CPU-bound while the predictor is GPU-bound) and versioned separately.
explainer: optional, and the one that has aged worst: the V2 protocol does not support the explain endpoint, which only exists in V1. Anyone who needs explainability with a standard contract has a real gap here.

All three share ComponentExtensionSpec: minReplicas, maxReplicas, scaleTarget, scaleMetric, containerConcurrency, timeout and canaryTrafficPercent.

`ServingRuntime` and `ClusterServingRuntime`: the catalogue of trains

A ServingRuntime is a pod template that knows how to serve one or several formats. It declares supportedModelFormats (with name, version, autoSelect and priority), protocolVersions (v1, v2) and containers, with args parameterised by templates in the {{.Name}} style that are substituted with metadata from the InferenceService.

Selection is what makes this scale. If the InferenceService names a runtime, the controller looks for it first in the namespace and then at cluster level; if it does not name one, it picks among those with autoSelect: true for that modelFormat, breaking ties by priority. That is what lets a platform publish a curated catalogue (“here, huggingface is served with this vLLM and these flags”) without the product teams having to know what --gpu-memory-utilization is. In multi-tenant setups, the ClusterServingRuntimes belong to the platform and the InferenceServices to the teams: the cleanest separation-of-duties line KServe offers.

The storage initializer and `storageUri`

The storageUri field triggers the storage initializer: an init container that downloads the artifact to /mnt/models before the server starts. Schemes supported as of July 2026: s3://, gs://, https://, pvc://, hf://, oci:// and git repositories; credentials are attached via a ServiceAccount with annotated secrets. That oci:// connects with the second article of the series: the weights can live in the same registry as the images, with the same access control and the same signing capability.

The deployment modes

This is where nearly everything operational is decided. It is selected with the serving.kserve.io/deploymentMode annotation.

	KNative (formerly Serverless)	Standard (formerly RawDeployment)	ModelMesh
Resources emitted	Knative `Service` + `Revision`, `VirtualService` with Istio	`Deployment`, `Service`, `HPA`, Gateway API `HTTPRoute`	Its own high-density runtime
Dependencies	Knative Serving + Istio (or an alternative gateway)	Kubernetes 1.32+, cert-manager 1.15.0+, Gateway API v1.2.1 and a controller	Its own components
Autoscaling	KPA (concurrency / RPS)	HPA or KEDA	—
Scale-to-zero	Yes, with an activator that holds the request	No for HTTP requests	—
`canaryTrafficPercent`	Yes, by revision weights in Istio	Silently ignored (issue #5335, open since 2 April 2026)	—
Sidecars in the pod	`queue-proxy` (+ `istio-proxy`)	None	—
Status as of July 2026	Current	Current and the default mode of the LLM branch	Archived on 14 April 2026

That table has an uncomfortable reading: the two most quoted KServe capabilities, scale-to-zero and canary by percentage, live in the mode that drags Knative and Istio along, while the mode the project has pivoted towards is the other one.

The LLM branch: `LLMInferenceService`

Since v0.16 KServe has a separate CRD for LLMs: LLMInferenceService, in serving.kserve.io/v1alpha1. It is not an extension of the classic InferenceService and does not share its API version, and confusing the two is the first mistake made by anyone arriving from old documentation.

It is built on llm-d, the project that Red Hat, Google, IBM, CoreWeave and NVIDIA donated to the CNCF. The division of roles the project itself declares is clean: KServe is the control plane (lifecycle, scaling and operational governance) and llm-d contributes distributed scheduling, with GPU utilisation, queue depth, cache residency and SLA. In the analogy: llm-d decides which line each train takes; KServe decides which trains exist and when they are withdrawn.

The fields that matter: spec.model (URI, invocation name, scheduling criticality and LoRA adapters, with static adapter reconciliation from v0.19, which connects with multi-LoRA serving); spec.template (single node, or the decode pool); spec.worker (multi-node, which triggers LeaderWorkerSet); spec.prefill (the disaggregation we analysed in disaggregated serving turned into a declarative field); spec.parallelism (tensor, data and expert parallelism); and spec.router (gateway, route, scheduler), where KServe leans on the Gateway API Inference Extension, v1.5.0 of 19 April 2026, already GA, with automatic migration to InferencePool v1 from v0.19.

The endpoint it exposes is OpenAI-compatible: /v1/chat/completions with streaming, and since v0.18 also /v1/responses. That detail underpins the next section: the LLM branch of KServe does not speak V2, it speaks OpenAI.

For multi-node there is the kserve-huggingfaceserver-multinode runtime, based on vLLM over Ray, with workerSpec.tensorParallelSize and workerSpec.pipelineParallelSize. Its restrictions are hard: it requires a ReadWriteMany PVC, it only works in Standard mode, it does not support autoscaling and it requires exactly one head pod. A reference 4×H100 SXM node with NVLink needs none of this, because a 70B fits with tensor parallel 4 on a single node; you need it from there upwards.

The Open Inference Protocol: the contract itself

The routes and the payload

The Open Inference Protocol (historically the “V2 Inference Protocol”) lives in kserve/open-inference-protocol, with REST and gRPC specifications and versioning under SemVer 2.0. Its HTTP surface is deliberately small:

Resource	Verb	Route
Server metadata	GET	`/v2`
Liveness	GET	`/v2/health/live`
Server readiness	GET	`/v2/health/ready`
Model metadata	GET	`/v2/models/<model>[/versions/<v>]`
Model readiness	GET	`/v2/models/<model>[/versions/<v>]/ready`
Inference	POST	`/v2/models/<model>[/versions/<v>]/infer`

The payload is tensorial: id, a list of inputs (each with name, shape, datatype and data) and optionally the desired outputs. The types are BOOL, signed and unsigned integers from 8 to 64 bits, 16-, 32- and 64-bit floating point, and BYTES. In gRPC the same contract is GRPCInferenceService, with six RPCs: ServerLive, ServerReady, ModelReady, ServerMetadata, ModelMetadata and ModelInfer. For high-volume embeddings, the serialisation difference is not cosmetic.

Who actually implements it

The specification repository declares KServe, NVIDIA Triton, Seldon MLServer and Core v2, OpenVINO Model Server, AMD Inference Server and TorchServe as adopters. Not all entries are worth the same as of July 2026:

Triton implements it in a documented way (“Triton exposes HTTP/REST and gRPC endpoints based on the standard inference protocols proposed by the KServe project”), plus its own extensions. It is the de facto reference implementation.
OpenVINO Model Server exposes the KServe API over gRPC and REST, the TensorFlow Serving one and OpenAI-compatible endpoints: the clearest example of a server that speaks both track gauges.
TorchServe forces you to be honest. Its official documentation carries the warning: “This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches.” An adopter on zero maintenance is not an adopter: it is a row nobody has updated.

On the health of the standard, the most eloquent data point is not in any press release: the specification repository does not have a single published release, has accumulated something like thirty commits and seventy-five stars, and its governance consists of monthly meetings and a Slack channel. There is also a “vendor neutral” fork at open-inference/open-inference-protocol with more commits than the original and no traction whatsoever. V2 is a stable contract, widely implemented and barely evolving. For a serialisation format that can be a virtue; for a standard aspiring to cover the generative world, it is a death sentence.

The `generate` extension and why it did not win

The standard did try to cover generation: the specification contains a generate_rest.yaml with two routes, /v2/models/<m>/versions/<v>/generate and .../generate_stream, and a minimalist schema: a request with text_input and parameters (temperature, top_p, max_tokens defaulting to 20, stop, details); a response with text_output, model_name, model_version and, if requested, finish_reason and logprobs; streaming over server-sent events. It is clean, it is enough for a simple completion and nobody uses it: the KServe generative inference documentation, as of July 2026, lists four endpoints, and none of them is generate:

Task	Endpoint in KServe
Text generation	`openai/v1/completions`
Chat	`openai/v1/chat/completions`
Embeddings	`openai/v1/embeddings`
Reranking	`openai/v1/rerank`

The openai/ prefix exists, according to the documentation itself, “to avoid confusion with the V1 API protocol”, and it is changed with KSERVE_OPENAI_ROUTE_PREFIX. It is plumbing that says a lot: the in-house standard was relegated to a defensive prefix while the outside one takes all the traffic.

The tension, unvarnished

V2 solved the problem of 2021, and it solved it well. A tensorial, typed contract with metadata and health checks, in REST and gRPC, is what you need to serve a classifier, a vision model, a ranker or an embedding, and it is still the right answer for that half of the catalogue: swapping an embedding server from MLServer to OpenVINO Model Server without touching clients is the track gauge doing its job.

The LLM world standardised somewhere else, not by committee, but because the entire client ecosystem (SDKs, agent frameworks, gateways, evaluation) was written against the OpenAI API, and any engine that wanted to be used had to speak it. What that means for an architecture:

There is not one contract: there are two, and both are legitimate. Designing as if there were one does violence to half the catalogue.
LLM portability is superficial. It covers chat and completion well; as soon as you use detailed logprobs, guided decoding, priority headers, prefix caching parameters or tool calling with specific schemas, you leave the common subset and get tied back to the engine. It is the warning from structured output: what is portable is the envelope, not always the contents.
The gauge changer is the gateway. What makes the set switchable is the translation and routing layer of choosing an OSS LLM inference gateway and the router as an L7 gateway, not a single standard.
Crossing over loses the versioning. V2 has /versions/<v> in the route; the OpenAI API only has the model field. The version has to be encoded in the name (asistente-70b-2026-07) or managed in the control plane. That is the difference between being able to audit which version answered and not being able to.

Four manifests you can actually apply

Platform runtime for the blog’s reference node, 4×H100 SXM 80 GB with NVLink, with tensor parallel 4:

apiVersion: serving.kserve.io/v1alpha1
kind: ClusterServingRuntime
metadata:
 name: vllm-h100-tp4
spec:
 annotations:
 prometheus.kserve.io/port: "8080"
 prometheus.kserve.io/path: "/metrics"
 supportedModelFormats:
 - name: huggingface
 version: "1"
 autoSelect: true
 priority: 10
 protocolVersions:
 - v2
 containers:
 - name: kserve-container
 # ALWAYS pin by digest in production; the tag is only for readability
 image: vllm/vllm-openai:v0.11.0
 command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
 args:
 - --port=8080
 - --model=/mnt/models
 - --served-model-name={{.Name}}
 - --tensor-parallel-size=4
 - --max-model-len=32768
 - --gpu-memory-utilization=0.92
 - --enable-prefix-caching
 - --kv-cache-dtype=fp8
 resources:
 requests: { cpu: "16", memory: 128Gi, nvidia.com/gpu: "4" }
 limits: { cpu: "16", memory: 128Gi, nvidia.com/gpu: "4" }
 # the probe that stops the pod dying while the model loads
 startupProbe:
 httpGet: { path: /health, port: 8080 }
 periodSeconds: 15
 failureThreshold: 40 # up to 10 minutes of loading
 readinessProbe:
 httpGet: { path: /health, port: 8080 }
 periodSeconds: 10
 livenessProbe:
 httpGet: { path: /health, port: 8080 }
 periodSeconds: 30

The InferenceService that consumes it, with KEDA on vLLM metrics:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
 name: asistente-70b
 namespace: inferencia
 annotations:
 serving.kserve.io/deploymentMode: "Standard"
 serving.kserve.io/autoscalerClass: "keda"
spec:
 predictor:
 minReplicas: 1
 maxReplicas: 4
 model:
 runtime: vllm-h100-tp4
 modelFormat: { name: huggingface }
 # the OCI registry from article 2/4 of this series
 storageUri: "oci://registry.example.internal/modelos/asistente-70b:2026-07"
 autoScaling:
 metrics:
 - type: External
 external:
 metric:
 backend: "prometheus"
 serverAddress: "http://prometheus.monitoring.svc.cluster.local:9090"
 query: 'sum(vllm:num_requests_waiting{model_name="asistente-70b"})'
 target:
 type: Value
 value: "8"

The canary, with its small print:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
 name: asistente-70b
 namespace: inferencia
 annotations:
 # WATCH OUT: canaryTrafficPercent only takes effect in KNative mode.
 # In Standard mode it is silently ignored (issue kserve/kserve#5335, open).
 serving.kserve.io/deploymentMode: "KNative"
spec:
 predictor:
 canaryTrafficPercent: 10
 minReplicas: 1
 maxReplicas: 4
 model:
 runtime: vllm-h100-tp4
 modelFormat: { name: huggingface }
 storageUri: "oci://registry.example.internal/modelos/asistente-70b:2026-08"

KServe keeps three references in the status (LatestReadyRevision, LatestRolledoutRevision and PreviousRolledoutRevision): promoting means setting the percentage to 100, reverting means pinning traffic on the previous one, and a revision that does not become ready automatically receives zero traffic.

And the two calls. Against the V2 contract, for a classifier served with MLServer or Triton:

# metadata: which model it is, what inputs it expects, what types
curl -s http://clasificador.inferencia.example/v2/models/clasificador-tickets | jq .

# readiness of the specific model, not of the server
curl -s -o /dev/null -w '%{http_code}\n' \
 http://clasificador.inferencia.example/v2/models/clasificador-tickets/ready

curl -s -X POST \
 http://clasificador.inferencia.example/v2/models/clasificador-tickets/infer \
 -H 'Content-Type: application/json' \
 -d '{ "id": "req-4711",
 "inputs": [ { "name": "input-0", "shape": [1, 4], "datatype": "FP32",
 "data": [5.1, 3.5, 1.4, 0.2] } ],
 "outputs": [ { "name": "output-0" } ] }'

Against the same control plane, but on the OpenAI-compatible surface of the LLM:

curl -s -N -X POST \
 http://asistente-70b.inferencia.example/openai/v1/chat/completions \
 -H 'Content-Type: application/json' \
 -d '{ "model": "asistente-70b",
 "messages": [ {"role": "user",
 "content": "Summarise last night'\''s incident report."} ],
 "max_tokens": 256,
 "stream": true }'

Two contracts and two payload shapes, with the same InferenceService as the unit of governance. That is the value proposition of a control plane: the contract is chosen by the model and the management is single.

Autoscaling and the problem of zero

Three autoscalers coexist, and choosing the wrong one is the most common cause of disappointment. KPA, in KNative mode, scales by concurrency or RPS and is the only one that goes down to zero in a usable way, because Knative provides an activator that holds the request while the pod starts; its metric does not distinguish a 30-token request from an 8,000-token one. HPA, in Standard, scales by CPU or memory: for LLMs it is useless, because the CPU of a saturated vLLM pod and that of an idle one look too much alike. KEDA, in Standard, is the right option, the one we develop in autoscaling with KEDA: it is enabled with serving.kserve.io/autoscalerClass: "keda" and autoScaling.metrics accepts external metrics against Prometheus (vllm:num_requests_running or vllm:num_requests_waiting) or, with the OpenTelemetry add-on, pushed metrics. Since v0.18 there is also Workload Variant Autoscaling, which uses HPA or KEDA as a backend.

That leaves the serious problem, which no autoscaler solves: zero. A 70B model in fp8 is on the order of 70 GB to move into HBM, and that path, from disk to HBM, is measured in minutes: the first request after idling takes that long, or it fails.

Worse: the installation documentation is explicit as of July 2026: “Scale from Zero is currently not supported in Standard mode for HTTP requests”. The mode the project has pivoted towards, and the only one that supports KEDA, does not know how to come up from zero on an HTTP request: KEDA can go down to zero (default cooldown of 300 seconds), but the way back requires an activator that Standard does not have. Anyone who wants real zero with HTTP activation still needs Knative.

The levers, in order of effectiveness: do not scale the hot model to zero (minReplicas: 1 for anything with an SLA; zero is for the long tail); attack the load time, the only solution that is not a patch, with the techniques from speeding up cold start; use LocalModelCache, available since v0.15 and extended to LLMInferenceService in v0.19, to preload artifacts into a node PVC; and set correct probes, which is what comes next.

Operational traps

The probe that kills the pod while the model loads. The most frequent and the most expensive to diagnose, because the symptom is a CrashLoopBackOff with no error in the server logs. Until early 2026 the KServe vLLM templates used initialDelaySeconds of 120 to 300 seconds on liveness, with the result described in issue #5062 (12 February 2026, now resolved): the fast models waited too long and the large ones died before finishing loading. The fix applied: a startupProbe of up to ten minutes and probes with no initial delay. A legacy runtime has to be reviewed by hand. In KNative mode there is a worse variant (issue #3795, closed as not planned): what dies is not the model container but the queue-proxy, which does not respond OK within the deadline and takes the whole pod down with it.

canaryTrafficPercent silently ignored. It deserves its own trap because there is no error and no condition in the status to give it away: in Standard mode the field does nothing. The team believes it is serving 10 % with the new version and it is serving 100 %. While issue #5335 remains open, in Standard the canary is done with HTTPRoute weights by hand, or in the gateway.

Canary and scarce GPUs do not get along. A 10 % canary of a 70B with tensor parallel 4 costs a whole 4-GPU node, not 10 % of the resources, because the minimum granularity is a complete replica. During the window there are two versions occupying twice the hardware, or capacity has to be taken from the stable one. That is the difference between a microservice canary and a model canary, and the reason a shadow over duplicated traffic often pays off better.

storageUri and credentials. Storage initializer failures are always the same ones: a ServiceAccount without the annotated secret, a KMS-encrypted bucket with a role that has no decryption permission, an S3-compatible but not exactly S3 endpoint, or intermittent 403s, with a specific fix in v0.18. It is diagnosed in the init container logs, not in the server’s. And one standing recommendation: no downloading over https:// from the internet in production; that is precisely the problem of article 2/4.

The model size against the emptyDir. By default the artifact lands in an ephemeral volume on the node’s disk. Seventy gigabytes per replica fill a system disk with surprising ease, and then the kubelet evicts pods for disk pressure, including ones that have nothing to do with it. You have to set sizeLimit or use a dedicated pvc://. For multi-node the restriction is harder, with a mandatory ReadWriteMany PVC, which rules out a good part of local block storage and pushes you towards shared file systems, with the consequences covered in storage for AI.

The dependency on Knative and Istio. The real cost of KNative mode: two more control planes to version and correlate with Kubernetes, and two sidecars, queue-proxy and istio-proxy, in every pod, with their CPU and memory consumption and their network hop per request. On a platform that already has a mesh it is a marginal cost; on a cluster dedicated to inference it has to be justified by the capabilities you are actually going to use.

Observability: the name of the metric. The ServingRuntime declares prometheus.kserve.io/port and /path annotations, but the metrics that come out are the engine’s, not KServe’s: vllm:num_requests_waiting, vllm:time_to_first_token_seconds, vllm:gpu_cache_usage_perc. The control plane does not normalise names across runtimes, so a dashboard written against vLLM is no use for Triton, and correlating with DCGM is still your own work. v0.19 raises the HPA/KEDA scaling status into the service conditions and emits events on readiness transitions: exactly what was needed to alert on the control plane and not only on the engine.

Decision map: when KServe, and when it is overkill

The useful comparison is not “KServe versus OME, vLLM Production Stack, Dynamo or llm-d”. Those four are covered in LLM inference operators and compete on a different axis:

Axis	What gets decided	Who competes
Contract and control plane	Which CRD describes a service, which API clients see, how the model is versioned, how the rollout is done, how LLM and non-LLM coexist	KServe, Seldon Core v2
Engine and topology	Which engine runs, how prefill/decode is disaggregated, how routing is done with cache awareness, how multi-node tensor parallel is done	vLLM Production Stack, OME, NVIDIA Dynamo, llm-d

The axes are orthogonal, and LLMInferenceService proves it: KServe did not reimplement distributed scheduling, it leaned on llm-d. Choosing KServe is not choosing instead of an LLM operator: it is choosing the contract layer that operator lives on.

Choose KServe when the catalogue is heterogeneous, LLMs plus embeddings, rerankers, classifiers and vision, which is its decisive advantage and the one none of the four operators has: a service like the one in TEI in production fits into the same vocabulary as the LLM. Also when you need a contract decoupled from the engine because you have clients you do not control; when you need declarative, auditable governance (which artifact, from which origin and with which runtime, in an object versionable in git and subject to admission policies), the piece that makes the rest of this series possible; and when you want a catalogue of platform runtimes separated from the teams’ services.

KServe is overkill when you serve one model, on one node, with up to three replicas: the Deployment from vLLM on Kubernetes is the right answer. When your workload is LLM only and the priority is performance per GPU, because there the value is in the engine. When you are going to use neither scale-to-zero nor canary by revision, because what remains is a Deployment with more YAML. And when you already have an LLM operator and only serve LLMs: putting KServe on top for completeness is adding CRDs without adding decisions.

For an inference factory

First: separate the runtime catalogue from the service catalogue, and do it on day one. The ClusterServingRuntimes belong to the platform; the InferenceServices, to the teams. That boundary turns “serving the thirtieth model” into a five-line change instead of a negotiation about vLLM flags, and it fits with GitOps with Flux: the runtime is reviewed as platform code, the service as product configuration.

Second: accept the two contracts and put the gauge changer where it belongs. V2 for the tensorial world, the OpenAI API for the generative one, and the translation in the gateway, not in every client. And since the OpenAI API does not version in the route, encode the version in the model name and make that name the same identifier that appears in the registry, in the signature and in the traces. That is what makes a request traceable back to the artifact.

Third: measure start-up before promising elasticity. The time from the pod being scheduled to the readiness probe going green, with the real model and from the real origin, decides whether scale-to-zero is a FinOps lever or an incident machine. That number, and not the price of the GPU-hour, is the one to take into the cost conversation.

And here this piece falls short on purpose. KServe gives you the contract, the lifecycle and the rollout, but its whole trust model hangs off a text string: storageUri. The storage initializer downloads whatever is at that destination and puts it in front of the GPU without asking anything else; nothing in the InferenceService guarantees that the artifact is the one that was approved, that it has not changed since the last audit, or that the origin is legitimate. The next article goes there: where the model bytes come from, and why an OCI registry with ORAS, Harbor, MLflow or the Kubeflow Model Registry is the difference between distributing models and scattering them.

Sources

CNCF, KServe becomes a CNCF incubating project (11 de noviembre de 2025) — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
CNCF, KServe project page (incubating, aceptado el 29 de septiembre de 2025) — https://www.cncf.io/projects/kserve/
CNCF TOC, Project Moving Levels Checklist: KServe joining CNCF at Incubation level — https://github.com/cncf/toc/issues/1905
KServe, Releases (v0.15.0, v0.16.0, v0.18.0, v0.19.0) — https://github.com/kserve/kserve/releases
KServe, Inference Protocol V2 (Open Inference Protocol) — https://kserve.github.io/website/docs/concepts/architecture/data-plane/v2-protocol
KServe, open-inference-protocol (especificación, adoptantes, gobernanza) — https://github.com/kserve/open-inference-protocol
KServe, open-inference-protocol — generate_rest.yaml — https://github.com/kserve/open-inference-protocol/blob/main/specification/protocol/generate_rest.yaml
open-inference, Vendor neutral fork of kserve/open-inference-protocol — https://github.com/open-inference/open-inference-protocol
KServe, Generative Inference — Runtime Overview (endpoints con prefijo openai/) — https://kserve.github.io/website/docs/model-serving/generative-inference/overview
KServe, Understanding LLMInferenceService — https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview
KServe, Cloud-Native AI Inference at Scale using KServe and llm-d (5 de marzo de 2026) — https://kserve.github.io/website/blog/cloud-native-ai-inference-kserve-llm-d
KServe, Serving Runtime (ServingRuntime y ClusterServingRuntime) — https://kserve.github.io/website/docs/concepts/resources/servingruntime
KServe, Kubernetes Deployment Installation Guide (K8s 1.32+, cert-manager 1.15.0+, Gateway API v1.2.1, sin scale-from-zero en Standard) — https://kserve.github.io/website/docs/admin-guide/kubernetes-deployment
KServe, Canary Rollout Strategy — https://kserve.github.io/website/docs/model-serving/predictive-inference/rollout-strategies/canary
KServe, Autoscaling with KEDA — https://kserve.github.io/website/docs/model-serving/predictive-inference/autoscaling/keda-autoscaler
KServe, Multi-node/Multi-GPU Inference (PVC RWX, sin autoescalado) — https://kserve.github.io/website/docs/model-serving/generative-inference/multi-node
KServe, Storage Options for Model Artifacts — https://kserve.github.io/website/docs/model-serving/storage/overview
KServe, issue #5335 — Support canaryTrafficPercent for RawDeployment mode via Gateway API HTTPRoute weights (abierta, 2 de abril de 2026) — https://github.com/kserve/kserve/issues/5335
KServe, issue #5062 — Add startupProbe to vLLM main containers (12 de febrero de 2026) — https://github.com/kserve/kserve/issues/5062
KServe, modelmesh-serving (última release v0.12.0 de julio de 2023; repositorio archivado el 14 de abril de 2026) — https://github.com/kserve/modelmesh-serving
Red Hat, Converting ModelMesh and Serverless InferenceServices to RawDeployment (Standard) Mode — https://access.redhat.com/articles/7134025
NVIDIA, Triton Inference Server — Inference Protocols and APIs — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/customization_guide/inference_protocols.html
OpenVINO, What is OpenVINO Model Server (API KServe, TFS y endpoints compatibles con OpenAI) — https://docs.openvino.ai/2026/model-server/ovms_what_is_openvino_model_server.html
PyTorch, TorchServe — Notice: Limited Maintenance — https://docs.pytorch.org/serve/
Kubernetes SIG Network, Gateway API Inference Extension (v1.5.0, 19 de abril de 2026) — https://github.com/kubernetes-sigs/gateway-api-inference-extension

LLM serving engines compared on the Pareto frontier: vLLM, SGLang, TRT-LLM and Dynamo

Tue, 16 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter). Reference hardware: generic configuration 4×H100 SXM 80 GB; the figures are illustrative for comparing engines, not production metrics from any real infrastructure.

TL;DR

Four OSS engines dominate on-premise LLM serving in 2026: vLLM, SGLang, NVIDIA TensorRT-LLM and NVIDIA Dynamo. None is superior on every axis. In raw throughput with a shared prompt, SGLang beats vLLM by 29 % on Llama-3.3-70B FP8 over H100 (16,200 vs 12,500 tok/s). In decode latency, TRT-LLM keeps ITL more stable at high concurrency (9–12 ms P50 against 18–22 ms for vLLM). Dynamo is not an inference engine: it is the orchestration layer on top of the previous three for multi-node clusters, and it reports up to 7× additional throughput in disaggregated serving on Blackwell. The right choice depends on three variables: load pattern (batch/interactive), deployment scale (single node/multi-node) and operational constraints (OSS maturity, ease of operation). The Pareto decision table is at the end of the article.

Framing the article: what this B8 is and is not

This article (B8 of the benchmarking pillar) takes for granted the metrics from the introduction to benchmarking and the tool catalogue from the card-by-card piece. It applies those concepts to engine selection. The protocol from GuideLLM for SLO validation and AIPerf in depth detail how to run the harness; here the result is used. The fundamentals of continuous batching, disaggregated serving, the attention backend and quantisation are assumed known.

Card for each engine

vLLM

Attribute	Value
Reference version	v0.9.x (June 2026)
Licence	Apache 2.0
Maintainer / governance	vLLM Project (Linux Foundation AI), led by UC Berkeley + Red Hat
Supported hardware	NVIDIA (Ampere/Hopper/Blackwell), AMD ROCm, Intel Gaudi, TPU, CPU
Repository	github.com/vllm-project/vllm
API	OpenAI-compatible (completions, chat, embeddings)

Origin: PagedAttention (Kwon et al., 2023). The first implementation of continuous batching with a paged KV cache. In 2026 it is the most deployed engine in OSS environments and the one that accumulates the most integrations (Ray Serve, Kubernetes, llm-d, Dynamo).

SGLang

Attribute	Value
Reference version	v0.4.x (June 2026)
Licence	Apache 2.0
Maintainer / governance	LMSYS Org (Berkeley, CMU, UCSD)
Supported hardware	NVIDIA (Ampere/Hopper/Blackwell), AMD ROCm
Repository	github.com/sgl-project/sglang
API	OpenAI-compatible + SGLang runtime API

Origin: the SGLang paper, 2024 (Zheng et al.). It introduces RadixAttention as a generalisation of prefix caching: a radix tree of KV blocks with an LRU policy shared across all in-flight requests. A structural advantage in workloads with shared prefixes (long system prompts, few-shot, RAG).

NVIDIA TensorRT-LLM (TRT-LLM)

Attribute	Value
Reference version	v0.18.x (June 2026)
Licence	Apache 2.0
Maintainer / governance	NVIDIA (proprietary in practice, OSS in name)
Supported hardware	NVIDIA only: Ampere (A100), Hopper (H100/H200/GH200), Ada (L40/L40S), Blackwell (B200/GB200)
Repository	github.com/NVIDIA/TensorRT-LLM
API	Python + C++ runtime; integrates with Triton Inference Server

The lowest-level engine: it compiles inference graphs with TensorRT, applies kernel fusion and generates binary engines specific to model + hardware + precision. Maximum performance on NVIDIA, with no portability to other hardware. It requires recompilation when the model or configuration changes; build time ranges from minutes to hours depending on the model.

NVIDIA Dynamo

Attribute	Value
Reference version	v1.1.1 (May 2026)
Licence	Apache 2.0
Maintainer / governance	NVIDIA + OSS community (ai-dynamo org on GitHub)
Supported hardware	NVIDIA (through the underlying engine: vLLM, SGLang or TRT-LLM)
Repository	github.com/ai-dynamo/dynamo
Role	Orchestration layer, NOT an inference engine

Dynamo does not run inference directly: it is the scheduler and router that coordinates pools of workers (vLLM, SGLang, TRT-LLM) across multiple nodes. Implemented in Rust (performance) + Python (extensibility). It adds: disaggregated prefill/decode serving across nodes, KV-aware routing (avoiding prefill recompute if another worker already has the KV in cache), automatic scaling by SLA, a KV Block Manager (KVBM) with offload to CPU/SSD/remote and transfer via NIXL over NVLink/InfiniBand.

Honourable mentions: LMDeploy and HF TGI

LMDeploy (OpenMMLab/InternLM, Apache 2.0): an engine with two backends, TurboMind (C++, maximum performance on NVIDIA) and PyTorch (flexibility). KV cache blocking and persistent batching; 4-bit inference up to 2.4× faster than FP16 in its own benchmark. A narrower ecosystem; no AMD support.

HF TGI (Hugging Face, Apache 2.0): the reference engine for the HuggingFace ecosystem. TGI v3.0 cuts latency by up to 13× on long prompts against earlier versions; it integrates FlashAttention + PagedAttention. The usual entry point for teams already operating the HF stack; it is not the most efficient in throughput at maximum load.

Capability matrix

The table uses this notation: Yes = supported and stable, Beta = available but not recommended for production, No = not available or only with a manual workaround.

Capability	vLLM	SGLang	TRT-LLM	Dynamo (via backend)
Continuous batching	Yes	Yes	Yes	Yes (delegates to the engine)
PagedAttention	Yes	Yes	Yes	Yes
RadixAttention / prefix caching	Yes (chunked prefill + prefix)	Yes (native RadixAttention)	Yes (prefix caching)	Yes + KV-aware routing
Speculative decoding	Yes (n-gram, EAGLE, DFlash, suffix)	Yes (draft+verify)	Yes (EAGLE-3, MTP)	Yes
FP8 quantisation	Yes	Yes	Yes	Yes
MXFP4/NVFP4 quantisation	Yes	Beta	Yes (Blackwell)	Yes
INT4 AWQ quantisation	Yes	Yes	Yes	Yes
GPTQ quantisation	Yes	Yes	Not native	Yes (via vLLM)
INT8 quantisation (SmoothQuant)	Yes	Yes	Yes	Yes
Structured output	Yes	Yes (compressed FSM)	Beta	Yes
Dynamic multi-LoRA	Yes	Yes	Beta	Yes
Disaggregated prefill/decode	Beta (experimental)	Yes	Yes	Yes (first class)
Chunked prefill	Yes	Yes	Yes	Yes
Multimodal support (VLM)	Yes	Yes	Yes	Yes
Multi-node tensor parallelism	Yes	Yes	Yes	Yes (+ pipeline across nodes)
OpenAI API compatible	Yes	Yes	Yes (via Triton)	Yes
Non-NVIDIA hardware	Yes (AMD/Intel/TPU)	Yes (AMD)	No	No

Sources for the matrix: docs.vllm.ai, docs.sglang.io, nvidia.github.io/TensorRT-LLM, docs.nvidia.com/dynamo, github.com/ai-dynamo/dynamo (feature matrix, June 2026).

Why they are not compared lightly: the variables that break a comparison

Before the table of figures, the four decisions that invalidate any cross-engine benchmark:

1. A different harness. A benchmark run with each engine’s native micro-bench (vllm bench serve vs SGLang bench) measures the client, not the engine. The discrepancy can be up to 7.2× at high concurrency. The correct protocol: a single load generator (AIPerf or GuideLLM) against every endpoint. The engine is the variable, the tool is the constant.

2. Different ISL/OSL. Input Sequence Length and Output Sequence Length determine the prefill/decode ratio and therefore which engine is favoured. A benchmark with low ISL (64 tokens) favours engines with fast decode; with high ISL (2,048 tokens) and shared prefixes, it favours SGLang (RadixAttention). Fixing ISL and OSL is mandatory for the comparison to be valid.

3. Different precision. FP16 and FP8 are not comparable: FP8 gives between 1.5× and 2× more throughput on Hopper. Comparing vLLM FP16 with TRT-LLM FP8 says nothing about the engine.

4. Not reporting P99. Maximum throughput is reached at the point where TTFT P99 already violates the SLO. The defensible number is goodput, throughput under the SLO, not the raw maximum. The end-to-end latency formula:

$$\text{latency}_{e2e} \approx \text{TTFT} + (N_{\text{out}} - 1) \times \text{TPOT}$$

where $N_{\text{out}}$ is the number of output tokens, makes it clear that a high TTFT with low TPOT has the same effect as the inverse case only for one specific $N_{\text{out}}$: both must be reported separately.

The complete fair-comparison protocol is in the tools article.

Performance on the Pareto frontier: illustrative figures

Reference configuration for every row: 4×H100 SXM 80 GB NVLink, model Llama-3.1-70B-Instruct FP8, harness AIPerf (formerly genai-perf, the multi-process successor), dataset ShareGPT (realistic length distribution), mean ISL ~512 tok, mean OSL ~256 tok, SLO TTFT P99 < 500 ms. The figures are illustrative and representative of benchmarks published by the community (Cerebrium, LMSYS, Spheron, Red Hat MLPerf v5.1); they are not measurements from any real infrastructure. Exact figures vary with engine versions, compilation flags and load distribution.

Throughput vs latency: Pareto table

Engine	Config	Throughput (tok/s)	TTFT P50 (ms)	TTFT P99 (ms)	ITL P50 (ms)	Goodput (@SLO)
vLLM v0.9	FP8, chunked prefill	12,500	160	420	18	~96 %
SGLang v0.4	FP8, RadixAttention	16,200	140	390	21	~97 %
TRT-LLM v0.18	FP8, compiled engine	14,800	190	480	10	~93 %
vLLM + Dynamo	FP8, disagg. P/D 2+2	18,500	120	310	19	~98 %
SGLang + Dynamo	FP8, disagg. P/D 2+2	21,000	110	280	22	~99 %
TRT-LLM + Dynamo	FP8, disagg. P/D 2+2	22,500	130	350	11	~97 %

Reading notes:

“Disagg. P/D 2+2” = 2 prefill GPUs + 2 decode GPUs (the same 4×H100 budget).
Goodput falls against raw throughput when the latency tail exceeds the SLO of 500 ms P99. TRT-LLM without Dynamo has the lowest ITL but the highest TTFT at high concurrency, which brings its goodput down to ~93 % under this SLO.
Dynamo adds ~30–40 % of effective throughput over 4 GPUs thanks to disaggregated serving and KV-aware routing, in the scenario with reusable prefixes (ShareGPT).
In workloads without shared prefixes (short ISL, unique prompts), Dynamo’s advantage shrinks to ~10–15 % over the base engine.

Saturation point (concurrency sweep, vLLM FP8, 4×H100)

Concurrency	TTFT P99 (ms)	ITL P50 (ms)	Throughput (tok/s)	Goodput (tok/s)
8	210	14	5,200	5,200
16	370	18	9,800	9,800
32	420	18	12,500	12,200
48	680	26	13,800	7,400
64	1,200	41	14,100	2,100

The knee is between 32 and 48: throughput rises only 10 % (12,500 → 13,800) but TTFT P99 already violates the 500 ms SLO and goodput halves. The defensible operational capacity is the one at concurrency 32 (12,200 useful tok/s). Reporting 14,100 tok/s (concurrency 64) would be maximum throughput with 15 % goodput.

Fair-comparison methodology: the harness as the constant

The protocol that makes numbers comparable across different engines. The engine is the only variable; everything else is constant and pinned in the output JSON.

Variables that are fixed (constants of the experiment)

Variable	Value fixed in the reference experiment
Load tool	AIPerf v2.x (multi-process)
Model	Llama-3.1-70B-Instruct
Precision	FP8
Hardware	4×H100 SXM 80 GB NVLink
Dataset	ShareGPT (realistic ISL/OSL distribution)
SLO	TTFT P99 < 500 ms, ITL P50 < 30 ms
Warm-up	200 requests discarded before measuring
Sweep	concurrencies 1, 4, 8, 16, 32, 48, 64
Primary metric	goodput (useful tok/s under SLO)

Harness steps

Deploy the engine with the exact config (pinned flags, fixed version).
Warm-up: 200 requests; discard the results.
Concurrency sweep: aiperf profile --concurrency-range 1:64:step with AIPerf.
Extend the sweep past the knee (until goodput falls below 50 %).
Collect the JSON: TTFT P50/P99, ITL P50, raw throughput, goodput.
Change only the engine; repeat steps 1–5.
Compare the goodput column across rows.

The sweep with GuideLLM is equivalent for SLO validation; AIPerf is preferable for engine comparisons because its automatic estimatedCapacity normalises the saturation point. Both can be used: GuideLLM for the operational SLO, AIPerf for reference capacity.

Reproducible output format

{
 "harness": "aiperf", "harness_version": "2.x",
 "model": "Llama-3.1-70B-Instruct", "precision": "FP8",
 "hardware": "4xH100_SXM_80GB_NVLink",
 "dataset": "sharegpt",
 "slo": {"ttft_p99_ms": 500, "itl_p50_ms": 30},
 "motor": "vllm", "motor_version": "0.9.x",
 "concurrency_sweep": [8, 16, 32, 48, 64],
 "results_at_knee": {
 "concurrency": 32,
 "ttft_p50_ms": 210, "ttft_p99_ms": 420,
 "itl_p50_ms": 18, "throughput_tok_s": 12500,
 "goodput_tok_s": 12200
 }
}

This versioned JSON is the auditable datum: anyone can reproduce the figure with the same tool, version, model, hardware and load.

State of the art 2026: relative maturity

Dimension	vLLM	SGLang	TRT-LLM	Dynamo
Production maturity	High (3+ years)	Medium-high (2 years)	High (NVIDIA)	Medium (1.0 GA Mar 2026)
Release cadence	Weekly	Weekly	Monthly	Fortnightly
Integration ecosystem	Very broad (Ray, k8s, llm-d, Dynamo, NIM)	Broad (Dynamo, k8s)	NVIDIA-centric (Triton, NIM)	NVIDIA-centric
Ease of operation	High (pip install, OpenAI API)	High	Medium (requires compiling engines)	Medium-low (multi-node, etcd/NATS)
Non-NVIDIA hardware	Yes (AMD, Intel, TPU)	Yes (AMD)	No	No
Disaggregated serving	Beta	Stable	Stable	First class
Dynamic multi-LoRA	Stable	Stable	Beta	Stable (via vLLM/SGLang)
Structured output	Stable	Stable (FSM)	Beta	Stable (via the engine)
OSS community support	Very high (>30 K GitHub stars)	High (~20 K stars)	High (~10 K stars)	Growing (6.8 K stars)

Pareto decision table: when to choose each engine

No prose; the decision as a table of criteria and outcome.

Selection criterion	Recommended engine	Quantitative justification
Single node, quick prototype, team with no LLM serving experience	vLLM	pip install + OpenAI API in minutes; the broadest community support
Workload with long shared prefixes (RAG, few-shot, system prompts > 512 tok)	SGLang	RadixAttention: up to 6.4× more throughput vs a baseline without prefix caching
Maximum throughput on NVIDIA, no portability, team with TRT experience	TRT-LLM	Fused kernels + native FP8: ITL 9–12 ms vs 18–22 ms for vLLM on the same hardware
Multi-node (> 8 GPUs), mixed prefill-intensive/decode-intensive traffic	Dynamo + SGLang	Disaggregated P/D: up to 7× additional throughput on Blackwell; 2× TTFT with KV-aware routing
AMD ROCm or Intel Gaudi hardware	vLLM	The only SOTA engine with stable non-NVIDIA support
Structured output + high concurrency	SGLang	Compressed FSM: guided decode with no appreciable overhead
Stable dynamic multi-LoRA in production	vLLM or SGLang	TRT-LLM multi-LoRA still in beta (Jun 2026)
Hugging Face ecosystem, models < 13B	HF TGI v3	Direct HF Hub integration; competitive latency on small models
Maximum energy efficiency (J/token) on NVIDIA	TRT-LLM	Lower-overhead kernels → fewer J/token at iso-throughput

Quantisation: effect on throughput and quality

The four main options in order of decreasing speed and increasing quality:

Format	Relative throughput (H100)	Quality degradation (perplexity)	Engines with stable support
NVFP4 / MXFP4	~2.2× vs FP16	1–3 % on MMLU	TRT-LLM (Blackwell), vLLM (experimental)
FP8 (W8A8)	~1.7× vs FP16	< 1 % on MMLU	vLLM, SGLang, TRT-LLM
INT4 AWQ	~1.5× vs FP16	1–2 % on MMLU	vLLM, SGLang, TRT-LLM
GPTQ (INT4)	~1.4× vs FP16	1–3 % on MMLU	vLLM, SGLang
FP16 (baseline)	1.0×	0 %	All

FP8 is the de facto standard on Hopper for production: 1.7× throughput with degradation below 1 %. The article on quantisation in depth develops the quality/speed trade-off by format and model architecture.

Disaggregated serving: when the overhead is worth it

Separating prefill/decode into independent pools introduces transfer latency for KV tensors between nodes (NIXL over NVLink or InfiniBand). The overhead pays off when:

Condition	Effect of disaggregated serving
Mean ISL > 1,024 tokens	Prefill dominates; separate pools stop it blocking decode
Prefill/decode ratio > 3:1 in time	The decode pool sits idle waiting for prefill
Mixed traffic (prefill burst + decode queue)	Independent scaling of each pool by SLA
< 512 tokens ISL, unique prompts	Transfer overhead exceeds the benefit; an aggregated engine is enough

Dynamo quantifies the break-even point: disaggregated serving pays off when the transfer time for KV tensors (a function of ISL and NVLink/IB bandwidth) is lower than the time the decode worker would spend waiting for the prefill worker in aggregated mode. On 4×H100 with an ISL of 512 tok, break-even occurs at a prefill/decode ratio > 2.5:1. The article on disaggregated serving develops the full formula.

MLPerf Inference v5.1: cross-vendor reference figures

MLPerf Inference v5.1 (September 2025) is the standard for cross-vendor comparability. Server scenario (Llama-3.1-70B, tokens/s):

Submitter	Hardware	Scenario	Throughput (tok/s)
Red Hat (vLLM)	1×H100 80 GB	Server	5,103
Red Hat (vLLM)	1×L40S	Server	1,207
NVIDIA (TRT-LLM)	8×H100 SXM	Server	~52,000 (estimated)

The MLPerf figures are not directly comparable with the internal benchmarks above (a different harness, MLPerf’s Server scenario with a Poisson distribution and fixed SLOs). They are useful for comparing vendors under the same rules; not for sizing a specific case. For that, run your own sweep with AIPerf/GuideLLM against the target endpoint.

Variables the benchmark does not capture

Omitted variable	Effect in production
Network latency (gateway, load balancer)	Can add 20–100 ms to the TTFT measured on the server
Engine compilation time (TRT-LLM)	Hours per model×precision×hardware; penalises cold starts and CI
Weight loading time on the GPU	Relevant in autoscaling; ModelExpress (Dynamo) cuts it 7×
Operational cost of the system (Dynamo vs a simple engine)	Dynamo requires etcd/NATS, a Rust runtime, a more specialised team
Response quality	Performance does not imply quality; measure with lm-evaluation-harness (article B7)
Energy (J/token)	Does not come out of a throughput benchmark; requires DCGM running alongside

Sources

vLLM · documentación oficial — https://docs.vllm.ai/en/latest/
vLLM · blog de rendimiento (v0.6.0, 2,7× throughput) — https://blog.vllm.ai/2024/09/05/perf-update.html
vLLM · disaggregated prefill (experimental) — https://docs.vllm.ai/en/latest/features/disagg_prefill/
SGLang · documentación oficial — https://docs.sglang.io/
SGLang · paper RadixAttention (LMSYS, 2024) — https://arxiv.org/pdf/2312.07104
SGLang · blog LMSYS sobre Llama-3 serving — https://www.lmsys.org/blog/2024-07-25-sglang-llama3/
NVIDIA TensorRT-LLM · GitHub — https://github.com/NVIDIA/TensorRT-LLM
NVIDIA TensorRT-LLM · overview y release notes — https://nvidia.github.io/TensorRT-LLM/overview.html
NVIDIA TensorRT-LLM · quantización — https://nvidia.github.io/TensorRT-LLM/features/quantization.html
NVIDIA Dynamo · GitHub (ai-dynamo/dynamo) — https://github.com/ai-dynamo/dynamo
NVIDIA Dynamo · documentación oficial — https://docs.nvidia.com/dynamo/latest
NVIDIA Dynamo · disaggregated serving — https://docs.dynamo.nvidia.com/dynamo/design-docs/disaggregated-serving
Cerebrium · benchmark vLLM vs SGLang vs TRT-LLM (Llama 3.1-70B) — https://cerebrium.ai/blog/benchmarking-vllm-sglang-tensorrt-for-llama-3-1-api
Spheron · vLLM vs TRT-LLM vs SGLang H100 benchmarks 2026 — https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/
MLCommons · MLPerf Inference v5.1 results — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
Red Hat · MLPerf Inference v5.1 con vLLM — https://www.redhat.com/en/blog/efficient-and-reproducible-llm-inference-red-hat-mlperf-inference-v51-results
arXiv 2605.24217 · sesgo sistemático en benchmarks de inferencia LLM — https://arxiv.org/html/2605.24217
IETF Draft · LLM Benchmarking Methodology (enero 2026) — https://www.ietf.org/archive/id/draft-gaikwad-llm-benchmarking-methodology-00.html

GenAI-Perf in depth: LLM inference profiling with real data

Mon, 15 Jun 2026 03:00:00 +0200

Notation: amounts in euros (N EUR), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

TL;DR

GenAI-Perf (the genai-perf package, part of NVIDIA’s Triton ecosystem) is the reference LLM inference profiler for OpenAI-compatible endpoints. Installable with pip install genai-perf. A single genai-perf profile command produces a console table with TTFT, ITL, request latency (avg/min/max/P75/P90/P99), output sequence length, output token throughput and request throughput, plus profile_export_genai_perf.json and .csv artefacts. The analyze subcommand automatically sweeps concurrencies from 1 to 256 (powers of 2) and generates a summary analyze_export_genai_perf.csv. GenAI-Perf was declared retired in April 2026 (replaced by AIPerf); it remains the reference benchmark for historical runs and documented comparisons, and the NVIDIA tool with the most public documentation for LLMs before 2026.

What GenAI-Perf is and where it fits

GenAI-Perf is a command-line tool for measuring the throughput and latency of generative AI models served through an inference server. It is part of the triton-inference-server/perf_analyzer GitHub repository and is distributed as a pip package (genai-perf) and inside the Triton SDK container (nvcr.io/nvidia/tritonserver:YY.MM-py3-sdk).

Position in the NVIDIA ecosystem:

Relationship with perf_analyzer: GenAI-Perf internally uses Triton’s perf_analyzer binary to generate load and measure latencies. perf_analyzer is the low-level load generator; GenAI-Perf is the high-level layer that adds prompt synthesis, OpenAI payload construction and the LLM-specific metrics (TTFT, ITL, sequence lengths).

Installation:

# Option 1: pip (requires CUDA 12, Ubuntu 24.04, Python 3.10+)
pip install genai-perf

# Option 2: Triton SDK container (recommended for reproducibility)
export RELEASE="25.01"
docker run -it --net=host --gpus=all \
 nvcr.io/nvidia/tritonserver:25.01-py3-sdk
# Inside the container:
genai-perf --help

Metrics GenAI-Perf produces

The metrics cover the full cycle of a streaming generation request. All of them are reported with avg, min, max, P75, P90 and P99 except the throughput ones (avg only).

Metrics glossary

Metric	Symbol	Exact definition (GenAI-Perf)	Unit
Time to First Token	TTFT	Time from sending the request to receiving the first token (includes queueing, prefill and network)	ms
Inter-Token Latency	ITL (= TPOT)	$\frac{e2e\_latency - TTFT}{output\_tokens - 1}$ — decode phase only, excludes the first token	ms/token
Request Latency	e2e_latency	$e2e = TTFT + Generation\_time$ — from the first request to the last response	ms
Output Token Throughput	OTT	$\frac{total\_output\_tokens}{T_y - T_x}$ where $T_x$ = first request, $T_y$ = last token received	tok/s
Request Throughput	RPS	$\frac{total\_completed\_requests}{T_y - T_x}$	req/s
Input Sequence Length	ISL	Mean length in tokens of the input prompt	tokens
Output Sequence Length	OSL	Mean length in tokens of the generated response	tokens

Methodological note on ITL versus LLMPerf: GenAI-Perf excludes TTFT from the ITL calculation (pure decode only). LLMPerf includes TTFT in its ITL mean. The numbers are not directly comparable.

Methodological note on TPS versus LLMPerf: GenAI-Perf measures throughput between the first request and the last token received ($T_y - T_x$). LLMPerf uses the total duration of the benchmark including prompt generation and response storage; in a concurrency-1 scenario, that difference can amount to up to 33% in the reported throughput figure.

Console output table (real example, TRT-LLM backend)

The table GenAI-Perf prints for a profile with ISL=200, OSL=100, concurrency=1:

 NVIDIA GenAI-Perf | LLM Metrics
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
┃ Statistic ┃ avg ┃ min ┃ max ┃ p99 ┃ p90 ┃ p75 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
│ Time to first token (ms) │ 13.68 │ 11.07 │ 21.50 │ 18.81 │ 14.29 │ 13.97 │
│ Inter token latency (ms) │ 1.86 │ 1.28 │ 2.11 │ 2.11 │ 2.01 │ 1.95 │
│ Request latency (ms) │ 203.70 │ 180.33 │ 228.30 │ 225.45 │ 216.48 │ 211.72 │
│ Output sequence length │ 103.46 │ 95.00 │ 134.00 │ 122.96 │ 108.00 │ 104.75 │
│ Input sequence length │ 200.00 │ 200.00 │ 200.00 │ 200.00 │ 200.00 │ 200.00 │
│ Output token throughput (per sec) │ 504.02 │ N/A │ N/A │ N/A │ N/A │ N/A │
│ Request throughput (per sec) │ 4.87 │ N/A │ N/A │ N/A │ N/A │ N/A │
└───────────────────────────────────┴────────┴────────┴────────┴────────┴────────┴────────┘

(Source: docs.nvidia.com/deeplearning/triton-inference-server — LLM tutorial)

Invocation against OpenAI-compatible endpoints

`profile` mode: one load point

The profile subcommand measures a fixed operating point (concurrency or request-rate, synthetic ISL/OSL).

Example 1: vLLM endpoint (chat, streaming)

# 1. Bring up the vLLM server
docker run -it --net=host --rm --gpus=all \
 vllm/vllm-openai:latest \
 --model meta-llama/Llama-3.1-70B-Instruct \
 --dtype float16

# 2. Profile with GenAI-Perf
genai-perf profile \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --concurrency 16 \
 --synthetic-input-tokens-mean 1024 \
 --synthetic-input-tokens-stddev 128 \
 --output-tokens-mean 256 \
 --output-tokens-stddev 0 \
 --output-tokens-mean-deterministic \
 --request-count 200 \
 --warmup-request-count 20 \
 --tokenizer meta-llama/Llama-3.1-70B-Instruct

Example 2: completions endpoint (no streaming)

genai-perf profile \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type completions \
 --concurrency 32 \
 --synthetic-input-tokens-mean 512 \
 --synthetic-input-tokens-stddev 64 \
 --output-tokens-mean 128 \
 --request-count 200 \
 --warmup-request-count 20

Example 3: NVIDIA NIM (TRT-LLM backend endpoint, via Triton)

genai-perf profile \
 -m meta/llama-3.1-70b-instruct \
 --backend tensorrtllm \
 --streaming \
 --concurrency 8 \
 --synthetic-input-tokens-mean 2048 \
 --synthetic-input-tokens-stddev 0 \
 --output-tokens-mean 512 \
 --output-tokens-mean-deterministic \
 --request-count 100 \
 --warmup-request-count 10 \
 --generate-plots

Example 4: request-rate instead of concurrency (Poisson)

genai-perf profile \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --request-rate 5.0 \
 --synthetic-input-tokens-mean 1024 \
 --output-tokens-mean 256 \
 --request-count 300 \
 --warmup-request-count 30

Main flags and what they do

Flag	Typical values	Function
`--endpoint-type`	`chat`, `completions`, `embeddings`	Type of OpenAI-compatible endpoint
`--streaming`	(boolean)	Enables SSE streaming; required to measure real TTFT and ITL
`--concurrency`	1, 8, 16, 32, 64	Number of concurrent requests maintained; GenAI-Perf guarantees N active at all times
`--request-rate`	1.0, 5.0, 10.0	Constant arrival rate (req/s); does not guarantee N active
`--synthetic-input-tokens-mean`	128–8192	Mean of synthetic ISL tokens
`--synthetic-input-tokens-stddev`	0–512	Standard deviation of the ISL (0 = fixed)
`--output-tokens-mean`	64–2048	Mean of target OSL tokens
`--output-tokens-mean-deterministic`	(boolean)	Sets the minimum output tokens = target mean (more precise with TRT-LLM)
`--request-count`	100–2000	Number of requests to benchmark
`--warmup-request-count`	10–50	Warm-up requests discarded from the metrics
`--generate-plots`	(boolean)	Generates PNG plots of TTFT vs ISL, ITL vs token position, etc.
`--tokenizer`	HF model id	Tokenizer for counting tokens; mandatory when ISL/OSL matter
`--backend`	`tensorrtllm`, `vllm`	Triton backend (for serving directly via Triton without an OpenAI endpoint)
`--extra-inputs`	`ignore_eos:true`	Extra request parameters; `ignore_eos:true` guarantees a consistent OSL

Note on --concurrency versus --request-rate: NVIDIA recommends using --concurrency. With --request-rate, if the rate exceeds the engine’s throughput, the queue grows without bound and the metrics stop being stable. With --concurrency, there are always exactly N active requests, which gives a clean latency-throughput curve.

Output artefacts

GenAI-Perf dumps all results into an artifacts/ directory:

Artefact	Format	Content
`profile_export_genai_perf.json`	JSON	Full metrics (avg, min, max, P75, P90, P99) + CLI arguments used
`profile_export_genai_perf.csv`	CSV	Exported console tables, ready to import into Excel/pandas
`profile_export.json`	JSON	Raw perf_analyzer data (per-request traces)
`inputs.json`	JSON	Synthetic payloads sent (for reproducibility)
PNG plots	PNG	TTFT analysis, Request latency, TTFT vs ISL, ITL vs position, ISL vs OSL (with `--generate-plots`)

The `analyze` subcommand: automatic concurrency sweep

The analyze subcommand sweeps multiple values of a parameter in a single command and generates a summary CSV.

Concurrency sweep (the most used)

# Sweeps concurrencies 1, 2, 4, 8, 16, 32, 64, 128, 256
genai-perf analyze \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --sweep-type concurrency \
 --sweep-range 1:256 \
 --synthetic-input-tokens-mean 1024 \
 --output-tokens-mean 256 \
 --request-count 200 \
 --warmup-request-count 20

Request-rate sweep

# Sweeps 2, 4, 6, 8, 10, 12 req/s
genai-perf analyze \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --sweep-type request_rate \
 --sweep-list 2,4,6,8,10,12 \
 --synthetic-input-tokens-mean 1024 \
 --output-tokens-mean 256 \
 --request-count 200 \
 --warmup-request-count 20

ISL sweep (to study the effect of prefill)

genai-perf analyze \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --sweep-type input_sequence_length \
 --sweep-list 256,512,1024,2048,4096 \
 --concurrency 16 \
 --output-tokens-mean 256 \
 --request-count 200 \
 --warmup-request-count 20

Artefact structure of `analyze`

For a sweep with --sweep-type concurrency --sweep-range 1:32:

artifacts/
llama70b-openai-chat-concurrency1/
inputs.json
profile_export.json
profile_export_genai_perf.json
profile_export_genai_perf.csv
llama70b-openai-chat-concurrency2/
...
llama70b-openai-chat-concurrency4/
...
[...]
analyze_export_genai_perf.csv ← summary of every scenario
checkpoint.json ← allows interrupted sweeps to be resumed

Summary CSV format

Config Name,Concurrency,ISL,p99 TTFT (ms),p99 ITL (ms),p99 Request Latency (ms),Avg. OTT (tok/s),RPS
llama70b_run_0,1,1024,45.2,6.8,1823.4,128.3,0.55
llama70b_run_1,4,1024,48.1,7.1,1901.2,510.7,2.18
llama70b_run_2,8,1024,61.3,7.9,2142.5,989.2,4.12
llama70b_run_3,16,1024,98.7,9.4,2891.3,1741.6,6.88
llama70b_run_4,32,1024,284.1,14.2,4320.8,2134.2,7.14
llama70b_run_5,64,1024,892.4,28.7,8741.2,2251.8,7.23

The summary CSV also includes a second table with GPU metrics (P99 power, energy, utilisation, memory) when they are captured via Triton’s telemetry URLs (--server-metrics-urls http://localhost:8002/metrics).

Comparison table: GenAI-Perf versus GuideLLM versus LLMPerf versus vLLM bench

The choice of tool changes the number. They are not comparable with each other without adjustment.

Dimension	GenAI-Perf	GuideLLM	LLMPerf	vLLM bench serve
Origin / maintainer	NVIDIA / Triton team (retired Apr 2026)	Red Hat / vLLM project	Anyscale / Ray	vLLM project
Class	multi-process load generator	multi-process load generator	multi-process load generator	single-process micro-bench
Supported endpoints	OpenAI-compatible + KServe + native Triton	OpenAI-compatible	OpenAI-compatible	native vLLM
Main load mode	fixed concurrency N (recommended) or constant request-rate	synchronous, concurrent, poisson, throughput, automatic sweep	batches of N concurrent (with a draining period at the end)	request-rate, concurrency
Draining period	NO — guarantees N active at all times	NO (poisson)	YES — at the end of each batch the system empties, concurrency drops to 0	N/A
Automatic sweep	`analyze` (concurrency, request-rate, ISL, OSL)	`--rate-type sweep` (idle to saturation, 10 rounds)	manual (several runs)	`vllm bench sweep serve`
LLM metrics	TTFT, ITL, e2e latency, OTT, RPS, ISL/OSL	TTFT, ITL/TPOT, throughput, goodput under SLO	TTFT, ITL (includes TTFT in the mean), TPS (total benchmark duration)	TTFT, TPOT, throughput
ITL difference	excludes TTFT: $(e2e - TTFT) / (N_{tok}-1)$	excludes TTFT (same as GenAI-Perf)	includes TTFT in the mean	excludes TTFT
TPS difference	$total\_tokens / (T_y - T_x)$	throughput + goodput under SLO	$total\_tokens / (T_{end} - T_{start})$ — up to 33% lower	$total\_tokens / (T_y - T_x)$
Goodput / SLO	NO	YES (the key differentiator)	NO	NO
Exports	JSON + CSV + PNG + checkpoint	JSON + YAML + CSV + interactive HTML	JSON	console / JSON
Built-in warm-up	YES (`--warmup-request-count`)	YES (warm-up requests)	NOT native	YES
Real datasets	OpenOrca, CNN DailyMail, JSONL file, moon_cake	file with a traffic distribution	JSONL file	sharegpt, random
GPU telemetry	YES (via Triton’s server-metrics-urls → summary CSV)	NOT native (integrate DCGM separately)	NO	NO
When to choose it	fine profiling of one operating point, comparing configs with the same harness, integration with NIM/Triton	SLO-driven sweep, finding the knee, sizing replicas	quick endpoint validation in the Ray ecosystem	iterating on vLLM flags

Fair-comparison rule: never cross numbers from different tools for the same conclusion. If vLLM and TRT-LLM are being compared, use GenAI-Perf against both, with the same ISL/OSL, the same concurrency and the same --warmup-request-count. The tool is the constant; the engine is the variable.

Profiling on 4×H100 SXM 80 GB: a full sweep example

Generic reference hardware: 4×H100 SXM 80 GB NVLink, model Llama-3.1-70B-Instruct FP16, ISL=1024 tokens, OSL=256 tokens, vLLM endpoint.

Illustrative result of a concurrency sweep (1→64)

Concurrency	TTFT P50 (ms)	TTFT P99 (ms)	ITL P50 (ms)	ITL P99 (ms)	OTT (tok/s)	RPS
1	38	45	6.4	7.1	130	0.51
4	40	51	6.8	8.2	510	2.00
8	46	62	7.3	9.4	980	3.83
16	68	98	8.9	12.1	1,720	6.72
32	145	284	13.7	21.4	2,130	8.32
48	380	740	22.3	38.6	2,280	8.90
64	890	1,820	41.2	78.3	2,340	9.14

How to read it: throughput saturates around concurrency 48–64, while TTFT P99 already exceeds 500 ms from concurrency 32 onwards. The knee (the last point with TTFT P99 < 500 ms) is at concurrency 16 with 1,720 useful tok/s. Anyone reporting “2,340 tok/s” (concurrency 64) is describing the maximum throughput, at a point where the system meets no interactive chat SLO at all.

Formulas for the key metrics

$$ \text{ITL} = \frac{e2e\_latency - TTFT}{N_{output\_tokens} - 1} $$ $$ \text{OTT} = \frac{\sum output\_tokens}{T_y - T_x} $$ $$ \text{RPS} = \frac{N_{requests}}{T_y - T_x} $$

where $T_x$ is the timestamp of the first request sent and $T_y$ the timestamp of the last token received from the last request.

Synthetic ISL/OSL profiling: what to configure for each use case

The ISL/OSL parameters determine which engine of the system is stressed: a long ISL stresses prefill (KV-cache, TTFT); a long OSL stresses decode (ITL, memory bandwidth).

Use case	Typical ISL	Typical OSL	`--synthetic-input-tokens-mean`	`--output-tokens-mean`
Short interactive chat	~300 tok	~100 tok	300	100
Code copilot	~800 tok	~200 tok	800	200
Document summarisation	~2000 tok	~256 tok	2000	256
RAG with long context	~4096 tok	~512 tok	4096	512
Batch without latency	~512 tok	~1024 tok	512	1024

The --extra-inputs ignore_eos:true option disables the engine’s EOS token, forcing exactly the requested tokens to be generated. Essential for measuring a consistent OSL on TRT-LLM; optional on vLLM.

Honest methodology: how not to measure badly

1. Warm-up is mandatory

Without warm-up, the first requests include the KV-cache loading latency, the engine’s autotuning and the initialisations. GenAI-Perf applies a sliding window to detect stability (max/min ratio within a margin over the last 3 measurements), but the explicit --warmup-request-count discards the first N requests from the metrics. Recommended minimum value: 10 requests (or 5% of the total if the total is large).

2. The concurrency knee: extend the sweep until you see it

Throughput saturates once the engine’s effective max batch size is reached. If the sweep stops before saturation is reached, the “maximum capacity” reported is the last concurrency tested, not the real knee. The rule: the sweep must reach a point where TTFT P99 has doubled relative to the low-load value (or where OTT stops growing by more than 5% between steps). Only then is the knee visible.

For a sweep with --sweep-range 1:256, GenAI-Perf sweeps 1, 2, 4, 8, 16, 32, 64, 128, 256 (powers of 2). On most 70B models over 4×H100, saturation occurs between concurrency 32 and 64; the range 1:128 covers the knee with room to spare.

3. Concurrency versus request-rate: choosing correctly

Mode	Behaviour	When to use
`--concurrency N`	Always N active requests; when one finishes, the next is sent	Clean latency-throughput curve; comparison between configs
`--request-rate R`	1 request is sent every 1/R seconds; the queue can grow without bound	Simulating traffic with arrivals at a constant rate

NVIDIA recommends --concurrency for benchmarking. With --request-rate, if R exceeds the engine’s maximum throughput, the queue grows indefinitely and the metrics become unstable.

4. Single-process versus multi-process bias

GenAI-Perf is multi-process: it maintains N concurrent requests using perf_analyzer’s internal manager. There is no risk of client saturation. This sets it apart from vllm bench serve (single-process), which can saturate on the client at high concurrency and give inflated throughput metrics.

5. `ignore_eos` and an inconsistent OSL

Without --extra-inputs ignore_eos:true, the engine may generate fewer tokens than requested (if the model produces an EOS earlier). This makes the real OSL vary between requests and the ITL become inconsistent between runs. For reproducible benchmarks with a controlled OSL: always enable ignore_eos:true.

6. The correct tokenizer

The --tokenizer flag must point to the model being served. If it is omitted, GenAI-Perf uses a default tokenizer that can give different ISL/OSL in the real model’s tokens, making the throughput numbers in tok/s not comparable between models with different vocabularies.

7. One run = a data point; three or more = a number

Variance between runs under identical conditions can be 5–10% on tail latency metrics (P99). A single profile is not a defensible number; the mean of three runs with the same config is.

Connection with the cost and energy track

GenAI-Perf can capture GPU metrics in the same summary CSV if --server-metrics-urls http://localhost:8002/metrics is passed (Triton’s metrics endpoint, which exposes power and utilisation via DCGM). The analyze summary CSV then includes a second table with P99 power (W), energy (MJ), utilisation (%) and memory (GB) per GPU and per scenario.

With that data, a single sweep yields the three axes of the scorecard:

Axis	Source	Metric
Performance	GenAI-Perf	OTT (tok/s), TTFT P99, ITL P99
Energy	DCGM via server-metrics-urls	Power (W) → $J/token = W / OTT$
Cost	node price	EUR/hour → $EUR/token = EUR_{hour} / (OTT \times 3600)$

The formula for cost per million tokens:

$$ CPM = \frac{EUR_{hour}}{OTT \times 3{.}6 \times 10^3} $$

where $OTT$ is the output token throughput in tok/s and $CPM$ is the cost per million tokens in EUR.

It connects with GPU observability with DCGM for the power capture, and with anatomy of an LLM request for the breakdown of phases (prefill versus decode) that TTFT and ITL quantify.

Status 2026: GenAI-Perf → AIPerf

NVIDIA announced in April 2026 that GenAI-Perf moves to passive maintenance mode (no new features) and that the active successor is AIPerf (github.com/ai-dynamo/aiperf). AIPerf adds automatic detection of the saturation point (estimatedCapacity) and is integrated into the NVIDIA Dynamo pipeline.

For new benchmarks in production: migrate to AIPerf. To reproduce historical runs documented with GenAI-Perf, or to use NVIDIA’s NIM benchmarking artefacts (which use GenAI-Perf): stay on GenAI-Perf 25.x.

The article LLM benchmark tools, one by one covers the full comparison including AIPerf. The article GuideLLM: SLO validation under load details the SLO-driven sweep that GenAI-Perf does not do natively. For the general methodology context: LLM benchmarking frameworks, state of the art.

Sources

NVIDIA · GenAI-Perf — README oficial — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/README.html
NVIDIA · Tutorial LLM con GenAI-Perf (comandos y tablas de salida reales) — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/docs/tutorial.html
NVIDIA · GenAI-Perf Analyze subcommand (sweep, CSV resumen, checkpoint) — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/docs/analyze.html
NVIDIA · NIM LLM Benchmarking — Métricas (definiciones TTFT, ITL, TPS, RPS, fórmulas exactas) — https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html
NVIDIA · NIM LLM Benchmarking — Parámetros y buenas prácticas (ISL/OSL, concurrencia vs request-rate, ignore_eos) — https://docs.nvidia.com/nim/benchmarking/llm/latest/parameters.html
NVIDIA Technical Blog · Measuring Generative AI Model Performance Using NVIDIA GenAI-Perf and an OpenAI-Compatible API — https://developer.nvidia.com/blog/measuring-generative-ai-model-performance-using-nvidia-genai-perf-and-an-openai-compatible-api/
GitHub · triton-inference-server/perf_analyzer (genai-perf) — https://github.com/triton-inference-server/perf_analyzer/blob/main/genai-perf/README.md
GitHub · ray-project/llmperf (métricas y diferencias metodológicas) — https://github.com/ray-project/llmperf
PyPI · nvidia-genai-perf-eval — https://pypi.org/project/nvidia-genai-perf-eval/
Macnica · Benchmarking LLM Applications Part 1: What is GenAI-Perf? — https://www.macnica.co.jp/en/business/semiconductor/articles/nvidia/145977/

GuideLLM in depth: validating the SLO under load and sizing from the knee

Sun, 14 Jun 2026 04:30:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Third article of the benchmarking track (B3). B2 covered the tool catalogue; here we go into GuideLLM in depth, because it answers the operational question that sizes a platform: how far can I load this engine without breaking the SLO? It is not a catalogue benchmark (“X tokens/s”), it is an SLO-driven sweep that finds the knee, the safe capacity, and from which the real replica count and cost per token come out. We will look at its load modes, how the SLO is defined, how to read the output and how to turn the knee into sizing and into euros. With real commands; no recommendations, only the mechanics.

Why GuideLLM

GuideLLM (a vLLM project) generates realistic, configurable traffic patterns and captures full distributions of TTFT, ITL and end-to-end behaviour, for SLO-driven evaluation, with reproducible sweeps that identify the safe operating range (Red Hat). Unlike a micro-benchmark, it measures the engine, not the client (multi-process load generation), and unlike MLPerf, it sizes your specific case (your model, your SLO, your load). It is the middle tool: not for tuning a flag, not for comparing vendors, but for deciding how many GPUs you need.

The load modes (rate-types)

GuideLLM offers several modes through --rate-type, and picking the right one is half the work:

`--rate-type`	What it does	When
synchronous	one request at a time	baseline latency, no concurrency
concurrent	keeps N fixed simultaneous requests	measuring at a specific concurrency
poisson	requests per second following Poisson	simulating real traffic (random arrivals)
throughput	saturates the engine to the maximum	raw peak capacity
sweep	automatic ramp from idle to maximum	finding the knee (the most useful one)

In concurrent mode, --rate "1,2,4" launches three benchmarks at those concurrencies (Medium · GuideLLM on OpenShift). The poisson mode is the most realistic for online inference (requests do not arrive at regular intervals), and the sweep is the one that automates the search for the saturation point.

The automatic sweep: from idle to the knee

The flagship mode. When running a sweep profile, GuideLLM automatically increases the request rate from idle to maximum throughput across 10 rounds, and produces an interactive HTML report with detailed latency and throughput (Red Hat · GuideLLM on Kubernetes). During the sweep, on detecting saturation it identifies the previous iteration, the knee, and returns it as estimated capacity; if it does not detect saturation, the sweep has to be extended beyond the knee.

A worked example: reading a 10-round sweep

An illustrative output from a sweep on a 70B on 8×H100 (SLO: TTFT P99 < 500 ms), per round:

Round	Rate (req/s)	TTFT P99 (ms)	ITL P50 (ms)	Throughput (tok/s)	Goodput (tok/s)
1	2	95	19	480	480
3	8	140	20	1,900	1,900
5	14	240	22	3,100	3,060
6	17	460	24	3,400	3,330
7	20	760	31	3,700	2,520
9	26	1,500	48	3,950	900
10	30	2,400	71	4,000	380

How to read it: up to round 6 (17 req/s) the TTFT P99 meets the SLO (460 ms) and goodput ≈ throughput (3,330 ≈ 3,400). In round 7 P99 already breaks (760 ms) and goodput drops to 2,520. The knee sits between 6 and 7: the safe capacity is that of round 6, ~3,330 tok/s of goodput. Rounds 9 and 10 give more raw throughput (3,950, 4,000) but with goodput of 900 and 380; the system is “performing” by mostly serving requests that miss the target. Whoever reports “4,000 tok/s” is describing round 10, where the system is broken. The defensible number is 3,330 tok/s under TTFT P99 < 500 ms, and that is the one that goes into the sizing. Note as well that the sweep had to reach round 10 (P99 of 2,400 ms) to see where it broke: that is why it has to be extended beyond the knee.

Defining the SLO: the number that decides the knee

The “knee” is not absolute: it depends on the SLO you define. A typical online inference SLO:

Metric	Example threshold	What it protects
TTFT P99	< 500 ms	the wait for the first token (interactivity)
TPOT / ITL P95	< 50 ms/token	the perceived “typing speed”
Error rate	< 0.1 %	reliability

Goodput, the throughput that meets that SLO, is the metric that defines the knee: the safe capacity is the maximum load where goodput ≈ throughput. Changing the SLO moves the knee: a stricter TTFT SLO (200 ms) gives an earlier knee (less safe capacity) than a loose one (1 s). That is why the SLO is fixed before the sweep, and reported alongside the result: a capacity “of 3,330 tok/s” without saying under which SLO means nothing.

Execution: the command

An SLO-driven sweep against a vLLM endpoint:

guidellm benchmark \
 --target "http://vllm:8000" \
 --rate-type sweep \
 --max-seconds 120 \
 --data "prompt_tokens=1024,output_tokens=256" \
 --output-path resultados.json

Key parameters: --rate-type sweep (the automatic ramp), --data (the length distribution of the load, which must resemble your real traffic), --max-seconds (duration per round), --target (the endpoint). For the SLO, GuideLLM allows the latency constraints that define goodput to be set. The output goes to --output-path in JSON, YAML or CSV, in addition to the interactive HTML report.

Defining the load: `--data` decides the result

The --data parameter (the distribution of prompt and output lengths) changes the knee as much as the SLO does. Three forms, from least to most faithful:

Load	`--data`	Fidelity
Fixed length	`prompt_tokens=1024,output_tokens=256`	low: real traffic is not fixed
Synthetic distribution	length ranges	medium
Real traces	a dataset of your traffic	high: the knee that applies

The trap: a sweep with short fixed-length prompts gives an optimistic knee that does not resemble production, where long prompts dominate the prefill cost and bring the knee forward. To size properly, feed GuideLLM the real distribution of your traffic (prompt and output lengths measured in production). The knee of a sweep is only as realistic as the load you put into it; with an unrealistic --data, you are sizing for traffic that does not exist.

The arrival profile matters too: --rate-type poisson simulates random arrivals (like real online traffic), while concurrent keeps N fixed (like a batch). An online service measured with concurrent can give a knee different from the real one; for interactive inference, poisson is the honest profile.

The output: what to read

GuideLLM produces standardised, exportable reports for dashboards, analysis and regression tracking, in JSON, YAML and CSV (Red Hat). What matters to read, per sweep round:

TTFT (P50, P95, P99) — the latency to the first token.
ITL/TPOT (P50, P95, P99) — between tokens.
Throughput (req/s and tok/s) — the raw figure.
Goodput — the part that meets the SLO (the honest number).

The interactive HTML report lets you see the full distributions (not just means), which is where the latency tail that a mean hides becomes visible. For sizing and comparison, the JSON is what gets versioned and fed into the reproducible harness.

From the knee to sizing and cost

This is where GuideLLM connects with the rest of the series. The replica count and the cost per token come out of the knee:

The calculation: if the knee gives a goodput of 3,330 tok/s per replica and your peak target load is 20,000 tok/s, you need 6 replicas (20,000 ÷ 3,330 ≈ 6.0). And the cost per token comes from the replica cost (from OpenCost, ~11 €/h) divided by its goodput: ~0.92 €/1M tokens. It is the bridge to capacity planning and to cost per token: the correct denominator is the goodput at the knee, not catalogue throughput. Sizing with raw throughput leaves you short of useful capacity and breaks the SLO in production.

Deployment as a Kubernetes Job

GuideLLM runs as a Kubernetes Job inside the cluster to benchmark models served by the orchestration platform (Red Hat). This matters for fidelity: the load generator runs inside the cluster, on the same network as the endpoint, so it measures the engine without the network latency of an external client. The Job is parameterised with the target, the rate-type and the load, and dumps the JSON to a volume or a store for versioning. Running it as a Job also makes automation easier (a CronJob for periodic benchmarking, or triggered from CI).

SLO validation in CI: catching regressions

The most valuable use in the medium term: a Job that runs a short sweep on every change (a vLLM release, a config change) and compares the goodput against the baseline. If goodput drops by more than a threshold or the knee moves earlier, the pipeline fails. Since GuideLLM exports standardised JSON for regression tracking, comparing two runs is trivial. That way a performance regression, which is a cost and capacity regression, is caught at the commit, not in production. The full sweep is not needed on every commit: a short sweep covering the knee is enough to catch the regression; the exhaustive one is for releases.

Comparing configurations with sweeps

GuideLLM shines for deciding between configurations by running the same sweep against each one. The fair protocol: same load (--data), same SLO, same hardware, varying only the config. Examples of decisions that a sweep settles with data:

Decision	What to compare	What the knee reveals
FP16 vs FP8	two deployments of the same model	FP8 usually gives a higher knee (more goodput)
`max-num-seqs`	different values of the flag	the one that maximises goodput under the SLO
vLLM vs SGLang	two engines, same model	which engine gives more goodput at your SLO
KV size/precision	KV cache configs	the effect on capacity

Each sweep gives a knee; the highest knee under the same SLO wins. It is the controlled experiment that fills the scorecard row (article B8): not “X is faster” in the abstract, but “X gives Y tok/s more goodput under this specific SLO, which translates into Z fewer replicas and W € less per million tokens”.

Watching the three axes during the sweep

A trick that connects with the cost and energy track: while the sweep runs, capture the GPU metrics with DCGM. That way each round yields the goodput (GuideLLM), the average power (DCGM → J/token) and the cost (node price → €/token) at once. In a single sweep campaign you measure the three axes of the scorecard for every operating point, and they come out coherent by construction (same instant, same load). Instead of an isolated performance benchmark, you get the complete row, cost, performance, energy, for the knee, which is exactly what the proposal needs. Align the time windows (the DCGM power and the GuideLLM metrics must cover the same interval) or the J/token does not correspond to the measured goodput.

Interpreting the distributions, not just the knee

The knee is the headline, but the GuideLLM HTML report gives full distributions, and there is information there that the knee summarises away. Three additional readings:

The shape of the tail. Two configs with the same P99 can have different tails: one with P99 at 500 ms and P99.9 at 600 ms is stable; another with P99 at 500 ms and P99.9 at 3,000 ms has a long tail that will hit some users severely. The mean, and even P99, hide it; the distribution shows it.
The TTFT/ITL split. Seeing the two distributions separately tells you whether the bottleneck is prefill (high TTFT) or decode (high ITL), which steers the optimisation (more concurrency vs chunked prefill, and so on).
The spread across rounds. If goodput varies a lot between rounds at the same rate, the system is unstable under load, something a single capacity number does not reveal.

The knee sizes; the distributions diagnose. For a proposal, the knee is the datum; for operating and optimising, the distributions are where you see what to fix.

GuideLLM in the reproducible harness

GuideLLM fits as the load engine of the reproducible harness (article S4). The pattern:

A script/Job that deploys the engine with the pinned config, warms up, runs the sweep and dumps the JSON with all the metadata (model, version, hardware, --data, SLO).
Named by date and config to version the runs.
A store (a git repo of JSONs or a bucket) to reproduce and compare.
DCGM capturing in parallel to add the energy to each point.

That way “reproduce the knee of this config” is one command, not an afternoon, and comparing two releases is a JSON diff. It is what turns benchmarking from a one-off activity into a continuous capability, and what lets the capacity figure in the proposal come with the bench to reproduce it.

The cost of a sweep (in euros)

A 10-round sweep at ~2 minutes per round occupies the node for ~20–30 minutes. At an amortised cost of ~11 €/h, that is ~4–5.5 € per full sweep of a model. Not much per run, but a continuous benchmarking programme (every release, every config, several models) adds up; hence the value of automating the sweep as a reproducible Job and running short sweeps in CI (only around the knee), reserving the exhaustive one for releases. The cost of measuring is part of the cost of the platform, and small next to the cost of sizing badly.

The SLO by use case: the knee moves with it

Since the SLO defines the knee, different use cases give different capacities on the same hardware. Worth keeping in mind when sizing:

Use case	Typical SLO	Effect on the knee
Interactive chat	TTFT P99 < 500 ms, ITL < 50 ms	early knee (latency rules)
Code copilot	TTFT P99 < 300 ms	even earlier knee
Batch / summarisation	no TTFT SLO, maximise throughput	late knee (almost pure throughput)
Agent (multi-step)	bounded per-step latency	depends on the number of steps

The same GPU serves more batch load than interactive chat load, because the batch SLO is loose. That is why sizing requires one sweep per load profile: there is no “the node’s capacity”, there is “the node’s capacity for this SLO”. Mixing loads with different SLOs in the same pool without separating them is a classic source of broken SLOs, with batch saturating and interactive chat paying for it. The fix connects with scheduling: separate pools or priorities by SLO.

GuideLLM vs the others: when to use which

To place it against the tools from B2:

Question	Tool	Why
“How far can I load without breaking the SLO?”	GuideLLM	SLO-driven sweep, the knee
“What is the endpoint’s maximum capacity?”	AIPerf	automatic saturation detection
“Does my vLLM flag change help?”	vllm bench serve	fast, native to the engine
“Which hardware/engine is better in the abstract?”	read MLPerf	cross-vendor comparability

GuideLLM is the tool for sizing under an SLO: when the question is operational (how many replicas, what cost per token at my SLO), it is the most direct answer. AIPerf and GuideLLM overlap (both multi-process); the practical difference is that GuideLLM is more oriented to the SLO and the reproducible report, and AIPerf to maximum capacity with automatic detection. Many teams use both; what must not be done is to compare a result from one with a result from the other as if they were the same measurement.

Checklist for a defensible sweep

For the knee of a sweep to support a sizing decision:

Step	Check
Realistic load	`--data` with your traffic’s distribution (or poisson)
Declared SLO	TTFT/TPOT with a fixed percentile and threshold
Extended sweep	reaches the point where P99 breaks (past the knee)
Goodput, not throughput	capacity is the goodput under the SLO
Job in the cluster	no network latency from an external client
Versioned output	JSON with all the metadata
DCGM in parallel	energy per point, optional but recommended

If you can tick the seven boxes, the knee is an auditable datum; if any is missing, it is an anecdote. The sizing in the proposal hangs on these seven being green.

Limits and traps (data-driven)

Not extending the sweep beyond the knee. If saturation is not detected, the estimated capacity is the last round tested, not the real knee. Extend it until P99 breaks.
Unrealistic load. A fixed-length --data does not resemble your traffic; use the real distribution (or poisson) for a knee that applies.
Reporting throughput, not goodput. The knee is defined by goodput under the SLO; raw throughput overestimates capacity.
Undeclared SLO. A capacity without the SLO it was measured under is neither comparable nor defensible.
Client outside the cluster. Running GuideLLM from outside puts network latency into the measurement; run it as a Job inside the cluster.

With GuideLLM mastered, you have the tool that turns performance into an actionable datum: the knee, the sizing and the cost per token, all from a reproducible sweep. The next article (B4) goes into NVIDIA’s AIPerf; this one closes SLO validation, which is what sizes the proposal.

Closing

GuideLLM answers the only performance question that sizes a platform: how far can I load without breaking the SLO? And it answers it with the right discipline: a sweep from idle to saturation that finds the knee, defined by goodput under a declared SLO, over a load that resembles the real one. Everything the proposal needs comes out of that knee: the replica count, the cost per token (with goodput as the denominator, not catalogue throughput) and, if you capture DCGM during the sweep, the energy per token as well. The mistake that invalidates the exercise is the usual one: reporting peak throughput instead of goodput, measuring with an unrealistic load, or not extending the sweep until you see where it breaks. Done properly, as a Job inside the cluster, with the real distribution, the SLO fixed and the output in versioned JSON, GuideLLM turns performance from a marketing figure into a reproducible datum that survives an audit and sizes a sovereign architecture with numbers. The knee is not the fastest point; it is the point where your platform still delivers what it promised.

Sources

Red Hat · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
Red Hat · desplegar y benchmarkear vLLM con GuideLLM en Kubernetes — https://developers.redhat.com/articles/2025/12/24/how-deploy-and-benchmark-vllm-guidellm-kubernetes
GuideLLM · PyPI — https://pypi.org/project/guidellm/
GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
Medium · GuideLLM en OpenShift (rate-types, ejemplo) — https://medium.com/@jajodia.nirjhar/exploring-guidellm-benchmarking-a-live-llm-on-openshift-ccc2d0841794

The second cost vector of AI agents: durable execution with Temporal

Sun, 14 Jun 2026 04:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

The thesis: agents have two cost vectors, not one

Almost all the cost discourse around agentic AI focuses on a single vector: the cost of inference (the tokens). And that one is solvable: FastMCP + vLLM on local hardware eliminates the cost of inference — you serve the models on your own iron, the cost per token drops to your amortised cost, with no API bill. But there is a second vector that almost nobody measures and that kills more projects: the cost of faulty execution. An agent is not a call, it is a sequence of steps (reason, call a tool, reason, call another one, and so on), and when that sequence fails halfway through with no persisted state, the cost is not linear: it doubles, it spirals with retries and it inflates on every recovery. This article is about that second vector, how durable execution attacks it (with Temporal as the reference), what the state of the art and the papers say, and why it is the real cause of so many cancellations.

The framing that backs the urgency comes from Gartner: it predicts that more than 40 % of agentic AI projects will be cancelled before the end of 2027, because of rising costs, unclear business value or inadequate risk controls (Gartner). “Rising costs” is exactly this second vector: not the cost of serving the model, but the cost of running it badly.

The three expenses of a faulty agent

The cost of faulty execution breaks down into three concrete sources, and it is worth seeing them with numbers:

Vector 1: failures charged twice

With no persisted state, a failure at step 15 of 20 costs you the tokens of the 14 previous steps (which you already paid for) plus the full retry from scratch. The wasted cost of an agent with no checkpoint, per failure, is:

$$\text{wasted cost} = \text{tokens of the completed steps} \times \text{cost per token} \times (\text{retries})$$

The longer the sequence and the later it fails, the more expensive: a failure at 75 % of the work throws away 75 % of the cost. Durable execution checkpoints every step: if the worker dies halfway through, another one picks the workflow up exactly where it stopped, not from scratch (Temporal). Temporal uses its Event History as the record of past decisions; on a failure it replays the progress to date and resumes at the exact point (CallSphere). Step 15 fails → step 15 is retried, not the previous 14. The wasted cost goes from “the 14 steps” to “the step that failed”.

Vector 2: uncontrolled retries

Agents that retry with no policy blow up the API or compute bill: a retry loop against a rate limit can generate dozens of billable calls for a single task. Durable execution defines retry policies per activity: hard limits on retries, exponential backoff and timeouts that bound the maximum possible cost of each call. Instead of “retry until it works” (unbounded cost), you define “at most N retries with backoff and a timeout of T” (bounded cost).

The state of the art backs this with data: the ReliabilityBench benchmark evaluates agent reliability under controlled failures and finds that rate limiting is the most damaging failure of all (arXiv 2601.06112). That is, exactly the failure that an unpolicied retry turns into a cost spiral. Bounding retries is not hygiene, it is the defence against the failure mode the literature singles out as the worst.

Vector 3: inflated context on recovery

With no persisted state, the agent rebuilds the full context on every recovery: it reloads the whole conversation/task history to “remember” where it was, paying the tokens of that enormous context on every recovery. With durable execution the state lives in the workflow history, so the next step receives only the context needed for that step, not the whole conversation. The saving is direct: fewer input tokens per step, and no penalty for rebuilding the context after each failure.

A worked example: the cost of one failure

Let us put numbers on the three vectors. A 20-step agent consuming ~3,000 tokens per step (reasoning plus tool call), on in-house inference at ~1.09 €/1M tokens, with a failure at step 15 and a failure rate of 10 % of executions:

Scenario	Tokens recomputed per failure	Cost per failure	Over 10,000 executions (10 % fail)
No state (full retry)	14 steps × 3,000 = 42,000	~0.046 €	~46 € wasted
With checkpoint (resumes at 15)	0 (only step 15 is retried)	~0.003 €	~3 € wasted

The absolute figure looks small because inference is in-house and cheap; but the ratio is ~15× less waste per failure, and it scales with the length of the sequence and with the cost per token (were it an external API at 10-30 €/1M, the waste without a checkpoint would be multiplied by 10-30). And this is only vector 1: add uncontrolled retries (vector 2), which can turn one failure into dozens of calls, and the context rebuilt on every recovery (vector 3). The cost of faulty execution is not the inference cost × something small; it is a multiplier that grows with the agent’s complexity, and it is exactly what does not show up in the demo and does show up in the production bill.

How Temporal works: workflows, activities and determinism

To understand why Temporal achieves this, its two primitives:

Workflow: the agent’s orchestration logic (the sequence of steps, the decisions). It has to be deterministic: given the same Event History, it produces the same decisions. That is what allows the execution to be replayed after a failure and to arrive at the same point.
Activity: every step with side effects (a call to the LLM, to a tool, to an API). Activities are not deterministic (the LLM is probabilistic), so their result is persisted in the Event History: on replay they are not executed again, they are read from the history.

The trick is that separation: the deterministic logic (workflow) is replayed cheaply from the history; the expensive, non-deterministic steps (activities, the model calls) are persisted and not repeated. That is why “completed work is never repeated”: completed activities are read, not recomputed. And retry policies are defined per activity, so each call to the model or to a tool has its own retry limit, backoff and timeout, which is the vector 2 cost control at the right level.

Durable execution: what it is and the landscape

Durable execution is a programming model that guarantees the code completes despite failures: every step is checkpointed automatically, and if a worker dies, another one picks the workflow up where it stopped (Temporal). It crossed into the “early majority” in 2025, with offerings from AWS, Cloudflare and Vercel, driven precisely by AI agent infrastructure (Inngest).

The tooling landscape (the 2026 reference architecture for durable LLM agents): Temporal, AWS Step Functions Express, Restate, DBOS and Inngest as durable execution primitives; and the checkpointer model of LangGraph (PostgresSaver, RedisSaver, DynamoDBSaver) on the agent framework side (render.com). Temporal stands out for its workflow-as-code model and its Event History as the source of truth.

Option	Model	Self-hosted	Note for sovereignty
Temporal	workflow-as-code, Event History	yes (open source)	the most mature; full on-prem
Restate	durable RPC/handlers	yes	lightweight, modern
DBOS	durable on top of Postgres	yes	state in your own Postgres
Inngest	event-driven, steps	SaaS / self-host	focus on DX
AWS Step Functions Express	state machine	no (AWS SaaS)	tied to AWS, not sovereign
LangGraph checkpointer	framework checkpointer	yes (your backend)	at framework level, not platform level

For a sovereign platform, the ones that are self-hosted (Temporal, Restate, DBOS) keep state in your cluster; the SaaS ones (Step Functions) move execution into someone else’s jurisdiction. Temporal is the choice for maturity and for its complete on-prem deployment.

The cost of durable execution itself

Honesty about the data: durable execution is not free. It has two costs to watch:

Execution overhead: persisting state and replaying history adds latency and compute. The reference architectures put it at around 5-20 % overhead on a load of 100,000 executions/day (render.com).
Price per step: in the offerings with per-step pricing, orchestrating many model calls, rate limit retries and complex interactions generates a large number of billable steps, and the cost can run away (render.com).

The conclusion for a sovereign platform: self-host the durable execution engine (Temporal is open source and deploys on-prem) and the SaaS per-step price disappears, leaving only the execution overhead, a 5-20 % that is paid back many times over by what it saves on the three vectors above. The arithmetic: if a failure halfway through a sequence used to cost you 50-75 % of the recomputed cost, a 5-20 % overhead is a bargain.

Observability: from manual post-mortem to real-time traceability

The fourth element of the equation: the observability layer. The integration with Braintrust and the complete history of every execution (Temporal’s Event History) turn cost analysis from a manual post-mortem, reconstructing by hand what happened and what it cost after a failure, into real-time traceability: every execution is recorded step by step, with its tokens, its retries and its cost, ready to analyse. This closes the FinOps loop of the cost-per-token article, but at the agent level: not just how much a token costs, but how much an execution costs, and where it is wasted.

The complete equation

Putting the pieces together, the cost architecture of a sovereign agent:

FastMCP + vLLM running locally eliminates the cost of inference. Temporal eliminates the cost of faulty execution. They are complementary, attacking different vectors of the same problem. Whoever solves only the first (serves cheaply but executes badly) keeps bleeding through the second; whoever solves only the second (executes robustly but pays for the API) keeps bleeding through the first. A sustainable platform needs both.

State of the art and papers

The reader asked for papers, and there are some: agent reliability and cost is an active research area in 2026.

Paper	Contribution
ReliabilityBench (2601.06112)	an agent reliability benchmark: consistency, robustness and tolerance to tool/API failures; rate limiting is the most damaging failure; ReAct more robust than Reflexion under stress
Cost-Efficient LLM Agents (arXiv)	the reliability-cost trade-off: routing every decision through the LLM improves accuracy but blows up cost; pre-coded graphs are cheaper but brittle against compound failures
Multi-agent Byzantine Fault Tolerance (2511.10400)	BFT consensus (CP-WBFT) to stabilise multi-agent systems under extreme failure rates (85.7 %)
ACRFence (2603.20625)	security of agent checkpoint-restore: preventing semantic rollback attacks
The Six Sigma Agent (2601.22290)	enterprise reliability via consensus-driven decomposed execution

Two readings for the proposal. First, the reliability-cost trade-off is real and is characterised: you cannot maximise accuracy ignoring cost, or the other way round; durable execution shifts the frontier by making reliability cheaper (checkpoint plus bounded retry instead of recomputing). Second, agent checkpoint-restore has its own security surface (ACRFence): persisting state introduces rollback risks that have to be controlled — durability is not only a cost decision, it is a security one too.

The sovereign angle

For a European sovereign platform the combination fits perfectly and is entirely on-prem: vLLM serves the models (no API cost and no sovereignty given away), FastMCP exposes the tools, and self-hosted Temporal orchestrates with durability (no SaaS per-step price and no execution data leaving for a foreign jurisdiction). The execution history, which describes in detail what your agent does and with which data, stays in your cluster, just like the inferences. Against a stack of agents on external APIs plus a US SaaS orchestrator, the sovereign stack removes both cost vectors and keeps the data under EU jurisdiction. It is the same argument as the rest of the series, applied to the agent layer: bounded, measured and sovereign cost.

FastMCP and the tool layer

The other component of the equation is FastMCP, the layer that exposes tools to the agent via MCP (Model Context Protocol). The agent does not call the APIs directly: it calls MCP servers that encapsulate each tool (a database, a search engine, an internal system) with a uniform contract. The connection with cost and durability: every call to an MCP tool is a Temporal activity, so it inherits its retry policy, its timeout and its persistence. That is, FastMCP defines which tools exist and Temporal governs how they are executed with durability and bounded cost. And because the MCP servers are your own, the tools and the data they touch stay on-prem, never exposed to an external provider. The triad vLLM (model) + FastMCP (tools) + Temporal (orchestration) is a fully sovereign agent stack: the model, the tools and the execution, all under your control and with the cost of both vectors bounded.

Patterns: human-in-the-loop and long sequences

Where durable execution shines most, by agent pattern:

Human-in-the-loop (HITL): human approval patterns, essential for an agent’s safety, map directly to the suspend/resume primitives of durable execution, allowing workflows that pause for hours or days waiting for an approval without losing state (Inngest). Without durability, keeping an agent “waiting” consumes resources or loses the context; with it, the workflow sleeps at no cost and wakes up with all its state.
Long sequences: the longer the chain of steps, the greater the checkpoint’s saving (a late failure throws away more work). Multi-step agents with tools are the canonical use case.
Multi-agent: when several agents collaborate, reliability gets complicated (the BFT papers study this); durable execution provides the consistent state substrate on which to build that coordination.

The pattern that does not need it: a single-step, stateless call (a classifier, a simple extraction). There the durability overhead buys nothing. The rule: durable execution for stateful sequences that fail halfway through; for everything else, a direct call.

How to measure the second vector

To manage the cost of faulty execution you have to measure it, and with the Event History plus observability (Braintrust) the KPIs are within reach:

KPI	What it indicates	Where it comes from
Tokens wasted per failure	the cost of vector 1	tokens recomputed after a failure
Retries per activity	the cost of vector 2	retry count from the Event History
Context size on recovery	the cost of vector 3	input tokens after a recovery
Failure rate per workflow	how often the problem occurs	failed executions / total
Cost per complete execution	the business figure	sum of tokens × cost/token per run

The last one closes the circle with cost per token: not just “how much a token costs”, but “how much an agent execution costs, and how much of that was useful work versus waste”. Without these KPIs the second vector is invisible, and what is invisible does not get optimised, it gets cancelled when the bill surprises you. With them, the cost of every execution is a real-time figure, and the decision to make each workflow durable (or not) is taken with numbers.

Limits and traps (data-driven)

Durability is not free. 5-20 % overhead; on SaaS with per-step pricing it can run away. Self-host Temporal to remove that price.
Not every agent needs it. A single-step, stateless task gains nothing from durable execution; the value is in the long sequences that fail halfway through.
Bounded retries can hide real failures. A retry limit avoids the cost spiral, but you have to alert when it is exhausted, not swallow the failure in silence.
Checkpoint = security surface. Persisting state opens the door to rollback attacks (ACRFence); protect it.
The first vector still exists. Temporal does not lower the inference cost; you need local vLLM for that. They are complementary, not substitutes.

The conclusion the data and the papers support: the cost of agentic AI has two vectors, and the industry has obsessed over the first (inference) while ignoring the second (faulty execution), which is precisely where Gartner locates the cause of 40 % of cancellations. FastMCP + vLLM solves the first; Temporal the second. A sovereign and sustainable agent platform needs to attack both, with observability that turns the cost of each execution into a real-time figure rather than an autopsy.

Closing

The dominant narrative about agent cost is half-lame: it treats the inference cost as if it were the only one, when it is only the one you see in the demo. The one that kills projects, the one Gartner files under “rising costs” inside its 40 % of cancellations, is the cost of faulty execution: failures charged twice, retries with no ceiling and the context rebuilt on every recovery. The engineering that attacks it is not a prompt trick or a better model, it is an execution model: durable execution, which checkpoints every step, bounds every retry and passes only the necessary context. Temporal, open source and self-hosted, does this while keeping state in your cluster, and combined with vLLM (model) and FastMCP (tools) it composes a fully sovereign agent stack where both cost vectors are bounded and the data never leaves the EU. The state of the art and the papers confirm the problem is real and characterisable: the reliability-cost trade-off, rate limiting as the worst failure, checkpoint-restore as a new security surface. For anyone who wants to be in the 60 % of agentic projects that are not cancelled, the recipe is the one in this article: solve both vectors, not one, and measure the second as seriously as the first.

Sources

Gartner · más del 40 % de proyectos de IA agéntica cancelados antes de 2027 — https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
Temporal · durable execution — https://temporal.io/
CallSphere · Temporal para workflows de agentes (Event History, replay) — https://callsphere.ai/blog/temporal-ai-agent-workflows-durable-execution-workflow-as-code
Inngest · durable execution, clave para agentes en producción — https://www.inngest.com/blog/durable-execution-key-to-harnessing-ai-agents
render.com · plataformas de workflow durables para agentes/LLM (overhead, precio por paso) — https://render.com/articles/durable-workflow-platforms-ai-agents-llm-workloads
ReliabilityBench (arXiv 2601.06112) — https://arxiv.org/abs/2601.06112
Cost-Efficient LLM Agents (arXiv 2603.01548) — https://arxiv.org/pdf/2603.01548
Byzantine Fault Tolerance multi-agente (arXiv 2511.10400) — https://arxiv.org/abs/2511.10400
ACRFence · checkpoint-restore de agentes (arXiv 2603.20625) — https://arxiv.org/pdf/2603.20625
The Six Sigma Agent (arXiv 2601.22290) — https://arxiv.org/html/2601.22290

LLM benchmark tool catalogue: the practical deep dive

Sun, 14 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

What this article covers

Second article in the benchmarking track (B2). The B1 introduction pinned the metrics and the methodology; this article is the practical manual: for each tool, what it measures, how it is actually invoked (with the command), what output format it produces and when to choose it. The goal is that, after reading it, you know what to run to measure your engine and you can reproduce the number. No universal recommendations; only the mechanics of each tool and the criterion for choosing.

How to choose: the tool map

Recalling the split from B1, the tools are ordered on two axes: micro-bench (single-process, for tuning an engine) versus load generator (multi-process, for measuring real capacity), and engine-native versus endpoint-agnostic.

Datasets: the synthetic load matters as much as the tool

Before the profiles, one point that changes results without anyone noticing: what load you feed it. The tools accept different kinds of dataset, and each one measures something different:

Dataset	What it simulates	Bias it introduces
random (fixed lengths)	controlled uniform load	unrealistic: traffic is not fixed length
sharegpt (real conversations)	realistic prompt distribution	the de facto standard for comparison
your own traces	your real traffic	the most faithful, but specific to your case

The trap: a benchmark with short, fixed-length prompts gives a very high throughput that looks nothing like production, where lengths vary and long prompts dominate the prefill cost. For a defensible figure, use sharegpt (comparability) or, better, traces from your own traffic (fidelity). And always declare the length distribution (prompt/output) alongside the number, because two benchmarks with different datasets are not comparable even if they use the same tool.

vLLM bench serve

What it measures: TTFT, TPOT, throughput and latencies of the vLLM server under a synthetic load. Class: micro-bench (single-process). It is the tool for tuning vLLM and seeing the effect of its optimisations (decode, prefill).

Typical invocation (vLLM · benchmark CLI):

vllm bench serve \
 --backend vllm \
 --model meta-llama/Llama-3.1-8B-Instruct \
 --endpoint /v1/completions \
 --dataset-name sharegpt \
 --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
 --num-prompts 1000

Key parameters: --num-prompts (load), --dataset-name (sharegpt, random, etc.), --request-rate (requests/s). The output is a console summary with TTFT, TPOT, throughput and percentiles, and it can be dumped to JSON. To sweep concurrencies there is vllm bench sweep serve, which automates the sweep (vLLM · sweep).

Limit: single-process, it saturates on the client at high concurrency; less flexible than GuideLLM in datasets and load patterns. Use it to iterate quickly on the vLLM config, not to measure maximum capacity at scale.

SGLang bench

What it measures: the equivalent for the SGLang engine (TTFT, TPOT, throughput). Class: micro-bench. Use: tuning SGLang and comparing it against itself across configurations. The mechanics are analogous to vllm bench serve: a synthetic load, console/JSON output with the same metrics. If you are evaluating SGLang against vLLM, do not compare them with their respective micro-benches (same class but different implementations): use an endpoint-agnostic load generator (GuideLLM/AIPerf) against both endpoints, so that the tool is not the variable.

AIPerf (NVIDIA, formerly genai-perf)

What it measures: TTFT, ITL, throughput and latency against any compatible endpoint (vLLM, NIM, TGI, SGLang). Class: multi-process load generator. It is the successor to genai-perf (retired on 15 Apr 2026). Its distinguishing feature: during the sweep it detects GPU saturation and returns the previous iteration as estimatedCapacity, which is why the sweep has to extend beyond the knee (AIPerf).

Output: structured metrics (JSON) with TTFT/ITL distributions and the estimatedCapacity. Use it when you want the real capacity of an endpoint, whatever the engine, with automatic detection of the saturation point.

GuideLLM (vLLM project)

What it measures: full distributions of TTFT, ITL and end-to-end behaviour, for SLO-driven evaluation. Class: multi-process load generator. It is the recommended tool for benchmarking vLLM servers in production: more flexible than vllm bench serve in dataset loading, request format and traffic patterns, with live progress and automatic report generation (Red Hat).

Typical invocation:

guidellm benchmark \
 --target "http://localhost:8000" \
 --rate-type throughput \
 --max-requests 1000 \
 --data "samples=1000,prompt_tokens=1024,output_tokens=256"

Key parameters: --rate-type (synchronous, concurrent, throughput, or rate-based), --data (the load specification: number of samples and prompt/output lengths), --target (the endpoint). It generates reproducible sweeps to find the safe operating range under SLO, with full distributions (not just means). Output: a report with percentiles and, often, exportable to JSON/HTML. It is the default option for answering “how hard can I load this engine without breaking the SLO?”.

LLMPerf (Anyscale/Ray)

What it measures: throughput and latency at the inference level. Class: load generator. Use: endpoint validation, historically very widespread in the Ray/Anyscale ecosystem. It is a solid and well-known option, though less focused on distributions and sweeps than GuideLLM/AIPerf. It fits if you already operate on Ray or want a simple tool to validate an endpoint.

inference-benchmarker (Hugging Face)

What it measures: latency and throughput of inference endpoints, aimed at producing comparable reports. Class: load generator. Use: an OSS alternative inside the Hugging Face ecosystem, useful if you already work with TGI or the HF stack. Like the others in its class, its value lies in measuring real capacity with distributed load; the choice between this one, GuideLLM and AIPerf usually comes down to the ecosystem you already operate in rather than to deep differences in what they measure.

GuideLLM versus AIPerf: which of the two load generators

They are the two serious multi-process load options, and they overlap a great deal. The practical differences that tip the choice:

Criterion	GuideLLM	AIPerf
Origin	vLLM project (Red Hat)	NVIDIA (successor to genai-perf)
Focus	SLO-driven evaluation, reproducible sweeps	real capacity, automatic estimatedCapacity
Load patterns	synchronous, concurrent, rate-based	sweep with saturation detection
Report	live progress + automatic report	structured metrics (JSON)
Ecosystem	vLLM / OpenShift AI	NVIDIA NIM / Triton / vLLM

In practice: if your question is “how hard can I load it without breaking the SLO?”, GuideLLM answers it more directly with its SLO-driven sweeps; if your question is “what is the maximum capacity of this endpoint?”, AIPerf gives it with its automatic knee detection. Many teams use both: GuideLLM for the operational SLO, AIPerf for the reference capacity. What you must not do is compare a GuideLLM result with an AIPerf one as if they were the same measurement: although both are multi-process, their sweep methodology differs; pick one for a given comparison and stick to it.

MLPerf Inference (MLCommons)

What it measures: performance under standardised scenarios (Offline, Server, Interactive) with strict rules. Unlike the previous ones, MLPerf is not “run” for your day-to-day case: it is a vendor comparison suite whose results are read (published by NVIDIA, AMD, Intel and others). Use it to compare hardware and engines against each other under identical rules, not to size your specific workload. For that, GuideLLM/AIPerf against your endpoint.

Practical comparison table

Tool	Class	Base command	Output	When
vllm bench serve	micro	`vllm bench serve`	console/JSON	tune vLLM
vllm bench sweep	micro (sweep)	`vllm bench sweep serve`	console/JSON	sweep concurrency in vLLM
SGLang bench	micro	bench from the SGLang repo	console/JSON	tune SGLang
AIPerf	load	`aiperf profile …`	JSON + estimatedCapacity	real capacity, multi-endpoint
GuideLLM	load	`guidellm benchmark …`	report + JSON/HTML	SLO, reproducible sweep
LLMPerf	load	Ray/LLMPerf script	JSON	validate endpoint (Ray)
MLPerf	suite	(results are read)	official results	compare vendors

Summary: what you run for each question

So as not to get lost in the catalogue, the direct mapping from question to tool:

Your question	Tool	Why
“Does my vLLM config change help?”	`vllm bench serve` (+ sweep)	fast, engine-native
“How hard can I load without breaking the SLO?”	GuideLLM	SLO-driven sweeps
“What is the maximum capacity of the endpoint?”	AIPerf	automatic knee detection
“vLLM or SGLang for my workload?”	GuideLLM/AIPerf against both	same tool, engine as the variable
“Which hardware/engine is better in the abstract?”	read MLPerf	cross-vendor comparability
“Is there a regression in this release?”	short sweep in CI against a baseline	continuous detection

The rule underlying the whole table: micro-bench to iterate on an engine, load generator to measure capacity and decide, MLPerf to compare vendors. And, for any comparison between systems, the same tool for every candidate. If these three categories are clear to you, the specific choice is secondary.

Step-by-step benchmark methodology

A reliable run is not “launching the command”: it is a procedure. The steps, with the load tool (GuideLLM/AIPerf) against your endpoint:

Deploy the engine with the exact config to be measured (model, precision, flags), pinned.
Warm up: send a few requests and discard them, so that the prefix cache and the autotuning do not inflate the first numbers.
Sweep: sweep rising concurrencies (1, 8, 16, 24, 32…) beyond the knee, to see where latency takes off.
Collect the output in JSON with percentiles (TTFT/ITL/throughput/goodput) and the metadata.
Compare: read the goodput under the SLO, not the maximum throughput.

Skipping step 2 (warm-up) or not going past the knee in step 3 are the two mistakes that bias the result the most.

Comparing two engines: the fair protocol

If the goal is to choose between vLLM and SGLang (or TRT-LLM), the protocol that avoids false conclusions:

Same load tool (GuideLLM or AIPerf) against both endpoints, never each engine’s own micro-bench.
Same dataset and length distribution.
Same hardware and precision.
Same SLO to compute the goodput of both.
Vary only the engine; everything else fixed.

Only then is the difference you measure down to the engine and not to the tool, the dataset or the hardware. It is the controlled experiment that supports the scorecard row (article B8).

Output format and comparability

What you record alongside the number is what makes it comparable. A useful output includes, in JSON so it can be versioned:

{
 "tool": "guidellm", "version": "x.y.z",
 "model": "Llama-3.1-70B", "precision": "FP16",
 "hardware": "8xH100 SXM NVLink",
 "load": {"prompt_tokens": 1024, "output_tokens": 256, "concurrency": 16},
 "results": {"ttft_p50_ms": 180, "ttft_p99_ms": 460,
 "itl_p50_ms": 22, "throughput_tok_s": 3400, "goodput_tok_s": 3330}
}

Saving this for every run turns a benchmark into an auditable datum: anyone can reproduce the figure with the same tool, version, model, hardware and load. It is the material of the reproducible harness (article S4), and the difference between a defensible number and a console screenshot.

Worked example: reading the output of a sweep

An illustrative output from a GuideLLM sweep on a 70B on 8×H100 (SLO: P99 TTFT < 500 ms), just as you would read it:

Concurrency	TTFT P99 (ms)	ITL P50 (ms)	Throughput (tok/s)	Goodput (tok/s)
8	240	20	2,100	2,100
16	460	22	3,400	3,330
24	980	31	3,900	2,420
32	1,800	54	4,000	800

How to read it: the knee sits between 16 and 24. At concurrency 16, P99 (460 ms) meets the SLO and goodput (3,330 tok/s) ≈ throughput. At 24, throughput rises little (3,400 → 3,900) but P99 already violates the SLO and goodput falls to 2,420. At 32, throughput is at its maximum (4,000) but goodput collapses to 800: the system “performs well” while serving requests that do not comply. The defensible capacity is the one at concurrency 16 (3,330 useful tok/s), and that is the number that goes into capacity planning and into cost per token. Anyone reporting “4,000 tok/s” is describing the point where the system no longer meets its SLO.

Automating the harness

A manual run does not scale to a benchmarking programme. The minimum automation:

An idempotent script that deploys the engine, warms it up, runs the sweep and saves the JSON with all the metadata (model, version, hardware, dataset, SLO).
Naming by date and config, to version the runs and compare over time.
A results store (a git repo of JSONs, or a bucket) so that anyone can reproduce and compare.

The goal is that reproducing a number is one command, not an afternoon. It is the basis of the harness in article S4, and what turns benchmarking from a one-off activity into a continuous platform capability.

CI integration: continuous benchmarking

The next level is measuring on every change: a CI job that, when the engine or the config is updated, launches a short sweep against a test environment and compares against the baseline. If goodput falls by more than a threshold, the pipeline fails. That way a performance regression is caught at the commit, not in production. Watch out for two things: benchmarking in CI consumes GPU-hours (budget for it) and it needs a stable environment (same hardware) for the comparison to be valid. You do not need the full sweep on every commit: a short sweep covering the knee is enough to detect regressions; the exhaustive sweep is for releases.

The cost of measuring (in euros)

Benchmarking consumes GPU-hours, and that has a cost worth budgeting for. A serious sweep on an 8×H100 node can occupy the cards for a couple of hours; at an amortised cost of ~11 €/h, that is ~22 € per full sweep, more if you sweep several models and precisions. It is not much per run, but a continuous benchmarking programme (every release, every config change) adds up. The practical rule: automate the harness so that each run is cheap and reproducible, and measure what you are going to use to decide, not for the sake of exhaustiveness. The cost of measuring is part of the cost of the platform, small next to the cost of serving but real.

Observing during the benchmark: one run, three axes

A trick that saves work and connects the series: while the sweep runs, capture the GPU metrics too. With DCGM exporting to Prometheus during the run, you record at the same time:

Axis	Source during the sweep	Metric
Performance	the tool (GuideLLM/AIPerf)	TTFT, ITL, throughput, goodput
Energy	DCGM	power (W) → J/token
Cost	node price (OpenCost)	€/hour → CPM per point

That way, from a single run you get the three numbers for the same operating point: at concurrency 16, the goodput (3,330 tok/s), the average power (from DCGM, which divided by the throughput gives J/token) and the cost per token (with the node price). Instead of three separate campaigns, you measure the three axes at once, and they end up coherent by construction because they correspond to the same instant and the same load. It is what the harness in article S4 does, and the reason to export DCGM during the benchmark even if you are only after performance: energy and cost come almost free if you capture them in the same window.

The methodological warning: align the time windows. The DCGM power figures and the tool’s metrics have to cover exactly the same interval (no warm-up, no shutdown), or the J/token does not correspond to the throughput measured. Same window, same three numbers.

The benchmark is also FinOps and energy

An idea that connects this article with the rest of the series: measuring performance is, in fact, measuring cost and energy. By the identity of the opening article, goodput is the denominator of cost per token and of energy per token. When a sweep reveals that config A gives 3,330 tok/s of goodput and config B gives 4,000, you are not only measuring speed: you are measuring that B costs fewer euros and fewer watts per token. That is why the output JSON of a benchmark should be accompanied by the iron cost (from OpenCost) to compute the real CPM of every point in the sweep: throughput × node price = cost per token. Benchmarking is not an isolated axis; it is the tool that, indirectly, moves the cost and the energy of the platform the most, and the one that fills the performance column of the scorecard with numbers that translate directly into euros.

The operational consequence: do not benchmark performance in a vacuum. Every run you save with its throughput and its goodput should be crossable with the node cost (€/hour) and the energy (J/token) to give the three faces of the same operating point. That is how the benchmarking track feeds the FinOps and energy tracks instead of living apart.

State of the art 2026 and limits

Migration to multi-process: AIPerf (formerly genai-perf) and GuideLLM consolidate the measurement of real capacity; micro-benches are left for tuning engines.
GuideLLM as the OSS standard for SLO-driven evaluation with an automatic report.
Careful with comparing across classes: a micro-bench and a load generator are not comparable; pin the class and the tool.
Version and dataset matter: the same command with a different dataset or version gives a different number; pin them.
MLPerf does not size your case: it compares vendors, it does not replace a sweep against your workload.

With the practical catalogue covered, the next article in the track (B3) goes into GuideLLM and SLO validation under load in depth. The tool is the means; the reproducible datum is the end.

Mistakes that invalidate a benchmark

To finish, the list of what turns a run into rubbish, by frequency:

Mistake	Effect	Fix
Comparing different classes	up to 7× of spurious difference	same tool for everyone
Not discarding the warm-up	artificially low TTFT	warm up and discard
Not going past the knee	you do not know the safe capacity	extend the sweep
Unrealistic dataset (fixed length)	throughput that does not apply	sharegpt or your own traces
Reporting the mean, not P99	hides the tail	percentiles always
Tokenizer foreign to the model	biased tok/s and cost per token	count with the model’s own
Not pinning version/config	irreproducible	save everything in the JSON

Any one of these is enough for the number not to be defensible. A benchmark is only as good as its methodology: the tool matters less than running it properly and recording everything.

Closing

The catalogue of benchmark tools boils down to a simple decision (micro-bench to tune an engine, load generator to measure capacity, MLPerf to compare vendors) and to a discipline that weighs more than the choice: the method. The same guidellm benchmark gives a golden datum or a useless one depending on the dataset, the warm-up, how far the sweep goes and what you record in the output. For a sovereign architecture proposal, performance only counts if it comes with its command, its version, its load and its goodput under SLO, and, crossed with cost in euros and energy per token, it becomes the scorecard column that decides which engine and which configuration hold up the platform. You choose the tool in five minutes; the methodology is what makes the number survive an audit.

Sources

vLLM · CLI de benchmark (bench serve) — https://docs.vllm.ai/en/latest/cli/bench/serve/
vLLM · bench sweep serve — https://docs.vllm.ai/en/latest/cli/bench/sweep/serve/
Red Hat · desplegar y benchmarkear vLLM con GuideLLM en Kubernetes — https://developers.redhat.com/articles/2025/12/24/how-deploy-and-benchmark-vllm-guidellm-kubernetes
GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
NVIDIA AIPerf · guía de benchmarking — https://lucaberton.com/blog/nvidia-aiperf-llm-inference-benchmarking-guide/
Medium · Benchmarking LLM Serving Performance (guía) — https://medium.com/@kimdoil1211/benchmarking-llm-serving-performance-a-comprehensive-guide-db94b1bfe8cf

LLM inference benchmarking: frameworks, metrics and the state of the art (tool by tool)

Sat, 13 Jun 2026 02:30:00 +0200

Notation: amounts in euros (N €), decimals with a point. Performance is not very sensitive to the country, but its associated cost (cost per token) is expressed in € and links back to the opening article.

What this introduction covers

Third article in the data series, a deep dive into the performance axis. Measuring the performance of an inference engine looks trivial (“how many tokens per second?”) and it is exactly where people fool themselves most: two tools can report results that differ by a factor of 7 for the same system. This article inventories the metrics that matter and how they are defined, why the architecture of the tool biases the figure, how the saturation point is found with a concurrency sweep, and the profile of each framework. No recommendations: the choice of engine is settled in the Pareto article (B8); here there are only the facts and the methodology, because a benchmark without a published methodology is not comparable.

The performance metrics

There is no single performance metric, there are five, and mixing them is the first source of error:

Metric	Definition	Unit	Dominant phase
TTFT (Time To First Token)	time from sending the prompt to the first token	ms	prefill
TPOT / ITL (Time Per Output Token / Inter-Token Latency)	average time between output tokens once generation has started	ms/token	decode
Request throughput	complete request-response cycles per second at the concurrency tested	req/s	both
Token throughput	total tokens (input + output) per second across all concurrent requests	tok/s	both
Goodput	share of requests that meet the defined SLO	useful tok/s	both
P50 / P95 / P99	latency percentiles (not the mean)	ms	—

Precise definitions, because every tool computes them its own way (Anyscale · latency and throughput metrics): TTFT is what a user waits before seeing the first character, dominated by prefill compute; ITL is the average time between successive output tokens and sets the perceived “typing speed” of the answer; request throughput is complete cycles per second; token throughput is total tokens (input plus output) per second across all concurrent requests.

The latency decomposition

The total latency of a request with $N$ output tokens decomposes as:

$$\text{latency} \approx \text{TTFT} + (N-1)\times \text{TPOT}$$

That is why TTFT and TPOT are reported separately: a single mean latency hides very different profiles. A system with high TTFT and low TPOT (expensive prefill, fast decode) and another one the other way round can have the same mean latency for one particular length, yet behave in opposite ways when the response size changes. For an interactive chat experience TTFT and TPOT rule; for a batch of long summaries, token throughput does. Measuring the mean hides both realities.

Goodput: the honest metric

Raw throughput (TPS, RPS) says how much work the system does; goodput says how much of that work meets your quality-of-service standards (SLO) (Anyscale). An engine can boast 10,000 aggregate tok/s, but if half the requests violate the P99 TTFT SLO, its goodput is 5,000. The figure you defend in a proposal is the goodput, not the catalogue throughput: it is the only one that translates into satisfied users and an honest cost per token.

How each metric is instrumented (and where the error creeps in)

Before comparing numbers it helps to know where each clock starts and stops, because two tools can call different things “TTFT”:

Client-side versus server-side TTFT. The TTFT measured by the client includes network latency and gateway queueing; the one measured by the server does not. For an engine comparison the server-side figure is the one that matters; for user experience, the client-side one. Mixing the two invalidates the comparison.
Streaming is required. TTFT and ITL can only be measured if the response arrives as a stream (token by token). If the tool measures complete responses, there is no real TTFT: there is total latency in disguise.
Token counting. Throughput in tok/s depends on which tokenizer counts the tokens. If the tool uses a tokenizer different from the model’s, the token count (and therefore the tok/s and the cost per token) is biased. You have to count with the tokenizer of the model being served.
Warm-up and prefix cache. The first requests of a benchmark benefit from a warm prefix cache and give artificially low TTFT; the warm-up has to be discarded or the result inflates reality.

These four instrumentation decisions explain a good share of the discrepancies between tools. A performance figure that does not say where the clock is measured and with which tokenizer is not comparable, however precise it may look.

The architecture of the tool biases the figure

Here is the trap that invalidates half the published benchmarks. Tools split into two classes by the architecture of the client that generates the load, and that architecture determines whether the measurement is reliable at high concurrency:

Single-process micro-bench (vLLM bench, SGLang bench, genai-perf): a Python client with asyncio in a single process. Useful for quick experiments on one engine, but the single-process architecture introduces a bottleneck on the client side that biases the data at high concurrency (genAI-perf and vLLM): the client cannot generate enough load and you measure the limit of the client, not the limit of the engine.
Multi-process load (GuideLLM, AIPerf): they spread load generation across several processes, avoiding that limit. This is the class that has emerged for measuring at real scale.

The size of the bias is enormous: at 1,000 QPS, a single-process benchmark processed 75,574 tokens against the 545,733 tokens of a distributed architecture, a discrepancy of 7.2× in measurement capacity for the same system ([search]). Anyone comparing two engines with tools from different classes is not comparing the engines: they are comparing the benchmark clients.

Frameworks, one by one

vLLM bench and SGLang bench — engine micro-bench

What they measure: TTFT, TPOT and throughput of the engine itself (vLLM or SGLang). Class: single-process micro-bench. Use: quick experiments to tune one engine and see the effect of its optimisations (see decode and prefill). Limit: they saturate on the client at high concurrency; they are no good for measuring real capacity at scale.

AIPerf — the NVIDIA one (formerly genai-perf), multi-process

What it measures: TTFT, ITL, throughput and latency against vLLM, NIM, TGI and any compatible endpoint. Class: multi-process load. State-of-the-art datum: NVIDIA retired genai-perf and replaced it with AIPerf on 15 April 2026. During the sweep, AIPerf detects GPU saturation and identifies the previous iteration, returning it as estimatedCapacity; if it detects no saturation, estimatedCapacity is the last iteration tested, which is why the sweep has to extend beyond the knee (AIPerf).

GuideLLM — from the vLLM project, SLO-oriented

What it measures: full distributions of TTFT, ITL and end-to-end behaviour, for SLO-driven evaluation. Class: multi-process load. Differentiator: it generates realistic, configurable traffic patterns in synchronous, concurrent and rate-based modes, including reproducible sweeps to identify safe operating ranges (Red Hat · GuideLLM, GuideLLM · GitHub). It is the tool for answering “how hard can I load this engine without breaking the SLO?”.

LLMPerf — the Anyscale/Ray classic

What it measures: throughput and latency at the inference level. Use: endpoint validation, historically very widespread. Class: load generator. Limit: less focused on distributions and sweeps than GuideLLM/AIPerf.

MLPerf Inference — the industry standard

What it measures: performance under standardised scenarios with strict rules, for comparability across vendors. Maintainer: MLCommons. It is the gold standard of cross-vendor comparability; it is expanded on below.

Comparison table

Tool	Class	What it measures	Maintainer	When to use it
vLLM bench	single-process micro	TTFT, TPOT, throughput of vLLM	vLLM (OSS)	tune vLLM, quick experiments
SGLang bench	single-process micro	metrics of the SGLang engine	SGLang (OSS)	tune SGLang
AIPerf (formerly genai-perf)	multi-process load	TTFT, ITL, throughput; estimatedCapacity	NVIDIA (OSS)	real capacity, multi-endpoint
GuideLLM	multi-process load	distributions, SLO, sweeps	vLLM (OSS)	validate SLO, find the knee
LLMPerf	load	throughput and latency	Anyscale/Ray (OSS)	endpoint validation
MLPerf Inference	standard suite	server/offline/interactive scenarios	MLCommons	cross-vendor comparability

The concurrency sweep: finding the knee

The most useful measurement for sizing is not a number, it is a curve: how latency and throughput change as concurrency rises. Raising concurrency keeps the GPU busier and raises RPS, but past a certain point it sends TTFT, ITL and end-to-end latency through the roof ([search]). The goal of the sweep is to find the knee: the maximum concurrency where throughput still rises without latency breaking the SLO.

That is why AIPerf extends the sweep beyond the knee: only by seeing where latency takes off can it return the safe capacity (estimatedCapacity). This curve is the raw material of capacity planning: the knee yields the replica count and the cost per token at the target load.

Worked example: reading a sweep

An illustrative sweep on an example node (a 70B on 8×H100, SLO of P99 TTFT < 500 ms), to see how the knee is read:

Concurrency	RPS	TTFT P50 (ms)	TTFT P99 (ms)	Token tput (tok/s)	Goodput
1	2	80	110	350	100 %
8	14	110	240	2,100	100 %
16	22	180	460	3,400	98 %
24	26	320	980	3,900	62 %
32	27	540	1,800	4,000	20 %

Reading: up to concurrency ~16 throughput grows and P99 stays under the SLO (goodput ~100 %). Between 16 and 24 sits the knee: throughput barely rises any more (3,400 → 3,900 tok/s) but P99 takes off (460 → 980 ms) and goodput collapses (98 % → 62 %). At concurrency 32 raw throughput is at its maximum (4,000 tok/s) but goodput is 20 %: the system “performs well” while serving mostly requests that violate the SLO. The defensible safe capacity is the one at concurrency ~16, not the one at maximum throughput. This is the number that goes into capacity planning and into cost per token: at 3,400 useful tok/s, not 4,000 raw tok/s.

MLPerf Inference: the comparability standard

To compare across vendors and engines with identical rules there is MLPerf Inference (MLCommons). The datacenter category focuses on two scenarios plus an optional one (MLCommons · datacenter):

Scenario	What it simulates	Metric
Offline	raw throughput processing the whole dataset in batch	maximum throughput
Server	interactive environment: requests one at a time following Poisson at a mean RPS	RPS under TTFT and TPOT limits
Interactive (optional)	like server but with stricter latency limits	RPS under a hard SLO

The Server scenario is the realistic one for online inference: the load generator sends requests following a Poisson distribution and demands that specific TTFT and TPOT bounds be met. MLPerf Inference v5.0 (April 2025) introduced a large-scale 405B benchmark and a low-latency interactive 70B one, offering language benchmarks at every scale (7B to 405B), architectural diversity (including MoE) and scenarios (MLCommons · v5.0); v5.1 (September 2025) widened the results with record participation (MLCommons · v5.1).

The value of MLPerf is comparability: everyone measures the same thing under the same rules. Its limit is that those rules may not match your workload (your length distribution, your specific SLO), so it serves to compare hardware and engines against each other, not necessarily to size your case. For that, your own sweep.

Measurement bias and reproducibility

That two benchmarks give very different results for the same system is no accident: systematic measurement bias in production benchmarks is characterised in the literature (arXiv 2605.24217), and there is work dedicated to the meta-metrics and good practices of system-level performance benchmarking (arXiv 2508.10251). The most common sources of bias:

Source of bias	Effect	Mitigation
Single-process client	underestimates throughput at high concurrency	use multi-process load
Unrealistic length distribution	results that do not apply to your traffic	use realistic traces
Measuring the mean instead of percentiles	hides the latency tail	report P95/P99
Uncontrolled warm-up	the prefix cache inflates the first results	discard the warm-up
Not pinning engine/model version	irreproducible	pin everything and publish it

The methodological conclusion of this article: a benchmark without a published methodology is not comparable. For a performance figure to support a proposal it has to come with the tool, its version, the model and precision, the load distribution and the SLO. Synthesis article S4 builds a reproducible harness that pins all of that.

Checklist for a reproducible benchmark

For a performance figure to be defensible before a committee, it has to come with everything that allows it to be reproduced. The minimum published alongside the result:

What to pin	Why
Tool + version	each one measures differently; the version changes behaviour
Model + precision (FP16/FP8/INT4)	precision changes throughput and quality
Hardware (GPU, count, interconnect)	an 8×H100 NVLink is not an 8×H100 PCIe
Engine + version + flags	vLLM/SGLang/TRT-LLM and their configuration
Length distribution (in/out)	real traffic is not fixed length
Concurrency levels of the sweep	you have to go past the knee
SLO (which percentile, which threshold)	it defines the goodput
Warm-up handling	discard it or it biases the result
Tokenizer used for counting	it affects tok/s and cost per token

The practical rule: if you cannot hand over this table alongside the figure, the figure is not a datum, it is an anecdote. The reproducible harness of article S4 automates the recording of all these parameters so that anyone, including whoever challenges the proposal, can reproduce the exact number.

Performance ≠ quality

One warning that avoids the most expensive mistake: these tools measure speed and throughput, not accuracy. An engine can be blisteringly fast while serving a model that answers badly. Quality is measured with another family of tools (lm-evaluation-harness, HELM, task leaderboards) which is another axis of the scorecard (article B7). Confusing “fast” with “good” is how you build a platform that serves bad answers very quickly. On the final Pareto frontier, performance and quality are two distinct axes that must be looked at together, never one instead of the other.

The other family of tools, for reference (expanded in B7):

Tool	What it measures	Common trap
lm-evaluation-harness	accuracy on hundreds of standardised tasks	contamination of the test dataset
HELM	holistic evaluation (accuracy, robustness, bias, efficiency)	heavy to run
LiveBench / dynamic leaderboards	tasks that rotate to avoid contamination	comparability over time

The data point: contamination, the model having seen the test during training, inflates quality metrics just as warm-up inflates performance ones. That is why dynamic leaderboards rotate their questions. Quality and performance share that lesson: the measurement method biases the result as much as the system being measured.

The latency versus throughput trade-off

A property the sweep reveals and which is worth keeping in mind: latency and throughput pull in opposite directions. Batching groups requests to amortise the cost of moving the weights out of VRAM, which raises throughput, but each request waits for the batch to form, which raises individual latency. There are two operating regimes, and the benchmark serves to place you in the right one:

Regime	Optimises	Configuration	Case
Latency	low TTFT/TPOT	small batch, low concurrency	interactive chat, copilots
Throughput	maximum tok/s	large batch, high concurrency	overnight batch, ingestion

There is no single “best” point: there is the best point for your SLO. A benchmark that reports only maximum throughput is describing the throughput regime and ignoring whether that point meets the latency your case needs. That is why goodput, throughput under the SLO, is the metric that reconciles the two regimes: it measures how much throughput you get without stepping outside acceptable latency. The sweep traverses the curve between both regimes; your SLO marks where on that curve your system sits.

The connection with cost and energy

Performance is not an isolated axis: by the identity of the opening article, throughput is the denominator of cost per token and of energy per token. A sweep that finds a knee at 4,000 tok/s rather than 2,800 is not just “faster”: it lowers the CPM from ~1.09 to ~0.76 €/1M tok and the energy per token in the same proportion, on the same iron. That is why performance benchmarking is the tool that, indirectly, moves cost the most: every goodput improvement translates into euros and into watts per token. The number that connects the three axes is goodput, the throughput that meets the SLO, not catalogue throughput.

State of the art 2026

genai-perf → AIPerf migration (15 Apr 2026): NVIDIA consolidates its benchmarking into a multi-process tool with saturation detection.
GuideLLM as the OSS standard for SLO-driven evaluation with reproducible sweeps.
MLPerf Inference v5.0/v5.1 extends to 405B, interactive 70B and MoE, with record participation: cross-vendor comparability is maturing.
Measurement bias characterised: the community accepts that the measurement method matters as much as the system measured; the emphasis on reproducibility and meta-metrics is growing.

Limits and traps (data-driven)

Comparing tools from different classes. A single-process micro-bench and a multi-process load generator are not comparable; the difference can be 7×. Pin the class.
Catalogue throughput instead of goodput. The honest number is the one that meets the SLO.
Means instead of percentiles. The mean hides the tail; report P95/P99.
Not extending the sweep beyond the knee. Without seeing where latency takes off you do not know the safe capacity.
Confusing performance with quality. They are distinct axes; fast is not good.
Not pinning versions. Engine, model, precision and load left unpinned = irreproducible = not defensible.

The next article in the track (B2) goes into the tool catalogue in depth; this one pins the metrics and the methodology. With performance measured reproducibly, the scorecard can cross it with cost (in €) and energy for the final decision.

Closing

Performance benchmarking looks like the most “objective” of the three axes (they are tokens per second, after all) and it is exactly where the most manipulation happens, almost always without bad intent: a single-process tool here, a mean instead of a P99 there, a catalogue throughput instead of the goodput. The difference between a marketing number and a defensible datum lies not in the engine measured but in the methodology: the class of tool, where the clock is measured, with which tokenizer, how far the sweep goes and which SLO defines the goodput. For a sovereign architecture proposal, performance only counts if it is delivered with that reproducibility record, and, crossed with cost in euros and energy per token, it becomes the column of the Pareto frontier that decides which engine and which configuration hold up the platform. The number you defend is not the highest one: it is the reproducible goodput.

Sources

Anyscale · métricas de latencia y throughput de LLM — https://docs.anyscale.com/llm/serving/benchmarking/metrics
Red Hat · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
NVIDIA AIPerf · guía de benchmarking (ex genai-perf) — https://lucaberton.com/blog/nvidia-aiperf-llm-inference-benchmarking-guide/
MLCommons · MLPerf Inference datacenter (escenarios) — https://mlcommons.org/benchmarks/inference-datacenter/
MLCommons · MLPerf Inference v5.0 (405B + 70B interactivo) — https://mlcommons.org/2025/04/llm-inference-v5/
MLCommons · MLPerf Inference v5.1 — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
arXiv 2605.24217 · sesgo sistemático de medición en benchmarks de inferencia LLM — https://arxiv.org/html/2605.24217
arXiv 2508.10251 · meta-métricas y buenas prácticas de benchmarking de rendimiento — https://arxiv.org/pdf/2508.10251
Medium · LLM Inference Benchmarking (genAI-perf y vLLM) — https://kchandan.medium.com/llm-inference-benchmarking-genai-perf-and-vllm-5dd06b57428e

Serving reasoning models: the invisible scratchpad that decides your latency and your bill

Fri, 12 Jun 2026 05:30:00 +0200

TL;DR

A reasoning model generates, before the answer you see, a thinking block, hundreds or thousands of tokens enclosed between <think> and </think>, that the user does not read but that consumes the same compute, the same VRAM and the same bill as any other token. Serving them in production is not “serving a bigger model”: it is serving a model whose cost per request is variable and, by default, uncontrolled. Three levers change the equation: choosing well between think / non-think mode (the hybrid models of 2026 allow it within the same weights), setting a reasoning budget (thinking_token_budget in vLLM, s1-style budget forcing) and measuring the impact on the KV cache (a 30k-token reasoning chain can eat ~9 GB of KV in FP16). Managed well, you gain accuracy where it matters; managed badly, you pay up to 113× more energy for an answer that did not need that much thinking.

The analogy

Picture a candidate sitting a competitive exam. Before writing the good answer on the official sheet, they fill three sheets of scratch paper: they try approaches, correct themselves, discard. Those scratch sheets are not handed in, the board only reads the final answer, but the candidate has spent ink, time and half an hour of the exam on them.

A reasoning model does exactly that. The <think>…</think> block is the scratch paper: it reasons out loud, contradicts itself, backtracks, and finally emits the “clean” answer. The production problem is that you pay for the scratch paper too: every reasoning token takes a slot in the batch, grows the KV cache and adds to the bill just like an output token. And, like any nervous candidate, the model tends to overthink: it writes five sheets when one would have done.

The operational question in this post is not “does it reason well?”, but “how much scratch paper do I let it write, and when do I take the pen away?”.

What changes when the model “thinks”

In a normal model, a request is: prompt → output tokens. In a reasoning model the sequence has two stretches:

Reasoning block (<think> … </think>): the chain of thought (CoT). Long, variable, normally hidden from the end user.
Final answer: what the user sees.

vLLM models this explicitly. When started with a reasoning parser, the output carries a reasoning_content field separate from content:

vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
 --reasoning-parser deepseek_r1

The parser detects the thinking delimiters and separates both stretches in the API response (vLLM · Reasoning Outputs). This matters for serving for one practical reason: if your gateway or your UI does not separate reasoning_content from content, you will end up showing the scratch paper, or, worse, recording it in logs and traces without meaning to.

The 2026 landscape: from the model that “always reasons” to the switchable hybrid

The first generation (o1, DeepSeek-R1, QwQ-32B) were models that always reasoned. In 2026 the dominant pattern is the hybrid: a single set of weights that switches between think and non-think.

DeepSeek-V3.1 introduced a hybrid reasoning architecture in a single model: DeepSeek-V3.1-Think for complex problems and a faster direct mode for the trivial, switchable by the user (InfoQ). Important for serving: reasoning is off by default and is turned on with thinking=True in chat_template_kwargs.
Qwen3 brings Hybrid Thinking Modes and an explicit Thinking Budget to bound the reasoning compute; in its series thinking is on by default (like QwQ-32B).
gpt-oss shows a different reasoning structure from the R1/Qwen family, which in practice means that the parser and the delimiters are not universal: each family has its own.

The architectural consequence is clear: in 2026 you no longer decide “a model that reasons” vs “one that does not”, but when you let the same model reason. That turns reasoning into a routing and budgeting decision, not a model-catalogue one. It connects directly with the L7 inference router: the “cheap” route turns thinking off; the “expensive” one turns it on with a cap.

The economics: why the scratch paper drives up the bill

Here is the heart of the problem. Reasoning tokens are invisible to the user but real in compute and in billing. Providers charge them as output tokens; API providers bill those reasoning tokens at the output price, not for free (Test-Time Compute Quietly Changed the Economics of Inference). On-premise there is no external invoice, but there is a cost: they occupy batch slots, KV VRAM and watts.

The billable cost (or the equivalent cost in GPU·hours) of a request is:

$$\text{compute tokens} = \text{prompt} + \underbrace{\text{reasoning tokens}}_{\text{hidden, variable}} + \text{visible answer}$$

The middle term is the one that runs out of control. On hard tasks the reasoning/visible-answer ratio can be enormous, and empirical analysis has measured up to 113× the energy cost against standard inference on certain coding tasks (Towards Data Science · Inference Scaling).

And it does not always buy accuracy. The overthinking phenomenon is documented: more thinking does not guarantee a better answer, and beyond a certain point it makes things worse. There is work showing that shorter chains are up to 34.5 % more accurate than the longest one sampled for the same question (Don’t Overthink it), plus studies on the “mirage” of test-time scaling (Mirage of Test-Time Scaling) and on when more thinking hurts (When More Thinking Hurts). The first large-scale study of test-time scaling, more than 30 billion tokens generated with 8 open-source LLMs from 7B to 235B across 4 reasoning datasets, systematises where it scales and where it does not (The Art of Scaling Test-Time Compute).

The operational reading: reasoning is a cost you have to budget for, not a “more quality” switch you leave on full all the time.

Controlling the thinking budget

Three mechanisms, from coarsest to finest.

1. Switching think / non-think

The cheapest thing is not to reason when it is not needed. On hybrid models, turn thinking off for classification, extraction, formatting, trivial answers:

# DeepSeek-V3.1: reasoning OFF by default; turned on explicitly
chat_template_kwargs = {"thinking": True} # only on the "hard" route

This is 80 % of the saving for 20 % of the effort: route by difficulty and reserve thinking for what actually pays for it.

2. Setting a cap: `thinking_token_budget`

vLLM ships a sampling parameter thinking_token_budget that counts reasoning tokens and, on reaching the limit, forces the block to close (reasoning_end_str, typically </think>), making the model answer straight away. If it is not specified, there is no limit other than max_tokens (PR #37112, implementation note). Underneath, a ReasoningBudgetLogitsProcessor injects something like “Let me stop thinking and answer now.</think>” when the cap is reached.

# Define the delimiters of the reasoning block
vllm serve Qwen/Qwen3-32B \
 --reasoning-parser qwen3 \
 --reasoning-config '{"reasoning_start_str":"<think>","reasoning_end_str":"</think>"}'

// per request: cut the scratch paper off at 1024 tokens
{"messages":[...], "thinking_token_budget": 1024}

3. Budget forcing (state of the art)

The technique that popularised fine-grained control is budget forcing, from the s1 paper: to spend less, thinking is terminated by forcing the end delimiter; to spend more, the end is suppressed and the word “Wait” is appended, which pushes the model to review and often correct its own reasoning. With this, s1-32B (a Qwen2.5-32B-Instruct fine-tuned on just 1,000 examples) beat o1-preview by as much as 27 % on AIME24/MATH (s1: Simple Test-Time Scaling). The serving lesson: the budget is not only a cost cap, it is a bidirectional quality lever.

$$\text{accuracy} \approx f(\text{reasoning tokens}) \quad\text{with a maximum, not monotonic}$$

The curve rises, saturates and on many tasks falls. Your job is to operate it near the knee, not at the extreme.

The impact almost nobody measures: the KV cache

This is where reasoning really bites serving. Every token of scratch paper is one more token in the KV cache. A long reasoning chain inflates the working memory just like a long input context.

The KV size grows linearly with the sequence (reasoning included):

$$\text{KV bytes} = 2 \times L \times h_{kv} \times d_{head} \times s \times b$$

where $L$ is layers, $h_{kv}$ KV heads (few, with GQA), $d_{head}$ the per-head dimension, $s$ the sequence length and $b$ the bytes per element. The term reasoning blows up is $s$: a “normal” 500-token answer and one with 8,000 tokens of thinking do not take up the same space, not by a long way.

The real numbers are frightening: for a distilled reasoning model with standard attention such as DeepSeek-R1-Distill-Llama-70B, a single 30,000-token reasoning chain consumes ~9 GB of KV in FP16, which quantising the KV to FP8 reduces to ~4.5 GB, one of the highest-impact optimisations for chain-of-thought workloads (Spheron · KV Cache Optimization). Multiply that by concurrency and you understand why a cluster that served 200 simultaneous requests of a “normal” model drops to 40 with the same model reasoning.

Two state-of-the-art nuances worth knowing:

Prefix caching is not free here. The empirical study of reasoning-model serving finds that prefix caching clearly improves speed on models of 14B or more, but hurts on 7B models (cache management costs more than it saves) (Reasoning Language Model Inference Serving Unveiled). And since the bulk of the reasoning cost sits in the decode (generating tokens one by one), prefix caching, which only speeds up the prefill, barely touches the expensive phase. It connects with prefix cache: engineering the hit rate.
Reasoning-specific KV compression. Techniques such as R-KV have appeared, compressing the KV by exploiting the redundancy typical of chains of thought (the model repeats itself and rambles), recovering memory without touching quality (R-KV).

The most profitable and available lever today is still KV in FP8 (see FP8 end to end) combined with a reasoning budget that avoids 30k-token tails in the first place.

Reasoning + structured output: order matters

If you serve structured output (JSON with a schema) and reasoning at the same time, there is a subtle trap: the grammar must not be applied during the thinking block. The model needs free text to reason; forcing the schema inside the <think> breaks the reasoning. vLLM solves it by having the structured-output engine (xgrammar) use the reasoner’s end_token_id to skip the constraint while the block lasts and apply it only to the final answer (vLLM · Reasoning Outputs). If you build this by hand on another engine, remember: first let it think, then impose the form. It links with structured output.

Reference architecture

On the blog’s example cluster, a node of 4×H100 SXM (80 GB, NVLink), a sensible topology for serving reasoning without wrecking the SLO:

A cheap difficulty classifier up front (a small model or rules) that picks the route. Most of the traffic does not need to think.
Fast route (non-think): the same hybrid model with reasoning off. Low TTFT, predictable cost.
Reasoning route (think): reasoning on with thinking_token_budget by default (say 1–2k tokens) and the option to raise it per request for the genuinely hard cases.
KV in FP8 in the engine on the reasoning route, plus per-request reasoning-length metrics exported to your observability (instrumenting vLLM with OTel).

To prototype this routing outside the cluster, an RTX 5090 (Blackwell, 32 GB) serves a 7–14B reasoning model and lets you validate parsers, budgets and the reasoning_content/content separation before touching the H100s. Do not expect to serve a 32B in think mode at high concurrency on a consumer card: the reasoning KV eats the 32 GB in no time.

Sizing starts, as always, from an SLO; the difference is that now the “answer size” in your capacity planning has to count the scratch paper, not just the visible answer.

Operational pitfalls (and honest scepticism)

Leaking reasoning_content. If the gateway does not separate it, you end up showing or logging the scratch paper, which may contain wrong attempts, sensitive data or language you do not want in your UI. Always separate it at L7.
Serving reasoning “just in case”. Overthinking is real and sometimes lowers accuracy. Do not turn think on by default: turn it on per route, with a budget.
Forgetting the KV. The most common mistake: sizing concurrency from the length of the visible answer. The KV is dictated by the total sequence, reasoning included. Measure p50/p95 reasoning tokens per endpoint.
Prefix caching as a silver bullet. On small models it can make things worse, and in any case it does not touch the decode, which is where the reasoning cost goes.
Assuming universal delimiters. R1, Qwen3 and gpt-oss do not share a reasoning structure. Use the right --reasoning-parser per family and test streaming (the parser has to separate correctly token by token).
Believing more compute = more intelligence. The state of the art (s1, short-m@k, “When More Thinking Hurts”) points the other way in many cases: there is a knee, and operating beyond it is burning watts. Test-time compute is a tool with diminishing, and sometimes negative, returns, not a linear lever.

A note of caution for June 2026: the field of reasoning control (budget forcing, reasoning-specific KV compression, thinking early-exit) moves fast and almost all of it is very recent. The specific figures, the 113×, the 34.5 %, the 9 GB, depend on model, task and configuration; take them as an order of magnitude for design, not as constants. Measure on your own workload before committing to SLOs.

Closing

Serving a reasoning model means serving a model whose cost per request you decide with the thinking budget, not the catalogue. The chain goes: route by difficulty → turn think off where it adds nothing → set thinking_token_budget by default on the route that thinks → quantise the KV to FP8 → measure reasoning length as a first-class metric. Do that, and reasoning becomes a capability you pay for only when it buys accuracy. Do not, and your cluster will serve a fifth of the users while the model fills sheets of scratch paper nobody is going to read.

Sources

vLLM · Reasoning Outputs — https://docs.vllm.ai/en/stable/features/reasoning_outputs/
vLLM PR #37112 · reasoning_budget — https://github.com/vllm-project/vllm/pull/37112
Adding Reasoning Budget to vLLM — https://shuyo.wordpress.com/2026/03/26/adding-reasoning-budget-to-vllm-reasoning-token-limit/
s1: Simple Test-Time Scaling — https://arxiv.org/abs/2501.19393
The Art of Scaling Test-Time Compute — https://arxiv.org/abs/2512.02008
Don’t Overthink it (short-m@k) — https://arxiv.org/pdf/2505.17813
Mirage of Test-Time Scaling — https://arxiv.org/pdf/2506.04210
When More Thinking Hurts — https://arxiv.org/html/2604.10739v1
Reasoning Language Model Inference Serving Unveiled — https://arxiv.org/pdf/2510.18672
R-KV: Redundancy-aware KV Cache Compression — https://arxiv.org/pdf/2505.24133
Spheron · KV Cache Optimization Guide (2026) — https://www.spheron.network/blog/kv-cache-optimization-guide/
Test-Time Compute Quietly Changed the Economics of Inference — https://medium.com/@Elongated_musk/test-time-compute-quietly-changed-the-economics-of-inference-9d0b8d77641c
Inference Scaling (Test-Time Compute) · Towards Data Science — https://towardsdatascience.com/inference-scaling-test-time-compute-why-reasoning-models-raise-your-compute-bill/
DeepSeek-V3.1 Hybrid Reasoning · InfoQ — https://www.infoq.com/news/2025/09/deepseek-v31-hybrid/

Long context and KV offloading: when the notebook does not fit on the desk

Fri, 12 Jun 2026 05:00:00 +0200

TL;DR

Serving long context is not serving a smarter model, it is managing a notebook that does not fit on the desk. The context window is set by the model (and extended with RoPE scaling / YaRN), but the cost of serving it is set by the KV cache, which grows linearly with sequence length and, in production, explodes: a 300k-token contract on Llama 3 70B eats ~93 GB of KV, more than an entire H100, and a million tokens asks for ~125 GB. When the KV does not fit in HBM, there are only two ways out: recompute (extremely expensive, attention is quadratic) or offload the KV to a cheaper memory hierarchy: DRAM, NVMe, network. The 2026 OSS state of the art, LMCache, Mooncake (Kimi’s platform) and NVIDIA Dynamo/KVBM, turns that offload into a first-class KV layer, with reuse across requests, prefill/decode disaggregation and cache-aware routing. Reported results: 3×–10× less latency with LMCache and up to +525 % throughput in long-context scenarios with Mooncake. The price: every memory hop adds transfer latency, so offloading only pays off when what you save by not recomputing beats what it costs to move the bytes.

The analogy

A researcher works at a small desk. On it fit the papers they are pulling from right now, which is the GPU’s HBM, blazingly fast but tiny. When the case is long (a thousand-page file, a million-token context), the papers do not fit. There are three options:

Throw papers away and ask the archive for them again every time they are needed. That is recomputing the KV: correct, but agonisingly slow, because “asking the archive” in an LLM means running the whole prompt through attention again, and attention is quadratic.
Put a shelf next to the desk (the CPU’s DRAM) and, further out, a store room in the basement (NVMe) and a depot in another building (network/object). Moving papers between the desk and the shelf costs seconds, not hours. That is hierarchical KV offloading.
Have two researchers split the work: one reads and marks up the whole file (prefill), then passes their notes to the second, who only drafts (decode). That is the KVCache-centric disaggregated architecture.

This post is about options 2 and 3, the ones that make long context viable in production. Option 1 is what you pay by default if you do nothing.

Part 1 · Why long context is a memory problem

The KV cache grows with the sequence

For every token that goes in or out, the model stores its key and value vectors in each layer so it does not have to recompute them. The size is:

$$\text{KV bytes} = 2 \times L \times h_{kv} \times d_{head} \times s \times b$$

with $L$ layers, $h_{kv}$ KV heads (few, with GQA), $d_{head}$ the per-head dimension, $s$ the sequence length and $b$ the bytes per element. Everything is a model constant except $s$: the KV is linear in context length. Doubling the context doubles the KV. That is the basis of KV cache: the working memory of inference.

The numbers are frightening

At production scale, that linear term turns brutal:

A 300k-token contract on Llama 3 70B consumes ~93 GB of KV alone, more than the 80 GB of an entire H100 (DigitalOcean · Long-Context Inference Cost).
A 1M-token context needs ~125 GB of KV, which exceeds both an RTX 4090 (24 GB) and an 80 GB A100 (Introl · Long-Context LLM Infrastructure).
Even a 7B at 128k climbs to ~14 GB of KV (against ~6 GB at 4k).

The KV stops being an implementation detail and becomes the resource that dictates your concurrency.

And on top of that, attention is quadratic

The KV is linear, but the compute of attention is quadratic in length. When the prompt reaches 1M tokens, generating each token can take on the order of 1.765 seconds, with more than 96 % of the latency spent in attention (Introl). The aggregate effect is a throughput collapse of 10×–100× against short contexts. This is why long context is not “fixed” with more VRAM alone: there is a memory problem (KV) and a compute problem (attention) at the same time.

Extending the window ≠ serving it cheaply

A common confusion: “my model supports 1M of context” does not mean “I can serve 1M cheaply”. The window is extended with RoPE scaling techniques such as YaRN, the practical option for fine-tuning open source models to long context: it needs 10× fewer training tokens and 2.5× fewer steps than naive RoPE interpolation, and it took LLaMA-2 from 4k to 32k and 128k (YaRN/LongRoPE). But that solves the model attending to long context, not the KV fitting on your GPU. According to the 2026 round-ups, May 2026 marked the first generation of open million-token models (with families supporting 256k extensible to 1M via YaRN) (letsdatascience). Treat it as a trend, not a closed spec, and verify capabilities per concrete model.

Part 2 · The KV memory hierarchy

The core idea of offloading is old in systems: hierarchical caching. “Hot” KV (what is in use) lives in HBM; “warm” KV drops to DRAM; “cold” KV to NVMe; “shared across nodes” to network or object storage. Every hop multiplies capacity and divides bandwidth.

Working out when offloading pays off

Offloading is not magic: moving KV from DRAM/NVMe back to HBM costs time. The rule is simple. Offloading pays off when the cost of recomputing exceeds the cost of transferring:

$$t_{recompute}(s) \;>\; t_{transfer} = \frac{\text{KV bytes}}{BW_{enlace}}$$

Since $t_{recompute}$ grows with attention (quadratic) and $t_{transfer}$ grows only with the size of the KV (linear) divided by the link bandwidth, the longer the context, the more the balance tips in favour of offloading. That is why offloading is the lever for long context: it is exactly where recomputing becomes unbearable. The LMCache documentation says it without hedging: offloading only helps when the recomputation it avoids is larger than the overhead it introduces (LMCache benchmark).

Part 3 · KV offloading in practice (OSS)

The limit of out-of-the-box prefix caching

vLLM already caches prefixes in HBM (see prefix cache: hit rate engineering and PagedAttention). The problem: it only lives in HBM, which is small. When the useful KV exceeds HBM, it is evicted and, on the next request, has to be recomputed. There is no speedup even with prefix caching enabled, because the cache is no longer there (vLLM vs LMCache benchmark). Native prefix caching is necessary but insufficient for long context.

LMCache: the persistent KV layer

LMCache adds persistent storage backends to vLLM’s prefix cache (and also supports SGLang and NVIDIA Dynamo as engines). It pulls the KV out of HBM and shares it across engines and across queries, covering a three-level hierarchy: GPU HBM, CPU DRAM and NVMe SSD, with additional backends for Redis/Valkey, Mooncake, InfiniStore, S3 and NIXL/GDS (GitHub LMCache, arXiv 2510.09665). Because DRAM/NVMe hold far more KV than HBM, the hit ratio rises, and LMCache reports 3×–10× less latency combined with vLLM. Against vLLM’s basic CPU offload, LMCache loads the KV at chunk level with high-performance CUDA kernels, which cuts transfer overhead.

A minimal start, offloading to CPU:

# vLLM with LMCache as the KV layer (offload to DRAM)
pip install lmcache
LMCACHE_CONFIG_FILE=lmcache.yaml \
vllm serve meta-llama/Llama-3.1-70B-Instruct \
 --tensor-parallel-size 4 \
 --kv-transfer-config '{"kv_connector":"LMCacheConnector","kv_role":"kv_both"}'

# lmcache.yaml — hierarchy HBM -> DRAM -> NVMe
chunk_size: 256
local_cpu: true # warm KV in DRAM
max_local_cpu_size: 200 # GB
local_disk: "file:///nvme/lmcache" # cold KV on NVMe
max_local_disk_size: 2000 # GB

Offloading to CPU adds moderate overhead but retains a good part of the benefit against recomputing (LMCache docs). Mind the physical path: that HBM↔DRAM↔NVMe traffic goes over PCIe and across NUMA nodes. It is worth reading PCIe topology, GPUDirect and ACS and NUMA, hugepages and CPU isolation, because a badly wired offload can eat its own gain.

Combine it with FP8

Offloading reduces where the KV lives; quantization reduces how much it weighs. They are orthogonal and they add up: KV in FP8 halves the size (see FP8 end-to-end), which means half the bytes to transfer on every offload. In long context, FP8 + offload is the default combination.

Part 4 · KVCache-centric / disaggregated architecture

The next level is not just offloading KV, but redesigning serving around the KV.

Mooncake (Kimi’s platform)

Mooncake, the serving platform behind Kimi (Moonshot AI), is the reference KVCache-centric disaggregated architecture: it separates the prefill and decode clusters, and exploits the underused CPU, DRAM and SSD resources of the GPU cluster to build a distributed KV cache. Its core is a KVCache-centric scheduler that maximises throughput while respecting SLOs, with a prediction-based early rejection policy for overloaded scenarios (arXiv 2407.00079, USENIX FAST'25). The long-context numbers are striking: up to +525 % throughput in certain simulated scenarios while respecting the SLO, and in production Kimi handles 115 % and 107 % more requests on A800 and H800 clusters respectively against previous systems. The title of their FAST talk sums it up: “Trading More Storage for Less Computation”, exactly the rule from Part 2.

NVIDIA Dynamo and KVBM

NVIDIA Dynamo is the open source distributed serving framework with prefill/decode disaggregation, dynamic GPU scheduling and KV-aware routing: the PrefillRouter computes overlap scores between the incoming request and the KV blocks already cached, and routes to the GPUs that already have the relevant KV, avoiding recomputation (Dynamo · Disaggregated Serving, Dynamo · KV-aware routing). KV transfer between prefill and decode is handled by NIXL directly GPU-to-GPU over the best available transport (NVLink, InfiniBand). And its offload manager, KVBM (KV Block Manager), has a three-layer architecture (LLM runtime, logical block management, NIXL transport) that offloads cold KV to CPU RAM, NVMe or network storage to free up HBM (Dynamo · KV Cache Offloading). This connects with disaggregated serving: prefill and decode in specialised pods.

The llm-d project walks the same road, from vLLM’s prefix caching to KV-aware distributed scheduling (llm-d · KV-Cache Wins You Can See).

Part 5 · Reference architecture

On the blog’s example cluster, a node with 4×H100 SXM (80 GB, NVLink), local NVMe and plenty of DRAM, a sensible stack to serve long context without recomputing pointlessly:

KV in FP8 in the engine (vLLM), to start with half the bytes.
LMCache as the KV layer with a DRAM→NVMe hierarchy, so that long prefixes (a manual, a code base, a case file) are cached once and reused across sessions without recomputation.
KV-aware routing (Dynamo or llm-d) if you have several nodes: send the request to the GPU that already holds the prefix, not to any GPU at all.
Prefill/decode disaggregation once the prefill of enormous contexts starts stealing decode from the other requests. Prefilling 300k tokens monopolises the GPU and kills everyone’s latency.
Metrics for KV hit ratio, bytes transferred per level and prefill/decode ratio, exported to your observability stack (instrumenting vLLM with OTel). Without those three metrics, offloading is faith, not engineering.

To prototype the stack, validating the LMCache config, the backend formats, the hit/miss behaviour, an RTX 5090 (Blackwell, 32 GB) with NVMe is more than enough to serve a 7–14B and watch offloading work at contexts of 32–128k. Do not expect to serve 1M on a consumer card: the long-context problem is precisely that neither the KV nor the attention fit on small hardware. Real sizing starts from the SLO and from the context-length distribution of your traffic, not from the model’s maximum window, and that is where it links to capacity planning.

Part 6 · Operational pitfalls (and honest scepticism)

Believing that native prefix caching is enough. It only lives in HBM; in long context it is evicted and you recompute. You need a persistent layer (LMCache/KVBM) for the cache to survive.
Offloading without measuring the break-even. If your prefixes are not reused, offloading only adds transfer latency without saving recomputation. Offloading shines with shared prefixes (RAG over the same corpus, agents with the same system prompt, long sessions), not with one-off, never-repeated requests.
Forgetting that attention is still quadratic. Offloading fixes the memory of the KV, not the compute of attention. For real 1M tokens you also need efficient attention, sparse attention or retrieval approaches (RetrievalAttention, ShadowKV); offloading on its own does not bring down the 1.765 s/token.
Wiring the offload badly. HBM↔DRAM↔NVMe crosses PCIe and NUMA. An offload that crosses the wrong NUMA node or saturates a PCIe lane can be slower than recomputing. Measure the link’s real bandwidth, not the datasheet figure.
Treating the figures as constants. The 93 GB, the +525 %, the 3–10× all depend on model, length, hardware and reuse pattern. They are orders of magnitude for design; measure on your own load.
Trusting million-token model specs without verifying. The landscape of open 1M models is very recent (2026) and moves fast; the specific names and limits change between versions. Verify the window and the cost of serving it per model, do not take the headline on trust.

A note of caution for June 2026: the distributed KV layer (LMCache, Mooncake, KVBM, NIXL, llm-d) is setting right now, with APIs and backends that change release by release. It is exactly the kind of piece worth assembling behind an abstraction (vLLM’s kv_connector) rather than coupling to a single vendor.

Closing

Long context is a problem of memory and compute at the same time, and the KV cache is the bottleneck of both. The window is a gift from the model; serving it cheaply is systems engineering: quantise the KV (FP8), offload it by tiers (HBM→DRAM→NVMe→network) with a persistent layer, and reorganise serving around the KV (aware routing, separated prefill/decode) when scale demands it. The rule that governs it all is Mooncake’s: trade storage for computation. You store more KV in cheap places so you do not recompute expensive attention. Do it well and a 4×H100 node serves contexts that, by recomputing, would not even get started. Do it badly, or not at all, and every long request reads the whole case file from scratch again, while the GPU drowns in a quadratic attention you had already paid for once.

Sources

LMCache · GitHub — https://github.com/LMCache/LMCache
LMCache: An Efficient KV Cache Layer (arXiv 2510.09665) — https://arxiv.org/pdf/2510.09665
LMCache · Offload KV cache to CPU (docs) — https://docs.lmcache.ai/getting_started/quickstart/offload_kv_cache.html
vLLM Prefix Caching vs. LMCache: Benchmarking KV Reuse Tradeoffs — https://levelup.gitconnected.com/vllm-prefix-caching-vs-lmcache-benchmarking-kv-reuse-tradeoffs-944fbaf98b56
Mooncake: A KVCache-centric Disaggregated Architecture (arXiv 2407.00079) — https://arxiv.org/abs/2407.00079
Mooncake · USENIX FAST'25 (Trading More Storage for Less Computation) — https://www.usenix.org/conference/fast25/presentation/qin
Mooncake · GitHub — https://github.com/kvcache-ai/Mooncake/
NVIDIA Dynamo · Disaggregated Serving (docs) — https://docs.dynamo.nvidia.com/dynamo/design-docs/disaggregated-serving
NVIDIA Dynamo · KV-aware routing (docs) — https://docs.nvidia.com/dynamo/latest/user-guides/kv-cache-aware-routing
NVIDIA Dynamo · KV Cache Offloading / KVBM (docs) — https://docs.nvidia.com/dynamo/backends/v-llm/kv-cache-offloading
llm-d · KV-Cache Wins You Can See — https://llm-d.ai/blog/kvcache-wins-you-can-see
Long-Context Inference at Scale: The Hidden Infrastructure Cost · DigitalOcean — https://www.digitalocean.com/community/tutorials/long-context-inference-production-cost
Long-Context LLM Infrastructure · Introl — https://introl.com/blog/long-context-llm-infrastructure-million-token-windows-guide
YaRN / LongRoPE (arXiv 2402.13753) — https://arxiv.org/html/2402.13753v1

Speeding up model cold start: from minutes to seconds

Thu, 11 Jun 2026 13:00:00 +0000

This is the third batch of an operational series on squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB with NVLink. The sister pieces in this batch are Multimodal: serving a VLM on-premise with vLLM, adding vision to the same engine, and FinOps and GPU multi-tenancy with LiteLLM, sharing out and charging for the GPU across teams. This post takes the problem that From disk to HBM laid out conceptually and turns it into a runbook: how you actually bring the cold start down, knob by knob, so that elasticity is usable.

TL;DR

The post From disk to HBM left an uncomfortable idea behind: moving the weights from disk to HBM is only one of the five line items of cold start. The other four, starting the process, creating the CUDA context, setting up the allocator and, the most expensive one that hardly anyone looks at, capturing the CUDA graphs and compiling the JIT kernels, are untouched by any fast loader. That is why a server that loads the weights in 6 s can still take 90 s to be ready. This runbook attacks all five. For the weights: safetensors (mmap, zero-copy, no bounce through a host FP32 buffer) versus the pickle of torch.load; Tensorizer (CoreWeave), which serialises the model into one file and streams it tensor by tensor straight to the GPU from object storage/HTTP/S3; and the Run:ai Model Streamer (from NVIDIA), which reads concurrently and overlaps the read with the H2D copy. All three are switched on with one flag, --load-format, in vLLM. For the non-weight part: torch.compile/Inductor compilation cache (persisted and reused across start-ups), bounding the CUDA graph capture to the batch sizes you really use (the default capture can eat ~54 s), and sleep mode as a shortcut that skips all five line items. The maths: $t = W/B$ for the weights, the saving from overlapping read and H2D instead of doing them in series, and a 70B cold start broken down item by item. And the economics: when scale-to-zero with pre-warm beats keeping a warm replica. On the generic 4×H100 SXM 80 GB cluster.

The analogy: opening the kitchen in the morning

A restaurant does not open by flicking a switch. Anyone who has worked in one knows it: opening time is decided by everything that happens beforehand, and almost none of it is “bringing in the food”.

The first cook arrives at a cold, dark place. They have to: turn on the lights and start the systems (the process init); fire up the ovens and let them come up to temperature (the CUDA context, which takes its time to become operational); get the boards out, sort the cold room, organise the work space (the memory allocator, which reserves and structures the HBM); go down to the store and bring up the produce, the crates of vegetables, the meat, the fish, and put it in its place (loading the weights from disk to HBM, the entire subject of From disk to HBM); and, the most forgotten of all, sharpen every knife, set up the mise en place, make the stocks, pre-heat the mother sauces, all the preparation that is not an ingredient but without which not a single dish goes out (CUDA graph capture and JIT compilation of the kernels).

Here is the trap that From disk to HBM already anticipated: if you only optimise the produce, you still open late. You can hire the best delivery service in the world, a blazingly fast service lift, crates that put themselves away. But if the cook is still sharpening knives and waiting for the ovens to come up to temperature, the kitchen does not serve the first dish. The produce arrived early and the restaurant is still shut.

Today’s runbook is, literally, the list of everything that has to be done for the kitchen to open on time, and which of those tasks can be sped up, which can be left done from the night before (cached), and when it pays better never to switch the kitchen off (warm replica) than to light it again every morning (scale-to-zero).

The five line items of cold start

From the moment an inference pod is born until it returns its first token, the clock runs in five different places. It is worth naming them, because each optimisation attacks one or two, never all of them:

Process init. Starting the Python interpreter, importing the engine (vLLM and its dependencies), parsing the configuration. Seconds, and it grows with the size of the environment and the CUDA/PyTorch imports.
CUDA context. The first cudaSetDevice / cudaFree(0) initialises the CUDA runtime against the driver: it loads the context and maps the GPU. It is not instant, on the order of one to several seconds per GPU depending on driver and number of devices.
Allocator. vLLM sets up its memory manager over the HBM (PyTorch’s caching allocator plus the KV-cache layout). Reserving and structuring tens of GB has its cost.
Weight loading disk→HBM. The journey that From disk to HBM dissected: disk → page cache → host buffer → PCIe → HBM. What almost everyone believes cold start is, and it is only one of the five.
CUDA graph capture + kernel JIT. vLLM captures CUDA graphs for the various batch shapes and compiles kernels with torch.compile/Inductor (and backends such as DeepGEMM or FlashInfer). This item is the great ignored one: the default graph capture in vLLM takes on the order of 54 s because it covers a wide range of batch sizes (vLLM, torch.compile integration; Red Hat, vLLM with torch.compile).

The lesson the diagram is shouting: on a modern server with torch.compile and CUDA graphs, item 5 can be as big as or bigger than item 4. Speeding up weight loading from 40 s to 6 s is a huge advance, but if you leave graph capture at 50 s, the cold start falls from 104 s to ~70 s, not to 12 s. You have to attack both halves.

Speeding up item 4: loading the weights

safetensors versus pickle: the format rules

The first knob is the on-disk format. The old torch.load uses pickle, which has two problems. The security one is well known: deserialising a pickle executes arbitrary Python code, since the __reduce__ protocol lets the file invoke any callable on load, so every model download was a remote execution vector (HuggingFace, Safetensors). The performance one is what concerns us: loading a pickle rebuilds Python objects and usually passes through an intermediate host buffer before reaching the GPU.

safetensors, now under the PyTorch Foundation (HuggingFace, Safetensors joins PyTorch Foundation), solves both. The file is just a JSON header + raw tensor bytes: loading it cannot do anything except populate tensor buffers. And the key property for us is physical: the data region is aligned to a page boundary (the header is padded so that the first tensor starts at a multiple of the OS page size). That is what makes zero-copy possible: a loader can mmap the file, cudaHostRegister over the mapped region and DMA straight from the page cache into VRAM, with no torch.load deserialisation and no temporary host FP32 buffer (HuggingFace, Safetensors). It is the base format; everything else is built on top of it.

The mmap trap (already flagged in From disk to HBM): mmap reads nothing immediately, it defers the cost to the first access to each page. If you do not force the read, the cold start looks short and the first token pays the page faults. And the “blazingly fast second load” is the page cache lying to you: in production pods are born cold, on nodes where those files are not cached.

Tensorizer: streaming tensor by tensor straight to the GPU

Tensorizer, from CoreWeave, serialises the model’s weights and its tensors into a single file and, instead of loading the whole model into RAM before moving it to the GPU, streams the data tensor by tensor from disk, an HTTP/HTTPS endpoint or an S3 bucket, deserialising on the fly straight onto the GPU (vLLM, Loading Models with CoreWeave’s Tensorizer). The operational advantage: near-instant loading and low host RAM usage during initialisation, which matters especially in serverless and autoscaling scenarios, exactly the scale-to-zero case. And it decouples the weights from the container image: the model lives in object storage, not inflating the image pull.

In vLLM it is enabled with --load-format tensorizer (or load_format="tensorizer" via the API). It requires serialising the model once into Tensorizer’s format; from then on, any pod streams it.

Run:ai Model Streamer: reading and copying at the same time

The Run:ai Model Streamer (from NVIDIA) attacks the bottleneck from another angle: it reads the tensors concurrently, N OS threads reading from storage into the CPU buffer, while streaming them to VRAM, so that the storage read and the H2D copy overlap instead of happening in series (vLLM, Loading models with Run:ai Model Streamer; NVIDIA, Reducing Cold Start Latency). It reads safetensors directly, with no reconversion.

The numbers published by NVIDIA give the order of magnitude: the streamer reaches 4.88 s reading from S3 at concurrency 32 and 7.53 s from an SSD IO2 at concurrency 8; integrated into vLLM, the total time to ready drops to 23.18 s from S3, 28.28 s from SSD IO2 and 35.08 s from GP3 (NVIDIA, Reducing Cold Start Latency). Note the gap between the weight loading time (~5–8 s) and the total to ready (~23–35 s): those ~18–27 s of difference are the other four line items, above all item 5. The streamer fixed item 4 and the rest is still there, exactly what this post warns about.

Concurrency is the tuning parameter: it controls how many OS threads read tensors into the CPU buffer (and, for S3, how many client connections the host opens) (vLLM, Run:ai Model Streamer). 16 is usually enough for local NVMe; 32 for high-throughput object storage. One thread does not saturate a Gen5 NVMe; overlapping does.

# default safetensors (the slowest of the fast ones)
vllm serve <model> --load-format safetensors

# Run:ai Model Streamer (concurrent reads, overlaps read + H2D)
vllm serve <model> --load-format runai_streamer \
 --model-loader-extra-config '{"concurrency": 32}'

# Tensorizer (tensor-by-tensor stream straight to GPU from object storage/S3)
vllm serve <model> --load-format tensorizer

The maths of the weights: $t = W/B$ and the saving from overlapping

The floor of item 4 is simple: moving $W$ gigabytes at a bandwidth $B$ takes

$$t = \frac{W}{B}$$

Take a large model, a 70B. In BF16 ($b=2$ bytes/param) that is $W = 70 \times 10^9 \cdot 2 = 140$ GB; in FP8 ($b=1$), $W = 70$ GB. Bandwidth depends on the tier (see the table below): local Gen5 NVMe reads on the order of ~14 GB/s per disk; PCIe Gen5 x16 copies host→GPU at ~50 GB/s; network storage, ~1–3 GB/s. The theoretical floor for reading 140 GB from an NVMe is $140/14 = 10$ s; from the network at 2 GB/s, 70 s. The bottleneck rules.

The overlapping trick. The default loader does the two steps, reading from disk and copying H2D, in series: first it fills a buffer, then it copies, then the next one. The time is the sum:

$$t_{\text{series}} = t_{\text{read}} + t_{\text{H2D}}$$

Tensorizer and the Run:ai Model Streamer overlap them: while one thread copies a chunk to VRAM, another is already reading the next one from disk. With enough concurrency, the total time tends towards the maximum of the two, not the sum:

$$t_{\text{overlapped}} \approx \max(t_{\text{read}},\, t_{\text{H2D}})$$

With the 70 GB of the 70B in FP8, from NVMe at 14 GB/s and PCIe at 50 GB/s: $t_{\text{read}} = 70/14 = 5.0$ s, $t_{\text{H2D}} = 70/50 = 1.4$ s. In series, $5.0 + 1.4 = 6.4$ s; overlapped, $\max(5.0,\,1.4) = 5.0$ s. The saving here is modest (~22%) because the disk dominates comfortably. Overlapping shines when the two steps are comparable: from network storage at 5 GB/s, $t_{\text{read}} = 70/5 = 14$ s and $t_{\text{H2D}} = 1.4$ s, series 15.4 s vs overlapped 14 s; but with many concurrent streams saturating a fast disk and a matched H2D, going from the sum to the maximum can almost double the effective throughput. The flip side: overlapping only gives you what the physical bottleneck allows. If the disk only delivers 14 GB/s, no streamer brings you below 5 s for 70 GB, and that is what the other lever is for, moving fewer bytes (FP8 versus BF16 halves $W$).

The storage tier

Tier	Typical bandwidth	140 GB (70B BF16)	70 GB (70B FP8)	Notes
Local Gen5 NVMe	~14 GB/s per disk	~10 s	~5 s	the short path; one disk
Local Gen5 NVMe (several, RAID0)	~28–50 GB/s	~3–5 s	~1.5–2.5 s	if the loader saturates several streams
Object storage (S3/RGW)	~1–3 GB/s per stream	~47–140 s	~23–70 s	streamer concurrency helps a great deal
NFS / shared network	~1–2 GB/s	~70–140 s	~35–70 s	puts the network and its contention in the path

The operational conclusion is the same as in From disk to HBM: the weights that serve the cold start live on the node’s local NVMe, not on the network. Network storage is the repository; the inference node has a hot local copy (pre-pull with an initContainer or a per-node cache DaemonSet). Tensorizer is the interesting exception: it streams from object storage so efficiently that it sometimes makes serving from S3/RGW viable with no local copy, at the price of depending on the bandwidth of the storage network.

Speeding up item 5: everything that is not weights

Here is the half of the cold start that fast loaders do not touch. Three fronts.

Caching the torch.compile/Inductor compilation

torch.compile has a built-in cache system: the compiled artefacts are saved after the first start-up and reused across start-ups, even across machines if it is configured properly. In vLLM, the result of the Dynamo compilation is stored in ~/.cache/vllm/torch_compile_cache/ (vLLM Blog, Introduction to torch.compile; vLLM, torch.compile integration). The consequence for the runbook: if that directory is a persistent volume shared between pods (a PVC, or a hostPath on the node’s NVMe populated by the first start-up), subsequent pods skip the compilation and start faster. The first pod pays for the compilation; the rest inherit the cache.

It is exactly “sharpening the knives the night before”: the expensive preparation is done once and reused. The nuance: the cache is sensitive to the vLLM/PyTorch version, the GPU, the quantisation and the configuration, and changing any of them invalidates the cache and you pay for the compilation once again.

Bounding the CUDA graph capture

vLLM captures CUDA graphs for a wide range of batch sizes by default, and that is expensive: the default capture is around 54 s, and on large configurations up to 294 s has been measured. By limiting the capture to the batch sizes your workload really uses (for example 1, 2, 4, 8, 16, 24, 32, 64) the cold start is cut by more than 70%, from 294 s to ~82 s in the measured case (Red Hat, vLLM with torch.compile). The knob is the engine’s compilation/CUDA graphs configuration; the principle is not to capture graphs for batch sizes you will never see.

A real trade-off: fewer captured batch sizes = faster start-up but less coverage. A batch outside the captured set falls back to the eager path (slower per request). You bound to the frequent sizes of your traffic, not of any traffic. And there is always the extreme of disabling CUDA graphs (enforce_eager): near-instant start-up on item 5 at the cost of throughput in the hot state, valid for debugging or for very low-traffic services where cold start weighs more than steady state.

Warm pools: not paying for item 5 on the critical path

The definitive shortcut for all five items is not to run them while the user waits. A warm pool, a replica already started, with weights loaded, graphs captured and kernels compiled, waiting for traffic, turns the cold start into zero for that request. It is GPU cost (partly or wholly idle) in exchange for start-up latency; we analyse it in the economics section below.

Sleep mode: the shortcut that skips all five items

And there is a shortcut halfway between “start from scratch” and “keep a whole replica warm”: vLLM’s sleep mode, the subject of Serving several models on a single GPU. Instead of killing the process, it puts it to sleep: it parks the weights in host RAM (level 1) or discards them (level 2), but keeps the process alive, and with it the CUDA context, the allocator, the CUDA graphs and the already compiled JIT kernels. The wake pays for neither item 1, 2, 3 nor 5; it only puts the weights back into VRAM (item 4, and from RAM, not from disk). That is why a wake is 18–200× faster than a full cold start, and even level 2, which reloads the weights from the same disk, is still 23–45× faster, because it skips the other four items (vLLM Blog, Sleep Mode). Sleep mode is living proof of this post’s thesis: if preserving items 1, 2, 3 and 5 gives you 18–200×, then moving bytes was never the whole cost.

An example cold start, item by item

Let us put numbers on a cold start of 70B in FP8 (70 GB) from local NVMe and see which optimisation attacks each item:

#	Line item	Default	Optimisation	Optimised
1	Process init	~4 s	(little room; minimal environment)	~3 s
2	CUDA context	~6 s	(driver; fixed)	~6 s
3	Allocator	~4 s	(fixed)	~4 s
4	Weight loading	~25 s	Run:ai streamer / Tensorizer (concurrency + overlap)	~5 s
5	CUDA graphs + JIT	~54 s	torch.compile cache + bounded capture	~10 s
	Total	~93 s		~28 s

Two readings. First: optimising only item 4 (a fast loader and nothing else) brings the total down from 93 to ~73 s, a real but disappointing improvement, because item 5 is still intact. Optimising only item 5 (cache + bounded capture) brings it down to ~49 s. Attacking both brings it down to ~28 s, and that is what turns theoretical scale-to-zero into usable scale-to-zero. Second: items 2 and 3 are practically irreducible, they depend on the driver and the hardware; they constitute the floor of the cold start, on the order of 10 s, which no software optimisation crosses. Below that floor, the only answer is not to start from scratch: sleep mode (wake from sub-second to a few seconds) or a warm replica (zero).

The economics of scale-to-zero

The capacity question all of this serves is: switch off or keep warm? Scale-to-zero saves GPU while there is no traffic, but pays the cold start when it comes back. The decision is a balance between the cost of an idle GPU and the latency SLA.

The basic criterion: scale-to-zero is only viable if the cold start fits inside the start-up SLA your users tolerate, or if you have a pre-warming mechanism that starts the replica before the request arrives (load prediction, an early signal from the queue). Without pre-warm, a 90 s cold start means the first user after an idle period waits 90 s, unacceptable for almost any interactive SLO. With the cold start brought down to ~28 s it is still a lot for an interactive request, but it is already tolerable for batch workloads or for a pre-warm triggered by autoscaling with KEDA on queue depth.

The decision rule, in one sentence: keep a warm replica when the cost of the idle GPU during the troughs is lower than the cost of missing the SLA on every start-up; do scale-to-zero (with pre-warm) when the troughs are long and deep and the optimised cold start fits in your tolerance. For a 24×7 interactive service with irregular but continuous traffic, a floor of warm replicas almost always wins. For a large model invoked a few times a day (a 70B for hard tasks), scale-to-zero with an optimised cold start, or sleep mode if it shares a GPU with another model, is the sensible option. It is the same capacity planning boundary: cold start is a parameter of the replica cushion, not a start-up detail.

Applied to the generic 4×H100 cluster

Let us bring the runbook down to the 4 H100 SXM 80 GB with NVLink. The concrete decisions:

Loader and format. Weights in safetensors on disk (never pickle). For the short path from local NVMe, the Run:ai Model Streamer (--load-format runai_streamer, concurrency 16–32) is the default option: concurrent reads, overlapping read and H2D, and you only change one flag. Keep Tensorizer for the case where you serve from object storage (RGW/S3) and want to decouple the weights from the image with no local copy, since its tensor-by-tensor streaming straight to the GPU is what makes that pattern viable. In both cases, FP8 versus BF16 halves item 4 almost for free (measure the quality, do not assume it).

Storage tier. Model repository on the network (Ceph RGW), hot copy on the node’s local NVMe populated by pre-pull. The cold start that counts is the cold one, on a node where the files are not in page cache; serving that start-up from the network puts its latency and contention into the critical path.

Caching item 5. The torch_compile_cache directory on a persistent volume or a hostPath NVMe shared between pods of the same model: the first pod compiles, the rest inherit. And bound the CUDA graph capture to the real batch sizes of each service’s traffic.

When scale-to-zero with pre-warm vs a warm replica. The agent LLM (main service, continuous traffic) never does scale-to-zero: a floor of warm replicas on one or two GPUs. The occasional 70B does: scale-to-zero with an optimised cold start, or, if it shares a GPU with a mid-sized model, sleep mode so the wake is sub-second instead of a 28 s start-up (exactly the play in Serving several models on a single GPU). The optimised cold start in this post is what makes swap and failover between replicas tolerable: a replica that falls over and is replaced in 28 s degrades less than one that takes 90 s. And it connects directly with multi-tenancy: a short cold start lets you share GPUs between teams with per-tenant scale-to-zero without the first user of each tenant paying for an endless start-up.

The cross-cutting principle: cold start is the ceiling on elasticity. The five line items, with both halves attacked, a fast loader for the weights and cache plus bounded capture for the graphs, bring it down from minutes to seconds. And below the irreducible floor of ~10 s (CUDA context + allocator), the only way out is not to start from scratch: sleep mode or a warm replica.

Traps and things that are not what they seem

“I switched to a fast loader and start-up barely improved.” Your bottleneck probably was not item 4. If CUDA graph capture eats 54 s, bringing weight loading down from 25 to 5 s takes 20 s off 93, which is barely noticeable. Measure where the time goes before optimising: vLLM’s start-up logs break down weight loading versus graph capture.

“The second time it started in a flash.” The page cache (for the weights) and the torch.compile cache (for the graphs) lying to you at the same time. The start-up that matters is the cold one, on a clean node. Benchmarking the second start-up measures a situation that almost never occurs in the peak that triggers the autoscaler.

“Tensorizer/streamer will give me the 18–200× of sleep mode.” No. Those loaders speed up only item 4. The 18–200× of sleep mode comes from preserving the other four by keeping the process alive. They are tools for different problems: a fast loader is for a real cold start (a new process); sleep mode is for alternating models without killing the process.

"enforce_eager fixes the cold start." It fixes item 5 (no graph capture, near-instant start-up) but degrades throughput in the hot state, since CUDA graphs exist because they speed up the steady state. It is a trade: it wins for very low traffic services where start-up weighs more than the hot path; it loses for a service with sustained load.

“I bound the CUDA graphs to a single batch size and that is it.” You bound to the sizes your traffic uses, not to one. A batch outside the captured set falls back to the eager path and runs slower. Too aggressive and you penalise the steady state to gain a few seconds of start-up.

“Moving the weights to FP8 also speeds up item 5.” Not directly. FP8 halves the bytes (item 4) and doubles inference throughput, but graph capture and kernel compilation do not depend on the size of the weights, they depend on the shape of the graph and the batch sizes. Item 5 is attacked with caching and bounded capture, not with quantisation.

Conclusion

From disk to HBM said it conceptually: moving the weights is one of the five line items of cold start. This runbook turns it into action. For the weights, item 4, there is a clear menu: safetensors as the base (mmap, zero-copy, no pickle), Run:ai Model Streamer for the short path from NVMe (concurrent reads overlapping read and H2D), Tensorizer for streaming from object storage with no local copy, and the cross-cutting lever of moving fewer bytes with FP8. But the forgotten half of the start-up is item 5, CUDA graph capture and JIT compilation, which on a modern server can be as big as the weight loading: you attack it by caching the compilation across pods and bounding the graph capture to the real batch sizes. Attacking only one half leaves the cold start half done; attacking both brings it from ~90 s to ~28 s. And below the irreducible floor of ~10 s imposed by the CUDA context and the allocator, the only way out is not to start from scratch: sleep mode (sub-second wake) or a warm replica (zero). The kitchen opens on time when somebody took care of the produce and of sharpening the knives the night before. Optimising only the delivery from the store leaves the ovens cold and the restaurant shut.

References

HuggingFace, Safetensors (format, zero-copy, security versus pickle): https://huggingface.co/docs/safetensors/index
HuggingFace, Safetensors is Joining the PyTorch Foundation: https://huggingface.co/blog/safetensors-joins-pytorch-foundation
vLLM, Loading Models with CoreWeave’s Tensorizer: https://docs.vllm.ai/en/stable/models/extensions/tensorizer/
vLLM, Loading models with Run:ai Model Streamer: https://docs.vllm.ai/en/stable/models/extensions/runai_model_streamer/
NVIDIA, Reducing Cold Start Latency for LLM Inference with NVIDIA Run:ai Model Streamer (S3/SSD benchmarks, concurrency): https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer/
vLLM Blog, Introduction to torch.compile and How It Works with vLLM (compilation cache): https://blog.vllm.ai/2025/08/20/torch-compile.html
vLLM, torch.compile integration (cache in ~/.cache/vllm/torch_compile_cache/): https://docs.vllm.ai/en/latest/design/torch_compile/
Red Hat Developer, vLLM with torch.compile: Efficient LLM inference on PyTorch (CUDA graph capture ~54 s, bounding batch sizes −70%): https://developers.redhat.com/articles/2025/09/03/vllm-torchcompile-efficient-llm-inference-pytorch
vLLM Blog, Zero-Reload Model Switching with vLLM Sleep Mode (wake 18–200×): https://blog.vllm.ai/2025/10/26/sleep-mode.html

Multimodal on-premise: serving a VLM with vLLM (vision + language)

Thu, 11 Jun 2026 11:40:00 +0000

Third batch of an operational series on squeezing a generic on-premise LLM cluster of 4×H100 SXM. Its sibling pieces in this batch are FinOps and GPU multi-tenancy with LiteLLM, which puts a price on every token, and a visual token costs the same as a text one, and Speeding up cold start with Tensorizer, which matters twice as much here because a VLM loads two models (the vision encoder and the LLM). If the VLM ends up feeding a document assistant, that end-to-end setup with LibreChat and RAG is another story (in preparation).

TL;DR

A vision-language model (VLM) is not magic: it is a normal LLM with a sensory organ bolted on the front. Three pieces in series: a vision encoder (a ViT) that looks at the image, a projector or connector (typically an MLP) that translates the encoder’s output into the LLM’s embedding space, and the usual LLM, which receives those embeddings as if they were just more tokens. The consequence that governs the entire operational design is brutal in its simplicity: an image becomes visual tokens, and those tokens cost exactly the same as text tokens. A high-resolution document page is not “a couple of image tokens”: it is hundreds or thousands of tokens entering the context, inflating the prefill (compute-bound) and occupying KV cache (memory-bound) just as if you had pasted a thousand words. In Qwen2.5-VL the arithmetic is literal: the number of visual tokens is $H\times W / (14\times14\times4)$, so an $896\times896$ image is 1,024 tokens and an A4 page scanned at a decent resolution easily reaches 1,500–2,500 tokens for the image alone. This post opens up the anatomy, works through the token-cost maths and translates it into TTFT and VRAM on an 80 GB H100, explains vLLM’s multimodal support (--limit-mm-per-prompt, pixel budget via mm_processor_kwargs, chat template) and answers the question that really matters: classic OCR or VLM. The short answer: use Tesseract/PaddleOCR for the bulk of documents with clean text, and reserve the VLM, expensive in tokens and in GPU, for the subset that genuinely needs it: complex layout, nested tables, handwriting, stamps, charts. On the generic 4×H100 SXM cluster, with FP8 so it fits.

The analogy: the reader who sees the page versus the one who only hears the transcript

Picture two experts you ask to review a scanned contract. To the first you read the contract aloud, a blind transcript, word by word. It is fast, efficient, and for 90 % of the clauses it is enough: the text is the text. But there are things you cannot read out: that the signature at the end is handwritten and does not match the typed name; that there is a “PAID” stamp struck diagonally across the amount; that the table on page 3 has a merged cell that changes who each row applies to; that there is a pen annotation in the margin. The blind transcript loses all of that or flattens it into gibberish.

To the second expert you hand the whole page, to look at. They see the signature, they see the stamp, they understand the structure of the table because they perceive the lines and the cells, they read the marginal note. They grasp the document as a visual object with layout, not as a river of characters. They are strictly more capable.

Why not always use the second one, then? Because seeing costs more mental bandwidth. The first expert receives the contract as a flow of words: cheap, linear. The second receives it as an image that their brain has to parcel up, attend to region by region, and reconstruct. They process a great deal more information per page, most of which, in a clean text document, is redundant with what the transcript would already have given them. If you only need to read the clauses, hiring the expert-who-sees for every page is overpaying for capability you do not use.

That is the entire thesis of this post. The VLM is the expert who sees; OCR is the blind transcript; and “mental bandwidth” is tokens. The engineering consists of sending the expert-who-sees only the pages where seeing changes the answer, and resolving the rest with the cheap transcript.

Anatomy of a VLM: encoder, projector, LLM

A modern VLM of the Qwen-VL family (Qwen2.5-VL model card; Qwen3-VL Technical Report) is made up of three blocks in series:

Vision encoder (ViT). A Vision Transformer that receives the image, cuts it into patches and produces an embedding per patch (or per group of patches). In Qwen2.5-VL it is a dynamic-resolution ViT: it accepts images of arbitrary size, resizing them to multiples of 28 and splitting them into 14×14 pixel patches, with window attention to speed up the processing of large images. There is no “canonical size” everything is squashed to: a large image produces more patches than a small one.
Projector / connector. The ViT’s embedding space is not the LLM’s. The projector acts as the translator. In Qwen2.5-VL it is elegant and cheap: it groups every 2×2 block of adjacent patches (four ViT tokens), concatenates them and projects them with a two-layer MLP into a single fused token in the LLM’s dimension. That fusion factor of 4 is the reason for the $4$ that appears in the denominator of the token formula below: four ViT patches collapse into one visual token that the LLM sees.
The LLM. The usual language transformer. It receives a sequence of embeddings that is a mixture of text tokens (from your prompt) and visual tokens (from the image), all in the same space, and does what it knows how to do: attention over the complete sequence and autoregressive generation. To the LLM, a visual token and a text token are indistinguishable in cost: both go through the same QKV projections, both occupy an entry in the KV cache, both take part in the $O(C^2)$ attention.

The idea to internalise: the encoder and the projector are the sensory organ, but the inference cost lives in the LLM, and that cost is paid in tokens. The ViT adds a fixed prefill overhead (processing the image once), but the recurring bill, the LLM prefill and the KV cache throughout the whole generation, is set by the number of visual tokens the projector spits out. That is why the control lever is not “which encoder”, but how many pixels you let through.

The maths: what it costs to see a page

Let us put numbers on “an image costs many tokens”, with the real Qwen2.5-VL formula (Qwen team, Qwen2.5-VL blog):

$$N_{\text{tok}} = \frac{H \times W}{14 \times 14 \times 4} = \frac{H \times W}{784}$$

where $H$ and $W$ are height and width in pixels (rounded to the multiple of 28 the ViT imposes). The $14\times14$ is the patch size; the $\times4$ is the projector’s 2×2 fusion. A sanity check against the model card figure: an $896\times896$ image gives

$$N_{\text{tok}} = \frac{896 \times 896}{784} = \frac{802{,}816}{784} = 1{,}024 \text{ tokens.}$$

It checks out. Now an A4 page scanned at a resolution reasonable for reading small text, say $1120 \times 1568$ px (approximate height $\times$ width, already rounded to multiples of 28):

$$N_{\text{tok}} = \frac{1120 \times 1568}{784} = \frac{1{,}756{,}160}{784} \approx 2{,}240 \text{ tokens.}$$

A single page is ~2,240 visual tokens before you write a word of prompt. For context: that is roughly what 1,700 words of Spanish text would take up (at ~1.3 tokens/word). If that A4 page contains 600 words of actual text, the VLM is paying almost 4× in tokens to see it rather than read its transcript. If the page is a dense table or a handwritten document that OCR cannot read, that 4× is justified. If it is a paragraph of clean text, it is pure waste.

Qwen2.5-VL lets you control this with min_pixels / max_pixels: the number of tokens per image is dynamic and ranges from about 4 up to 16,384 by default, adjustable with those parameters (HF Qwen2.5-VL docs). Lowering max_pixels cuts tokens at the expense of resolution, and of the ability to read the fine print. It is exactly the lever of “how much mental bandwidth do I give the expert”.

From token cost to TTFT

Visual tokens are not free in latency. The prefill, processing the whole context before the first token, is compute-bound and its cost grows with the context length $C$ (linear in the projections, quadratic in the attention); this is worked through in detail in vLLM’s attention backend. Every visual token enters that $C$.

Let us model the TTFT as the time to process the prefill tokens at a given prefill throughput. Take a VLM of around 7–8B on an H100 with an illustrative prefill throughput of ~12,000 tok/s (an example figure; measure it, do not assume it, and it depends on how much the ViT forward pass weighs on top). For a request with an 80-token text prompt plus one page at 2,240 visual tokens:

$$\text{TTFT} \approx \frac{80 + 2240}{12{,}000 \text{ tok/s}} \approx \frac{2320}{12{,}000} \approx 0{.}19 \text{ s}$$

plus the fixed cost of the vision encoder’s forward pass over the ~9,000 ViT patches (another compute-bound slice of prefill). Now five pages in the same request (a short PDF):

$$\text{TTFT} \approx \frac{80 + 5 \times 2240}{12{,}000} = \frac{11{,}280}{12{,}000} \approx 0{.}94 \text{ s}$$

The TTFT multiplies almost fivefold, because the visual tokens completely dominate the context. And attention scales as $O(C^2)$: five pages are not “5× more expensive” in the attention component, but ~25× relative to one. The operational lesson: the number of images and their resolution are your TTFT budget, not a detail. Lowering max_pixels from 2,240 to, say, 1,100 tokens/page (lower resolution, enough for large text) cuts the prefill almost in half.

VRAM: the VLM’s weights and what is left for KV

On an 80 GB H100 SXM, let us budget a VLM of the Qwen-VL family. Take an 8B (LLM) plus the vision encoder (a ViT of a few hundred million parameters; call the whole thing ~8.5B effective parameters for weight purposes). In BF16 (2 bytes/parameter):

$$M_{\text{BF16}} \approx 8{.}5 \times 10^9 \times 2 \text{ B} \approx 17 \text{ GB.}$$

In FP8 (1 byte/parameter), the weights halve (Qwen3-VL-8B-Instruct-FP8, with reported quality almost identical to BF16):

$$M_{\text{FP8}} \approx 8{.}5 \times 10^9 \times 1 \text{ B} \approx 8{.}5 \text{ GB.}$$

For an 8B model this fits comfortably in BF16 on an H100; FP8 matters more for leaving VRAM free for KV cache, and here the KV is the problem, because visual tokens inflate it. The detail of end-to-end FP8 (weights and KV) is in FP8 end to end. What is relevant: one page is ~2,240 KV entries; serving several sessions that send long documents consumes KV at a rate a text-only case would never see. With a 32B VLM in FP8 (~32 GB of weights) on an 80 GB H100 you are left with ~45 GB for KV and activations, which with long multimodal contexts run out sooner than “text-only” intuition suggests. The rule: with a VLM, plan the VRAM counting the visual tokens in the KV, not just the weights.

Serving a VLM with vLLM

vLLM supports image multimodal input natively for the Qwen-VL family and many other VLMs (Multimodal Inputs, vLLM docs). The gears that matter:

--limit-mm-per-prompt. Limits how many multimodal items each prompt accepts, per modality. By default the limit is generous (999 per modality), but in production it is worth setting it low, --limit-mm-per-prompt '{"image": 2}', because every accepted image reserves token and KV budget. If you set a limit of 8 images and someone sends 8 A4 pages, that is ~18,000 visual tokens in a single request: it blows up max_model_len or the KV. The limit is a budget firebreak, not a cosmetic restriction. (A curious case: setting it to 0 for a modality lets you pass precomputed embeddings without loading the encoder module, saving VRAM, useful if you do the vision encoding outside vLLM.)

Pixel budget via mm_processor_kwargs. Here the token formula connects with the configuration. When serving Qwen2.5-VL you pass, for example, --mm-processor-kwargs '{"min_pixels": 12544, "max_pixels": 254016}' (vLLM Qwen2.5-VL discussion). With $254{,}016$ maximum pixels per image, the token ceiling per image is $254{,}016 / 784 \approx 324$ tokens, an aggressive trim that keeps the cost in check at the price of not being able to read very small print. This is the dial that really governs your token bill: raise it for dense documents, lower it for images where fine detail does not matter.

The chat template and the image placeholder. The model’s processor (loaded via AutoProcessor) inserts special tokens marking where the image goes in the sequence: in Qwen2.5-VL the sequence <|vision_start|><|image_pad|><|vision_end|>, where <|image_pad|> expands to the exact number of visual tokens the image produced. vLLM applies this template automatically; it is worth knowing there is a documented quirk, the image block tends to be placed before the user’s text in the sequence order, regardless of where you put it in the content (issue #15125, vllm-project/vllm), which rarely causes trouble but is worth bearing in mind if the image/text order is semantically important for your prompt.

Current models. As of June 2026, the reference open family is Qwen3-VL (dense 2B/4B/8B/32B variants and MoE 30B-A3B / 235B-A22B, interleaved contexts of up to 256K tokens combining text, image and video), with published FP8 checkpoints of almost identical quality to BF16 (Qwen3-VL Technical Report, arXiv 2511.21631; Qwen3-VL Usage Guide, vLLM recipes). Qwen2.5-VL remains perfectly serviceable and its token formula is the one we have used because it is public and clean; the cost figures are of the same order in both generations.

Classic OCR vs VLM: when to use which

Here is the decision that pays the bill. Parsing documents to feed them into a document ingestion pipeline has two paths, and choosing the split between them well is the difference between a healthy cluster and a choking one.

Classic OCR (Tesseract, PaddleOCR). Text-recognition engines that run on CPU, are cheap, fast and mature. Tesseract is the portable workhorse (edge, offline kiosks, embedded scanners). PaddleOCR is more capable on layout and Asian scripts, with its PP-Structure module for preserving structure, extracting tables and detecting key-value fields (PaddleOCR vs Tesseract, CodeSOTA 2026). On clean, well-scanned text, classic OCR is hard to beat on cost per page: zero GPU, millisecond latency, very high throughput. Its weakness is the “hard” document: complex multi-column layout, tables with merged cells, handwriting, stamps, charts, poor scan quality. There, pipeline-based OCR either fails or returns gibberish that destroys the structure.

VLM. It takes the image of the document and generates structured markdown or HTML in a single end-to-end pass, understanding the layout because it sees it. It shines exactly where classic OCR struggles: nested tables, reading charts, handwriting, understanding the spatial relationship between elements. The price is the one we have been calculating all through this post: expensive in tokens and in GPU. Worth noting a 2026 nuance: the frontier has moved. VLM models specialised in document parsing have appeared, compact (~1–3B), open source, scoring very high on benchmarks such as OmniDocBench (PaddleOCR-VL in the 94–96 point band; dots.ocr 3B at ~88 with full markdown including tables) (LLM-OCR vs Traditional OCR, Parsli 2026; Open-Source OCR Models, E2E 2025). In other words: for pure parsing, today you may not want a 32B generalist VLM, but a small specialised VLM-OCR. But it is still GPU and it still pays tokens per image: the cost calculation does not disappear, the model just gets cheaper.

The split rule. It is not “OCR or VLM” as a global choice, but per-document triage. Run everything through a cheap first filter: classic OCR already returns a confidence score; if it is high and the document has no tables or visual elements, the OCR transcript will do and you have spent CPU. If confidence is low, there are tables, there is handwriting or there are graphical elements, escalate that page to the VLM. In a typical corpus, 80–90 % of pages are resolved with cheap OCR and only the rest consumes GPU. This is what makes an on-premise VLM sustainable: you do not serve everything with it, you use it as the expensive specialist you send only the hard cases.

Applied to the generic 4×H100 cluster

How this fits into a cluster of 4×H100 SXM 80GB:

Where to place the VLM. A VLM deserves a whole H100 or a large slice, for a VLM-specific reason that text-only intuition does not anticipate: the token cost per image means not many concurrent sessions fit. If each request brings one or several pages, each consumes thousands of KV tokens; the KV cache runs out with few simultaneous sessions. Compared with a text-only chat service where dozens of short conversations fit per GPU, a document service saturates the KV with a handful of heavy requests. That is why it makes no sense to carve the GPU finely for the VLM with aggressive MIG (GPU sharing is covered in sharing a GPU: time-slicing, MPS and MIG): the bottleneck is not idle compute but the KV, and a small slice runs out of KV before it makes use of the compute.

FP8 so it fits and to free up KV. Serve the VLM in FP8: at equal reported quality, the weights take up half and you leave the maximum VRAM for the KV cache, which is the scarce resource with multimodal contexts. For a 32B in FP8 (~32 GB) on an H100, those ~45 GB free for KV are what determine how many concurrent pages you can handle.

The VLM only for the subset that needs it. The most important architectural piece: do not put the VLM on the critical path of every document. The triage from the previous section lives upstream; the bulk (clean text) is resolved with OCR/PaddleOCR on CPU, and in a cluster with 4×H100 there is probably spare CPU to run it in parallel with the GPUs, and only the hard subset reaches the VLM. This frees the H100s for what really needs them and keeps a reasonable cost per document. It connects directly with FinOps and GPU multi-tenancy with LiteLLM: when you put a price on every token, a visual token costs the same as a text one, so a page through the VLM can cost 4× its transcript, and that should show up in the chargeback of the team that decides to send everything to the VLM for convenience.

Double cold start. An operational detail that bites: a VLM loads two models into memory, the vision encoder and the LLM, so its cold start is heavier than that of an equivalent LLM. If you are going to do model swapping or on-demand scaling, fast loading matters twice as much; that is where speeding up cold start with Tensorizer comes in. If instead you serve several models on one GPU by rotating them, the swap+sleep pattern (serving several models on one GPU) has to account for the VLM taking up more room and loading more slowly.

Consumer GPU, for reference. If someone wants to prototype outside the cluster, an RTX 5090 (Blackwell, 32 GB) comfortably serves a 7–8B VLM in FP8 (~8.5 GB of weights) with KV to spare for a few pages per request; a 32B in FP8 (~32 GB) no longer fits with useful KV on that card and calls for the H100. The 5090 is good for validating the triage pipeline and the prompt; the real service belongs on the H100s.

What we have not covered

Video. Qwen3-VL handles video with dynamic FPS sampling; each frame is an image and the token cost multiplies by the number of sampled frames. One minute of video can be tens of thousands of tokens. It is the most expensive multimodal beast and deserves its own analysis.
Prefix caching of images. If the same image (a logo, a template) appears in many requests, can its KV be cached? It depends on the backend and on the stability of the placeholder; it connects with the attention backend.
Evaluation. How to measure that the VLM extracts the table correctly (it is not enough for it to “look right”): faithfulness against a set with structured ground truth.
Fine-tuning the VLM for a specific document domain (invoices, particular forms), which can put part of the domain’s “knowing how to see” into the weights and reduce what the prompt has to explain.

References

vLLM, Multimodal Inputs (limit_mm_per_prompt, precomputed embeddings): https://docs.vllm.ai/en/stable/features/multimodal_inputs/
vLLM, Vision Language Multi Image (mm_processor_kwargs, min/max pixels): https://docs.vllm.ai/en/latest/examples/offline_inference/vision_language_multi_image/
vLLM, Qwen3-VL Usage Guide (recipes): https://docs.vllm.ai/projects/recipes/en/latest/Qwen/Qwen3-VL.html
Qwen team, Qwen2.5-VL (blog, token formula, dynamic resolution): https://qwenlm.github.io/blog/qwen2.5-vl/
Qwen2.5-VL (Transformers docs, min/max pixels, token range): https://huggingface.co/docs/transformers/model_doc/qwen2_5_vl
Qwen3-VL Technical Report, arXiv 2511.21631: https://arxiv.org/abs/2511.21631
Qwen3-VL-8B-Instruct-FP8 (FP8 with quality almost identical to BF16): https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct-FP8
vLLM issue #15125 (image placeholder order): https://github.com/vllm-project/vllm/issues/15125
PaddleOCR vs Tesseract, CodeSOTA 2026: https://www.codesota.com/ocr/paddleocr-vs-tesseract
LLM-Based OCR vs Traditional OCR, Parsli 2026: https://parsli.co/blog/llm-ocr-vs-traditional-ocr
Open-Source OCR Models 2025, E2E Networks: https://www.e2enetworks.com/blog/complete-guide-open-source-ocr-models-2025

Serving several models on a single GPU: co-residency, model-swapping and sleep mode

Thu, 11 Jun 2026 02:40:00 +0000

This is the second piece of an operational series on squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB with NVLink. Its sibling, Sharing a GPU: time-slicing, MPS and MIG, splits the GPU so that several processes can use it at once; this one splits the complementary problem: you have more models than VRAM and you need them to coexist in time, not just in space. The third, The RAG data plane on CPU, moves off the GPU everything that does not need to be there; and the end-to-end sovereign assistant (fourth instalment, in preparation) assembles all of this behind LibreChat and LiteLLM. Here we assume that VRAM is the scarce resource and that keeping everything loaded at once is not an option.

TL;DR

You have an agent LLM serving almost all the traffic, one or two rerankers for RAG, an alternate model (another family, another language, a fine-tune) and, now and then, a large model for hard tasks. Added up, they do not fit in the 80 GB of an H100. There are three ways to make them coexist, and the key to not confusing them is to understand that they are three different states of where the weights live:

Co-residency: several models loaded at the same time in HBM, if the sum of their VRAM budgets fits in the physical memory. It is the ideal when they fit: zero switching latency. The catch: each engine must cap its own VRAM; if two engines both believe they own the whole 80 GB, the second one to ask for memory dies with OOM. On a GPU shared by time-slicing, sharing time does not protect memory (the processes still compete for the same HBM) and this problem gets worse, as we see in the sibling piece.
Model-swapping (with llama-swap): only one (or very few) resident at a time. The proxy looks at the request’s model field and, if another one is needed, unloads the current model and loads the requested one. Since there is only one inside, each model can use almost the whole GPU. The cost is a full cold start: seconds from local NVMe, minutes from network storage. It mixes engines: it exposes OpenAI-compatible endpoints including /rerank with llama-server on GGUF.
vLLM sleep mode (--enable-sleep-mode, endpoints /sleep and /wake_up): the model is not unloaded, it is put to sleep. The weights are parked in host RAM (level 1) or discarded while the process stays alive (level 2). The wake is 18–200× faster than a cold start because the process is still alive and keeps the CUDA allocator, the CUDA graphs and the compiled JIT kernels. The only thing rebuilt is the KV cache, which is discarded on sleep. It needs enough RAM for the sleeping weights.

The decision tree is short. Do they all fit? Co-residency. They do not fit, switching is infrequent and you mix engines? llama-swap. Everything is vLLM, there is RAM to spare and wake latency matters? Sleep mode. Sustained load on all of them at once? That is not swapping, it is more GPUs or replicas (see one big vs N small). This post puts numbers on each of them on an 80 GB H100.

The analogy: one stage, one spotlight

Picture a theatre with a single stage and a single spotlight, the GPU. You have more actors (models) in the company than fit under the light at once. There are three ways to stage the show:

Co-residency: several small actors on stage. If the cast for this scene is three slight actors, the main LLM, one reranker, another reranker, they all fit under the spotlight at the same time. Switching lines between them is instantaneous: they are already on stage. But the stage has a fixed size; add a fourth, burly actor (the large model) and they do not fit, so someone falls off the edge. That is the OOM.
Model-swapping: the actor goes home. When an actor finishes their part, they go home (the disk). If the next scene needs them again, they have to come back from home: get dressed, put the make-up on, travel to the theatre. That takes minutes. In exchange, while they are on stage they have the whole stage to themselves. That is the llama-swap model: maximum VRAM per actor, but re-entry costs a full round trip.
Sleep mode: the actor waits in the wings. Instead of going home, the actor steps out of the spotlight but stays backstage (host RAM), already dressed and made up. When their turn comes, they step in in two strides. No journey, no wardrobe: just crossing the curtain. That is the vLLM wake: the weights come back not from disk but from RAM, and everything that goes into “preparing the actor”, the allocator, the CUDA graphs, the kernels, is already done because they never left the building.

The operational moral is the same as in the theatre: the cost of an actor is not only their talent, it is how long they take to walk on when you need them. Co-residency pays for the permanent spot; swapping pays for the journey; sleep pays for the RAM of keeping them waiting nearby. The rest of this post is about choosing which, with a VRAM budget and a stopwatch in hand.

The three states: where the weights live

Co-residency: the VRAM budget

Co-residing is the default option when they fit: zero switching latency because every model is already in HBM. The whole question is do they fit?, and it is answered with a VRAM budget. The VRAM of an inference engine splits, roughly, into three items:

$$\text{VRAM}_{\text{model}} = \underbrace{P \cdot b}_{\text{weights}} + \underbrace{\text{KV}}_{\text{cache}} + \underbrace{A}_{\text{activations + overhead}}$$

where $P$ is the number of parameters, $b$ the bytes per parameter given the quantisation, KV the KV-cache budget (which scales with concurrency and context length) and $A$ the overhead of activations, CUDA buffers and fragmentation. The hard rule of co-residency is:

$$\sum_i \text{VRAM}_{\text{model } i} + \text{headroom} \;<\; \text{VRAM}_{\text{physical}}$$

If the sum overflows, no sharing scheme saves you: the first engine to ask for memory beyond the free gap dies with OOM. And here is the critical nuance that connects to the sibling piece: splitting the GPU by time does not split the memory. Under time-slicing, two processes take turns on compute but share the whole HBM with no isolation; if both assume they own the 80 GB, they collide. Only MIG gives memory partitions with a real boundary, as we see in Sharing a GPU. To co-reside properly, each engine must cap itself: in vLLM, --gpu-memory-utilization 0.45 tells it “do not use more than 45 % of the GPU”; in llama-server, you control the layers on GPU and the KV size. If you do not set those limits, vLLM claims 90 % of the GPU for itself by default, and there is no room left for anyone else.

A worked example: what fits in 80 GB

Take the realistic scenario. A 32B agent LLM in FP8 ($b = 1$ byte/param):

$$P \cdot b = 32 \times 10^9 \cdot 1 \text{ B} = 32 \text{ GB of weights}$$

On top of that comes the KV cache. To serve with decent concurrency and agent contexts (which are long: history, tools, documents), a KV budget of ~12 GB is reasonable, plus some ~2 GB of activations and overhead. Total for the main LLM: ~46 GB. That leaves ~34 GB of the 80. Let us see what fits there:

Service	Size (FP8/INT8)	KV + overhead	Total VRAM
32B agent LLM	32 GB	~14 GB	~46 GB
Reranker A (cross-encoder ~0.5B)	~0.5 GB	~0.5 GB	~1 GB
Reranker B (cross-encoder ~0.5B)	~0.5 GB	~0.5 GB	~1 GB
8B alternate model	~8 GB	~4 GB	~12 GB
Sum			~60 GB
Free headroom			~20 GB

With this they co-reside comfortably: the 32B LLM, the two rerankers (which are featherweights, a few hundred MB to 1 GB each) and even an 8B alternate, leaving 20 GB of cushion. Rerankers are the textbook case for co-residency: so small that they always fit alongside the main LLM, and rotating them would be absurd. Switching between them is instantaneous because they are all loaded.

When does it break? When the occasional large model arrives, a 70B in FP8:

$$70 \times 10^9 \cdot 1 \text{ B} = 70 \text{ GB of weights}$$

The 70B’s weights alone already eat 70 of the 80 GB. There is no way to co-reside it with the 32B, not remotely. That is where co-residency ends and rotation begins: the 70B can only come in if we evict the 32B from the GPU. That is the exact boundary: co-reside while the sum of budgets fits with headroom; rotate as soon as one model needs, on its own, more VRAM than is left over.

Model-swapping with llama-swap: the actor goes home

When they do not fit at once, the first answer is to rotate: keep one resident model and change it on demand. The canonical tool for this in the on-premise world is llama-swap, a Go proxy that sits in front of your inference servers (llama.cpp, vLLM, TabbyAPI and so on) and starts and stops them as needed.

The mechanism is elegant in its simplicity. Every OpenAI-compatible request carries a model field. llama-swap reads that field, looks at which upstream server it has configured for that model, and:

If the requested model is already loaded, it routes the request straight through.
If another one is loaded, it stops it (freeing its VRAM) and starts the right one.
When the new server reports ready, it forwards the request.

The cost of a swap is exactly a full cold start of the new model: unloading the current one (fast, just freeing memory) plus loading the new one (slow, moving the weights from disk to HBM). That second term is the painful one, and its magnitude is decided by where the weights are: from local NVMe it is seconds; from network storage (Ceph RGW, NFS) it can be minutes. The whole analysis of the load path, and why the default loader makes it slow, is in From disk to HBM; here the consequence is enough: a swap is only viable if it is infrequent, because every switch pays that toll in full.

The advantage in exchange: since there is only one resident, that model can use almost the whole GPU. The 70B that co-resides with nobody fits comfortably if it is the only one inside. And llama-swap mixes engines: you can have a vLLM configured for the big LLM, a llama-server with GGUF for the alternate, and two llama-server instances for the rerankers, which expose /rerank, /v1/rerank and /v1/reranking natively. That heterogeneity (GGUF + rerank + vLLM behind a single OpenAI endpoint) is exactly what vLLM alone does not give you, and the main reason to choose llama-swap.

Example ConfigMap: two rerankers rotating on the same port

A concrete and useful case: you have two rerankers, one multilingual and one specialised in code, that you do not need at the same time and would rather not keep both resident. With llama-swap they rotate on the same endpoint, triggered by the model field. The ConfigMap (mounted as the proxy’s config.yaml in a Kubernetes deployment) would be:

apiVersion: v1
kind: ConfigMap
metadata:
 name: llama-swap-rerankers
data:
 config.yaml: |
 # How long a model stays loaded after the last request
 # before llama-swap unloads it to free VRAM
 healthCheckTimeout: 60

 models:
 # Multilingual reranker (GGUF, via llama-server)
 "reranker-multilang":
 cmd: >
 /usr/bin/llama-server
 --model /models/bge-reranker-v2-m3.Q8_0.gguf
 --reranking
 --port ${PORT}
 --n-gpu-layers 99
 --ctx-size 8192
 # ttl: after 300 s idle, it is unloaded and the GPU is freed
 ttl: 300

 # Code reranker (GGUF, via llama-server)
 "reranker-code":
 cmd: >
 /usr/bin/llama-server
 --model /models/codereranker.Q8_0.gguf
 --reranking
 --port ${PORT}
 --n-gpu-layers 99
 --ctx-size 8192
 ttl: 300

A request to /v1/rerank with "model": "reranker-multilang" starts that server; the next one with "model": "reranker-code" stops the multilingual one and starts the code one on the same ${PORT} that llama-swap manages. Since both are small, the swap between them takes one or two seconds: quantised GGUFs weigh a few hundred MB and come from local NVMe. The ttl controls how long a model stays loaded after its last request: raising it avoids swaps if requests arrive in bursts; lowering it frees the GPU sooner for other uses.

An honest caveat: this pattern of two rerankers that rotate makes sense when they do not fit alongside the rest, or when you want to reserve the VRAM for something else. If they do fit (and a couple of 0.5 GB rerankers almost always fit, as we saw above), co-residing them is strictly better: zero swap latency. llama-swap shines when you rotate large models or mix engines that do not coexist well, not for juggling tiny models that would fit together.

vLLM sleep mode: the actor in the wings

Swapping has a problem: every switch pays the entire cold start, and a cold start is not just moving weights. As we saw in From disk to HBM, starting an inference engine includes initialising the Python process and the CUDA context, setting up the memory allocator, capturing the CUDA graphs and compiling the JIT kernels (DeepGEMM, FlashInfer, TorchInductor). Moving the weights is only one of five items, and often not the largest.

vLLM’s sleep mode (--enable-sleep-mode) attacks exactly that: instead of killing the process when a model stops being used, it puts it to sleep, leaving the process alive. There are two levels, and the difference is where the weights go:

Level 1: offloads the weights to host RAM (CPU) and discards the KV cache. The process stays alive. The wake copies the weights from RAM to VRAM, not from disk. Typical wake: ~0.1–0.8 s for small models, ~3–6 s for large ones. It needs enough RAM for the sleeping weights (on the order of the model’s size in GB).
Level 2: discards the weights entirely, keeping only small buffers (rope scaling tensors and the like). The wake does reload the weights from disk, but everything else, process, allocator, CUDA graphs, JIT kernels, is already done. Typical wake: ~0.8–2.6 s for small models. RAM use is almost nil (megabytes).

The key that explains the numbers: at both levels, keeping the process alive preserves the expensive infrastructure. That is why the benchmark in the vLLM blog (Oct 2025) reports that a wake is 18–200× faster than a full reload, and, most counter-intuitively, level 2 is still 23–45× faster than a cold start despite reloading the weights from the same disk, because it skips the other four items. In their measurements on an A100, a full cycle of 5 model switches goes from 357 s without sleep (≈48 s per switch) to 112 s with level 1 (wake of 0.26 s / 0.82 s) or 125 s with level 2 (0.85 s / 2.58 s).

The KV cache is always discarded on sleep. That is not a minor detail: it means the first response after the wake rebuilds the KV from scratch, paying a full prefill. That is why the wake is not “entirely free”: the model is available in sub-second time, but the first request is a little slower until the KV is repopulated. A one-request warm-up after the wake hides that cost.

# Start vLLM with sleep mode (admin endpoints, trusted networks only)
export VLLM_SERVER_DEV_MODE=1
vllm serve <model> --enable-sleep-mode --port 8001

# Sleep (level 1: weights to host RAM)
curl -X POST 'localhost:8001/sleep?level=1'

# Wake up
curl -X POST 'localhost:8001/wake_up'

Security warning (from vLLM’s own blog): the /sleep, /wake_up, /collective_rpc and /reset_prefix_cache endpoints require VLLM_SERVER_DEV_MODE=1 and should only be exposed on trusted networks, since they can take the service down. They are for internal orchestration (a controller that sleeps and wakes models according to the queue), not for the public plane.

The latency maths: why the wake wins

Let us put numbers on “the wake comes from RAM, not from disk”. The cost of having a model available is, in each strategy, the time to move its weights from wherever they are to HBM, plus, in the swap case, the other four cold-start costs. Take the 34 GB of weights of the 32B LLM in FP8 and compare the three paths.

Cold start from NVMe (swap). A reasonable Gen4/Gen5 NVMe gives on the order of ~5 GB/s effective per stream with the default loader (the disk’s theoretical floor is higher, but single-threaded deserialisation does not saturate it, see From disk to HBM). For 34 GB:

$$t_{\text{NVMe→HBM}} \approx \frac{34 \text{ GB}}{5 \text{ GB/s}} \approx 6.8 \text{ s just moving bytes}$$

And that is before adding CUDA graph capture and kernel compilation, which add several more seconds. The real cold start of a 32B is around 15–40 s depending on loader and storage. From the network (Ceph RGW), multiply.

Level 1 wake from RAM. The weights come not from disk but from host RAM, and travel over PCIe Gen5 x16, whose practical host→GPU bandwidth is ~50–64 GB/s. For the same 34 GB:

$$t_{\text{RAM→HBM}} \approx \frac{34 \text{ GB}}{55 \text{ GB/s}} \approx 0.62 \text{ s}$$

And there is nothing else to pay: the allocator, the graphs and the kernels are already there. The real wake of a model this size lands in the sub-second to a few seconds range that vLLM reports. The speed-up against an NVMe cold start is on the order of:

$$\frac{t_{\text{cold start}}}{t_{\text{wake}}} \approx \frac{15\text{–}40 \text{ s}}{0.6\text{–}3 \text{ s}} \approx 10\text{–}60\times$$

consistent with the 18–200× from the blog, whose measurements include smaller models, where the relative weight of the preserved CUDA graphs is even larger and the factor goes up.

Why the bandwidth difference explains almost everything. The key jump is not 5 vs 55 GB/s (a ~11× in transport). It is that the cold start also pays for rebuilding infrastructure, which the wake skips entirely. The table:

Path	Weight source	Bandwidth	34 GB bytes only	+ CUDA graphs / JIT	Realistic total
Cold start (swap) NVMe	disk	~5 GB/s	~6.8 s	yes (several s)	15–40 s
Cold start (swap) network	network	~1–2 GB/s	17–34 s	yes	30 s – min
Level 1 wake	host RAM	~50–64 GB/s	~0.6 s	no (preserved)	0.6–3 s
Level 2 wake	disk	~5 GB/s	~6.8 s	no (preserved)	7–10 s

Look at the level 2 row: it reloads the weights from the same disk as the swap (~6.8 s of bytes), but since it does not rebuild graphs or kernels, its total (~7–10 s) still beats a full cold start (15–40 s). It is the proof that moving bytes is only part of the cost, and the part sleep mode exploits.

Assumptions, honestly: the bandwidths are indicative. NVMe “5 GB/s effective” assumes the default loader; with a concurrent streamer it goes up. PCIe “55 GB/s” assumes Gen5 x16 with a pinned buffer and NUMA locality; if the buffer lands on the wrong socket, it drops. And the “15–40 s” cold-start range depends on the model, the quantisation and whether the files are in page cache (the “the second time it loaded fast” trap). The numbers are there to reason about orders of magnitude, not to size anything without measuring on your hardware.

The decision tree

The three strategies do not compete: each wins in a different regime. The tree, in order:

The node most often ignored is the one at the top right: “sustained load on all of them at once?”. If your four models get constant and simultaneous traffic, neither swapping nor sleep helps, since both assume the models take turns in time. Rotating under sustained load only adds switching latency without solving the underlying problem: there is not enough compute. The answer then is to scale horizontally (more replicas) or spread across more GPUs, a capacity decision analysed in One big vs N small. Swapping and sleep are tools for temporally unbalanced workloads: many models, but rarely active at the same time.

Applied to the generic 4×H100 cluster

Let us bring this down to the 4 H100 SXM 80 GB with NVLink. The winning strategy is not to pick one of the three, but to spread the models across GPUs according to their usage pattern and apply to each GPU the strategy it calls for. A reasonable split:

H100 #0, the workhorse (co-residency). The 32B agent LLM (main service, constant traffic) co-resides with the two rerankers and, if they fit, the 8B alternate. It is the GPU that never rotates: everything living here is used continuously and fits comfortably in 80 GB (the ~60 GB of the example above). Zero switching latency between the LLM and its rerankers, which is exactly what RAG needs, since a fast reranker is useless if you have to wait for a swap every time.

H100 #1, light services with MIG. If you have many small heterogeneous services, an embedding model, a classifier, a guardrail, an STT/TTS, splitting this GPU with MIG into isolated instances (each with its slice of HBM with a real boundary) gives co-residency with memory isolation, preventing a service that inflates its KV from taking the others down. The detail of when MIG beats time-slicing is in Sharing a GPU; the rule here: co-residing light services on one GPU makes sense when they fit and when isolating them is worthwhile, and MIG is the tool for the second part.

H100 #2, the occasional large model (sleep mode or swap). The 70B that is only invoked for hard tasks does not deserve a dedicated GPU kept awake, since it would sit idle most of the time, burning 700 W for nothing. Two options:

If this GPU also serves a medium model routinely and you only occasionally need the 70B, use sleep mode: put the medium one to sleep (level 1, weights to RAM), wake the 70B. Note that level 1 sleep frees the VRAM, the weights go to RAM, so yes, it fits even though the 70B would not fit alongside an awake medium model. Waking the medium one again is sub-second.
If the 70B comes as GGUF or you mix engines, llama-swap rotates between the medium model and the 70B by the model field. Each 70B invocation pays its cold start (seconds from local NVMe), acceptable if it is occasional.

H100 #3, replica / overflow. The fourth GPU absorbs peaks: a replica of the main LLM for when #0’s queue grows, or reserve capacity. There is no swapping here: it is pure capacity, the answer to the tree’s “sustained load” node.

The cross-cutting principle: co-reside what is used together and continuously (LLM + rerankers); isolate the light and heterogeneous with MIG; sleep or rotate the large and occasional; replicate what saturates. The four GPUs do not all do the same thing, each runs the strategy its load pattern calls for. And the NVLink between them matters for something else (tensor parallelism for the 70B if it would not fit even on one; see One big vs N small), but for this post’s problem, many models, one GPU, the lever is when each model needs to be awake.

Traps and things that are not what they seem

“Co-residing is always better if they fit.” Almost, but watch the KV cache: co-residing two models means splitting the KV budget between them. If the main LLM needs a large KV for high concurrency and long contexts, adding a flatmate cuts that KV and lowers its throughput. Sometimes it is better to give the whole GPU to the main model and rotate the secondary one. Co-residency is not free: the tenant takes room away from the cache of the one that matters.

“Sleep mode is like swapping but faster.” Not exactly. Swapping releases the process; you can have N models configured and only pay RAM/disk for the resident one. Sleep mode keeps one live process per sleeping model, so each sleeping vLLM still occupies its process slot, its RAM (level 1) and its management footprint. Sleep scales well to a few rotating models; for many (10+), level 2 (minimal RAM) or plain swapping fit better. Do not put 15 models into level 1 sleep and expect the RAM to hold.

“The wake is instantaneous, I lose nothing.” The model’s wake is sub-second, but the KV cache was discarded on sleep. The first request after the wake pays a full prefill to repopulate the KV, slower than normal. If your SLA is strict on the first response after an idle period, add an automatic warm-up after the wake. Prefix caching helps make that reprefill cheaper if there are stable prefixes.

“llama-swap with a low ttl saves me VRAM for free.” It saves you VRAM while nobody uses that model, but every time it comes back it pays the cold start. An aggressive ttl on a model with bursty traffic turns every burst into a load wait. The right ttl depends on the temporal pattern of the requests, not on how much VRAM you want to free. Measure it.

“Time-slicing lets me co-reside more models.” False and dangerous. Time-slicing shares out compute time, not memory, and all the processes still compete for the same HBM with no isolation. Co-residing via time-slicing does not give you more effective VRAM; it gives you more processes fighting over the same VRAM, and an OOM when the sum overflows. For real memory partitioning, MIG. The detail is in the sibling piece.

Conclusion

Having more models than VRAM is not a problem of insufficient hardware: it is a problem of managing a scarce resource over time. The intuition of “I need one GPU per model” is expensive and almost always wrong, because models are rarely used all at once. The three strategies are three answers to the same question, where do the weights of the models you are not currently using live: in HBM if they fit (co-residency, zero latency but they cost space), on disk if switching is rare (swapping, maximum VRAM per model but a return trip of seconds to minutes), or in RAM if switching latency matters (sleep mode, sub-second wake at the cost of tied-up RAM). Sleep mode is the most interesting addition of 2025 because it breaks the false dilemma of “everything loaded vs reload every time”: by keeping the process alive and preserving the allocator, the CUDA graphs and the kernels, it turns a 30–100 s cold start into a sub-second wake, and it does so even when it reloads the weights from the same disk (level 2), because moving bytes was never the whole cost. On the four-H100 cluster, the play is not to pick one strategy but to spread: co-reside what goes together, isolate the light stuff with MIG, sleep or rotate the large and occasional, replicate what saturates. The GPU is the stage with a single spotlight; the art is knowing which actor walks on, which goes home and which waits in the wings.

References

vLLM Blog (Embedded LLM), Zero-Reload Model Switching with vLLM Sleep Mode, 26 oct 2025: https://blog.vllm.ai/2025/10/26/sleep-mode.html
vLLM Docs, Sleep Mode: https://docs.vllm.ai/en/latest/features/sleep_mode/
mostlygeek, llama-swap (OpenAI/Anthropic-compatible model-swapping proxy): https://github.com/mostlygeek/llama-swap
llama-swap, Configuration: https://github.com/mostlygeek/llama-swap/blob/main/docs/configuration.md
NVIDIA, H100 Tensor Core GPU (HBM3 specs, 80 GB, ~3.35 TB/s): https://www.nvidia.com/en-us/data-center/h100/
NVIDIA, Reducing Cold Start Latency for LLM Inference with NVIDIA Run:ai Model Streamer: https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer/

QLoRA runbook: from dataset to adapter served in multi-LoRA (operational procedure)

Tue, 09 Jun 2026 03:00:00 +0000

This is the operational companion to QLoRA and multi-LoRA at the limit on small models. That post takes apart the why, NF4, double quantisation, paged optimizers, the adapter maths; this is the how, with commands you copy and paste. If you have not read the fundamentals one, read it first: here we assume you know what an adapter is, why the base lives in 4-bit and why the gradient only touches the adapter.

TL;DR

A reproducible procedure in five phases: (1) pin the environment with fixed versions; (2) prepare the dataset in chat format; (3) train the QLoRA adapter with TRL + PEFT on an RTX 4090 (24 GB, Ada Lovelace) using gradient checkpointing, gradient accumulation and paged_adamw_8bit; (4) validate and version the adapter as an artefact of megabytes; (5) serve it in vLLM with --enable-lora, loading it hot without restarting the server and resolving it from object storage. All on-premise, on consumer hardware, without a single data item leaving the perimeter. What follows are the exact commands and the memory budget that separates “it fits” from “OOM”.

The end-to-end flow

Phase 0 — Environment and versions

QLoRA is sensitive to the versions of bitsandbytes, transformers, peft and trl: misaligned combinations give dequant errors or adapters that do not load in vLLM. Pin the environment and do not touch it mid-campaign. Reference versions as of June 2026 (check the specific ones in your index; the exact pin matters less than the coherence between them):

python -m venv .venv && source .venv/bin/activate
pip install --upgrade pip

# Training (producer)
pip install "torch>=2.4" \
 "transformers>=4.50" \
 "peft>=0.14" \
 "trl>=0.15" \
 "bitsandbytes>=0.45" \
 "accelerate>=1.2" \
 "datasets>=3.2"

# Serving (consumer) — in its own environment/image
pip install "vllm>=0.8"

What each piece does and why it is pinned:

Package	Role in the flow	Why the version matters
`torch`	tensor runtime and CUDA kernels	the CUDA ABI has to match the driver and `bitsandbytes`; a major jump breaks the 4-bit kernels.
`transformers`	loads the base, the tokenizer and the `chat_template`	it has to know the architecture of the SLM you use; a new model needs a version that supports it.
`peft`	implements LoRA/QLoRA: injects the `A,B` matrices and writes the `adapter_config.json`	that `adapter_config.json` is what vLLM reads when serving; old versions write fields the serving side does not understand.
`trl`	the `SFTTrainer`: the supervised training loop	it integrates `peft` natively; its API (`SFTConfig`) changes between versions, hence the pin.
`bitsandbytes`	the NF4 quantisation and `paged_adamw_8bit`	the most sensitive piece: a badly compiled binary gives corrupt dequant or hangs on the first step.
`accelerate`	orchestrates device, mixed precision and `device_map`	the silent backend of almost everything; misaligning it with `transformers` gives cryptic errors.
`datasets`	loads the JSONL (and allows streaming if the corpus is large)	not very sensitive; any recent 3.x will do.
`vllm`	the multi-LoRA serving	separate environment or image: do not mix its stack with the training `bitsandbytes`.

The golden rule: coherence between the four above (transformers, peft, trl, bitsandbytes) matters more than the exact number of each. Pin them when starting a campaign and do not move them until it closes.

Check that the GPU and CUDA are healthy before starting; a badly compiled bitsandbytes shows up late:

python -c "import torch, bitsandbytes; print(torch.cuda.get_device_name(0), torch.cuda.is_available())"
nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv

For 100 % sovereignty: download the base once from your internal Hugging Face mirror (or a MinIO with the weights) and export HF_HOME to a local volume. Nothing in this flow needs to leave the perimeter.

Phase 1 — Preparing the dataset

The canonical format for a conversational task is JSONL, one conversation per line, with the model’s chat template. Do not invent your own format: use the base tokenizer’s chat_template, because any mismatch between how you train and how you serve degrades quality silently.

{"messages":[{"role":"system","content":"You are a network support assistant."},{"role":"user","content":"The north wing AP does not come up after the outage."},{"role":"assistant","content":"First confirm the PoE on the port..."}]}
{"messages":[{"role":"user","content":"Generate the VLAN change for customer 42."},{"role":"assistant","content":"interface GigabitEthernet0/3\n switchport access vlan 42..."}]}

What each field is and why:

Field	What it is	Operational note
`messages`	the complete conversation, a list of turns	one conversation per JSONL line; it is what `apply_chat_template` turns into tokens.
`role`	who is speaking: `system`, `user`, `assistant`	the adapter learns to produce the `assistant` turns; the `user`/`system` ones are context, not target.
`content`	the text of the turn	the `system` one sets the persona/task; keep it identical to the one you will use in production or the adapter drifts out of alignment.

Operational rules that save grief: watch the ratio of examples (a well-curated narrow-task dataset of 2,000-20,000 examples performs better than 200,000 noisy ones), deduplicate, and set aside 5-10 % as a validation split that does NOT go into training. Building the corpus from production signal is covered by Retrain: closing the loop.

Phase 2 — The training script

A minimal, complete script with TRL + PEFT. It trains an r=8 adapter on an 8B SLM quantised to NF4. Each block has its rationale commented.

# train_qlora.py
import torch
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

BASE = "Qwen/Qwen3-8B" # or whichever SLM you serve; ALWAYS use the same one in train and serve
OUT = "adapters/soporte-redes-v1"

# 1) Base frozen and quantised to 4-bit NF4 with double quantisation
bnb = BitsAndBytesConfig(
 load_in_4bit=True,
 bnb_4bit_quant_type="nf4", # NormalFloat, quantile-optimal for Gaussian weights
 bnb_4bit_use_double_quant=True, # quantises the scale constants -> ~0.37 bits/param less
 bnb_4bit_compute_dtype=torch.bfloat16 # the matmuls run in BF16 after dequant on the fly
)

tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(
 BASE, quantization_config=bnb, torch_dtype=torch.bfloat16, device_map={"": 0}
)

# 2) The adapter: low rank, attention projections only (aggressive). Raise target_modules if the eval asks for it.
peft_cfg = LoraConfig(
 r=8, lora_alpha=16, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
 target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)

ds = load_dataset("json", data_files={"train": "data/train.jsonl",
 "eval": "data/eval.jsonl"})

# 3) Training config designed to fit in 24 GB
cfg = SFTConfig(
 output_dir=OUT,
 per_device_train_batch_size=1, # small real batch
 gradient_accumulation_steps=16, # EFFECTIVE batch = 1*16 = 16, without paying its VRAM all at once
 gradient_checkpointing=True, # recomputes activations in backward: trades compute for memory
 optim="paged_adamw_8bit", # paged optimizer: the airbag against VRAM peaks
 learning_rate=2e-4, lr_scheduler_type="cosine", warmup_ratio=0.03,
 num_train_epochs=3, bf16=True,
 max_length=2048, # bounds the sequence: activations scale with it
 logging_steps=10, eval_strategy="steps", eval_steps=100, save_steps=200,
 report_to="none",
)

trainer = SFTTrainer(model=model, args=cfg, peft_config=peft_cfg,
 train_dataset=ds["train"], eval_dataset=ds["eval"],
 processing_class=tok)
trainer.train()
trainer.save_model(OUT) # saves ONLY the adapter (MB), not the base

`BitsAndBytesConfig` — how the base is quantised

Option	What it does	Why this value / when to change it
`load_in_4bit=True`	loads the base weights in 4-bit	it is the foundation of QLoRA: without it the 8B does not even fit to train.
`bnb_4bit_quant_type="nf4"`	uses the NF4 format (quantile-optimal for Gaussian weights)	`"fp4"` exists, but NF4 performs better on transformer weights; leave NF4.
`bnb_4bit_use_double_quant=True`	quantises the scale constants themselves	saves ~0.37 bits/param (hundreds of MB on an 8B); the margin separating “it fits” from “OOM”. Leave it at `True`.
`bnb_4bit_compute_dtype=torch.bfloat16`	matmul precision after undoing the quantisation on the fly	BF16 on Ada/Hopper (4090, H100); use `float16` only on GPUs without BF16.

`LoraConfig` — the shape of the adapter

Option	What it does	Why this value / when to change it
`r=8`	adapter rank: its correction capacity	4-8 for a narrow task (aggressive); raise it to 16-64 only if the eval shows underfitting.
`lora_alpha=16`	scale factor of the delta (effective `α/r`)	common convention `α=2r`; it modulates how much the adapter “weighs” on the base.
`lora_dropout=0.05`	regularisation on the adapter	0.05-0.1 with small datasets (avoids overfit); 0 if the corpus is large.
`bias="none"`	does not train the bias terms	`"none"` is the standard; `"all"`/`"lora_only"` rarely help and cost params.
`task_type="CAUSAL_LM"`	objective/head type	fixed for a generative LLM.
`target_modules=[q,k,v,o]`	which matrices get an adapter	attention only = cheap and aggressive; add `gate_proj`/`up_proj`/`down_proj` (MLP) if the task demands rewriting more behaviour and the eval asks for it.

`SFTConfig` — the memory budget and the loop

Option	What it does	Why this value / when to change it
`per_device_train_batch_size=1`	microbatch per GPU	1 on 24 GB; the real batch is built by `gradient_accumulation_steps`.
`gradient_accumulation_steps=16`	accumulates 16 microbatches before updating	effective batch = 1×16 = 16 without paying its VRAM all at once; raise it if you shorten the sequence and want a bigger effective batch.
`gradient_checkpointing=True`	recomputes activations in the backward pass instead of storing them	essential on a 4090: ~20-30 % slower in exchange for much less VRAM.
`optim="paged_adamw_8bit"`	Adam optimizer in 8-bit + states pageable to RAM	less state VRAM and the airbag that avoids the OOM at the peaks.
`learning_rate=2e-4`	learning rate of the adapter	1e-4-3e-4 is the typical QLoRA range; adapters tolerate a higher LR than a full fine-tune.
`lr_scheduler_type="cosine"`	LR decay curve	`cosine` or `linear`; cosine usually gives a smooth drop at the end.
`warmup_ratio=0.03`	warms the LR over the first 3 % of steps	avoids the instability of the first steps.
`num_train_epochs=3`	complete passes over the dataset	1-3; watch the eval loss so as not to overfit.
`bf16=True`	compute and adapter precision	BF16 on Ada/Hopper; `fp16=True` if your GPU has no BF16.
`max_length=2048`	maximum sequence length	the #1 VRAM lever for activations: shorten it first if there is an OOM.
`eval_strategy`/`eval_steps`/`save_steps`	validation and checkpoint cadence	tune them to the dataset size; evaluating often costs time.

The four pieces that make it fit on a 4090 are: per_device_train_batch_size=1 + gradient_accumulation_steps (a large effective batch without its memory cost all at once), gradient_checkpointing=True (recomputing activations instead of storing them) and optim="paged_adamw_8bit" (paging states to RAM at the peaks). Remove any of the three with long sequences and you will see the OOM.

A declarative alternative with Axolotl if you prefer YAML over Python (same result):

base_model: Qwen/Qwen3-8B
load_in_4bit: true
adapter: qlora
lora_r: 8
lora_alpha: 16
lora_target_modules: [q_proj, k_proj, v_proj, o_proj]
sequence_len: 2048
micro_batch_size: 1
gradient_accumulation_steps: 16
gradient_checkpointing: true
optimizer: paged_adamw_8bit
learning_rate: 0.0002
num_epochs: 3
bf16: true
datasets:
 - path: data/train.jsonl
 type: chat_template

Phase 3 — Launch and monitor

# Simple launch on one GPU
python train_qlora.py

# In another terminal: watch VRAM. If it gets close to the ceiling, lower max_length or raise grad accumulation.
watch -n 2 nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv

Approximate VRAM budget when training the 8B on the 4090, and what to touch when it gets tight:

Component	Approx. VRAM	Lever if there is an OOM
8B NF4 base (frozen)	~4.0 GB	— (fixed)
Adapter + grad + Adam states	~0.3-0.7 GB	lower `r`
Activations (batch × sequence)	~6-14 GB	lower `max_length`, `batch_size`; raise `grad_accum`
Dequant buffers / workspace	~1-2 GB	—

Table of quick OOM remedies, in order of cost: (1) lower max_length; (2) confirm gradient_checkpointing=True; (3) raise gradient_accumulation_steps and lower per_device_train_batch_size to 1; (4) use paged_adamw_8bit (already in the script); (5) as a last resort lower r. If after all that it still does not fit, the sequence or the model is too large for 24 GB: either you bound it, or you move up the hardware.

Phase 4 — Validating the adapter

Never promote an adapter on the strength of the training loss. Measure against the validation split you set aside and against a handful of real prompts.

# quick_eval.py
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
 bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", quantization_config=bnb, device_map={"": 0})
model = PeftModel.from_pretrained(base, "adapters/soporte-redes-v1") # base + adapter

msgs = [{"role": "user", "content": "The north wing AP does not come up after the outage."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(0)
print(tok.decode(model.generate(ids, max_new_tokens=256)[0], skip_special_tokens=True))

For a serious verdict, run the adapter through your eval suite (the layer described by LLM evals) and compare against the base without an adapter and against the previous version of the adapter. Promote only if it wins on the task metric without regressing on safety/format.

Phase 5 — Versioning the adapter as an artefact

The adapter is a pair of files of a few MB (adapter_model.safetensors + adapter_config.json). Treat it as a versioned, signed, traceable artefact, not as a loose file.

# Reproducible checksum + upload to internal object storage (MinIO/S3)
sha256sum adapters/soporte-redes-v1/adapter_model.safetensors > adapters/soporte-redes-v1/SHA256
aws --endpoint-url https://minio.interno s3 cp \
 adapters/soporte-redes-v1/ s3://adapters/soporte-redes/v1/ --recursive

A convention that works: s3://adapters/<task-or-customer>/<version>/. Immutable per version, with its SHA256. Deleting a customer is deleting a prefix of a few MB, not retraining anything. Versioning 500 adapters costs what versioning 500 heavy configuration files costs.

Phase 6 — Serving in multi-LoRA with vLLM

The consumer loads one shared base and applies the adapter delta per request. Startup with static adapters declared:

VLLM_ALLOW_RUNTIME_LORA_UPDATING=True \
vllm serve Qwen/Qwen3-8B \
 --enable-lora \
 --max-loras 8 \  # max nº of DIFFERENT adapters per batch (not the total loadable)
 --max-lora-rank 8 \  # = the maximum rank of your adapters; do not inflate it (wastes memory)
 --max-cpu-loras 64 \  # adapters cached in RAM for fast swap to VRAM
 --lora-modules soporte-redes=/srv/adapters/soporte-redes/v1

Each flag, what it controls and how to size it:

Flag / variable	What it controls	How to size it
`--enable-lora`	turns on adapter support	mandatory; without it, vLLM ignores any `model` that is an adapter.
`--max-loras 8`	nº of different adapters in a single batch	more adapters per batch makes the SGMV kernels more expensive; 8-32 is reasonable. It is not the total loadable.
`--max-lora-rank 8`	maximum rank the server reserves	set it equal to the real rank of your adapters (8 here); inflating it wastes VRAM and performance.
`--max-cpu-loras 64`	adapters cached in RAM ready to page to VRAM	≥ nº of active adapters; it is the “bench” from which fast swapping happens.
`--lora-modules name=path`	declares static adapters at startup	useful for the fixed ones; leave it out if everything goes through dynamic loading/Resolver.
`VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`	enables the hot load/unload endpoints	essential for `/v1/load_lora_adapter`; without it, the server is static.

--max-loras limits the different adapters per batch, not how many you can have loaded; the bulk lives on CPU (--max-cpu-loras) and is paged to VRAM on demand. Set --max-lora-rank to the real rank (8 here): inflating it wastes memory and performance. Requests pick the adapter through the model field:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
 "model": "soporte-redes",
 "messages": [{"role":"user","content":"The north wing AP does not come up after the outage."}]
}'
# model:"Qwen/Qwen3-8B" (no adapter) uses the bare base on the same server

Hot loading of a new adapter without restarting (thanks to VLLM_ALLOW_RUNTIME_LORA_UPDATING=True):

curl -X POST http://localhost:8000/v1/load_lora_adapter -H "Content-Type: application/json" -d '{
 "lora_name": "cliente-42",
 "lora_path": "/srv/adapters/cliente-42/v3"
}'
# and to free VRAM/CPU when a customer goes idle:
curl -X POST http://localhost:8000/v1/unload_lora_adapter -H "Content-Type: application/json" -d '{"lora_name":"cliente-42"}'

For multi-tenant at scale, avoid declaring hundreds of adapters by hand: the LoRAResolver resolves and loads the adapter from local storage or S3 the first time an unknown model arrives, so the server stays lean and the adapters are pulled lazily from your MinIO. The internals of how thousands of concurrent adapters are batched (SGMV kernels, unified paging, the heterogeneous gather/scatter) are in Multi-LoRA serving; this runbook only switches them on. To squeeze the base’s decode throughput on a 4090, combine this with Optimising decode in vLLM.

Serving multi-adapter vs merging per task

Two deployment architectures, and the procedure changes:

Serving multi-LoRA (the above). A shared base + N hot adapters. It is the default sovereign pattern: minimum footprint, per-customer isolation, hot-swap. Use standard QLoRA and merge nothing.

Merging per task. If you want a single quantised-and-adapted artefact per task (no adapter at runtime), do not merge a standard QLoRA adapter into the 4-bit base: the merge reintroduces precision that NF4 does not represent and on requantising you lose part of what was learnt. For that case, train with QA-LoRA (quantization-aware), which merges cleanly onto a quantised base. It is an architecture decision, not a quality one; the conceptual detail is in the fundamentals post.

Checklist of operational gotchas

Coherent chat template between training and serving. The most common and most silent mismatch: you train with one chat_template and serve with another. Use the base’s on both sides.
Exactly the same base (revision included) in train and serve. An adapter trained on Qwen3-8B is not valid on another revision of the model.
--max-lora-rank ≥ the rank of ALL the adapters served together, but no more: inflating it wastes VRAM.
KV budget vs --max-loras. The bottleneck in serving is not the adapters (MB), it is the KV cache and concurrency; see Inverted roofline for the SLM regime.
r too low = underfitting if the task demands rewriting a lot of behaviour. Raise r only if the eval asks for it.
Do not promote on training loss. Validate against the reserved split + real prompts + a safety regression.
Version and make immutable every adapter with its SHA256; never overwrite a served version.

Applied to on-premise infrastructure

On an RTX 4090 (24 GB) the same machine is producer and consumer: you train a customer’s adapter in hours and serve it on the same server on top of the shared base. It is the canonical case for multi-tenant demos and platform prototypes.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8) QLoRA stops being necessary in order to fit, but it serves to parallelise production (several adapter jobs at once) and to keep the quantised format consistent between training and serious serving of hundreds of concurrent adapters. The base can run in native FP8; the mechanics of the runbook do not change, only the scale.

References

Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. https://arxiv.org/abs/2305.14314
Hugging Face TRL — PEFT integration (SFTTrainer + QLoRA): https://huggingface.co/docs/trl/peft_integration
Hugging Face PEFT: https://github.com/huggingface/peft
bitsandbytes: https://github.com/bitsandbytes-foundation/bitsandbytes
vLLM — LoRA Adapters (serving, dynamic loading, LoRAResolver): https://docs.vllm.ai/en/stable/features/lora/
Axolotl: https://github.com/axolotl-ai-cloud/axolotl
Xu, Y. et al. QA-LoRA: Quantization-Aware Low-Rank Adaptation. ICLR 2024. https://arxiv.org/abs/2309.14717

The signature-dish specialist: vLLM's attention backend (FlashAttention, FlashInfer and the prefill/decode asymmetry)

Mon, 08 Jun 2026 05:40:00 +0200

Part of the under the engine series. The PagedAttention post explained where the KV lives (in paged blocks). This one explains who reads it and how: the attention kernel. And it connects with FlashAttention v1-v4, which took that kernel apart from the inside; here we look at the level above, how vLLM chooses between several kernels and why it needs more than one.

TL;DR

An LLM forward is, for the most part, standard matrix multiplications that any library does well. The exception that decides performance is attention, and having one good kernel is not enough: you need two, because the two phases of inference are physically opposite problems. Prefill processes the whole prompt: many queries against many keys, dense and compute-bound, the home ground of FlashAttention’s IO-aware tiling. Decode generates one token: a single query against all the accumulated KV, thin and memory-bound, where the only thing that matters is saturating HBM bandwidth while reading the paged KV. That is why vLLM does not have “the attention kernel” but a switchable backend (FLASH_ATTN, FLASHINFER, TRITON_ATTN and others) plus logic that picks according to the GPU: by default FA4 on Blackwell (SM100), FA3 on Hopper (SM90), FA2 elsewhere, with FlashInfer as the alternative that compiles bespoke kernels (JIT) and knows how to do cascade attention for shared prefixes. This post explains why prefill and decode are opposites (with arithmetic intensity), how the backend reads paged KV, how the engine chooses, what FlashInfer brings, the 10 knobs and the trap of pinning a backend blindly. On the generic 4×H100 SXM cluster.

Where you are: the specialist, not the commis

In the kitchen, almost all the work is chopping and sautéing: standard operations any competent commis chef executes, the matrix multiplications of the feed-forward layers and the projections. There is one dish that is not delegated: the signature dish, the one that defines the restaurant. That dish is attention, and it has a quirk: it is cooked in two radically different ways depending on the moment of service.

During prefill, when a new order arrives with its whole prompt, you cook big: a lot of raw material at once, a lot of fire, an intense operation that fills the stoves. During decode, when a table asks for “one more dish”, you cook à la carte: a single dish, but you have to go to the larder and bring back every ingredient that table has accumulated over the whole meal. One is a problem of firepower; the other, of larder speed. The same specialist does not do both well. That is why vLLM has several, and a chef who decides which one steps up depending on the GPU and the phase. That is the attention backend.

Why prefill and decode are opposite problems

This is the central idea, and one sum proves it: arithmetic intensity (FLOPs per byte read). An operation with high intensity is limited by compute; one with low intensity, by memory.

Prefill. We serve $N$ queries (the whole prompt) against $N$ keys. The $QK^\top$ operation and the $\text{softmax}\cdot V$ do on the order of $N^2 d$ FLOPs and read on the order of $N d$ of data. Intensity grows with $N$:

$$I_\text{prefill} \sim \frac{N^2 d}{N d} = N$$

With large $N$ (a prompt of thousands of tokens), intensity is high: compute-bound. This is where FlashAttention’s tiling squeezes the tensor cores and where you get close to the GPU’s peak TFLOPS.

Decode. We serve one query (the new token) against $L$ keys (all the accumulated KV). FLOPs on the order of $L d$; bytes read on the order of $L d s$ (the entire KV has to be read from HBM). Intensity is:

$$I_\text{decode} \sim \frac{L d}{L d s} = \frac{1}{s} \quad (\approx 0.5 \text{ FLOP/byte in FP16})$$

Constant and tiny: memory-bound. The decode kernel is not limited by how much the GPU can compute but by how fast it reads the KV from HBM. It makes no difference that the H100 has 132 idle SMs (see the SM post): the bottleneck is the 3.35 TB/s of bandwidth, and the decode kernel exists so as not to waste a single one of those bytes/s.

The design consequence: a kernel optimised for prefill (dense tiling, maximum tensor-core occupancy) is not the optimum for decode (coalesced reads of the paged KV, minimum latency). Serious servers have different kernels, or one kernel with two paths. In models with MLA (multi-head latent attention), vLLM goes as far as using separate backends for prefill and decode, selectable independently (attention backends, vLLM).

The scheduler’s trick: prefill and decode in the same forward

Here the circle closes with the scheduler post. Because vLLM V1 mixes requests in prefill and in decode in every step, one single forward has to serve both. The backend receives metadata telling it, for each sequence in the batch, how many queries it brings and how much KV it has to read, and applies the right path to each one. That is why the attention backend and the scheduler are coupled: the first has to digest the heterogeneous batch the second builds.

How the backend reads the paged KV

The kernel does not receive a contiguous KV tensor: it receives the block table from the block manager and does a gather over the physical blocks. This imposes a real constraint on the backend: it has to support the paged layout and vLLM’s block_size. Not every kernel in the world does; the ones vLLM integrates (FlashAttention, FlashInfer, Triton) are adapted to read KV in fixed-size blocks scattered across HBM. That is why you cannot plug in any attention kernel from a paper: it has to speak the language of the pigeonhole larder.

The backends and how the engine chooses

vLLM exposes a backend abstraction with several implementations (vLLM deepwiki):

FLASH_ATTN — the FlashAttention family. By default the version is picked by architecture: FA4 on SM100 (Blackwell), FA3 on SM90 (Hopper), FA2 elsewhere, configurable with flash_attn_version.
FLASHINFER — an attention engine with JIT compilation and specialisable kernels; strong on heterogeneous KV and shared prefixes.
TRITON_ATTN — written in Triton, portable and with no dependency on precompiled CUDA binaries (Triton backend deep dive, vLLM, mar-2026).
Specific backends for MLA and for non-NVIDIA hardware.

Selection is automatic unless you force it with VLLM_ATTENTION_BACKEND. The heuristic tries FlashAttention first; on Blackwell (SM100) the fallback order for MLA is TRT-LLM Ragged → FlashInfer → others; on other GPUs only FlashAttention is considered for the main path (attention backends, vLLM). The decision depends on: architecture (SM), dtype (FP16/BF16/FP8), head dimension, and whether the load needs a feature only one backend has (cascade attention, certain soft caps, FP8 in KV).

What FlashInfer brings: JIT and cascade attention

FlashInfer does not compete with FlashAttention on “being a bit faster”; it attacks a different problem: the heterogeneity of the KV in real serving (FlashInfer, arXiv 2501.01005). Two ideas:

JIT compilation. Instead of a monolithic kernel, FlashInfer generates bespoke kernels for the attention variant, the problem shape and the KV layout you have, injecting functors (query/key/logit transformations, masks). It specialises rather than generalises.

Cascade attention. Here is the jewel for serving with shared prefixes. If $R$ requests share a prefix of $P$ tokens (a common system prompt), naive attention would read that prefix $R$ times. Cascade attention computes it once against the shared prefix and then combines with each request’s own suffix:

$$\text{reads: } \underbrace{R \cdot (P + s_i)}_{\text{naive}} \;\longrightarrow\; \underbrace{P + \textstyle\sum_i s_i}_{\text{cascade}}$$

With $R=50$ requests and a prefix $P=1000$, that is reading 50,000 prefix tokens versus 1,000. It is the natural complement to prefix caching: the block manager shares the prefix’s memory, and cascade attention shares the compute of attending over it.

The arithmetic that matters: when switching backend gives you something

The backend only moves the needle where attention is the bottleneck. In memory-bound decode, a kernel that exploits HBM bandwidth better gives a real improvement; in compute-bound prefill with long sequences, FA3/FA4 getting close to the tensor-core peak gives a real improvement. But if your bottleneck is in another layer, launch overhead, a badly sized scheduler, the cold start, then switching backend does not touch that part. The rule, once again: measure the regime before optimising.

The 10 knobs

#	Knob	What it controls	Cost / risk
1	`VLLM_ATTENTION_BACKEND`	force a backend	mismatch with hardware/feature
2	`flash_attn_version` (2/3/4)	FA version	version unsupported on your SM
3	enable FlashInfer	JIT + cascade	initial JIT compilation time
4	cascade attention	prefix compute reuse	only helps with a heavily shared prefix
5	`kv_cache_dtype` (FP8)	FP8 support in the kernel	not every backend/SM supports it
6	`block_size`	layout the kernel must read	consistency with PagedAttention
7	MLA prefill backend	kernel for the dense phase	MLA models only
8	MLA decode backend	kernel for the thin phase	MLA models only
9	soft cap / sliding window	features that restrict backends	fewer kernel options
10	head_dim / variant	which kernels are eligible	exotic models with no support

How it connects to the rest of the stack

With FlashAttention. The FA post explains the kernel from the inside (tiling, online softmax, FA1-4); this is the level above, how vLLM chooses between kernels and why it needs more than one.

With PagedAttention. The backend reads the KV that the block manager places in blocks; it has to speak block-table.

With the scheduler. The scheduler builds mixed prefill+decode batches; the backend has to serve both regimes in a single forward.

With CUDA graphs. Attention kernels are captured in the CUDA graphs; a backend that launches many small kernels benefits more from capture.

With prefix caching. Cascade attention is the compute side of what prefix caching does in memory.

With FP8. Attending over KV in FP8 requires the backend to have the FP8 path; not all of them do on every architecture.

Traps and things that are not what they look like

“FlashInfer is always faster than FlashAttention.” No. FlashInfer wins when its specialisation (cascade, heterogeneous KV, a particular attention variant) applies to your load; in classic dense prefill, FA3/FA4 usually does as well or better. It depends on the regime, there is no universal winner.

“A good attention kernel works for everything.” The underlying error of this post. Prefill and decode are compute-bound and memory-bound respectively; a kernel tuned for one wastes effort on the other. That is why there are separate paths (and separate backends in MLA).

“Decode is compute-bound because the GPU is at 100%.” nvidia-smi at 100% is misleading (see the SM post): decode is memory-bound, the GPU is moving KV, not computing. Optimising decode’s compute is polishing what is not the bottleneck.

“I pin VLLM_ATTENTION_BACKEND and forget about it.” Pinning a backend by hand can leave you on a suboptimal one when you change GPU or version, or force a slow fallback if your hardware does not support what you asked for. Autoselection usually gets it right; pin it only with a measurement that justifies it.

“Cascade attention always helps.” Only with a heavily shared prefix across many concurrent requests. If each request has its own context, there is nothing to share and the overhead of organising the cascade is not amortised.

“The attention backend is the bottleneck, that is why I am slow.” Almost always the bottleneck is higher up (launch, scheduling, memory) or lower down (bandwidth). The backend matters where attention dominates; measure it with nsys/DCGM before changing it.

Conclusion

Of everything an LLM does while generating text, almost all of it is matrix multiplications that any library resolves. Performance is decided in a single kernel, attention, and the surprise is that it is not even one kernel: it is two opposite problems wearing the same name. Prefill wants fire, dense compute over thousands of tokens, and decode wants a fast larder, reading a token’s entire KV with the minimum waste of bandwidth. That is why vLLM did not pick a winning kernel but an abstraction that switches: FlashAttention tuned to each architecture for the general case, FlashInfer compiling bespoke kernels when there is heterogeneity or prefixes to share, Triton for portability. The head chef does not cook the signature dish just one way: he looks at who is ordering and at what point of service, and sends out the right specialist. The lesson for whoever is tuning is the usual one in this series: before changing specialist, make sure the signature dish really is what is holding you back.

References

vLLM, Attention Backends (selección, FA2/3/4 por arquitectura, MLA): https://docs.vllm.ai/en/latest/design/attention_backends/.
vLLM / DeepWiki, FlashAttention and FlashInfer: https://deepwiki.com/vllm-project/vllm/8.2-flashattention-and-flashinfer.
vLLM, Triton Attention Backend Deep Dive (mar-2026): https://vllm.ai/blog/2026-03-04-vllm-triton-backend-deep-dive.
Z. Ye et al., FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving (arXiv 2501.01005): https://arxiv.org/abs/2501.01005.
T. Dao, FlashAttention-2 / FlashAttention-3 (kernel IO-aware, async Hopper): https://github.com/Dao-AILab/flash-attention.

The pantry of pigeonholes: PagedAttention and the vLLM block manager

Mon, 08 Jun 2026 05:20:00 +0200

Part of the under the engine series. The scheduler post ended with a loose end: the scheduler has a second budget, the KV blocks, and when they run out it preempts. This post opens up that budget. It is the piece the KV cache post took for granted, what gets stored, in order to explain how it is managed in memory. And it is the one the FlashAttention post had been promising for months.

TL;DR

The KV cache grows a little with every token generated, and the problem was never its total size but the way it is reserved. The first servers asked, per request, for a contiguous chunk of HBM the size of the maximum possible context. Since almost no request reaches that maximum, the result was catastrophic: 60-80% of the HBM wasted on fragmentation. PagedAttention applies to the KV the oldest and most battle-tested idea in operating systems, paging: split the KV into fixed-size blocks (16 tokens by default), store them in non-contiguous HBM wherever there is room, and keep a block table that translates each sequence’s logical block into its physical one. Waste falls to ~4% (only the last block, half-filled). And since each block can be identified by the hash of its content, two requests that share a prefix point at the same physical block and share memory, with copy-on-write when one of them diverges: that is the engine of prefix caching. This post explains fragmentation with numbers, the block manager, the block table, COW, the block-size trade-off, the 10 knobs and the trap of confusing “fragmentation solved” with “zero waste”. On the generic 4×H100 SXM cluster.

Where you are: the pantry, underneath the scheduler

Go back to the kitchen of the previous post. The front-of-house manager assembles trays, but behind him there is a pantry where the ingredients each table accumulates over its meal are stored, its KV cache. The question in this post is how that pantry is organised.

The naive way: each table gets a whole contiguous shelf, sized for the greediest imaginable customer. The problem is obvious: a table that orders little leaves almost all of its shelf empty, but that shelf is already reserved and nobody else can use it. With many tables, the pantry fills up with half-empty shelves and no new table fits, even though the gaps added together leave plenty of room.

The PagedAttention way: the pantry is divided into small, identical pigeonholes. Each table is given the pigeonholes it needs, one at a time, wherever there is room, and they do not have to be next to each other. A map book records which physical pigeonholes each table has and in what order. When a table leaves, its pigeonholes go back to the pile. There are no half-empty shelves: the only waste is the last pigeonhole of each table, the half-filled one. That is, almost literally, the virtual memory of an operating system applied to the KV cache.

Why contiguous memory fragmented

Reserving contiguously and up front produces three distinct kinds of waste:

Reservation fragmentation. You set aside space for max_model_len (8192 tokens, say) even though the request will use 800. Reserved and never used.
Internal fragmentation. Within what you reserved, everything above what you actually use at any given moment.
External fragmentation. Gaps between contiguous reservations that are too small for a new request, even though added together they would be plenty.

The original vLLM paper measured that earlier systems wasted 60% to 80% of the KV memory through these three routes (Kwon et al., SOSP 2023). That is: on a GPU with room for 100 real requests, only 20-40 fitted. Paging attacks all three at once. It eliminates reservation waste (on-demand allocation) and external waste (blocks need not be contiguous), and leaves only a remainder of the internal kind: the last partial block.

The mechanism: blocks, block table and the kernel gather

The KV of a sequence is chopped into logical blocks of $b$ tokens (by default $b = 16$). Each logical block is mapped, via the block table, to a physical block somewhere in the HBM. The block table is the “map book”: a per-request list of which physical block corresponds to each logical one (vLLM implementation).

The key point is that the attention kernel knows how to read like that. Instead of assuming a contiguous KV tensor, the PagedAttention kernel receives the block table and does a gather: for each sequence it walks its physical blocks in logical order and reads K and V as if they were together. That is why PagedAttention is not just a data structure: it is a kernel that knows how to attend over paged memory. And that is why the attention backend and the block manager are tied together, the second deciding where the KV lives and the first knowing how to read it from there.

The block manager: the pantry librarian

The block manager (in V1, the KVCacheManager) is the one that keeps the map book. Its responsibilities:

Maintain a pool of free physical blocks (a queue of available blocks).
Allocate blocks to a sequence as it grows (a new block every $b$ tokens).
Free the blocks when the sequence finishes or is preempted.
Maintain the block tables (logical→physical) of each request.
Manage prefix caching: detect blocks with identical content and share them.
When free blocks run out, tell the scheduler to preempt (see the scheduler post).

When the block manager says “no blocks left”, the scheduler has to take somebody off the train. That is why the two budgets, tokens and blocks, are the two hands of the same engine.

Here is the elegant part. If two requests start with the same prefix, the same system prompt, the same context document, the first KV blocks of both are identical byte for byte. Why compute and store them twice?

vLLM gives each block a hash summarising its content (the tokens that formed it, plus the hash of the previous block, so that the hash captures position). It keeps a global table of blocks by hash. When a new request produces a block whose hash already exists, it allocates no new memory: it points its block table at the physical block that was already there (automatic prefix caching, vLLM).

Copy-on-write is the safeguard: while A and B share a block, neither can modify it. The moment one of the two needs to write something different into that block (because their sequences diverge, or in parallel sampling / beam search where several branches share a prefix), the block manager copies the block for that branch and only then writes (details, vLLM). It is the same COW that fork() uses in an OS: share until somebody writes.

The saving is direct: if 50 requests share a system prompt of 1,000 tokens, instead of 50 copies of that prefix’s KV there is one. How to maximise that saving in practice is the subject of the prefix cache hit rate post.

The maths that matter: how much KV, how many blocks

KV bytes per token. For a transformer block with $L$ layers, $h_{kv}$ KV heads (GQA), per-head dimension $d$ and $s$ bytes per element (2 in FP16):

$$\text{KV/token} = 2 \cdot L \cdot h_{kv} \cdot d \cdot s$$

For a Llama-70B ($L=80$, $h_{kv}=8$, $d=128$, FP16):

$$\text{KV/token} = 2 \cdot 80 \cdot 8 \cdot 128 \cdot 2 = 327680 \text{ bytes} \approx 320 \text{ KB}$$

A block of 16 tokens takes up $16 \times 320\,\text{KB} = 5.12$ MB.

How many requests fit. If after loading the weights ~120 GB of the node’s 320 are left for KV:

$$\text{KV tokens} = \frac{120 \cdot 10^9}{327680} \approx 366000 \text{ tokens} \approx 22900 \text{ blocks}$$

With average contexts of 4,000 tokens (250 blocks each), that is ~90 concurrent requests. That number, not max_num_seqs, is the real concurrency ceiling, and it is exactly the “block budget” of the scheduler.

The waste that remains. PagedAttention does not get to zero: each sequence wastes, on average, half a block (the last one, half-filled). With 16-token blocks and sequences of 4,000, that is $8 / 4000 = 0.2\%$ per sequence, and the famous aggregate “~4%” from the paper includes other overheads. The lesson: the waste does not disappear, it is bounded to the size of one block.

The block-size trade-off

The block_size (16 by default) is a trade-off, not a magic constant:

Block	Advantage	Drawback
Small (8)	less internal waste; finer-grained prefix sharing	more block table entries; more management and gather overhead
Large (32)	fewer metadata; more efficient gather	more waste in the last block; prefix caching shares at a coarser grain (fewer hits)

A large block shares worse: prefix caching can only reuse complete and identical blocks, so with 32-token blocks two prompts that agree on 20 tokens share nothing (they do not fill a common block), whereas with 8-token blocks they share two blocks. The default of 16 is the point vLLM found reasonable for most workloads; it is worth testing if your workload has short, heavily repeated prefixes.

The 10 knobs

#	Knob	What it controls	Cost if you overdo it
1	`block_size`	tokens per block	waste / overhead (see table)
2	`enable_prefix_caching`	sharing blocks by hash	almost none; usually on
3	`gpu_memory_utilization`	how many physical blocks there are	OOM if too high
4	`kv_cache_dtype` (FP8)	bytes per KV element	quality (measure, do not assume)
5	`swap_space`	blocks that fit on the host (SWAP)	PCIe traffic on preemption
6	`max_model_len`	maximum length per request	fewer requests if too high
7	eviction policy	who gets blocks taken away	prefix cache hit rate
8	sliding window	discarding old KV	quality on long contexts
9	TP / KV sharding	splitting the KV across GPUs	NVLink traffic
10	num_gpu_blocks (override)	forcing the block count	OOM or underuse

How it connects to the rest of the stack

With the scheduler. The “block budget” of the scheduler is administered by this block manager. When it says there are no blocks, the scheduler preempts (RECOMPUTE by default).

With the KV cache. The KV cache post explains what each token stores; this one, how it is placed in memory without fragmenting.

With prefix caching. COW and block hashes are the mechanism; hit rate engineering is how to squeeze it (prompt structure, prefix-aware routing).

With KV quantisation. Moving the KV to FP8 halves the bytes/token: the same node holds twice the tokens. It is the most direct lever on concurrency.

With the attention backend. The FlashAttention/FlashInfer kernel has to know how to attend over paged blocks; the block manager decides where they live, the kernel knows how to read them.

With disaggregated serving. Moving a request from a prefill pool to a decode pool in disaggregated serving is, at bottom, transferring its KV blocks between engines, over NVLink or over the network.

With multi-LoRA. In multi-LoRA serving, the base shares prefix KV between requests from different adapters as long as the prefix is identical.

Traps and things that are not what they seem

“PagedAttention eliminates waste.” It bounds it, it does not eliminate it. What remains is the last partial block per sequence (~half a block) plus the block table metadata. It is ~4% instead of 60-80%, but it is not zero. Sizing as if it were zero leaves you without a cushion.

“Bigger blocks always perform better.” The gather is somewhat more efficient, yes, but you lose sharing granularity: prefix caching shares worse and the waste in the last block grows. On workloads with many short repeated prefixes, small blocks can win.

“Prefix caching shares KV between users, that is a privacy problem.” It shares only blocks that are identical token by token (same system prompt, same document). It does not expose one user’s content to another: if the tokens do not match, there is no common block. What is worth watching is information leaking through timing side channels (a hit is faster than a miss), relevant only in very adversarial multi-tenant scenarios.

“FP8 in the KV is free: twice the concurrency.” It does double the tokens that fit, yes, but KV in FP8 degrades quality measurably on long contexts. It is a real lever, not a free lunch: quality has to be measured (FP8 end-to-end), not assumed.

“Going back to contiguous memory would be simpler and almost as good.” That is nostalgia for the contiguous tensor. The “simple” option reintroduces 60-80% fragmentation: on a GPU, that is the difference between 30 and 90 concurrent requests. The complexity of the block table pays for itself many times over.

SWAP versus RECOMPUTE when preempting. Configuring a lot of swap_space “so as not to lose KV” puts gigabyte transfers over PCIe on the critical path. In V1, RECOMPUTE is usually better; swap is for specific cases.

Conclusion

The bottleneck in serving an LLM was never only how much memory you have, but how you hand it out. The first servers treated the KV cache as a contiguous shelf per client and threw two thirds of the HBM in the bin without it showing up on any dashboard. PagedAttention stole from the operating system its best idea of fifty years ago, paging, and applied it at the exact place where it hurt: small pigeonholes, a map book, on-demand allocation and, as a bonus, the possibility that two requests starting the same way share the same pigeonholes until they stop resembling each other. The result is not magic: the waste is still there, but bounded to the size of one block instead of to the size of the worst imaginable case. And that difference, from 70% to 4%, is what turned a GPU serving thirty clients into one serving ninety, without touching the hardware. The pantry did not get bigger; it got better organised.

References

W. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023): https://arxiv.org/pdf/2309.06180.
vLLM, Automatic Prefix Caching (design, block hashing): https://docs.vllm.ai/en/v0.8.1/design/automatic_prefix_caching.html.
vLLM, Automatic Prefix Caching — Implementation (block table, COW): https://docs.vllm.ai/en/v0.6.1/automatic_prefix_caching/details.html.
H. Elshafie, Paged Attention from First Principles: A View Inside vLLM: https://hamzaelshafie.bearblog.dev/paged-attention-from-first-principles-a-view-inside-vllm/.
vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention (alternative, critical context): https://arxiv.org/pdf/2405.04437.

The pass: the maître d' who builds every round — vLLM's scheduler step

Mon, 08 Jun 2026 05:00:00 +0200

Part of the under the engine series. Earlier posts looked at the silicon that runs the kernels (SMs and CUDA graphs) and at loading the weights (from disk to HBM). This one goes up a floor: who decides what runs in each forward. Before the GPU launches a single kernel, someone has had to build this round’s order. That someone is the scheduler, and it is the heart of continuous batching.

TL;DR

An LLM server does not serve one whole request and then the next: it advances every live request at once, a little on each engine iteration. The piece that decides how far each one advances in each step is the scheduler, and its output is deceptively simple: a dictionary {req_id: number of tokens} that the model runner turns into a single forward on the GPU. vLLM V1’s most important decision was to erase the distinction between prefill and decode: for the scheduler, a prompt token and a freshly generated token are the same thing, tokens that need processing, and that is why it can put a 4000-token prompt in the same batch as 200 one-token decodes. There are four pieces: the token budget (max_num_batched_tokens), a fixed-size tray filled every round; chunked prefill, which splits a huge prompt into chunks so it does not hog the tray and blow up everyone else’s latency; the two queues (waiting, in arrival order, and running); and preemption, because when the KV cache runs out somebody has to get off the train. This post explains the loop, the arithmetic of the budget and of concurrency, the 10 knobs and the headline trap: raising the budget improves throughput but worsens ITL (inter-token latency), and almost nobody measures both at once. On the generic 4×H100 SXM cluster.

Where you are: the pass, before the kitchen starts

Picture the pass of a restaurant with a busy dining room. There is no one cook per customer; there is a shared kitchen and a maître d’ who, every few seconds, looks at all the open orders and builds a tray to send to the stoves. That tray holds, say, 8000 “units of work”. The maître d’ decides what goes in: tables already eating (requests in decode) get one more dish each; new tables that have just ordered (requests in prefill, with their whole prompt still to process) get as much of their order as fits in whatever is left of the tray. The tray goes out, the kitchen executes it in one go, and it starts again. Hundreds of times per second.

That maître d’ is the scheduler. The kitchen is the GPU running a forward pass. And the house rule is that the kitchen never stops to wait for a single table: if a new order is enormous (a 30,000-token prompt), it is not sent whole in one go blocking everyone else, it is sent in chunks. That is, in one sentence, the whole mechanics of vLLM’s scheduler.

The engine loop: one dictionary per iteration

The inference engine is a very short loop. On each pass:

The scheduler looks at the queues and produces a decision.
The model runner runs a forward with that batch on the GPU.
The sampler draws one new token for each active sequence.
State is updated (KV cache, positions, finished requests) and it goes back to 1.

The surprising part is the shape of the decision in step 1. In vLLM V1 it is not a complex structure with phases: it is literally a dictionary

$$\text{schedule} = \{\, \text{req id} \rightarrow n_\text{tokens} \,\}$$

that says, for each request entering this round, how many tokens of it get processed. For a request that is generating text, n_tokens = 1 (one autoregressive step). For a new request, n_tokens can be the entire length of its prompt. And it can be anything in between, a chunk of prompt, under chunked prefill, prefix caching or speculative decoding (vLLM V1 docs).

That the decision fits in a {id: number} dictionary is not a neat implementation detail: it is why continuous batching works. Because the scheduler does not think in “phases” but in “how many tokens for each”, it can mix requests at any point of their life in the same forward. The GPU receives a single heterogeneous token tensor and processes it in one go.

The death of the prefill/decode distinction

This is the idea that costs the most effort and matters the most. In the first LLM server architectures, a request lived in two separate phases: first prefill (process the whole prompt and fill the KV cache), then decode (generate token by token). The scheduler had to choreograph the move from one phase to the other, and mixing them was hard.

vLLM V1 removed the distinction (V1 design). The scheduler treats prompt tokens and generated tokens uniformly: they are all tokens the model has to process in a forward. The practical consequence is enormous. A 4000-token prompt and a sequence that has generated 800 tokens and needs one more are, to the scheduler, “4000 tokens of request A” and “1 token of request B”. They fit together on the same tray. There is no phase choreography, just a budget to hand out.

This unlocks the pattern that really pays: mixing prefill and decode in every step. Prefill is compute-bound work (a lot of matrix to multiply); decode is memory-bound (little compute, a lot of KV to move). Mixing them in the same batch fills the gaps: while the GPU is busy with the heavy prefill, it advances the light decodes “for free”. It is the same efficiency principle as continuous batching, taken to its cleanest form.

The token budget: the fixed-size tray

The tray has a size: max_num_batched_tokens. It is the maximum number of tokens the scheduler can put in a single step (vLLM optimisation). The policy, with chunked prefill on (it always is in V1), is clear:

Decodes first. Room is reserved for one token per request in the running queue. They are cheap, and they are diners already eating: they are not left waiting.
What is left goes to prefills. With the remaining budget, prompt tokens from waiting requests are added, in arrival order (FCFS), split into chunks if needed.

An example with numbers. Budget max_num_batched_tokens = 8192, and at this instant there are 200 requests generating:

$$\text{decode} = 200 \times 1 = 200 \text{ tokens}$$ $$\text{free budget} = 8192 - 200 = 7992 \text{ tokens for prefill}$$

If a new request arrives with a 4000-token prompt, it fits whole in this step (4000 < 7992) and 3992 are left for another. If one arrives with 30,000 tokens, it does not fit: the scheduler sends it a 7992-token chunk this step, and the remaining 22,008 in later steps. That is chunked prefill.

The budget matters because it fixes how many forwards are needed to swallow a prompt. A 30,000-token prompt with an 8192 tray takes 4 steps in prefill alone before it emits its first token. With a 2048 tray it takes 15 steps, but each of those steps leaves more room for other people’s decodes, so the other diners feel the jam less.

The two queues and preemption: when someone gets off the train

The scheduler handles two queues. The waiting queue holds requests that have not started yet (their prompt has not been processed), served in arrival order, FCFS by default, though there is a priority policy. The running queue holds those already alive and generating (vLLM scheduling).

There is a second budget, harder than the token one: the KV cache. Every live token occupies KV blocks in HBM, and they are finite (gpu_memory_utilization fixes them). When the scheduler wants to advance the running requests but there are no free blocks for the next token’s KV, someone has to get off: that is preemption.

vLLM V1 preempts by RECOMPUTE by default, not by SWAP (V1 guide). The difference:

SWAP: copy the victim’s KV to host RAM and bring it back later. It moves gigabytes across PCIe (see the PCIe and P2P post).
RECOMPUTE: throw the victim’s KV away and, when there is room again, redo its prefill from scratch. It sounds expensive, but in the V1 architecture it works out cheaper than swapping, because prefill is work the GPU does very fast and you save the round trip across the bus.

The victim is usually the newest request in the running queue (so as not to penalise whoever has been waiting longest for their answer). The danger is thrashing: admit too many requests at once and the system enters a preempt-recompute-preempt cycle that drives throughput into the floor. That is why the second ceiling exists.

The arithmetic that matters: concurrency and the budget trade-off

How many requests fit at once. The real concurrency limit is not max_num_seqs (the nominal ceiling on simultaneous sequences); it is usually the KV cache. If a node has $B$ free KV blocks, each block holds $b$ tokens (16 by default), and each request occupies $L$ context tokens on average, the maximum sustainable concurrency is:

$$N_\text{max} \approx \frac{B \cdot b}{L}$$

Take a 70B model in FP16 on 4×H100 SXM (320 GB), with the bulk of HBM in weights and, say, ~120 GB free for KV. With a KV of ~0.3 MB/token (an order-of-magnitude figure, it depends on layers and heads), that is ~400,000 KV tokens. With average contexts of 4000 tokens:

$$N_\text{max} \approx \frac{400000}{4000} = 100 \text{ concurrent requests}$$

Raising max_num_seqs to 400 does not give you 400 concurrent: it gives you preemption and thrashing as soon as contexts grow. The KV rules.

The budget trade-off. Raising max_num_batched_tokens puts more work in each forward, so fewer forwards for the same total work, and more throughput. But a large budget lets a huge prefill occupy almost the whole tray in one step, and that step takes longer, so everyone else’s decodes wait for that entire step, and ITL (inter-token latency) rises for everybody. The rule of thumb (vLLM optimisation):

Budget	Effect	At the cost of
Low (e.g. 2048)	more interleaving, stable ITL	less peak throughput
High (e.g. 16384)	maximum throughput	ITL spikes when a large prefill arrives

There is no “correct” value: there is a point for your load. And you only find it by measuring throughput and ITL at the same time, which is exactly what almost nobody does.

The scheduler’s 10 knobs

#	Knob	What it controls	Cost if you overdo it
1	`max_num_batched_tokens`	tray size per step	high ITL if too large
2	`max_num_seqs`	nominal concurrency ceiling	preemption if the KV falls short
3	`gpu_memory_utilization`	KV blocks available	OOM if too high
4	chunked prefill (threshold)	prompt chunk size	chunking overhead if too fine
5	policy (`fcfs`/`priority`)	who gets served first	starvation of low priority
6	preemption mode	RECOMPUTE vs SWAP	PCIe traffic / recompute
7	`enable_prefix_caching`	reuse prefix KV	little; almost always on
8	`max_model_len`	maximum context per request	pessimistic KV reservation
9	CUDA graph sizes	align batch with buckets	padding / capture (see below)
10	speculative tokens	extra tokens per step	wasted work if acceptance drops

How it connects to the rest of the stack

With continuous batching. The scheduler is continuous batching turned into code. The batching post explains the what (advance every request at once); this one explains the how (the per-step token dictionary).

With the KV cache and the block manager. The second budget, the blocks, is managed by the PagedAttention block manager. The scheduler asks for blocks; if there are none, it preempts. The two pieces are coupled through memory.

With CUDA graphs. CUDA graphs are captured for specific batch sizes (buckets). The scheduler should produce batches whose size lands in those buckets to avoid padding; otherwise part of the graph’s benefit is lost.

With chunked prefill and the prefix cache. Chunking a prompt interacts with prefix caching: chunks that match an already cached prefix skip the compute, and the scheduler reflects that by lowering that request’s n_tokens.

With speculative decoding. Speculative decoding makes a step verify several tokens at once; the scheduler models it as n_tokens > 1 for a request in decode.

With disaggregated serving. In disaggregated prefill/decode there are two schedulers, one per pool, each with its own budget; the phase distinction that V1 erased inside an engine reappears between engines.

With autoscaling. The metrics that drive autoscaling with KEDA, waiting-queue length and preempted requests, come straight out of the scheduler’s state.

Traps and things that are not what they look like

“Raising max_num_batched_tokens always helps.” It improves throughput and worsens ITL. If you only look at tokens/s in a large-batch benchmark, you “confirm” that more is better; in interactive production, your users feel the stutter. Measure both metrics or you are not measuring.

“The engine does all the prefills first and then the decodes.” That is the old architecture’s intuition. In V1 there are no phases: each step mixes prefill and decode according to the budget. Reasoning with the phase model leads to wrong conclusions about why latency rises.

“More max_num_seqs = more throughput.” Only until the KV cache runs out. Past that, more nominal concurrency produces preemption, and cascading preemption (thrashing) lowers throughput. The real ceiling is the KV, not the parameter.

“RECOMPUTE is a waste, SWAP is better.” In V1, RECOMPUTE usually wins: prefill is extremely fast on GPU and SWAP puts a gigabyte-scale round trip across PCIe on the critical path. Switching to SWAP “to avoid recomputing” can make latency worse.

“The scheduler is the bottleneck.” Almost never. The decision is a dictionary built in microseconds; the cost of the round is the forward on the GPU, three orders of magnitude above. If your scheduling CPU shows up in the profiler, the problem is usually host-thread jitter (see NUMA and CPU isolation), not the scheduler logic.

Chunked prefill that is too fine. Tiny chunks make a large prompt take many steps and add fixed per-step overhead. Chunking is there to bound the ITL impact, not to pulverise the prompt.

Conclusion

All the power of a modern LLM server, swallowing hundreds of requests at once without any of them blocking the rest, rests on a decision that fits in a {request: how many tokens} dictionary, taken hundreds of times per second. The idea that made it possible was not a faster kernel or a bigger GPU: it was to stop thinking in phases. When a prompt token and a generated token are the same thing, the scheduler can fill every tray by mixing the heavy and the light, and the kitchen never stops. The rest is two budgets, tokens and KV blocks, and a rule for when the second runs out. The maître d’ does not cook; he only decides what goes to the stoves in each round. But that decision, repeated without pause, is what separates an idle GPU waiting for orders from a kitchen running flat out. And the uncomfortable lesson for whoever is tuning: throughput and latency meet in the budget, and optimising one blindly means worsening the other without noticing.

References

vLLM, vLLM V1: A Major Upgrade to vLLM’s Core Architecture: https://openlm.ai/vllm-v1/.
vLLM, vLLM V1 User Guide (chunked prefill por defecto, preemption RECOMPUTE): https://docs.vllm.ai/en/v0.9.2/usage/v1_guide.html.
vLLM, Optimization and Tuning (max_num_batched_tokens, presupuesto y trade-off): https://docs.vllm.ai/en/stable/configuration/optimization/.
A. Wong, Understanding vLLM Scheduling: Token Budgets, Chunked Prefill, and Policies: https://audreywongkg.medium.com/understanding-vllm-scheduling-token-budgets-chunked-prefill-and-policies-2c879e3980e3.
W. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023): https://arxiv.org/pdf/2309.06180.

The chef who calls out every order: SMs, CUDA streams and CUDA graphs, or why the GPU gets bored generating tokens

Sun, 07 Jun 2026 09:00:00 +0200

This closes the “outside the API” pair. The previous post moved the weights from disk into HBM; here we look at what happens once they are inside, in the silicon that executes them. It is the floor below the kernel launch that the NUMA post mentioned without opening: who launches those kernels, how, and why in decode the GPU spends more time waiting for orders than computing.

TL;DR

An H100 has ~132 streaming multiprocessors (SMs), the “hobs” that execute the compute, and occupancy measures how many warps (groups of 32 threads) it has active to hide latency. But the decode bottleneck is rarely the power of those SMs. Each decode step launches hundreds of tiny kernels (several projections per layer × ~80 layers), and each kernel launch costs 5-10 µs of serial CPU. Since decode kernels are small (small batch, a single token), the GPU finishes them before the CPU calls out the next one: bubbles appear and the GPU sits bored waiting for orders. That regime is called launch-bound, and it is the deep reason, not power, not memory, why --enforce-eager delivers 54 tok/s where with optimisations you reach 89-140. The solution is CUDA graphs: record the entire kernel sequence once and replay it as a single submission, removing the per-launch overhead (~28% of the latency per iteration). vLLM captures ~102 graphs at startup and pads the real batch to the nearest bucket so it can replay a graph with fixed shapes. This post explains SM, occupancy, streams, launch overhead with the maths, CUDA graphs, the 10 knobs, and the trap that this capture is the second half of the cold start from the previous post. With scepticism about what actually moves the needle. On the generic 4×H100 SXM cluster.

Where you are: the silicon, below the kernel launch

The analogy: the chef who calls out every order

Last scene in the restaurant of the series. The kitchen is set up, the pantry is stocked (the previous post). Now the food has to be plated. The hobs are the SMs: 132 stations cooking in parallel. The head chef is the CPU: he calls out the orders, each kernel launch is a shout of “one matrix multiplication, coming up!”. The cooks (the SMs) execute what the chef calls out.

In prefill, processing the whole prompt, each order is a huge dish: a giant matmul over hundreds of tokens at once. The chef calls out an order and the hobs take a good while to produce it. The chef has plenty of time to call out the next one. The hobs are flat out: compute-bound.

In decode, generating one token at a time, each order is minuscule: a matmul over a single token. The hob finishes it in an instant… and sits staring at the chef waiting for the next one. But the chef can only call out one order every 5-10 µs, and there are hundreds of orders per token. The hobs, lightning fast, get bored between shouts. The restaurant is not slow because the cooks are bad: it is slow because the chef does not call out fast enough. That is the launch-bound regime.

The solution is not more hobs or faster cooks. It is to stop calling out order by order. If the chef prints the whole night’s sequence on a single sheet and hands it to the line (“do this, in this order, without waiting for me”), the hobs run without pauses. That is a CUDA graph: record the kernel sequence once and replay it in one go, without the CPU calling out each one. And --enforce-eager is exactly the opposite: forcing the chef to call out order by order, all night long.

The mechanism: SM, warps and occupancy

An H100 SXM has ~132 SMs. Each SM executes threads in groups of 32 called warps, and can have several warps “in flight” at once. Occupancy is the fraction of active warps against the maximum the SM supports. What is the point of having many active warps? To hide latency: while one warp waits for data from HBM (hundreds of cycles), the SM executes another ready warp. With few warps, the SM runs out of anyone to give a turn to and stalls.

But, and this is key, occupancy is a necessary, not sufficient condition, and it only matters if the SM has work to do. In decode, the typical problem is not low occupancy inside a kernel: it is that between kernels the SM has nothing, because the CPU has not launched the next one yet. Raising the occupancy of a kernel that lasts 8 µs does not help if the GPU spends 6 µs waiting for it to be launched.

The mechanism: streams, the order queue

A CUDA stream is a queue of operations that the GPU executes in order. Operations in the same stream are sequential; operations in different streams can overlap. That is what allows, for instance, copying data H2D in one stream while another stream computes, the compute/copy overlap. vLLM uses streams to overlap work, but a stream on its own does not remove the cost of launching each kernel: it only decides ordering and parallelism. The launch cost is still there, order by order, until the graphs come in.

The maths that matters: when the GPU is left waiting

The number that governs everything: a kernel launch costs 5-10 µs of CPU, serially. Take a Llama-70B with ~80 layers. Each layer, without fusion, launches on the order of ~10 kernels (Q/K/V projections, attention, output projection, the two or three MLP matmuls, the normalisations, RoPE and so on). That gives:

$$ N_{\text{kernels}} \approx 80 \text{ layers} \times 10 \approx 800 \text{ launches per token} $$

At 5 µs per launch, serially:

$$ T_{\text{launch}} \approx 800 \times 5\,\mu s = 4{.}0 \text{ ms per token} $$

Those 4 ms are CPU calling out orders only, not counting how long the SMs take to cook. If the GPU could compute instantaneously, the launch ceiling would be ~250 tok/s, and with synchronisation points between kernels, worse. Now compare with the memory ceiling of decode: each token reads the 140 GB of weights once from HBM:

$$ T_{\text{mem}} = \frac{140 \text{ GB}}{3{.}35 \text{ TB/s}} \approx 42 \text{ ms} \;\Rightarrow\; \approx 24 \text{ tok/s (one sequence, no batch)} $$

Here is the subtlety almost nobody has in mind. For a single sequence, decode is memory-bound at ~24 tok/s, and the 4 ms of launch fit inside the 42 ms of reading: the launch hides. But batching changes everything. When serving a batch of B sequences, the weights are read once and serve all B, so the memory cost per token amortises and falls. The GPU stops being memory-bound… and what was underneath emerges: the launch cost, which does not amortise with batch because you have to launch the same kernel sequence regardless. The result: the better you batch, the more launch-bound you become, and the more CUDA graphs pay off. That is why the raw measurement confirms it, --enforce-eager gives 54 tok/s where graphs give 89, and up to 8× in configurations where decode is very small and launch dominates completely.

vLLM’s CUDA graphs, specifically

vLLM does not capture a single graph: it captures ~102 at startup, on the order of 51 piecewise (for the mixed prefill+decode steps) and 51 full (for pure decode). Each one is recorded for a fixed batch size (a bucket: 1, 2, 4, 8… up to a maximum). In service the real batch almost never falls exactly on a bucket, so vLLM pads with zeros up to the next bucket up, replays that graph, and trims the output to the real size. That is the price of graphs: they need static shapes, and padding is what makes them static.

This has two consequences that show up in the knobs and the traps:

Capture costs time and memory. Recording 102 graphs at startup adds seconds to the cold start, the second half of the startup that the previous post left pending, and consumes HBM (each graph retains its buffers). The FULL_AND_PIECEWISE mode (the default) is the fastest in service but the one that asks for the most memory and the longest capture; FULL_DECODE_ONLY saves both in exchange for not accelerating the mixed steps.

Padding wastes some compute. Padding a batch of 33 up to the bucket of 64 computes 31 phantom sequences. It is a small cost against what removing the launch overhead saves, but it exists, and it grows if the buckets are badly chosen.

The 10 knobs worth touching

Knob 1 — Measure whether decode is launch-bound

Before touching anything: is the GPU computing or waiting? With nsys (Nsight Systems) you can see the gaps between kernels on the timeline; if there are gaps in decode, it is launch-bound and graphs will help. If the GPU is at 100% with no gaps, the bottleneck is something else (memory or compute) and graphs will not work miracles. nvidia-smi dmon showing low utilisation in decode but poor TPS is the cheap signal.

Knob 2 — Do not use `--enforce-eager` in production

--enforce-eager disables CUDA graphs. It is a debugging tool (to isolate which kernel fails), not a production one. Leaving it on “because it used to start up” throws away 26-50% of decode throughput. If it is in your production command, take it out and measure.

Knob 3 — Capture buckets (`cudagraph_capture_sizes`)

Which batch sizes to capture. Buckets too far apart make padding expensive; too many of them make capture slow and eat HBM. Tuning them to the real distribution of batch sizes you see in production is the fine adjustment, but only after measuring that distribution.

Knob 4 — CUDA graph mode

FULL_AND_PIECEWISE (default, fastest, more memory/capture), FULL_DECODE_ONLY (saves memory and capture, ideal for pure decode pods in disaggregated serving), PIECEWISE, or NONE (= eager). The right mode depends on whether the pod does pure or mixed decode.

Knob 5 — `torch.compile`

vLLM leans on torch.compile to fuse and optimise kernels before capturing them in graphs. Fewer kernels (fusion) = fewer launches = less dependence on the graph and better decode even in eager. The compilation level is a knob, with its startup time cost.

Knob 6 — Batch size: fill the hobs

Memory-bound decode amortises with batching (as we saw in continuous batching): read the weights once for B sequences. More batch = more SM occupancy and more memory amortisation. The limit is set by the HBM available for the KV cache. It is the knob that moves aggregate throughput most.

Knob 7 — Do not break stream overlap

vLLM overlaps compute and copy with streams. Patching the code to “simplify” can serialise what was overlapped. If you do not know why there are several streams, do not collapse them.

Knob 8 — Persistence mode + locked clocks

nvidia-smi -pm 1 keeps the driver resident (avoiding reinitialisations that add launch latency). Locking clocks to the boost frequency stops the GPU dropping P-state between tiny decode kernels and paying the ramp-up latency. It is the same anti-jitter spirit as the NUMA post, applied to the GPU.

Knob 9 — Fused kernels (FlashAttention, FP8 kernels)

Fewer kernels = fewer orders to call out. FlashAttention fuses attention into one kernel instead of several; fused FP8 kernels reduce the count. Fusion attacks the problem at the root: it does not speed up the launch, it removes launches.

Knob 10 — Accept the capture cost in the cold start

Graph capture adds seconds to startup. In a pod that lives for hours, it amortises easily. In one that scales up and down every minute, that cost is paid over and over; there, FULL_DECODE_ONLY (shorter capture) or accepting slightly less throughput can be worth it. It is the same warm-vs-elastic tension as in the cold start.

Summary table

#	Knob	What it attacks	Risk / cost
1	`nsys` / `dmon`	knowing whether it is launch-bound	none; do it first
2	remove `--enforce-eager`	graphs disabled	it was for debugging; reactivate if a bug returns
3	capture buckets	expensive padding / slow capture	requires measuring the real distribution
4	graph mode	memory and capture	less coverage on mixed steps
5	`torch.compile`	unfused kernels	startup time
6	batch size	occupancy + memory	HBM for KV cache
7	streams	broken overlap	do not touch if not understood
8	persistence + clocks	jitter / P-states	electrical consumption
9	fused kernels	number of launches	kernel compatibility
10	capture vs cold start	slower startup	less throughput if trimmed

How this connects with the rest of the stack

With the cold start. CUDA graph capture is the second half of the startup that the previous post opened: loading weights + capturing graphs = the complete cold start.

With continuous batching. Continuous batching is what makes decode launch-bound (it amortises memory and leaves the launch exposed), and that is why graphs and batching reinforce each other.

With the KV cache. The KV cache decides how much batch fits in HBM, and the batch decides SM occupancy and how much launch overhead matters. Everything is coupled through memory.

With the interconnect. In TP, between the compute kernels there are all-reduces (NVLink/NCCL) that also get launched and synchronised. vLLM’s custom all-reduce is integrated into the same graph so the sequence is not broken by a CPU synchronisation.

With NUMA. Who launches the kernels is the CPU from the host post; if that thread suffers jitter or lands on the wrong socket, launch overhead gets worse. Graphs reduce the dependence on that thread, which is another reason they help.

With disaggregated serving. The pure decode pods of disaggregated serving are the ideal case for FULL_DECODE_ONLY: they maximise the graph benefit exactly in the most launch-bound phase.

Traps and things that are not what they look like

“Raising occupancy will fix slow decode.” Not if the problem is launch-bound. Occupancy matters inside a kernel that has work; if the GPU is idle between kernels waiting for the CPU, more occupancy does not touch that bubble. Measure before optimising what is not the bottleneck.

“CUDA graphs always speed things up.” They speed things up when decode is launch-bound. If the GPU is already at 100% (compute-bound in prefill, or memory saturated with a huge batch), graphs add little. Their territory is decode with small kernels.

"--enforce-eager gives more stable results." It gives slower results. The stability it appears to give is that it avoids graph capture bugs on new hardware (for example a newly supported architecture). It is a temporary patch, not a production configuration.

Capturing too many buckets “just in case”. Each bucket adds capture time and HBM. Capturing 30 sizes when in production you only see 4 means paying cold start and memory for graphs that are never replayed. Tune to the real distribution.

Confusing utilisation with efficiency. nvidia-smi at 100% “utilisation” only says there is a kernel running, not that the SM is full of useful work. A low-occupancy kernel keeps “utilisation” high while wasting the SM. nvidia-smi utilisation is a coarse thermometer; to know whether the silicon is performing you need nsys/DCGM and to look at real occupancy and gaps.

Optimising the silicon before memory. If decode is limited by HBM bandwidth (large batch, large model), fighting with graphs and occupancy is polishing what is not the bottleneck. The right order: measure the regime (memory / compute / launch) and attack the one in charge.

Conclusion

Intuition says that a GPU generating slow tokens is “working hard”. Almost never: in decode it is waiting for orders. The 132 SMs cook a tiny token in an instant and then sit staring at the CPU, which can only call out one order every 5-10 µs and has hundreds to call out per token. That bottleneck, neither power nor memory but launch, is invisible on any dashboard that looks at “GPU utilisation”, and it is the real reason why --enforce-eager performs at half speed. CUDA graphs solve it with a simple idea: stop calling out order by order and hand over the whole sheet for the night, so the silicon runs without pauses. And there is an uncomfortable truth that reorders the optimisation priority: the better you batch, the more launch-bound you become, because batching kills the memory bottleneck and exposes the launch one. That is why graphs and batching are not separate optimisations: they are the same lever seen from two sides. The chef who learns not to call out every dish is what finally makes the kitchen run as fast as the hobs always could.

References

vLLM, CUDA Graphs (diseño, modos FULL/PIECEWISE, captura): https://docs.vllm.ai/en/stable/design/cuda_graphs/.
vLLM, Inside vLLM: Anatomy of a High-Throughput LLM Inference System: https://blog.vllm.ai/2025/09/05/anatomy-of-vllm.html.
NVIDIA, Getting Started with CUDA Graphs: https://developer.nvidia.com/blog/cuda-graphs/.
NVIDIA, Achieved Occupancy (ocupación de SM): https://archive.docs.nvidia.com/gameworks/content/developertools/desktop/analysis/report/cudaexperiments/kernellevel/achievedoccupancy.htm.
PyTorch, torch.compile y CUDA Graphs para inferencia LLM: https://docs.vllm.ai/en/stable/design/cuda_graphs/.
Understanding the Overheads of Launching CUDA Kernels (ICPP 2019): https://www.hpcs.cs.tsukuba.ac.jp/icpp2019/data/posters/Poster17-abst.pdf.

The pantry lift: from disk to HBM, or why the kitchen opens late

Sun, 07 Jun 2026 08:30:00 +0200

This is a trip down to the basement. The under the engine series optimised the hot path, what happens with every token once you are already serving. This post looks at the journey that comes before serving: how the weights climb from disk to HBM. It is the first of a pair on the two things that happen outside the API and that almost nobody times: model loading (this one) and execution on the silicon (the next one).

TL;DR

Before an inference pod generates its first token, it has to push the whole model into HBM. A Llama-70B in FP16 is 140 GB travelling along a path nobody draws: disk → page cache → host buffer → PCIe → HBM. Intuition fails here: HBM is not the bottleneck, it moves 3.35 TB/s and swallows 140 GB in 42 ms; the bottleneck is the supply chain. The Gen5 NVMe disk reads at ~14 GB/s (10 s for 140 GB); PCIe Gen5 copies host→GPU at ~50 GB/s (2.8 s); and the default safetensors loader, which deserialises tensor by tensor and bounces every byte through a CPU buffer, inflates all of that up to 30-60 s. That time is the cold start, and it is the hidden tax paid by autoscaling (scale-from-zero), by canary/blue-green and by disaggregated serving every time a pod is born. There are three families of solution, GPUDirect Storage (direct DMA disk→HBM, no bounce through the CPU), fastsafetensors (4.8-7.5× over the default loader) and the Run:ai Model Streamer (concurrent reads that saturate the disk), plus the simplest lever of all: move fewer bytes (FP8 is half of FP16). This post explains the path, the maths, the 10 knobs, and the cruellest trap: “the second time it loaded fast” is not your loader being good, it is the page cache lying to you. On the generic 4×H100 SXM cluster.

Where you are: the basement, before opening

The analogy: the pantry lift

Let us stay in the restaurant of the series. The shared table was the NVSwitch, the floor next door was NUMA, the maître was the kubelet. All of that describes the restaurant running, with diners seated. But there is a moment no post looked at: before opening, somebody has to bring the entire pantry up from the basement store to the kitchen.

The model weights are the ingredients. They live in the basement store (the disk). The kitchen, the hot line where tokens are plated, is the GPU’s HBM. And between one and the other there is a service lift: the path disk → host → PCIe → HBM. The kitchen cannot serve the first dish until the pantry is upstairs and in place. That restocking time is the cold start.

The intuition trap: the kitchen (HBM) is enormous and blazingly fast, it places ingredients at 3.35 TB/s. So we blame the kitchen when the restaurant opens late. But the kitchen is standing idle waiting for the lift. The bottleneck is never the hot line: it is the lift and the store. And worse still, there is a porter (the default loader) who, instead of carrying whole crates, takes the ingredients out one by one, writes each one down in a notebook and packs them again before sending them up. That porter, not the lift, is half the problem.

The mechanism: what really happens when a model loads

When vLLM starts up with a model in safetensors, the 140 GB of the Llama-70B FP16 make this journey:

Disk → page cache. The kernel reads the .safetensors files from NVMe into the page cache (host RAM). If this is the first time after a reboot, it is a physical read from disk (~14 GB/s on Gen5). If the files are already in page cache from an earlier start-up, this is almost free, and here is where the trap we will see is born.
Deserialise. The default Hugging Face loader mmaps the file and builds the tensors one by one, copying them into a CPU tensor before moving them. This is single-threaded CPU work that saturates neither the disk nor PCIe: most of the “slow” loading time goes here, not on moving bytes.
Host → HBM (H2D). Each tensor is copied from the host buffer into HBM over PCIe Gen5 x16 (~50 GB/s in practice). For the DMA to be efficient, the host buffer should be pinned, which connects directly with the hugepages and pinned memory of the NUMA post.
Place in HBM. HBM receives the 140 GB. At 3.35 TB/s, this takes 42 ms. It is never the bottleneck.

The path has a shortcut: GPUDirect Storage (GDS). Instead of bouncing through the CPU buffer (steps 2-3), a DMA engine close to the NVMe controller writes straight from disk to HBM, without involving the CPU. It is the same principle as the GPUDirect RDMA of the network: get the CPU out of the way. fastsafetensors uses GDS and reaches 26.4 GB/s reading a Llama-70B from NVMe across 4 GPUs.

Why the problem exists: the economics of bytes

The size of the model in bytes is decided by quantisation, and that sets the floor of the cold start, because all those bytes have to move before the first token. For a 70B parameter model:

Format	Bytes/param	Size 70B	Read 1 NVMe @14 GB/s	H2D PCIe @50 GB/s	HBM @3.35 TB/s
FP16 / BF16	2	140 GB	10.0 s	2.8 s	42 ms
FP8	1	70 GB	5.0 s	1.4 s	21 ms
INT4 (GPTQ/AWQ)	0.5	~35 GB	2.5 s	0.7 s	10 ms

Three readings of this table:

HBM never shows up as a problem. The last column is always milliseconds. Anyone saying “the GPU is slow to load” is blaming the wrong place.

Quantising is the most underrated cold start lever. Going from FP16 to FP8 does not only double inference throughput (less HBM bandwidth per token, as we saw in quantization): it also halves the cold start, because there are half as many bytes to push up. It is a two-for-one that sizing usually ignores.

The disk is the bottleneck on bytes; the loader is the bottleneck on time. The figures in the table are the theoretical floor, just moving bytes. The default loader adds single-threaded deserialisation on top, which is the difference between the theoretical 10 s and the real 30-60 s. That is why the solutions attack on two fronts: fewer bytes (quantisation) and a better porter (concurrent loaders / GDS).

The maths that matter: cold start as a tax on autoscaling

Cold start is not paid once. It is paid every time a pod is born. And on an elastic platform, pods are born continuously.

Take an autoscaling setup with KEDA that scales from 2 to 6 replicas when the queue grows. The 4 new replicas take this long to be ready:

$$ T_{\text{ready}} = T_{\text{schedule}} + T_{\text{pull-image}} + T_{\text{load-weights}} + T_{\text{cuda-graphs}} $$

With a Llama-70B FP16 and the default loader, $T_{\text{load-weights}}$ dominates: it can be 40 s of the ~60 s total. During those 40 s, the queue that triggered the autoscaling keeps growing, since the new replicas absorb no traffic until they have loaded. The real number in the formula is not “how many replicas”, it is how long each one takes to start serving, and that number is written by the loading path.

This has a harsh operational consequence: scale-to-zero is unworkable for workloads with a latency SLO if the cold start is 40 s. Nobody waits 40 s for the first token. The real elasticity of an inference platform is not limited by GPU availability, it is limited by how long that GPU takes to have the model inside it. Bringing the cold start down from 40 s to 8 s (with streamer + FP8) is what turns “theoretical scale-to-zero” into “usable scale-to-zero”.

The published numbers give the order of magnitude of the improvement: fastsafetensors cuts start-up from 12.39 s to 4.74 s on a Llama-2-13B over 4×L40S, and from 16.04 s to 6.88 s on 1×A100, 4.8-7.5× over the default deserialiser. The Run:ai Model Streamer loads in 4.88 s from S3 at concurrency 32 and 7.53 s from SSD IO2 at concurrency 8; integrated into vLLM, total time to ready drops to ~23 s from S3. There is no magic here: it is getting the CPU out of the loop (GDS) and reading in parallel (concurrency) to saturate the disk instead of leaving it half idle while one thread deserialises.

The 10 knobs worth touching

Knob 1 — Measure where the time goes (read vs deserialise vs H2D)

Before touching anything: time it. Is the time going on reading from disk, on deserialising, or on the H2D? iostat -x 1 during the load tells you whether the NVMe is saturated (disk bottleneck) or nearly idle (loader/CPU bottleneck). If the disk is at 20%, the problem is not the disk: it is the porter. Changing disk would fix nothing; changing loader would.

Knob 2 — `--load-format`: choosing the porter

vLLM exposes several loaders via --load-format: safetensors (default), runai_streamer, fastsafetensors, tensorizer. The default is the slowest. Changing one flag can be the cheapest 4-7× there is.

# Run:ai Model Streamer (concurrent reads, saturates the disk)
vllm serve meta-llama/Llama-3.1-70B --load-format runai_streamer
# fastsafetensors (GPUDirect Storage, direct DMA disk→HBM)
vllm serve meta-llama/Llama-3.1-70B --load-format fastsafetensors

Knob 3 — Streamer concurrency

The Run:ai Model Streamer splits the read across N threads according to the size of each tensor in order to saturate the storage bandwidth. Concurrency is the key parameter: 16 is usually enough for local NVMe; 32 (sometimes 64) for high-throughput network storage. One thread does not saturate a Gen5 NVMe; 32 do.

vllm serve <model> --load-format runai_streamer \
 --model-loader-extra-config '{"concurrency": 32}'

Knob 4 — GPUDirect Storage (GDS) + fastsafetensors

If there is an nvidia-fs driver, a supported filesystem and local NVMe, GDS writes straight disk→HBM with no bounce through the CPU. It is the difference between the two paths in the diagram. But: it requires the stack to be in place, and it only wins if the bottleneck is the CPU bounce, not the disk itself. Check with gdscheck.

Knob 5 — Local NVMe for the weights, not network

Serving the weights from Ceph RGW / NFS is convenient (one shared place) but it puts the network into the loading path. For cold start, weights on the node’s local NVMe (or a local cache). Network storage is for the model repository; the inference node should have a hot local copy.

Knob 6 — Pre-pull / local model cache

An initContainer that downloads the model onto a local or hostPath NVMe volume before starting vLLM turns a cold start “from the network” into one “from local NVMe”. Combined with a per-node cache DaemonSet, new pods on an already warm node read from local disk, not from the network.

Knob 7 — Quantisation: move fewer bytes

Weights already in FP8 or INT4 on disk = half or a quarter of the cold start. It is the knob from the table above. And it composes with all the others: FP8 + streamer + GDS is multiplicative.

Knob 8 — Parallel loading across GPUs (vLLM V1)

vLLM’s V1 engine (the default since 0.19) loads the weight shards in parallel across the GPUs of a TP group, instead of sequentially. At TP=4, each GPU loads its quarter at the same time. Check it is active; in old versions loading was serial and the cold start of TP=4 was almost 4× that of TP=1.

Knob 9 — NUMA locality of the NVMe

The NVMe hangs off a PCIe root under a specific socket, exactly the same NUMA map as the host post. If the load’s host buffer lands on the wrong socket, the H2D crosses the UPI. The fifth list to align, alongside isolcpus, reserved-cpus and the NIC IRQs: which socket is local to the NVMe and to the destination GPU. nvidia-smi topo -m shows it.

Knob 10 — Do not pay the cold start: keep pods warm

Sometimes the answer is not to load faster, but not to unload. A floor of always-live replicas (no scale-to-zero), or a pool of preloaded warm standbys, swaps “wait 40 s” for “0 s”. It is the cost of idle GPU in exchange for start-up latency: a capacity planning decision, not a technical one.

Summary table

#	Knob	What it attacks	Risk / cost
1	`iostat` while loading	knowing whether the bottleneck is disk or loader	none; always do it first
2	`--load-format`	single-threaded deserialisation	format compatibility
3	streamer concurrency	underused disk	host RAM for buffers
4	GPUDirect Storage	bounce through the CPU	requires nvidia-fs + supported FS
5	local NVMe vs network	the network in the path	duplicating weights per node
6	pre-pull / node cache	network on every start-up	local disk space
7	FP8/INT4 quantisation	bytes to move	quality (measure, do not assume)
8	V1 parallel loading	serial loading across GPUs	none if V1 is active
9	NUMA of the NVMe	H2D crossing the UPI	align with the rest of the NUMA lists
10	warm pods / no zero	the whole cold start	idle GPU paid for

How it connects with the rest of the stack

With autoscaling. The whole of scale-from-zero with KEDA rests on this: real elasticity is limited by the cold start, not by GPU availability. An autoscaler with 40 s of loading reacts late to every peak.

With disaggregated serving. In disaggregated prefill/decode, bringing up a decode pool on demand pays the cold start of loading the model into each new pod. The elasticity of the pattern depends on how fast those pods start.

With canary/blue-green. Every canary deployment loads a new version of the model alongside the old one. The validation time of a canary includes its cold start; large models make deployments slower and more expensive.

With NUMA and hugepages. The load’s host buffer wants to be pinned and NUMA-local, the same thing the host post asked for on the hot path. The loading path is another client of the same NUMA map.

With quantisation. FP8/INT4 is not only inference throughput: it is the direct lever on the bytes of the cold start.

With capacity planning. Sizing that ignores cold start underestimates how many replicas are needed to absorb a peak: if they take 40 s to start, you need a bigger permanent cushion.

Traps and things that are not what they seem

“The second time it loaded really fast.” This is the star trap. The first load filled the page cache (host RAM); the second reads from RAM, not from disk, and flies. But in production pods are ephemeral and are born on different nodes: the start-up that counts is the cold one, on a node where those files are not in page cache. Benchmarking the second load is measuring a situation that almost never happens at the moment that matters (the peak that triggers the autoscaler).

“GDS always speeds things up.” No. GDS removes the bounce through the CPU; if your bottleneck is the disk itself (saturated NVMe) or the deserialisation, GDS does not touch that part. Measure first (knob 1). It also demands nvidia-fs, a supported filesystem, and sometimes it does not work over whatever network storage you have.

“mmap makes loading instant.” mmap maps the file but reads nothing yet: the cost is deferred to the first access to each page. The time does not disappear, it moves, and the first token pays the page faults that start-up did not. You have moved the cold start into the latency of the first request, which is probably a worse place to have it.

Weights on network storage “because it is cleaner”. Sharing a model repository on Ceph RGW is fine for storing them; serving the cold start from there puts the network (and its latency and its contention) into the critical path. Local NVMe cache on the inference node.

Loading FP16 and quantising at start-up. Quantising on the fly during loading (for example FP16→FP8 on the GPU) can be slower than having the weights already quantised on disk: you move twice the bytes and on top of that you do conversion work. If you are going to serve in FP8, store the weights in FP8.

Optimising the load and ignoring T_cuda-graphs. Bringing weight loading down to 8 s and forgetting that CUDA graph capture adds several more seconds leaves the cold start half done. That second half of start-up is the subject of the next post.

Conclusion

The whole series optimised what happens with every token: the cable, the host, the network, the silicon. But before the first token there is a journey almost nobody times and that decides whether an inference pod opens in 10 s or in 60 s: pushing the model from disk to HBM. Intuition blames the GPU, and the GPU is innocent: HBM swallows 140 GB in 42 ms. The bottleneck is the supply chain: a disk reading at 14 GB/s, a PCIe copying at 50, and above all a default loader that deserialises tensor by tensor with a single thread and turns 10 s of bytes into 60 s of waiting. The solutions attack the two right fronts, fewer bytes (quantisation) and better transport (GDS, concurrent streamers, local NVMe), and give 4-7× almost for free. And above the technique, an idea that reorders the priorities: on an elastic platform, cold start is not a start-up detail, it is the ceiling on elasticity. The fastest GPU in the world does not scale if it takes 40 s to have the model inside. The pantry lift, the one nobody timed, is what decides what time the kitchen really opens.

References

NVIDIA, Reducing Cold Start Latency for LLM Inference with NVIDIA Run:ai Model Streamer: https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer/.
vLLM, Loading models with Run:ai Model Streamer: https://docs.vllm.ai/en/stable/models/extensions/runai_model_streamer/.
vLLM, Loading model weights with fastsafetensors: https://docs.vllm.ai/en/stable/models/extensions/fastsafetensor/.
foundation-model-stack, fastsafetensors (high-performance loader, GDS): https://github.com/foundation-model-stack/fastsafetensors.
Speeding up Model Loading with fastsafetensors (arXiv 2505.23072): https://arxiv.org/html/2505.23072v1.
NVIDIA, GPUDirect Storage: A Direct Path Between Storage and GPU Memory: https://developer.nvidia.com/blog/gpudirect-storage/.
NVIDIA, Magnum IO GPUDirect Storage (benchmarking and configuration): https://developer.nvidia.com/gpudirect-storage.

The floor next door: NUMA, hugepages and CPU isolation, or why your GPU waits on the kernel

Sat, 06 Jun 2026 07:30:00 +0200

Second post in the “below the engine” series. The first opened up the cable between GPUs (NVLink/NCCL). This one goes down to the host: the cores, the memory and the kernel that surround those GPUs and that, badly configured, leave them waiting. The third will explain how Kubernetes automates all of this; here is the raw layer, the one you have to understand before delegating it.

TL;DR

A node with 4×H100 SXM is, physically, a two-socket server = two NUMA domains. Each socket has its own cores, its own memory channels and PCIe lanes towards half of the GPUs and NICs. Inference is not only GPU: the host does work on the hot path of every token, launching CUDA kernels, sampling the next token, tokenising, moving pinned buffers between host and GPU, running the NCCL threads. If those threads and their memory land on the socket that is not local to the GPU, every access crosses the inter-socket link (UPI/Infinity Fabric): 2-3× slower and with p99 spikes. There are three kernel levers that decide the latency tail and that almost nobody touches: locality (NUMA affinity: CPU, memory, GPU and NIC on the same “floor”), page tables (hugepages: a few large pages instead of millions of small ones, plus pinned memory for DMA), and jitter (CPU isolation with isolcpus/nohz_full/rcu_nocbs plus IRQ affinity, so that the kernel does not interrupt the thread launching the next decode kernel). This post explains the mechanism, gives the 10 real knobs, and connects with the interconnect and latency-bound decode. With scepticism about which knobs move the needle in inference and which are cargo cult inherited from low-latency trading.

Where you are: the host below the cable

The analogy: the two-floor office

Picture a consultancy in a building with two floors. On each floor there are desks (CPU cores), an archive with the files (the memory of that socket) and a loading bay connecting to the outside (the PCIe lanes towards the GPUs and NICs of that socket). An analyst works fast while everything they need is on their floor: they reach out and take the file from the archive next to them.

The problem starts when the analyst is on floor 1 but their file is in the archive on floor 2. Every time they need it, they take the lift. The work “functions”, but every lookup costs a trip. And if on top of that the loading bay through which their materials arrive (their GPU) is on the other floor, every delivery crosses the building. This is NUMA: local access (same floor) is fast; remote access (another floor, via the inter-socket link) is 2-3× slower.

And there are two more ways to ruin that analyst even when they are on the right floor:

Interrupting them constantly. Every few minutes, a tannoy announcement, a colleague with a question, a fire alarm test. Every interruption breaks their concentration right when they were about to deliver. This is kernel jitter: the scheduler tick, device IRQs, RCU callbacks, interrupting the host thread right when it was about to launch the next GPU kernel. CPU isolation is putting them in an office with a “do not disturb” sign.
Giving them an index of a thousand tiny tabs. If to find each file they have to search an index with a million minuscule entries, they lose time in the search. If the index has a few large entries, they find it instantly. These are hugepages: pages of 2 MB or 1 GB instead of 4 KB reduce the pressure on the TLB (the cache of the page index).

The thesis: the GPU is expensive and fast, but it spends a surprising fraction of decode waiting on the host. If the host is on the wrong floor, interrupted, and searching a giant index, the GPU, the 30,000 € resource, waits. The three levers in this post exist so that it does not wait.

The mechanism: what the host does on the token path

It is tempting to think that in inference “the GPU does everything”. It is not true. For every token, the host (CPU) does, at a minimum:

Launch the CUDA kernels for each operation. The GPU does not decide what to execute; the host keeps putting kernels in its queue. In decode, where each kernel is short, the host has to run ahead feeding the queue; if the host thread stalls, the GPU runs out of work: a bubble.
Sample the next token (argmax/top-p/top-k over the logits), which comes back from device to host.
Tokenise the input and detokenise the output.
Move pinned buffers (page-locked) between host and GPU by DMA: prompts, logits, and in offload configurations, part of the KV cache.
Run the NCCL threads that coordinate the collectives of the previous post.

All of that is CPU and host memory work. And all of it suffers if:

The process runs on the socket that is not local to its GPU → every DMA and every memory access crosses the inter-socket link.
The kernel interrupts the threads → bubbles in the GPU queue.
The memory is not pinned or uses small pages → page faults, TLB misses, and worse: if the memory of the DMA transfer is not pinned, the driver makes an intermediate copy.

The map: `nvidia-smi topo -m`

Everything starts by looking at the map. nvidia-smi topo -m shows, for each GPU, which NUMA node and which cores it is local to, and through what kind of path it talks to each NIC and to each other GPU:

 GPU0 GPU1 GPU2 GPU3 NIC0 CPU Affinity NUMA Affinity
GPU0 X NV18 NV18 NV18 PIX 0-31,64-95 0
GPU1 NV18 X NV18 NV18 SYS 0-31,64-95 0
GPU2 NV18 NV18 X NV18 SYS 32-63,96-127 1
GPU3 NV18 NV18 NV18 X SYS 32-63,96-127 1

Read it like this: GPU0 and GPU1 are local to NUMA node 0 (cores 0-31, 64-95); GPU2 and GPU3 to NUMA node 1. NV18 between GPUs = 18 NVLink links (the good stuff, from the previous post). In the NIC column: PIX = a single PCIe switch in between (optimal for GPUDirect RDMA); SYS = the path crosses the inter-socket link (the worst). The rule: the process serving on GPU0/1 must be pinned to cores 0-31/64-95 and to the memory of node 0; and if it also uses RDMA, you want the NIC that is on PIX with its GPU.

The maths that matter: a kernel tick is a GPU bubble

The figure worth internalising is not the bandwidth one, it is the jitter one. In decode, the host launches many short kernels per token. If the host thread that launches them is preempted by the scheduler (a timer tick, an IRQ, an RCU callback) for $t_{\text{stall}}$, and the GPU drains its queue in that time, a bubble appears: the GPU stops.

Put numbers on it. A typical timer tick or the handling of an IRQ costs of the order of tens of microseconds of diversion. If a decode kernel lasts around 50-100 µs and the queue carries 2-3 kernels in flight, a host stall of 50-100 µs drains the queue and the GPU sits idle until the host resumes. Multiply by the interrupt frequency of a non-isolated kernel (the default tick is 250-1000 Hz, plus network and disk IRQs): the p99/p999 tail of TTFT and of inter-token time fills up with these episodes.

$$ \text{jitter}_{p99} \approx f_{\text{interrupts}} \times t_{\text{stall}} \times \mathbb{1}[\text{GPU queue drained}] $$

The intuition: in average throughput it is barely noticeable (the bubbles average out), but in the tail, which is what an SLO measures, kernel jitter is a first-order contributor. That is why CPU isolation, which was born in low-latency trading, makes sense in LLM decode: it is the same physics, a critical thread that cannot afford the kernel stopping it.

And the NUMA cost, in parallel: a remote memory access (another floor) has latency around 1.5-2× the local one and half the bandwidth. For the pinned buffers moved by DMA on every step, and for the vLLM scheduler structures that live on the host, that penalty is paid token by token.

The three levers, one by one

Locality (NUMA): getting everything on the same floor

The goal is that the inference process using GPU0/1 has its cores, its memory and (if applicable) its NIC on NUMA node 0. Raw, without Kubernetes:

# Pin process to node 0 (cores and memory) to serve on GPU0/1
numactl --cpunodebind=0 --membind=0 \
 vllm serve meta-llama/Llama-3-70B --tensor-parallel-size 2

--membind=0 is the key: it forces all the process memory to be allocated on node 0. Without --membind, the kernel can place pages on node 1 under pressure, and you start paying for the lift without knowing it.

Page tables (hugepages): a few large pages and pinned memory

Two different things under the same umbrella. First, hugepages reduce TLB pressure for the large host buffers (pinned, KV offload). Second, pinned memory (page-locked) is what allows direct DMA with no intermediate copy. The silent trap is transparent hugepages (THP): their background compaction causes latency spikes, exactly what you do not want.

Jitter (CPU isolation): the office with “do not disturb”

Three kernel boot parameters, coordinated:

isolcpus=2-31,66-95 # take these cores out of the scheduler's balancing
nohz_full=2-31,66-95 # tickless: no timer tick if there is only 1 runnable thread
rcu_nocbs=2-31,66-95 # offload RCU callbacks to housekeeping cores

isolcpus sets the cores aside; you have to pin the inference threads there (the non-isolated cores, 0-1, are left for the system). nohz_full removes the periodic tick (it only works if there is a single runnable thread on the core). rcu_nocbs takes RCU work off those cores. And separately, IRQ affinity: moving device interrupts away from the inference cores.

The 10 knobs worth touching

Ordered by impact/frequency. Almost all of them are sysctls, kernel boot parameters or numactl. The low-level reference is Rigtorp’s low-latency guide and Red Hat’s real-time docs.

Knob 1 — `nvidia-smi topo -m`: look at the map before touching anything

Just as in the interconnect post: the map first. Which GPU is local to which NUMA node and to which cores, and what path (PIX/PHB/SYS) there is to each NIC. Without this, any pinning is blind. Half of the “inference has latency spikes” problems are processes running on the wrong socket without anyone having looked.

Knob 2 — `numactl --cpunodebind --membind`: pin to the local node

numactl --cpunodebind=0 --membind=0 <process> # cores AND memory on node 0
numactl --hardware # see nodes, distances, free memory

The --membind is what really matters: without it, the memory scatters. It is the highest-impact knob on sustained throughput.

Knob 3 — `kernel.numa_balancing=0`: turn off automatic migration

sysctl -w kernel.numa_balancing=0

The kernel’s automatic NUMA balancing migrates pages between nodes trying to “bring them closer”, but that background work causes jitter and, with the memory already pinned by knob 2, it adds nothing. On dedicated inference nodes, turn it off.

Knob 4 — Explicit hugepages (1 GB) for host buffers

# Kernel boot, for KV offload / large pinned buffers
default_hugepagesz=1G hugepagesz=1G hugepages=32

grep Huge /proc/meminfo # verify the reservation

Useful when there is host memory on the hot path (vLLM with --cpu-offload-gb, or large pinned buffers). If your deployment does not touch host memory in the hot path, explicit hugepages add little, so do not set them out of cargo cult.

Knob 5 — THP on `madvise` or `never`: avoid the compaction spikes

echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

Transparent hugepages on always save TLB but their compaction triggers unpredictable latency. For tail-sensitive workloads, madvise (only where the app asks for it) or never is what is recommended. It is one of the few knobs with clear consensus: THP always is bad for latency.

Knob 6 — `isolcpus`: set the inference cores aside from the scheduler

isolcpus=2-31,66-95

Takes those cores out of the scheduler’s load balancing; the system (kernel threads, daemons) stays on the non-isolated ones. You have to pin the inference threads explicitly to the isolated cores (via numactl/taskset or, on K8s, the CPU Manager of the next post). Isolating without pinning is useless.

Knob 7 — `nohz_full` + `rcu_nocbs`: tickless and no RCU on the critical cores

nohz_full=2-31,66-95 rcu_nocbs=2-31,66-95

Removes the periodic timer tick and the RCU callbacks from the inference cores. Two warnings from practice: nohz_full only eliminates the tick if there is a single runnable thread on the core (if you pin two threads there, the tick comes back); and nohz_full is not compatible with the intel_pstate driver in some configurations, which has to be validated, not assumed.

Knob 8 — IRQ affinity: interrupts, away from the inference cores

systemctl stop irqbalance # or configure it to respect isolcpus
# move a device's IRQs to the housekeeping cores (0-1)
echo 3 > /proc/irq/<N>/smp_affinity # mask for cores 0-1

A network or disk IRQ landing on an inference core is a direct interruption of the thread feeding the GPU. Move them to the housekeeping cores. (irqbalance can respect isolcpus automatically if it is configured to.)

Knob 9 — CPU governor `performance` + C-states

cpupower frequency-set -g performance
# stop idle cores entering deep C-states (wakeup latency)
cpupower idle-set -D 0 # or limit the C-state depth

With the powersave/ondemand governor, a core that was idle takes time to ramp up its frequency: wakeup latency exactly when work arrives. performance keeps it flat out. On servers dedicated to inference, the energy saving does not compensate for the latency tail.

Knob 10 — Memory locking + `swappiness=0`

sysctl -w vm.swappiness=0 # do not evict inference pages to swap
# and in the app / container: ulimit -l unlimited (memlock) for pinned memory

An inference page that the kernel decides to swap to disk is a page fault of milliseconds on the hot path. swappiness=0 and adequate memlock limits (so the driver can pin memory) close that door. On K8s, this translates into Guaranteed QoS and memory limits, the bridge to the next post.

Summary table

#	Knob	Mechanism	What it attacks
1	`nvidia-smi topo -m`	diagnosis	see GPU–NUMA–NIC affinity
2	`numactl --cpunodebind --membind`	pinning	locality (the biggest lever)
3	`kernel.numa_balancing=0`	sysctl	jitter from page migration
4	explicit 1G hugepages	boot param	TLB on host buffers
5	THP `madvise`/`never`	sysfs	compaction spikes
6	`isolcpus`	boot param	scheduler off the critical cores
7	`nohz_full`+`rcu_nocbs`	boot param	tick + RCU jitter
8	IRQ affinity	`/proc/irq`	device interrupts
9	`performance` governor	cpupower	frequency wakeup latency
10	`swappiness=0` + memlock	sysctl/ulimit	page faults on the hot path

How it connects with the rest of the stack

With the interconnect (previous post). The NCCL host threads want cores local to their GPU; and for multi-node, the RDMA NIC must be on the PIX path with its GPU (knob 1). A GPUDirect RDMA with the NIC under the other socket loses half its advantage. NUMA and NVLink are the same story seen from the host and from the cable.

With vLLM and decode. Decode is latency-bound: the host thread feeding the kernel queue is exactly the one CPU isolation protects. And vLLM’s --cpu-offload-gb puts host memory on the hot path, where NUMA locality plus hugepages (knobs 2, 4) go from “nice” to “critical”. Continuous batching helps here too: more tokens per iteration amortise both the collective latency and the fixed cost of the host launches.

With Kubernetes (next post). Everything in this post is done by hand (numactl, taskset, boot parameters). In production it is not done by hand: the kubelet automates it with CPU Manager, Memory Manager and Topology Manager. The next post is exactly how this is declared so that every vLLM pod is born pinned to the right NUMA node, with no scripts.

With observability. The p99 spikes from jitter or from remote access are visible: in DCGM, low GPU utilisation with a full queue (bubbles); in system metrics, inter-socket traffic and CPU migrations. GPU observability with DCGM is where you diagnose a “the GPU is at 60 % and I do not know why” that many times is the host waiting.

With capacity planning. Reserving cores for the system (housekeeping) and dedicating the rest to inference changes the calculation of how many pods/replicas fit per node. Capacity planning has to count those reserved cores, not assume that all 128 vCPUs are available for serving.

Traps and things that are not what they look like

Cargo cult from low-latency trading. Many isolcpus/nohz_full guides come from HFT, where the last microsecond is squeezed out. In LLM inference, CPU isolation does help on the decode tail, but do not expect a miracle: if your bottleneck is HBM bandwidth or the interconnect, isolating cores does not move the needle. Measure first; apply it where the host is the limit.

Isolating without pinning. isolcpus takes the cores out of the scheduler, but if you do not pin the inference threads there, those cores stay empty and inference runs on the housekeeping ones, worse than before. Isolating and pinning always go together.

Forgotten --membind. Pinning cores but not memory (--cpunodebind without --membind) lets the pages scatter to the other node under pressure. Memory pinning is the half that gets forgotten most and the one that pays off most.

THP always “because it saves TLB”. It saves TLB and throws in latency spikes from compaction. For workloads with a tail SLO, it is a bad trade. madvise/never.

nohz_full with two threads on the core. Tickless only works with a single runnable thread per core. If you pin two inference threads to the same isolated core, the tick comes back and you have complicated the kernel boot for nothing.

Assuming the topology instead of reading it. Different servers wire GPUs and NICs to different sockets. nvidia-smi topo -m and numactl --hardware are the truth; the manufacturer’s diagram is indicative. Read it on every node model.

Conclusion

The GPU is the expensive resource, but in decode it spends a surprising part of its time waiting on the host: waiting for it to launch the next kernel, to sample, to move a buffer. If that host is on the wrong floor (remote NUMA), interrupted (kernel jitter) or searching a giant index (4 KB pages), the GPU is left with an empty queue and p99 blows up, without any API dashboard saying why. Of the ten knobs, the first (reading the map with nvidia-smi topo -m) and the second (pinning cores and memory to the local node with --membind) solve most of it; CPU isolation (isolcpus/nohz_full/IRQ affinity) is the second layer, the one that trims the decode tail, and it makes sense where the host is the limit, not as a ritual. The idea that reorders the intuition: inference is not “all GPU”; it is a dance between GPU and host, and the host dances better close by, uninterrupted, and with few large pages. The next post shows how Kubernetes choreographs that dance for every pod without a single hand-written script.

References

Erik Rigtorp, Low Latency Tuning Guide (isolcpus, nohz_full, IRQ affinity, THP): https://rigtorp.se/low-latency-guide/.
Red Hat, Usage, constraints and implications of isolcpus=, nohz_full= and rcu_nocbs=: https://access.redhat.com/articles/3720611.
NVIDIA, nvidia-smi topo -m and the GPU–NUMA–NIC affinity matrix (CUDA docs / Developer Forums): https://forums.developer.nvidia.com/t/nvidia-smi-topo-m-revisited/216584.
Chaim Rand, The Crucial Role of NUMA Awareness in High-Performance Deep Learning: https://chaimrand.medium.com/the-crucial-role-of-numa-awareness-in-high-performance-deep-learning-99ae3e8eb49a.
SUSE Labs, CPU Isolation – nohz_full (part 3): https://www.suse.com/c/cpu-isolation-nohz_full-part-3/.
Linux kernel, Automatic NUMA Balancing and Transparent Hugepage Support (Documentation/admin-guide): https://docs.kernel.org/admin-guide/mm/transhuge.html.

The shared bench: NVLink, NVSwitch and NCCL, the cable every token crosses in tensor parallel

Sat, 06 Jun 2026 07:00:00 +0200

This post goes one floor below the engine. In the seven-layer inference stack and in one big replica or many small ones the decision was how many GPUs and how to split the model; here we explain the cable that makes that split work, or that strangles it. It is the first of a mini-series “below the engine”: interconnect (this one) → kernel and NUMA → Kubernetes resource managers.

TL;DR

Tensor parallelism (TP) does not split a model into four chunks that run on their own. It spreads each layer across the GPUs, but after attention and after the MLP the GPUs have to add up their partial results with an all-reduce. On a Llama-70B with 80 layers, that is around 160 all-reduces per generated token. That all-reduce travels over the interconnect, so the interconnect sits on the critical path of every token, not in the background plumbing. On an HGX H100 baseboard, the 8 GPUs all talk to each other at 900 GB/s bidirectional via four NVSwitch chips; without NVSwitch/NVLink, that same traffic falls back to the CPU over PCIe and loses an order of magnitude. NCCL is the library that decides how each collective is done (ring, tree, or NVLS = NVLink-SHARP, which offloads the summation to the switch itself). And there is an asymmetry almost nobody keeps in mind: decode is latency-bound (tiny messages, 16 KB) and prefill is bandwidth-bound (huge batched activations), which is why “more NVLink bandwidth” speeds up prefill but barely touches token-by-token decode. This post explains the mechanism, gives the 10 real NCCL/driver knobs worth touching, and connects with vLLM’s custom all-reduce, disaggregated serving and GPU observability. With scepticism about which levers move the needle.

Where you are: the floor below the engine

The analogy: four mechanics and a single bench

Four mechanics assemble one and the same car engine. It is not that each builds their own engine in parallel, that would be four cars (four replicas of the model, a different strategy). Here they build a single one, at the same time, splitting the parts: one does the pistons, another the cylinder head, another the crankshaft. The problem is that the parts fit into each other: before going on, all four have to put together what they have and check that it matches. That “put together and check” happens dozens of times during the build.

There are two ways to organise the workshop:

A single large bench, everyone around it (NVSwitch). Each mechanic reaches out and hands their part directly to any other, all at once, without standing up. It is instant and simultaneous. This is NVLink + NVSwitch: the GPUs form an all-to-all where anyone talks to anyone at 900 GB/s at the same time.
Four separate workshops with a courier (PCIe via CPU). Every part a mechanic wants to hand to another goes into a box, goes down to reception (host memory, via CPU), and from there up to the destination workshop. Slower, and serialised by reception. This is what happens when there is no NVLink: inter-GPU traffic falls back to PCIe and bounces through the CPU, around 14× slower than NVLink.

The thesis of the post follows on its own: tensor parallelism only makes sense if the mechanics share the bench. As soon as the “put together and check” (the all-reduce) has to go through reception, splitting the work costs more than it saves. That is why, on a serious platform, TP does not cross the NVLink boundary: TP=4 or 8 inside the baseboard where there is NVSwitch, and above that you replicate or use pipeline, never stretch TP over PCIe or over the network. When each option makes sense is covered in one big replica or many small ones; here we explain why the cable dictates that decision.

The mechanism: what an all-reduce really is and why there are 160 per token

Tensor parallelism splits the weight matrices by columns/rows across the $N$ GPUs. Each GPU computes a portion of the layer output. But the next operation needs the complete output, so it has to be recombined. That recombination is a collective operation: an all-reduce, which adds the partial tensors of all the GPUs element by element and leaves the result identical on all of them.

In a standard transformer block there are two synchronisation points per layer:

After the output projection of attention (the o_proj that recombines the split heads).
After the second matrix of the MLP (the down_proj that recombines the split feed-forward).

$$ \text{all-reduces per token} = 2 \times L_{\text{layers}} $$

For a Llama-70B ($L = 80$): $2 \times 80 = 160$ all-reduces per generated token. Not per request, not per sequence: per token. Multiply by decode throughput and you understand why the interconnect is not background infrastructure but a hot path.

How the all-reduce is done: ring, tree, NVLS

NCCL does not have a single way of doing an all-reduce; it picks an algorithm according to topology and message size:

Ring. The GPUs form a ring; each one passes a chunk to its neighbour, adds, and rotates. It takes $2(N-1)$ steps. It is bandwidth-optimal for large messages: the cost of moving the data is $\frac{2(N-1)}{N} \times M$ bytes over the link, almost independent of $N$. The downside: $2(N-1)$ latency hops, bad for small messages.
Tree. Tree reduction: $\log N$ levels. Better latency for small messages and many nodes, worse bandwidth utilisation.
NVLS (NVLink SHARP). The Hopper trick: the summation is not done by the GPUs, it is done by the NVSwitch. The switch has reduction units; the GPUs send their tensors, the switch adds them in flight and returns the result. It takes work away from the GPUs (frees up SMs) and reduces hops. Available only with a 3rd-generation NVSwitch (NVLink4) plus Hopper or newer.

The mental rule: decode (tiny messages) wants latency → tree/LL or vLLM’s custom kernel; prefill (huge messages) wants bandwidth → ring/NVLS. That is why there is no global “optimal NCCL_ALGO”; it depends on which phase you are looking at.

The maths that matter: why decode and prefill stress the cable the other way round

Here is the asymmetry almost everyone skips. The size of the tensor that gets all-reduced in each layer is, approximately:

$$ M \approx B \times S \times h \times 2\ \text{bytes (BF16)} $$

where $B$ = batch, $S$ = tokens processed in this forward pass, $h$ = hidden size.

In decode, you generate 1 token per sequence per iteration. For a single sequence ($B \times S = 1$) and $h = 8192$ (Llama-70B):

$$ M_{\text{decode}} \approx 1 \times 8192 \times 2 = 16\ \text{KB per all-reduce} $$

16 KB is tiny. At 900 GB/s, moving 16 KB takes around 18 nanoseconds of pure transfer, but the real cost is dominated by the launch latency of the collective (synchronisation, kernel launch), of the order of single-digit microseconds. With 160 all-reduces per token:

$$ t_{\text{comms/token}} \approx 160 \times (5\text{–}10\,\mu s) \approx 0.8\text{–}1.6\ \text{ms} $$

That is the communication floor per token, independent of bandwidth. An uncomfortable and counter-intuitive implication: buying more NVLink bandwidth does not speed up token-by-token decode of a single sequence. What helps in decode is lowering the latency per collective (LL protocol, vLLM’s custom all-reduce, NVLS to remove hops) and batching (raising $B$ amortises the fixed latency over more tokens, the deep reason continuous batching exists, covered in continuous batching).

In prefill, you process the whole prompt at once: $S$ can be thousands of tokens, and with batching $B \times S$ reaches tens of thousands. There:

$$ M_{\text{prefill}} \approx 8000 \times 8192 \times 2 \approx 131\ \text{MB per all-reduce} $$

131 MB does stress the bandwidth. At 900 GB/s (NVSwitch) the ring all-reduce moves $\frac{2 \cdot 3}{4} \times 131 \approx 196$ MB effective in around 0.22 ms; over PCIe (around 64 GB/s aggregate, bouncing through the CPU) it would be around 3 ms and serialised. Here the cable is the bottleneck and NVLS/bandwidth rule.

Summary in one line: prefill is bandwidth-bound, decode is latency-bound. Any interconnect tuning that does not say which phase it helps is noise.

The hardware: NVLink 4 and NVSwitch on the HGX baseboard

On the generic reference cluster, 4×H100 SXM inside an HGX baseboard, the concrete figures:

Each H100 SXM5 has 18 NVLink 4 links, each one 50 GB/s bidirectional ⇒ 900 GB/s bidirectional aggregate per GPU. That is >14× the bandwidth of a PCIe Gen4 x16 (around 64 GB/s bidir).
On an 8-GPU HGX H100 baseboard, the 18 links of each GPU are spread against four 3rd-generation NVSwitch chips (grouped 5+4+4+5). The result is all-to-all: any GPU talks to any other at 900 GB/s simultaneously, without going through CPU or PCIe.
A 4-GPU baseboard is half a board: same principle, via NVSwitch. Design key: if your 4 H100s are connected by NVSwitch, you have real all-to-all; if they are on different boards connected by PCIe (some “4×PCIe” configurations), you do not have NVLink between all of them and TP=4 suffers. Verify it, do not assume it.

The 10 knobs worth touching

Almost all of them are NCCL environment variables (injected into the inference engine process) or driver settings. Ordered by impact/frequency in an on-premise deployment. The canonical detail is in the NCCL env vars docs.

Knob 1 — `NCCL_DEBUG` + topology dump: see what is happening before touching anything

Do not optimise blind: first confirm which topology and algorithms NCCL picked. This tells you whether it really is using NVLink or whether, silently, it fell back to PCIe/SHM, the number one failure and the most expensive one.

NCCL_DEBUG=INFO # prints topology, rings/trees built, transport chosen
NCCL_DEBUG_SUBSYS=GRAPH,TUNING,NET # narrows it down to what matters
# look in the log for: "via NVLink" / "via P2P" (good) vs "via SHM" / "via PCI" (bad)

If you see via SHM or via PCI between GPUs that should have NVLink, you have a topology problem (PCIe ACS enabled, IOMMU, GPUs on different boards) and no other knob fixes it. This is knob 1 for a reason: half of the “NVLink is slow” cases are “NVLink is not being used”.

Knob 2 — `NCCL_ALGO`: ring vs tree vs NVLS

Forces or excludes algorithms. By default NCCL picks according to size, and it usually gets it right; touch it only with a measurement in front of you.

NCCL_ALGO=NVLS,Tree,Ring # order of preference
NCCL_ALGO=^Ring # exclude Ring (^ prefix)

Rule: prefill/training (bandwidth) ⇒ Ring/NVLS; decode (latency) ⇒ Tree or, better, vLLM’s custom kernel (knob 10/stack). In most inference workloads, leaving it on auto and validating with knob 1 is the right thing; forcing it “just in case” usually makes things worse.

Knob 3 — `NCCL_PROTO`: LL / LL128 / Simple

The protocol controls the latency/bandwidth trade-off at low level:

NCCL_PROTO=Simple # maximum bandwidth, more latency (large messages)
NCCL_PROTO=LL # low-latency, half-bandwidth (tiny messages: decode)
NCCL_PROTO=LL128 # compromise, default on platforms that support it

LL (low-latency) uses flags instead of barriers and wins on the 16 KB messages of decode; Simple wins on the 131 MB of prefill. The default LL,LL128,Simple lets NCCL choose by size, which again is normally the best option.

Knob 4 — `NCCL_NVLS_ENABLE`: offload the summation to the NVSwitch

NVLink SHARP (NVLS) makes the switch do the reduction, freeing up GPU SMs:

NCCL_NVLS_ENABLE=1 # default: ON where there is an NVLink4+ NVSwitch (Hopper)

An important sceptical caveat: NVLS requires NVSwitch (3rd gen, NVLink4). On a node with NVLink through direct GPU-to-GPU bridges (no switch) or on 4×PCIe, NVLS is not available and this knob does nothing. Before “enabling it”, confirm with knob 1 that your topology has a switch. Where it does apply, its biggest advantage is freeing SMs for compute, which is relevant when comms and kernels compete (knob 5).

Knob 5 — `NCCL_MIN_NCHANNELS` / `NCCL_MAX_NCHANNELS`: how many SMs communication steals

Each NCCL “channel” consumes GPU SMs to move data. More channels = more collective bandwidth, but fewer SMs for the inference kernel. It is a split of a fixed resource.

NCCL_MIN_NCHANNELS=4
NCCL_MAX_NCHANNELS=16 # raising it helps prefill (bandwidth); steals SMs from decode

In decode, where the GPU is under-used on compute but tied to latency, trimming channels rarely hurts and sometimes helps; in prefill, more channels squeeze out the bandwidth. A measurement knob, not a faith knob.

Knob 6 — `NCCL_BUFFSIZE`: the buffer size per channel

NCCL_BUFFSIZE=8388608 # 8 MB (default 4 MB); larger buffers → better BW on large messages

Raising it helps bandwidth-bound prefill at the cost of memory per channel. For workloads dominated by small messages (pure decode), the default is plenty.

Knob 7 — `NCCL_P2P_LEVEL` / `NCCL_P2P_DISABLE`: guarantee P2P over NVLink

P2P is what lets one GPU read another’s memory directly over NVLink without going through the host. If it is disabled or degraded, traffic falls back to SHM/PCIe.

NCCL_P2P_LEVEL=NVL # use P2P up to the NVLink level
# NCCL_P2P_DISABLE=1 ← only as a workaround if P2P HANGS (multi-NUMA PCIe, certain Blackwell)

Watch out for the trap: NCCL_P2P_DISABLE=1 and --disable-custom-all-reduce are recommended as a patch when vLLM hangs on PCIe-only multi-NUMA topologies. It is a robustness patch that sacrifices performance: use it if it hangs, never “by default”.

Knob 8 — GPUDirect RDMA for multi-node: `NCCL_NET_GDR_LEVEL`

When TP fits in one node, this does not apply. When you have to cross nodes (huge model, pipeline parallel between baseboards), GPUDirect RDMA lets the GPU talk to the NIC without bouncing through host memory:

NCCL_NET_GDR_LEVEL=PHB # enables GDR according to GPU–NIC proximity on the PCIe bus

Without GDR, every inter-node hop adds a host copy. With InfiniBand/RoCE plus GDR, the KV or the activations travel GPU→NIC→network→NIC→GPU. It is the basis of serious multi-node work and of mixed environments.

Knob 9 — `NCCL_IB_HCA` / `NCCL_SOCKET_IFNAME`: pin the right NIC

The most common and silent multi-node mistake: NCCL picks the management NIC (1 GbE) instead of the fabric one (InfiniBand/100 GbE). Result: collectives at a snail’s pace with no visible error.

NCCL_SOCKET_IFNAME=eth0 # control interface (bootstrap)
NCCL_IB_HCA=mlx5_0,mlx5_1 # the real InfiniBand HCAs of the fabric
NCCL_IB_GID_INDEX=3 # correct GID for RoCE v2

Pin them explicitly. “Auto” gets it right on clean clusters and fails as soon as there is more than one NIC.

Knob 10 — Driver: persistence mode, clocks and NVLink error counters

Below NCCL, the driver has levers and, above all, telemetry you have to look at:

nvidia-smi -pm 1 # persistence mode: avoids driver re-init (latency/jitter)
nvidia-smi nvlink --status # are all 18 links active and at full speed?
nvidia-smi nvlink -e # error/CRC counters per link
nvidia-smi -q -d ECC # memory errors that degrade silently

An NVLink link that negotiates at half speed or accumulates CRC errors degrades the all-reduce without raising any error: the system “works”, it is just slower. These counters are the difference between diagnosing in five minutes and chasing a ghost for days. They are integrated in DCGM (knob/stack: observability).

Summary table

#	Knob	Variable / command	Phase it helps
1	Topology diagnosis	`NCCL_DEBUG=INFO` + `SUBSYS=GRAPH`	always, first
2	Collective algorithm	`NCCL_ALGO` (NVLS/Tree/Ring)	depends on phase; auto usually wins
3	Protocol	`NCCL_PROTO` (LL/LL128/Simple)	LL=decode, Simple=prefill
4	NVLink SHARP	`NCCL_NVLS_ENABLE=1`	prefill; frees SMs (requires NVSwitch)
5	Channels (SMs)	`NCCL_MIN/MAX_NCHANNELS`	+prefill bandwidth / −SM theft in decode
6	Buffer	`NCCL_BUFFSIZE`	bandwidth-bound prefill
7	NVLink P2P	`NCCL_P2P_LEVEL=NVL`	critical; disable only if it hangs
8	GPUDirect RDMA	`NCCL_NET_GDR_LEVEL`	multi-node
9	Fabric NIC	`NCCL_IB_HCA`/`SOCKET_IFNAME`	multi-node (avoids mgmt NIC)
10	Driver + telemetry	`nvidia-smi -pm 1` / `nvlink -e`	jitter + silent diagnosis

How it connects with the rest of the stack

The interconnect is not an island; it touches almost every layer above.

With vLLM, the custom all-reduce. vLLM does not always use NCCL: for the tiny messages of decode (world_size==2 or a fully-connected NVLink topology, below a certain max_size) it uses its own all-reduce kernel that beats NCCL on latency, exactly the decode bottleneck we saw in the maths. It falls back to NCCL for large messages and for topologies without NVLink (where its custom kernel “adds little over NCCL”). The flag --disable-custom-all-reduce / VLLM_DISABLE_CUSTOM_ALL_REDUCE turns it off; it is the patch for hangs on multi-NUMA PCIe. Translation: the most effective decode latency knob is sometimes not an NCCL one, it is choosing well between vLLM’s custom kernel and NCCL.

With TP vs replicas. Everything in one big replica or many small ones rests on this: high TP is only viable inside the NVLink domain. The boundary of “TP=4 or 4 replicas at TP=1?” is drawn by the cable: crossing NVLink with TP means paying for the all-reduce at PCIe prices.

With disaggregated serving. In disaggregated prefill/decode, the KV cache generated in the prefill pool has to travel to the decode pool. That transfer is another consumer of the interconnect (NVLink intra-node, GPUDirect RDMA inter-node) and competes with the all-reduces. Designing the disaggregation without counting the cost of transferring the KV is the classic trap.

With MoE. Mixture-of-Experts models add expert parallelism: an all-to-all (not an all-reduce) that routes each token to its expert, possibly on another GPU. It is a different communication pattern and heavier on bandwidth; MoE in inference lives or dies by the same cable, with an even more demanding collective.

With GPU observability. The NVLink counters (nvidia-smi nvlink -e, TX/RX bytes per link, CRC errors) and NVSwitch utilisation are exposed via DCGM and land in Prometheus/Grafana. The question “is the interconnect healthy and saturated?” is answered there, alongside the rest of GPU observability with DCGM. A slow all-reduce shows up sooner in an NVLink error counter than in API latency.

With capacity planning. The inference sizing that assumes “TP=4 scales almost linearly” only holds inside NVLink. Outside it, scaling efficiency collapses and the capacity plan lies. The cable is a parameter of the capacity model, not a detail.

Traps and things that are not what they look like

“More NVLink bandwidth = faster decode.” False for a single sequence. Decode is latency-bound; bandwidth is barely touched by 16 KB messages. What speeds up decode is batching (amortising the fixed latency) and lowering latency per collective (LL, custom kernel, NVLS). Bandwidth rules in prefill.

“I have 4 H100s, so I have NVLink between all four.” Not necessarily. There are configurations where the GPUs are on different boards joined by PCIe, or with NVLink bridges only in pairs. Confirm it with nvidia-smi nvlink --status and knob 1 before planning TP=4. A TP=4 over P2P-over-PCIe performs far worse than the brochure says.

Forcing NCCL_ALGO/NCCL_PROTO “to go faster”. NCCL picks well by size in most cases. Forcing an algorithm without measuring usually makes one of the two phases worse. The correct sequence is: knob 1 (see what it is doing) → measure → touch only if there is evidence.

Disabling P2P/custom all-reduce by default. They are robustness patches for broken topologies (multi-NUMA PCIe, certain Blackwell). Leaving them on “for stability” on a node with healthy NVLink throws performance in the bin.

Stretching TP over the network. TP=8 crossing two nodes over InfiniBand because “there is bandwidth” ignores that the per-layer all-reduce now pays network latency ×160 per token. To cross nodes, pipeline parallel (which communicates once per micro-batch, not per layer) almost always wins. The communication pattern, not just the bandwidth, decides.

Ignoring the NVLink error counters. A degraded link does not raise an exception: the system works, it is just slow. Without watching nvlink -e and ECC, you chase a performance ghost that a counter would have pointed out in five minutes.

Conclusion

Tensor parallelism sells a simple promise, split the model, multiply the VRAM, serve models that do not fit in one GPU, but the small print is that every layer forces the GPUs to come together and add up, twice, dozens of times per token. That all-reduce is the real hidden protagonist of performance, and it lives in the cable: NVLink does it on the shared bench of the NVSwitch at 900 GB/s, or PCIe drags it through the CPU’s reception desk 14× slower. Of the ten knobs, the first one, looking with NCCL_DEBUG at what is really happening, solves half the problems, because half of the “NVLink is slow” cases are “NVLink is not being used”. The rest are refinements that only mean something if you know which phase you are in: prefill wants bandwidth (NVLS, Simple, channels, buffer), decode wants latency (LL, vLLM’s custom kernel, batching). And above all, an idea that reorders the intuition: in on-premise inference, the interconnect is not plumbing you install and forget, it is a hot path, a capacity parameter and, when it degrades silently, the root cause that no API dashboard will point out to you unless you look at the counters of the cable itself.

References

NVIDIA, NVIDIA Hopper Architecture In-Depth (NVLink 4, 900 GB/s): https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/.
NVIDIA, Introducing NVIDIA HGX H100 (4× NVSwitch, all-to-all): https://developer.nvidia.com/blog/introducing-nvidia-hgx-h100-an-accelerated-server-platform-for-ai-and-high-performance-computing/.
NVIDIA, NCCL Environment Variables (all the knobs in this post): https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html.
NVIDIA, The NVLink-Network Switch (Hot Chips 2022, NVLink SHARP): https://hc34.hotchips.org/.
vLLM, Why does vLLM use a custom all-reduce method? (discussion #6159) and custom_all_reduce.py: https://github.com/vllm-project/vllm/discussions/6159.
NVIDIA, NCCL Multi-Node NVLink Tuning Guide: https://docs.nvidia.com/multi-node-nvlink-systems/multi-node-tuning-guide/nccl.html.

Batch sizing in vLLM: the two-hour grid search worth weeks of hardware

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

max-num-seqs and max-num-batched-tokens are the two dials that control how much work vLLM processes on each scheduler iteration. Their default values are calibrated to be safe on any hardware, not to maximise throughput on yours. A systematic grid search over 25 configurations, runnable in two hours, identifies the combination that, for your specific workload and hardware, can double throughput without changing a single line of the model or adding a GPU. The OTel metrics that confirm you found the optimum are vllm:num_waiting_seqs, vllm:num_preemptions_total and vllm:time_per_output_token_seconds.

The analogy

An industrial kitchen with one chef and ten stoves. If the maître d’ only sends one order at a time, the chef works at 10% of capacity. If he sends a hundred simultaneous orders but there are ingredients for only twenty, the chef spends half his time waiting for restocking. The optimum is the point where every stove is lit and the resupply never runs dry.

max-num-seqs is how many orders the chef can have in preparation at once. max-num-batched-tokens is how many ingredients he can process in a single movement of the wok. Getting either of them wrong leaves stoves empty.

The problem: the defaults are not for your hardware

In vLLM V1 (≥ 0.6), the defaults are:

max-num-seqs = 1024 (V1) / 256 (V0)
max-num-batched-tokens = 8192

These values guarantee that vLLM starts on any GPU without OOM. They do not guarantee optimal throughput. The reason: the optimal point depends on three variables vLLM does not know at start-up:

The length distribution of your real workload — a RAG system with 2K-token prompts needs a different budget from a chat with 50-token messages.
VRAM available for KV cache — determined by the model, the quantisation and --gpu-memory-utilization.
Real expected concurrency — how many simultaneous users arrive at the 95th percentile.

The interaction between these three factors makes it impossible for a universal default to be optimal in specific cases.

The arithmetic that matters

On each iteration, vLLM’s scheduler decides which tokens to process. The total budget available per step is max-num-batched-tokens. That budget is split between:

Decode tokens: 1 for each active request in the generation phase. With 64 requests in decode, 64 tokens of budget are consumed.
Prefill tokens (in chunks): the rest of the budget goes to processing new prompts.

$$\text{prefill tokens per step} = \text{max num batched tokens} - \text{num requests decode}$$

If max-num-batched-tokens = 8192 and you have 512 requests in decode, each step can only process 8192 - 512 = 7680 prefill tokens. With 2000-token prompts, that is ~3.8 new prompts per iteration.

The problem appears when max-num-seqs is very high relative to the available KV cache. Each active request in decode occupies KV cache blocks. If the blocks run out, vLLM performs preemption: it pauses a request, frees its KV cache and re-queues it. Each preemption costs additional latency for the paused request and complexity for the scheduler.

$$\text{KV budget} = \frac{\text{free VRAM} \times \text{gpu memory utilization}}{\text{bytes per token} \times \text{max model len}}$$

For a Qwen2.5-14B on an RTX 4090 with Q4_K_M (9 GB of model, 15 GB free):

$$\text{KV budget} = \frac{15 \times 0.92 \times 10^9}{40\,000} \approx 345\,000 \text{ tokens}$$

With max-model-len = 8192, the maximum number of simultaneous requests with a full context is:

$$\text{real max seqs} = \frac{345\,000}{8192} \approx 42 \text{ requests}$$

Setting max-num-seqs = 1024 with those numbers guarantees constant preemptions. The optimum is at 40-50.

The grid search: methodology

Step 1: measure the real workload

Before hunting for the optimum, you need to know your traffic’s percentiles. From Langfuse or the vLLM logs:

# Extract the length distribution from Langfuse
import langfuse
client = langfuse.Langfuse()

traces = client.fetch_traces(limit=1000).data
prompt_lens = [t.input_tokens for t in traces if t.input_tokens]
output_lens = [t.output_tokens for t in traces if t.output_tokens]

import numpy as np
print(f"Prompt p50={np.percentile(prompt_lens,50):.0f} p95={np.percentile(prompt_lens,95):.0f} p99={np.percentile(prompt_lens,99):.0f}")
print(f"Output p50={np.percentile(output_lens,50):.0f} p95={np.percentile(output_lens,95):.0f} p99={np.percentile(output_lens,99):.0f}")

Step 2: work out the KV budget

Run once with --dry-run, or read vLLM’s start-up log:

INFO: # GPU blocks: 4521, # CPU blocks: 512

Each block is 16 tokens. 4521 × 16 = 72,336 tokens of total KV budget.

Step 3: the grid

With the KV budget known and the p95 of prompt/output length:

# grid_search_batch.py
import subprocess, json, time

MODEL = "Qwen/Qwen2.5-14B-Instruct-AWQ"
PROMPT_LEN = 512 # p50 of your workload
OUTPUT_LEN = 256
CONCURRENCY = 32 # simultaneous users expected at peak

seqs_values = [32, 64, 128, 256, 512]
tokens_values = [4096, 8192, 16384, 32768, 65536]

results = []
for seqs in seqs_values:
 for tokens in tokens_values:
 cmd = [
 "python", "-m", "vllm.entrypoints.benchmark_throughput",
 "--model", MODEL,
 "--max-num-seqs", str(seqs),
 "--max-num-batched-tokens", str(tokens),
 "--num-prompts", "200",
 "--input-len", str(PROMPT_LEN),
 "--output-len", str(OUTPUT_LEN),
 ]
 out = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
 # Parse throughput from the output
 for line in out.stdout.splitlines():
 if "Throughput" in line:
 tps = float(line.split(":")[1].strip().split()[0])
 results.append({"seqs": seqs, "tokens": tokens, "tps": tps})
 print(f"seqs={seqs} tokens={tokens} → {tps:.1f} tok/s")

# Save for analysis
with open("grid_results.json", "w") as f:
 json.dump(results, f, indent=2)

25 configurations × ~5 min = ~2 hours. Real execution time, not waiting time.

Step 4: read the surface

The result is a 5×5 throughput matrix. The typical shape:

max-num-batched-tokens → 4K 8K 16K 32K 64K
max-num-seqs ↓
32 180 310 380 390 385 ← max-num-seqs too low
64 185 350 480 510 508 ← optimal point for this workload
128 182 340 450 480 475
256 178 320 400 410 402 ← KV cache runs out, preemptions
512 170 290 360 370 368 ← high preemptions

The optimum in this example: max-num-seqs=64, max-num-batched-tokens=32768. Above that, preemptions cancel out the concurrency gain.

Confirmation with OTel

Once the optimal configuration is deployed, three Prometheus metrics confirm it is well calibrated:

# 1. Queued requests — should stay close to 0
# If it grows steadily: max-num-seqs too low or max-num-batched-tokens insufficient
vllm:num_waiting_seqs

# 2. Preemptions — should be 0 or very occasional (<1/min)
# If it grows: max-num-seqs too high for the available KV cache
rate(vllm:num_preemptions_total[5m]) * 60

# 3. ITL (inter-token latency) — should be stable, with no spikes
# Bimodality = badly calibrated batch size (some requests outside the CUDA graph bucket)
histogram_quantile(0.99, rate(vllm:time_per_output_token_seconds_bucket[5m]))

The optimal configuration produces:

num_waiting_seqs ≈ 0 under normal operation
num_preemptions_total stable (not growing)
time_per_output_token unimodal

If num_waiting_seqs is high with gpu_cache_usage_perc low: raise max-num-batched-tokens to process prefills faster. If num_preemptions_total grows: lower max-num-seqs or enable FP8 KV cache to free up blocks.

Reference configurations by profile

Based on the grid search for mid-range hardware (generic 4×H100, 14B-70B model):

Profile	Prompt p50	Output p50	max-num-seqs	max-num-batched-tokens
Conversational chat	150 tok	300 tok	256	16384
Enterprise RAG	1500 tok	200 tok	64	32768
Coding (completion)	800 tok	500 tok	128	32768
Summarisation	2500 tok	400 tok	32	65536
Batch processing	4000 tok	800 tok	16	65536

None of these is universal. They are starting points for the grid search on your real hardware and workload.

When not to touch the defaults

If your system sits below 50% KV cache utilisation (vllm:gpu_cache_usage_perc < 0.50) under real demand and with no num_waiting_seqs, the defaults are enough for your current load. The grid search contributes more when you are close to maximum capacity or when you want to extract the full performance of fixed hardware.

References

FP8 end to end: enable it, measure quality and decide with data

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

FP8 is the configuration change with the highest impact per unit of effort available on H100 and Ada Lovelace hardware. On an H100 it enables native FP8 tensor cores: +40-60% decode throughput and ×2 the VRAM available for the KV cache. On an RTX 4090 and an L40 the compute benefit is smaller, but the ×2 VRAM is real and translates directly into twice the concurrency. The risk is quality degradation, which on well-calibrated modern models is <0.5% on standard benchmarks but can be larger on formal reasoning. The correct workflow is not to switch it on and pray: it is to switch it on in staging, run the eval suite, correlate quality with throughput in OTel, and decide with data.

The analogy

A photographer who works with 35 mm negatives and moves to digital. Digital photos take up less space and are processed faster. But a low-resolution photo of a landscape can be indistinguishable from the high-resolution one to the human eye, while a low-resolution photo of text loses letters. Exactly the same trade-off applies to FP8: for tasks where numerical imprecision is averaged out over thousands of activations (conversation, summarisation, RAG), it is practically invisible. For tasks where a single wrong multiplication propagates into an incorrect answer (formal mathematics, critical code), it can be decisive.

The three layers of FP8 in vLLM

FP8 is not a single flag: it is three independent layers that are enabled separately and have different benefits.

Layer 1 — Model weights (--quantization fp8): The model weights are stored and computed in FP8 E4M3. The models have to be pre-quantised (available on HuggingFace with a -FP8 or -fp8 suffix) or quantised at load time with calibration. The benefit: the model takes up half the VRAM and the weight matmuls are 2× faster on an H100.

# Pre-quantised model (recommended for production)
vllm serve neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 \
 --quantization fp8

# Or on-the-fly quantisation (no extra files, somewhat slower on the first tokens)
vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
 --quantization fp8 \
 --kv-cache-dtype auto

Layer 2 — KV cache (--kv-cache-dtype fp8): The K and V tensors of the KV cache are stored in FP8 instead of BF16. It halves the size of the KV cache, doubling the number of tokens that fit in VRAM. It does not affect the model weights.

vllm serve my-model \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales # dynamic calibration, mandatory to minimise degradation

Layer 3 — Activations (automatic on an H100): On Hopper GPUs, vLLM automatically enables FP8 for the intermediate activations when both previous layers are active. No extra flag is required.

Full production configuration:

vllm serve neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 \
 --quantization fp8 \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales \
 --gpu-memory-utilization 0.92 \
 --max-model-len 16384

The measurable impact by hardware

H100 SXM (Hopper, native FP8 tensor cores)

Metric	BF16 baseline	FP8 enabled	Delta
Decode throughput (tok/s, 70B, batch 32)	~1,800	~2,700	+50%
Model VRAM (70B)	140 GB	70 GB	−50%
KV cache VRAM available (on 4×H100)	180 GB	250 GB	+39%
Maximum concurrency (ctx 8K)	~22,500 tok	~31,250 tok	+39%

This is equivalent to an extra replica for free in terms of KV cache capacity.

RTX 4090 (Ada Lovelace, FP8 in CUDA but without dedicated tensor cores)

Metric	BF16/Q4 baseline	FP8 KV cache added	Delta
Decode throughput (tok/s, 14B Q4)	~45	~47	+4%
KV cache VRAM available	15 GB	15 GB (same model)	—
Total cache tokens (ctx 8K)	~46,000	~92,000	+100%
Maximum concurrency (ctx 8K)	~5 users	~11 users	+120%

On Ada, the compute benefit is smaller (the FP8 tensor cores do not have the same width as on Hopper), but the ×2 in KV cache capacity is entirely real and translates into twice the possible concurrent users.

The correct workflow: enable, measure, decide

Enabling FP8 directly in production without validating quality is inadequate. The correct workflow has four steps.

Step 1: baseline in staging

Before enabling FP8, record the quality metrics of the current BF16 model. The most reproducible way is to run an eval suite over a fixed dataset and save the results:

# Install lm-evaluation-harness
pip install lm-eval

# BF16 baseline
lm_eval --model vllm \
 --model_args pretrained=meta-llama/Meta-Llama-3.1-70B-Instruct,dtype=bfloat16 \
 --tasks mmlu,hellaswag,gsm8k \
 --num_fewshot 5 \
 --output_path ./results/baseline_bf16.json

Step 2: enable FP8 and run the same eval suite

# FP8
lm_eval --model vllm \
 --model_args pretrained=neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8,quantization=fp8,kv_cache_dtype=fp8,calculate_kv_scales=true \
 --tasks mmlu,hellaswag,gsm8k \
 --num_fewshot 5 \
 --output_path ./results/fp8_full.json

Step 3: compute the degradation

# compare_eval.py
import json

with open("results/baseline_bf16.json") as f:
 baseline = json.load(f)
with open("results/fp8_full.json") as f:
 fp8 = json.load(f)

tasks = ["mmlu", "hellaswag", "gsm8k"]
print(f"{'Task':<15} {'BF16':>8} {'FP8':>8} {'Delta':>8} {'OK?':>6}")
print("-" * 50)
for task in tasks:
 b = baseline["results"][task]["acc,none"]
 f = fp8["results"][task]["acc,none"]
 delta = (f - b) / b * 100
 ok = "✓" if abs(delta) < 1.0 else "✗ REVIEW"
 print(f"{task:<15} {b:>8.3f} {f:>8.3f} {delta:>+7.1f}% {ok:>6}")

Decision thresholds documented in MLPerf Inference 2025:

< 0.5% degradation: enable in production without restrictions.
0.5% – 1.5%: enable with active quality monitoring via LLM-as-judge.
> 1.5%: investigate before enabling — a possible calibration problem or an incompatible model.

Step 4: domain eval with LLM-as-judge

Academic benchmarks measure what they measure. Your use case may be different. Adding 200 representative samples from your domain, evaluated by an LLM judge, closes the gap:

# domain_eval.py
from langfuse import Langfuse
from openai import OpenAI

client = Langfuse()
judge = OpenAI(base_url="http://judge-llm:8000/v1", api_key="token")

# Load the 200 curated production samples (prompt + expected answer)
samples = load_domain_samples("eval_dataset_200.json")

scores_bf16, scores_fp8 = [], []
for sample in samples:
 for model_type, endpoint in [("bf16", "http://staging-bf16:8000"), ("fp8", "http://staging-fp8:8000")]:
 response = call_model(endpoint, sample["prompt"])
 score = judge.chat.completions.create(
 model="Qwen/Qwen2.5-72B-Instruct",
 messages=[{
 "role": "user",
 "content": f"Rate this answer from 1 to 5 on accuracy and completeness.\nQuestion: {sample['prompt']}\nExpected answer: {sample['expected']}\nModel answer: {response}\n\nReply with a number from 1 to 5 only."
 }]
 ).choices[0].message.content.strip()

 if model_type == "bf16":
 scores_bf16.append(int(score))
 else:
 scores_fp8.append(int(score))

import numpy as np
print(f"Mean BF16 score: {np.mean(scores_bf16):.2f}")
print(f"Mean FP8 score: {np.mean(scores_fp8):.2f}")
print(f"Degradation: {(np.mean(scores_fp8)-np.mean(scores_bf16))/np.mean(scores_bf16)*100:.1f}%")

OTel + Langfuse correlation: the dashboard that decides

The moment of decision rests on a single dashboard with two signals on the same time axis:

Signal 1 — Throughput (Prometheus):

rate(vllm:generation_tokens_total[5m])

Signal 2 — Mean quality (Langfuse → Prometheus via exporter):

# If you have configured Langfuse with scores exported via OTel
langfuse_score_value{name="llm_judge_domain"}

The expected pattern after enabling FP8: throughput rises by 40-60% and quality stays within ±0.1 points. If quality drops by more than 0.3 points and stays low, there is a real problem.

# Alert: quality drops more than 0.2 points, sustained, after the change
ALERT FP8CalidadDegradada
 IF avg_over_time(langfuse_score_value{name="llm_judge_domain"}[30m])
 < (avg_over_time(langfuse_score_value{name="llm_judge_domain"}[1d] offset 2h) - 0.2)
 FOR 15m
 LABELS { severity = "warning" }
 ANNOTATIONS { summary = "Possible quality degradation after the FP8 configuration change" }

When NOT to enable FP8

FP8 is not always the right answer. The cases where degradation exceeds the acceptable threshold:

Formal mathematical reasoning: GSM8K and MATH are the benchmarks most sensitive to FP8. If your use case is solving mathematical problems or precise financial calculation, measure specifically on these benchmarks before enabling it.

Critical code with tests: numerical precision affects the probability of tokens at key positions in a function. The risk is not that the code “looks” bad, but that it passes superficial tests while having subtle bugs.

Very long contexts without --calculate-kv-scales: without dynamic scale calibration, the accumulated numerical error in the KV cache grows with the context. With --calculate-kv-scales active, the impact is minimal up to 32K tokens.

Small models (<7B): the FP8 conversion overhead can exceed the throughput benefit. The break-even point is around 7B parameters.

References

Instrumenting vLLM with OTel: measuring what the optimisations really do

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

vLLM exposes two independent observability signals: Prometheus metrics (pull, aggregated) and OTel traces (push, per request). To measure whether chunked prefill, prefix caching, speculative decoding, FP8 KV cache and concurrency are really working, you need both. The metrics tell you what is happening in the system; the traces tell you why one specific request was slow. This article configures the complete pipeline and maps each optimisation to its diagnostic metric.

The analogy

A Formula 1 driver and his telemetry engineers. The driver feels that the car “goes odd” in turn 3, but without the sensor data he does not know whether it is the rear tyre, the differential or the fuel. The engineers see exactly what happened in that turn, temperature per sensor, lateral load per millisecond.

vLLM without OTel is the driver on his own: you notice that the TTFT “looks high” but you do not know whether it is a long prefill, a prefix cache miss, or a KV cache preemption. With OTel you have the full picture: the metrics are the race summary (aggregated), the traces are the lap-by-lap telemetry (per-request).

Architecture of the two signals

vLLM deliberately separates its two observability channels:

 ┌─────────────────────────────┐
│ vLLM │
│ │
requests ────────────►│ SchedulerStats │
│ │ │
│ ├─► Prometheus /metrics │◄── scrape (pull)
│ │ (aggregated, ~15s) │
│ │ │
│ └─► OTLP exporter │──► push (per request)
│ (spans, immediate) │
└─────────────────────────────┘
│ OTLP gRPC/HTTP
▼
┌─────────────────────┐
│ OTel Collector │
│ │
│ receivers: │
│ otlp (traces) │
│ prometheus │
│ exporters: │
│ langfuse │
│ prometheus remote│
│ loki (logs) │
└─────────────────────┘

Prometheus pull exposes metrics with the vllm: prefix at :8000/metrics. They are histograms, gauges and counters updated on every scheduler iteration. Good for dashboards and alerts about the system as a whole.

OTLP push sends one span per request, immediately on completion. It contains attributes of the specific request: prompt tokens, generated tokens, TTFT, model. Good for debugging anomalous requests and for Langfuse.

Installation and basic configuration

# vLLM with OTel support
pip install "vllm[otel]"
# Installs: opentelemetry-sdk, opentelemetry-api,
# opentelemetry-exporter-otlp, opentelemetry-semantic-conventions-ai

# Start vLLM with OTel enabled
export OTEL_SERVICE_NAME="vllm-production"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://otel-collector:4317"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="grpc"
export OTEL_EXPORTER_OTLP_TRACES_INSECURE="true" # on an internal network without TLS

vllm serve Qwen/Qwen2.5-7B-Instruct \
 --otlp-traces-endpoint http://otel-collector:4317 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --speculative-model Qwen/Qwen2.5-0.5B-Instruct \
 --num-speculative-tokens 5

The Prometheus metrics need no extra configuration: they are always at :8000/metrics.

OTel Collector: minimum configuration

# otel-collector-config.yaml
receivers:
 otlp:
 protocols:
 grpc:
 endpoint: 0.0.0.0:4317
 http:
 endpoint: 0.0.0.0:4318

 prometheus:
 config:
 scrape_configs:
 - job_name: vllm
 scrape_interval: 15s
 static_configs:
 - targets: ["vllm:8000"]

processors:
 batch:
 timeout: 5s
 resource:
 attributes:
 - key: deployment.environment
 value: "production"
 action: upsert

exporters:
 otlphttp/langfuse:
 endpoint: "https://cloud.langfuse.com/api/public/otel"
 headers:
 Authorization: "Basic <base64(pk:sk)>"

 prometheusremotewrite:
 endpoint: "http://prometheus:9090/api/v1/write"

service:
 pipelines:
 traces:
 receivers: [otlp]
 processors: [batch, resource]
 exporters: [otlphttp/langfuse]
 metrics:
 receivers: [prometheus]
 processors: [batch]
 exporters: [prometheusremotewrite]

The five metrics that matter

Every optimisation has a primary diagnostic signal. If the metric does not move as expected after enabling the flag, there is a configuration or load problem.

1. Chunked prefill → `vllm:time_to_first_token_seconds`

Chunked prefill should reduce the variance of the TTFT, not necessarily the median. Its main goal is for the high percentiles (p99) to come down even if the p50 rises slightly.

# TTFT p50 and p99 — expect p99 to drop with chunked prefill enabled
histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))
histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))

Sign that it works: the p99/p50 ratio approaches 1. Without chunked prefill, one request’s long prefill blocks all the others and the p99 rises disproportionately.

Sign of a problem: p50 and p99 both rise. A --max-num-batched-tokens that is too low makes the chunks so small that the prefill takes many steps to complete even though the other requests are not blocked. Raise the budget.

It is also useful to look at the llm.usage.prompt_tokens attribute per span in the OTel traces: requests with many prompt tokens should have a proportional TTFT, not a blocking one.

2. Prefix caching → `vllm:gpu_prefix_cache_hit_rate`

# Prefix cache hit rate on the GPU (0.0–1.0)
vllm:gpu_prefix_cache_hit_rate

# Evolution over a 5-minute window
rate(vllm:gpu_prefix_cache_hit_rate[5m])

Sign that it works: a sustained hit rate > 0.5 in workloads with a shared system prompt. With a hit rate of 0.8, 80% of requests skip the prefix prefill; the TTFT of those requests falls to the cost of the variable suffix alone.

Sign of a problem: a hit rate close to zero despite system prompts that “look” identical. The usual causes:

# ❌ This breaks the prefix caching hash:
system_prompt = f"Today is {datetime.now()}. You are an assistant..."
# ^^^ a different timestamp on every request

# ✅ The system prompt has to be identical byte for byte:
system_prompt = "You are an assistant specialising in infrastructure..."

Any variation in the system prompt, timestamps, session IDs, interpolated prompt versions, produces a different hash and a cache miss. The OTel traces do not expose the hit/miss per request directly in the current implementation; use them to correlate a high llm.usage.prompt_tokens with a high TTFT in the same request.

3. Speculative decoding → `vllm:spec_decode_draft_acceptance_rate`

# Acceptance rate of the draft model (0.0–1.0)
vllm:spec_decode_draft_acceptance_rate

# Estimated effective speedup (with k=5 proposed tokens)
# speedup ≈ (1 + α·k) / (1 + overhead_draft)
# Simplified: if α=0.75 and k=5 → speedup ≈ 1 + 0.75×5×(1 - cost_ratio)

Sign that it works: a sustained acceptance rate > 0.70. Below 0.60, the draft model’s overhead outweighs the gain from the accepted tokens and speculative decoding is counterproductive.

Sign of a problem: an acceptance rate < 0.50. The usual causes:

A drafter from a different family than the verifier (for instance, Mistral 0.5B as the draft for Qwen 7B).
A high generation temperature (>0.9): the higher the temperature, the more the draft’s distribution diverges from the verifier’s.
A very large batch: at high concurrency, the draft can fall outside the domain of the current requests.

# Alert: inefficient speculative decoding
ALERT SpecDecodeInefficient
 IF vllm:spec_decode_draft_acceptance_rate < 0.60
 FOR 5m
 LABELS { severity = "warning" }
 ANNOTATIONS { summary = "Draft acceptance rate low: disable spec decode or change the drafter" }

In the OTel traces, the complete span of the request includes the total decode time. Without an acceptance rate per span, the way to detect that spec decode is working is to compare the total decode time divided by the tokens generated: if it is significantly lower than the baseline without spec decode, it is helping.

4. FP8 KV cache and concurrency → `vllm:gpu_cache_usage_perc` + `vllm:num_preemptions_total`

These two metrics are the two sides of KV cache management:

# KV cache utilisation (0.0–1.0)
# With FP8 enabled, the same hardware supports more requests before saturating
vllm:gpu_cache_usage_perc

# Cumulative preemptions (counter)
# Rises when vLLM cannot fit more requests and pauses one
rate(vllm:num_preemptions_total[5m])

Sign that FP8 works: with --kv-cache-dtype fp8 enabled, gpu_cache_usage_perc should saturate at concurrency levels around 2× higher than the BF16 baseline before num_preemptions_total starts to grow.

Sign of a problem: num_preemptions_total growing at rates > 1/minute with gpu_cache_usage_perc below 0.90. It indicates that max-num-seqs is too high for the available KV cache: requests enter the system but there are no free blocks to assign to them. Lower max-num-seqs or reduce max-model-len.

# Alert: KV cache saturated with preemptions
ALERT KVCacheSaturated
 IF rate(vllm:num_preemptions_total[2m]) > 0.5
 AND vllm:gpu_cache_usage_perc > 0.85
 FOR 3m
 LABELS { severity = "critical" }
 ANNOTATIONS { summary = "KV cache saturated: lower max-num-seqs or max-model-len" }

The impact of FP8 on capacity can be quantified:

$$\Delta\text{capacity} = \frac{\text{max tokens FP8}}{\text{max tokens BF16}} \approx 2\times$$

Measure before and after enabling --kv-cache-dtype fp8: the level of gpu_cache_usage_perc for a given concurrency should fall by half.

5. Effective concurrency → `vllm:num_running_seqs` + `vllm:num_waiting_seqs`

# Requests active in the engine (decode + prefill in progress)
vllm:num_running_seqs

# Requests queued waiting for a slot
vllm:num_waiting_seqs

# Waiting ratio: if > 0.2 sustained, there is a concurrency bottleneck
vllm:num_waiting_seqs / (vllm:num_running_seqs + vllm:num_waiting_seqs)

Healthy sign: num_running_seqs stable close to the configured --max-num-seqs value, num_waiting_seqs low (< 10% of running).

Sign of a problem: a high num_waiting_seqs with a low gpu_cache_usage_perc. It indicates that the scheduler is not filling the available slots because max-num-batched-tokens is too low: the token budget per step does not allow the pending prefills to be processed fast enough. Raise max-num-batched-tokens.

Reference dashboard: the 5 metrics in Grafana

{
 "panels": [
 {
 "title": "TTFT p50 / p99 (chunked prefill)",
 "targets": [
 {"expr": "histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))", "legendFormat": "p50"},
 {"expr": "histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))", "legendFormat": "p99"}
 ]
 },
 {
 "title": "Prefix cache hit rate",
 "targets": [{"expr": "vllm:gpu_prefix_cache_hit_rate", "legendFormat": "GPU hit rate"}]
 },
 {
 "title": "Spec decode acceptance rate",
 "targets": [{"expr": "vllm:spec_decode_draft_acceptance_rate", "legendFormat": "acceptance rate"}]
 },
 {
 "title": "KV cache usage + preemptions",
 "targets": [
 {"expr": "vllm:gpu_cache_usage_perc", "legendFormat": "cache usage"},
 {"expr": "rate(vllm:num_preemptions_total[2m]) * 60", "legendFormat": "preemptions/min"}
 ]
 },
 {
 "title": "Effective concurrency",
 "targets": [
 {"expr": "vllm:num_running_seqs", "legendFormat": "running"},
 {"expr": "vllm:num_waiting_seqs", "legendFormat": "waiting"}
 ]
 }
 ]
}

Connecting traces to Langfuse

vLLM’s OTel traces are GenAI semconv compatible spans. Langfuse accepts them directly via OTLP:

# In the OTel Collector (already configured above)
# The otlphttp/langfuse exporter sends traces to Langfuse Cloud or self-hosted

# For self-hosted Langfuse (ENS/sovereign):
exporters:
 otlphttp/langfuse:
 endpoint: "http://langfuse-internal:3000/api/public/otel"
 headers:
 Authorization: "Basic <base64(pk_xxx:sk_xxx)>"

In Langfuse, each vLLM request appears as a trace with:

gen_ai.system: the model served
gen_ai.usage.input_tokens: prompt tokens
gen_ai.usage.output_tokens: generated tokens
Span duration: end-to-end latency

What does not appear directly in the span: the speculative decoding acceptance rate, the prefix cache hit/miss, or the number of preemptions. That data is only in Prometheus. The correct workflow is:

Langfuse identifies an anomalous request by latency.
Prometheus/Grafana shows whether in that interval there were high preemptions, low spec decode, or a prefix cache miss.
The two are correlated by timestamp.

Quick diagnostic matrix

Observable symptom	Prometheus metric	Probable cause	Action
TTFT p99 very high	`ttft p99/p50 >> 2`	Long blocking prefills	Raise `--max-num-batched-tokens`
TTFT p50 high, p99 the same	`ttft p50 > 500ms`	Prefix cache not working	Check the system prompt hash
Slow decode with no improvement	`spec_decode_acceptance < 0.60`	Incompatible drafter	Change the drafter or disable it
Sporadic OOM / crash	`gpu_cache_usage_perc = 1.0` + preemptions	KV cache full	Lower `max-num-seqs` or enable FP8
High queue with free cache	`waiting >> 0` + `cache < 0.70`	Low token budget	Raise `--max-num-batched-tokens`

Implications for sovereign on-premise inference

In an ENS deployment where you cannot use Langfuse Cloud or DataDog, the complete self-hosted stack is:

# docker-compose.yml (or equivalent K8s manifests)
services:
 otel-collector:
 image: otel/opentelemetry-collector-contrib:latest
 volumes: [./otel-config.yaml:/etc/otel/config.yaml]

 langfuse:
 image: langfuse/langfuse:latest
 environment:
 DATABASE_URL: postgres://...

 prometheus:
 image: prom/prometheus:latest

 grafana:
 image: grafana/grafana:latest

The whole pipeline runs on-premise. The traces never leave the perimeter. ENS compliance does not depend on which observability you choose: it depends on the inference data not going out to third parties. With a local stack, both conditions are met.

References

One big replica or many small ones: the decision that defines your platform

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

With 4 GPUs available you have two basic options: one vLLM instance using all 4 (TP=4) or two independent instances using 2 each (TP=2 × 2 replicas). The first gives lower latency per individual request. The second gives higher aggregate throughput at high concurrency, better fault tolerance and finer-grained scaling. The crossover point, where the second overtakes the first, is typically between 16 and 64 concurrent requests for 70B models, much earlier than most people assume. The metric that decides it: goodput, the tokens generated within the latency SLO divided by the total.

The analogy

Two ways to organise a translation service: one senior translator with access to four specialised dictionaries at once (able to resolve any complex query in 30 seconds), or two junior translators with two dictionaries each (they take 45 seconds per complex query, but they can handle two at the same time).

For a client who arrives alone and expects a fast answer, the senior wins. For a queue of twenty clients arriving at once, the two juniors process twice as many queries per hour even though each one takes longer. The question is not who is better, but what kind of traffic you have.

The two architectures in vLLM

Architecture A: TP=4, one replica

# A single instance uses all 4 GPUs via tensor parallelism
vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
 --tensor-parallel-size 4 \
 --gpu-memory-utilization 0.92 \
 --port 8000

GPU-0 ─┐
GPU-1 ─┤─ vLLM instance 0 ──► port 8000
GPU-2 ─┤ (TP=4, the model is
GPU-3 ─┘ split across 4 GPUs)

Every attention and FFN operation is split across 4 GPUs. They require an all-reduce after each layer (on NVLink: ~50-200 µs; on PCIe: ~2-8 ms). The complete model is available in the aggregate VRAM.

Architecture B: TP=2 × 2 replicas

# Two independent instances, each with 2 GPUs
# Instance 0 on GPU 0-1
CUDA_VISIBLE_DEVICES=0,1 vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
 --tensor-parallel-size 2 --port 8000

# Instance 1 on GPU 2-3
CUDA_VISIBLE_DEVICES=2,3 vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
 --tensor-parallel-size 2 --port 8001

GPU-0 ─┐ ┌─► port 8000
GPU-1 ─┘─ vLLM instance 0 ─┘
← load balancer
GPU-2 ─┐ ┌─► port 8001
GPU-3 ─┘─ vLLM instance 1 ─┘

Each instance holds half the model. Requests are distributed across instances. No communication between instances (they are completely independent).

Why TP=4 has higher individual latency than TP=2

Tensor parallelism splits every transformer layer. After computing its fraction, each GPU needs to synchronise with the others via all-reduce before moving on to the next layer. The cost of that synchronisation:

$$\text{overhead TP} = n\_layers \times 2 \times \text{allreduce latency}$$

For Llama 3 70B (80 layers) on 4×H100 NVLink:

$$\text{overhead TP4} = 80 \times 2 \times 100\,\mu s = 16\,ms$$

On PCIe (without direct NVLink between GPUs):

$$\text{overhead TP4 PCIe} = 80 \times 2 \times 3\,ms = 480\,ms$$

That overhead is added to every decode step. With TP=2:

$$\text{overhead TP2} = 80 \times 2 \times 60\,\mu s = 9.6\,ms \text{ (NVLink)}$$

The difference between TP=2 and TP=4 on NVLink is ~6 ms per decode step, which matters for TPOT (inter-token latency) in streaming applications.

On PCIe without direct NVLink, TP=4 can be 400 ms slower per step than TP=2. For an output of 200 tokens, that is 80 extra seconds. In that scenario TP=4 over PCIe should never be used unless the model does not fit in 2 GPUs.

The crossover point: when TP=2×2 beats TP=4×1

For a 70B model on 4×H100 SXM (NVLink), aggregate throughput in tokens per second:

Concurrency | TP=4 × 1 instance | TP=2 × 2 instances | Winner
──────────────┼──────────────────────┼───────────────────────┼────────
1 | 200 tok/s | 170 tok/s | TP=4 (latency)
4 | 650 tok/s | 620 tok/s | TP=4 (slight)
16 | 1,800 tok/s | 2,100 tok/s | TP=2×2
32 | 2,400 tok/s | 3,600 tok/s | TP=2×2 (+50%)
64 | 2,800 tok/s | 5,200 tok/s | TP=2×2 (+86%)
128 | 2,900 tok/s | 5,800 tok/s | TP=2×2 (+100%)

Why they diverge at high concurrency: with TP=4, the scheduler of a single instance manages every request but the KV cache is shared. With TP=2×2, each instance has its own scheduler and KV cache: less contention, more real parallelism.

The crossover point on NVLink sits around 16-32 simultaneous requests for 70B. For smaller models (14B, 7B) the crossover happens earlier, because the TP communication overhead weighs more in relative terms.

The three implications nobody mentions

1. Fault tolerance

With TP=4 × 1 replica: if one GPU fails, the whole instance goes down. The service drops to 0% until the GPU recovers or the pod restarts on another node.

With TP=2 × 2 replicas: if one GPU fails, one instance goes down. The service keeps running at 50% capacity. For ENS/NIS2, where availability is a contractual requirement, this difference is decisive.

2. Autoscaling granularity

With KEDA or HPA based on vllm:num_waiting_seqs, autoscaling has to provision in multiples of the deploy unit:

TP=4 × 1: each new node requires 4 GPUs. The minimum scaling granularity is 4 GPUs.
TP=2 × 2: each new pod requires 2 GPUs. The minimum granularity is 2 GPUs, which is finer and more cost-efficient.

3. Quality degradation under load

TP=4 with many concurrent requests starts to suffer preemptions when the KV cache fills up. TP=2×2 spreads that pressure across two independent KV cache pools, so the probability of preemption is lower under the same total load.

Measuring the crossover point with OTel

Goodput is the right metric for comparing the two architectures. Not raw throughput (which ignores the SLO), but the tokens generated within the agreed TPOT SLO:

# Goodput: tokens generated with TPOT inside the SLO (e.g. <50ms/token)
# For TP=4:
rate(vllm:generation_tokens_total{instance="tp4"}[5m])
 * (1 - histogram_quantile(0.95, rate(vllm:time_per_output_token_seconds_bucket{instance="tp4"}[5m])) > 0.050)

# For TP=2×2 (sum of the two instances):
sum(rate(vllm:generation_tokens_total{instance=~"tp2-.*"}[5m]))
 * (1 - histogram_quantile(0.95, sum(rate(vllm:time_per_output_token_seconds_bucket{instance=~"tp2-.*"}[5m]))) > 0.050)

A direct comparison on the same dashboard, with synthetic traffic at different concurrency levels, determines the exact crossover point for your hardware and model.

The decision by workload profile

Profile	Recommended architecture	Reason
Single-user chatbot / low concurrency (<10 simultaneous)	TP=4 × 1	Lower p50 latency, better streaming experience
Enterprise API (20-100 concurrent)	TP=2 × 2	Higher goodput, fault tolerance, finer autoscaling
Batch processing (throughput > latency)	TP=2 × 2 (or more replicas)	Maximum throughput always lies in replicas
Very large model (>80B, does not fit in 2 GPUs)	TP=4 × 1	No structural alternative
ENS / contractual availability	TP=2 × 2 minimum	Losing one GPU is not catastrophic

Kubernetes configuration for both architectures

# Parallel deployments for an A/B test or different topologies

# TP=2 instances (2 replicas per deployment)
apiVersion: apps/v1
kind: Deployment
metadata:
 name: vllm-tp2
spec:
 replicas: 2
 template:
 spec:
 containers:
 - name: vllm
 args: ["serve", "meta-llama/Meta-Llama-3.1-70B-Instruct",
 "--tensor-parallel-size", "2",
 "--gpu-memory-utilization", "0.92"]
 resources:
 limits:
 nvidia.com/gpu: "2" # 2 GPUs per pod
---
# Service with load balancing across the 2 replicas
apiVersion: v1
kind: Service
metadata:
 name: vllm-tp2
spec:
 selector:
 app: vllm-tp2
 ports:
 - port: 8000
 sessionAffinity: ClientIP  # for prefix cache awareness
 sessionAffinityConfig:
 clientIP:
 timeoutSeconds: 10800 # 3 hours of affinity per session

The sessionAffinity: ClientIP setting in the Kubernetes Service is the simplest way to implement session-affinity routing: requests from the same client always go to the same replica, maximising the prefix cache hit rate on the conversation history.

References

Optimising decode in vLLM: squeezing every token out of small hardware

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

Decode is the phase in which vLLM generates output tokens one at a time. It is memory-bound, not compute-bound: the GPU spends more time waiting for the weights to arrive from VRAM than doing arithmetic. On small hardware, an RTX 4090 (24 GB) or an L40 (48 GB), a badly configured decode wastes half the card’s capacity. Five vLLM parameters change the equation: gpu-memory-utilization, max-num-seqs, speculative decoding, FP8 KV cache and a swap-space correctly set to zero. Well calibrated, the difference is real: from 15 tokens/s to 35–50 tokens/s on the same hardware.

The analogy

An assembly-line worker putting cars together. Every car needs exactly the same process: walk to the store for the part, come back, bolt it on, repeat. The travel time to the store, the VRAM latency, is fixed and cannot be eliminated. But there are ways to make it less painful:

Have several cars in parallel on the line (more concurrency, the same travel time amortised).
Have an assistant prefabricate common parts (speculative decoding: the draft model proposes, the verifier confirms).
Keep only the most-used parts in the workshop (quantised KV cache: more contexts fit in the same space).

Those three strategies are exactly the three axes of decode optimisation in vLLM.

Why decode is memory-bound

During prefill, the GPU processes N tokens in parallel: the attention operation is a large matmul and the compute units are busy. During decode, it processes 1 token per step: the matmul becomes a vector-matrix product, an operation that underuses the tensor cores.

The typical compute utilisation ratio during decode on an RTX 4090:

$$\text{MFU}_{decode} \approx 5–15\% \quad \text{(vs 40–60\% in prefill)}$$

The bottleneck is not compute power, it is bandwidth. To generate each token, the model has to read its complete weights from VRAM:

$$\text{time per token} \approx \frac{\text{weight size in bytes}}{\text{VRAM bandwidth}}$$

For Qwen2.5-7B in BF16 (14 GB of weights) on an RTX 4090 (1,008 GB/s):

$$t \approx \frac{14 \times 10^9}{1.008 \times 10^{12}} \approx 13.9 \text{ ms/token} \approx 72 \text{ tokens/s theoretical maximum}$$

The real value is lower (~30–50 tok/s) because of scheduler overhead, attention over the growing KV cache and other latencies. But the theoretical limit sets the ceiling.

With Q4_K_M (weights ~4 GB):

$$t \approx \frac{4 \times 10^9}{1.008 \times 10^{12}} \approx 3.97 \text{ ms/token} \approx 252 \text{ tokens/s theoretical}$$

Quantising the model is the most direct way to improve decode throughput on memory-bound hardware. Everything else optimises on top of that ceiling.

The five levers

1. Give vLLM all the VRAM you can — `--gpu-memory-utilization`

--gpu-memory-utilization (short form --gpu-mem-util) defines the fraction of available VRAM that vLLM may use for the KV cache, once the model weights are loaded. The rest is reserved for activations during the forward pass and for the CUDA context.

vllm serve my-model \
 --gpu-memory-utilization 0.92

The default value is 0.90. On bare metal where no other process uses the GPU, 0.92–0.95 is safe. Do not go above 0.95: vLLM needs headroom for activations during batch peaks, and running out of VRAM in the middle of an inference results in a process crash, not a clean error.

Why it matters: more KV cache available = more simultaneous requests in flight = better GPU utilisation during decode. PagedAttention allocates the KV cache in fixed-size blocks (16 tokens per block by default), and vLLM manages them like virtual memory pages. The more blocks available, the more requests it can serve without any of them waiting for space.

RTX 4090, Qwen2.5-7B-BF16 (14 GB weights):
Total VRAM: 24 GB
Weights: 14 GB
Available for KV cache: 10 GB
gpu-memory-utilization 0.90 → 0.90 × 10 GB = 9 GB for KV cache
gpu-memory-utilization 0.94 → 0.94 × 10 GB = 9.4 GB → ~4% more tokens in flight

The impact is modest with models that fit comfortably, but it is amplified with models that push VRAM to its limit.

2. Real concurrency — `--max-num-seqs`

--max-num-seqs is the maximum number of requests vLLM can have in process simultaneously (counting prefill and decode together). It is the parameter that controls the system’s effective concurrency.

vllm serve my-model \
 --max-num-seqs 128

The effect is direct: more requests in simultaneous decode = better amortisation of the fixed cost of reading the weights. When the decode batch grows from 1 to 8, the time to generate 8 tokens is almost the same as generating 1 (the weights are read once for all of them). Aggregate throughput scales almost linearly until the KV cache or the activation VRAM becomes the bottleneck.

$$\text{aggregate throughput}(B) \approx B \times \text{throughput}(1) \quad \text{for } B \ll B_{max}$$

Common mistake: raising --max-num-seqs without making sure there is enough KV cache in VRAM for all the requests. If vLLM cannot hold the KV caches of 128 simultaneous requests, it does preemption (pauses some request and frees its KV cache) at a latency cost. Monitor vllm:num_preemptions_total.

Interaction with --max-num-batched-tokens: the vLLM scheduler processes up to max-num-batched-tokens tokens per step. If you have 128 requests in decode generating 1 token each, that is 128 decode tokens. The decode budget consumes 128 tokens of the total budget; the rest goes to prefill in chunks. Tune both values together.

# For an RTX 4090 serving ~50 concurrent users with responses of up to 512 tokens
vllm serve my-model \
 --max-num-seqs 64 \
 --max-num-batched-tokens 8192
# 64 decode tokens per step + up to 8128 chunked prefill tokens

3. Speculative decoding — `--speculative-model` + `--num-speculative-tokens`

Speculative decoding is the highest-impact change for decode on small hardware. The idea is simple: a small draft model proposes several tokens at once, and the verifier model validates or rejects them in a single forward pass.

vllm serve Qwen/Qwen2.5-7B-Instruct \
 --speculative-model Qwen/Qwen2.5-0.5B-Instruct \
 --num-speculative-tokens 5 \
 --speculative-draft-tensor-parallel-size 1

Why it works: the 7B verifier has to read 14 GB of weights per step. With 5 proposed tokens, if the acceptance rate is 80%, an average of 4–5 tokens are generated per verifier step instead of 1. Effective throughput goes up without the GPU working any harder.

The acceptance rate (α) depends on how well the draft predicts the verifier’s distribution. Within the same domain, models from the same family usually have α > 0.75:

$$\text{speedup} \approx \frac{1 + \alpha \cdot k}{1 + \alpha \cdot k / \text{cost ratio}}$$

Where $k$ is the number of proposed tokens and cost_ratio is the draft/verifier cost ratio. For a 0.5B draft and a 7B verifier (ratio ~14×):

$$\text{speedup} \approx 1 + 0.8 \times 5 \approx 5 \text{ (theoretical maximum, not reachable)}$$

In practice, with α = 0.75 and k = 5 on hardware without NVLink: 1.8–2.5× more tokens/s compared with decode alone.

EAGLE-3 in 2026: the best drafters today are not small versions of the same model, but networks specialised in predicting the verifier’s distribution. EAGLE-3 reports a 3–6.5× speedup over vanilla decode in public benchmarks. In production with mixed batches the real speedup is more conservative (1.5–3×). vLLM supports EAGLE/EAGLE-2 via --speculative-model:

# With an EAGLE drafter (requires a drafter trained specifically for the base model)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
 --speculative-model yuhuili/EAGLE3-LLaMA3.1-Instruct-8B \
 --num-speculative-tokens 6

When speculative decoding does NOT help:

Very large batches (>32 requests): the acceptance rate varies between requests and the batch spends more time re-drafting than accepting.
High-entropy tasks (brainstorming, very creative code): the draft predicts worse, α falls below 0.5 and the draft overhead outweighs the gain.
If the draft model does not fit in the available VRAM alongside the verifier.

On an RTX 4090 with a 7B verifier and a 0.5B draft (BF16): 14 + 1 GB = 15 GB. That leaves 9 GB for the KV cache. It works.

4. Quantised KV cache — `--kv-cache-dtype fp8`

This was already covered in the prefill article for its effect on context capacity. From the decode point of view, the benefit is different: more tokens fit in the KV cache → more simultaneous requests without preemption → better aggregate throughput.

vllm serve my-model \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales

Precision warning in decode: the KV cache is read on every attention step of the decode. Quantisation introduces noise into the attention activations. For long texts (>4K tokens of context) it can accumulate. In quality benchmarks (MMLU, HellaSwag) the degradation with FP8 KV and --calculate-kv-scales is <0.5% on modern models. Without --calculate-kv-scales, the degradation can be larger because the scales are fixed statically.

Optimal combination for RTX 4090:

vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \
 --quantization awq \  # weights in INT4: 4 GB model
 --kv-cache-dtype fp8 \  # KV cache at half the size
 --calculate-kv-scales \
 --gpu-memory-utilization 0.94

# Available VRAM: 24 - 4 = 20 GB for KV cache
# With FP8: ~40 KB/token (vs 80 KB BF16) → 20 GB / 40 KB = 500,000 tokens of total context
# With max-num-seqs 64 and a 4K ctx: 64 × 4096 × 40KB = 10 GB → fits with room to spare

5. Remove the swap — `--swap-space 0`

--swap-space defines how much system RAM (not VRAM) vLLM may use to preempt KV caches. When vLLM has more active requests than fit in VRAM, it can “pause” some by moving their KV cache to RAM and reactivating them later.

The problem: moving a 4K-token KV cache from VRAM to RAM and back has a latency of tens of milliseconds over PCIe. For a system where you want predictable latency, the swap introduces unacceptable jitter.

vllm serve my-model \
 --swap-space 0

With --swap-space 0, when vLLM cannot hold any more requests in VRAM, it simply queues them instead of doing preemption. The queue adds waiting latency, but it is predictable and it does not interrupt the requests already in flight.

When should you have swap? If your workload has short demand peaks and you can tolerate occasional jitter in exchange for not rejecting requests, a swap of 4–8 GB can be useful. In ENS deployments where latency is a contractual SLA, --swap-space 0 is the right option.

The reference configuration per hardware

RTX 4090 (24 GB) — 7B model, internal use

vllm serve Qwen/Qwen2.5-7B-Instruct \
 --gpu-memory-utilization 0.92 \
 --max-model-len 8192 \
 --max-num-seqs 64 \
 --max-num-batched-tokens 8192 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales \
 --swap-space 0 \
 --speculative-model Qwen/Qwen2.5-0.5B-Instruct \
 --num-speculative-tokens 5 \
 --speculative-draft-tensor-parallel-size 1 \
 --dtype bfloat16

Expected throughput: 35–55 tokens/s per user, up to 64 simultaneous, TTFT <500ms for prompts <1K tokens.

L40 (48 GB) — 14B model, multi-user

vllm serve Qwen/Qwen2.5-14B-Instruct \
 --gpu-memory-utilization 0.90 \
 --max-model-len 16384 \
 --max-num-seqs 128 \
 --max-num-batched-tokens 16384 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales \
 --swap-space 0 \
 --speculative-model Qwen/Qwen2.5-1.5B-Instruct \
 --num-speculative-tokens 5 \
 --dtype bfloat16

Expected throughput: 25–40 tokens/s per user, up to 128 simultaneous with speculative decoding active, TTFT <800ms for prompts <2K tokens.

How to measure that decode is optimised

# Key metrics at vllm:8000/metrics

vllm:generation_tokens_total # total tokens generated → trend
vllm:e2e_request_latency_seconds_* # end-to-end latency per percentile
vllm:time_per_output_token_seconds_* # ITL (inter-token latency)
vllm:num_preemptions_total # if it rises, the KV cache is filling up
vllm:spec_decode_draft_acceptance_rate # speculative decoding hit rate

If spec_decode_draft_acceptance_rate < 0.6, the drafter is not helping: turn speculative decoding off or find a drafter better trained for your model or domain.

If num_preemptions_total grows, you have too many simultaneous requests for the available KV cache. Options: lower max-num-seqs, enable FP8 KV cache, lower max-model-len, or quantise the model further.

Implications for on-premise inference

In a sovereign deployment with fixed hardware, you cannot buy more GPUs at will. Every well-calibrated tenth of gpu-memory-utilization, every point of speculative decoding acceptance rate and every MB of KV cache freed by FP8 is real capacity you do not have to provision with another node.

The combination of quantised weights (AWQ/GPTQ), FP8 KV cache and speculative decoding lets a 14B serve on one L40 what without optimisations would need two L40s in tensor parallel. That is the economic argument for investing time in these parameters.

Decode cannot be accelerated indefinitely on memory-bound hardware: the theoretical limit is set by VRAM bandwidth. But the difference between the minimum and the maximum achievable on that hardware can be 3–4× with the right levers.

References

Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023
EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees — Li et al., 2024
Fast Inference from Transformers via Speculative Decoding — Leviathan et al., 2022
vLLM Optimization and Tuning — documentación oficial
vLLM Speculative Decoding
FP8 KV Cache en vLLM

Optimising prefill in vLLM: the knobs your TTFT will not forgive

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

Prefill is the phase in which vLLM processes your input prompt and produces the initial KV cache. It is compute-bound (unlike decode, which is memory-bound), it takes longer the longer the prompt is, and it blocks the decode of every other queued request. There are four levers in vLLM that radically change its behaviour: chunked prefill, prefix caching, FP8 KV cache and the per-batch token budget. With modest hardware, a 24 GB RTX 4090 or a 48 GB L40, the difference between ignoring them and using them well can be a 3× lower TTFT and 40% more aggregate throughput.

The analogy

Picture a printing shop from the early twentieth century. Setting the lead type (preparing the forme) is slow and blocks the press. Printing the pages already set is fast, but it needs the forme ready before it can start.

Prefill is setting the type. Decode is printing. A press that can only do one thing at a time, either set or print, leaves the machinery idle half the time. The historical solution was to have one worker setting the next page while the previous one was already on the press. That is, exactly, chunked prefill.

What prefill really is

When a request arrives at vLLM, the engine has to process every token of the prompt at once before it can emit the first token of the response. During that processing it computes, for each prompt token, its attention Key and Value vectors. The result, the initial KV cache, is stored in VRAM and used throughout the subsequent decode.

Unlike decode, where the model processes one new token per step, in prefill it processes N tokens in one go. That makes it far more efficient in FLOPs/token (GPUs are good at large matmuls), but it carries a quadratic cost in attention:

$$\text{prefill attention FLOPs} \approx 4 \cdot N^2 \cdot d_{model}$$

With a prompt of 1,000 tokens and $d_{model} = 4096$ (Qwen2.5-7B): $4 \cdot 10^6 \cdot 4096 \approx 16 \times 10^9$ FLOPs in attention alone. With 4,000 tokens, 256× more, because of the quadratic nature.

Prefill (compute-bound):
prompt tokens → [attention O(N²)] → [FFN] → initial KV cache
Decode (memory-bound):
1 new token → [cross-attention over KV cache] → next token

Why prefill is a problem on small hardware

On an H100 with 3.35 TB/s of bandwidth, a long prefill amortises quickly. On an RTX 4090 (1.008 TB/s) or an L40 (864 GB/s), the bottleneck shows up sooner and has concrete consequences:

The head-of-line blocking problem. By default, vLLM processes a complete prefill before doing any decode. If you have 10 requests queued, 9 in decode and 1 with an 8,000-token prompt, those 9 requests stall while the GPU chews through the long prefill. Their users see the streaming freeze. This is called head-of-line blocking and it is the number one enemy of TTFT in production.

The four levers

1. Chunked prefill — `--enable-chunked-prefill` + `--max-num-batched-tokens`

Chunked prefill splits a long prefill into pieces (chunks) and interleaves them with decode steps in the same batch. In vLLM V1 (≥ 0.6) it is on by default.

vllm serve my-model \
 --enable-chunked-prefill \
 --max-num-batched-tokens 4096

--max-num-batched-tokens is the total token budget vLLM can process in a single engine step, counting prefill and decode together. It is the most important parameter for controlling the trade-off:

`max-num-batched-tokens`	Effect
Low (512–2048)	More decode steps per cycle → better ITL, worse TTFT
High (8192–32768)	Large prefill chunks → better TTFT and throughput, worse ITL

For an RTX 4090 serving 7B–13B models with mixed contexts (256–4096 tokens):

--max-num-batched-tokens 8192 # a reasonable balance point

For an L40 (48 GB) with larger models and longer prompts:

--max-num-batched-tokens 16384

How it works internally: with a budget of 4,096 tokens and a prefill of 10,000, vLLM splits it into 3 chunks (4,096 + 4,096 + 1,808). Between chunks, it processes the pending decode steps. The requests in decode keep advancing; the long prefill takes longer to finish, but it freezes nothing.

Without chunked prefill:
t=0 [prefill 10k tokens]─────────────────────────────┐
t=1 └─[decode r1,r2...r9]
With chunked prefill (budget 4096):
t=0 [prefill chunk 4096][decode r1..r9]
t=1 [prefill chunk 4096][decode r1..r9]
t=2 [prefill chunk 1808][decode r1..r9]
t=3 [decode all, including the new one]

The TTFT of the long request rises slightly (3 steps instead of 1), but the ITL of the other 9 requests is not interrupted.

2. Prefix caching — `--enable-prefix-caching`

If several requests share the same prefix, a system prompt, a few-shot block, a context document, vLLM can compute the KV cache of that prefix once and reuse it.

vllm serve my-model \
 --enable-prefix-caching

This is called Automatic Prefix Caching (APC). Internally, vLLM splits the KV cache into fixed-size blocks (16 tokens per block by default) and assigns them a content-based SHA hash. When a new request arrives, it checks whether any initial block is already in the cache. On a hit, that prefill is skipped.

The numerical impact: suppose a 512-token system prompt that appears in 80% of your requests, and a rate of 100 req/min:

Without APC: 80 req/min × 512 tokens × prefill_cost = 41,000 tokens/min of redundant prefill
With APC (80% hit rate): 20 req/min × 512 = 10,240 tokens/min of prefill → a 75% reduction

The TTFT of those 80 requests falls to whatever it costs to process only the new suffix.

Limitation with chunked prefill: when chunked prefill is active, only the first chunk of the prefill benefits from APC in the current vLLM implementation. For workloads where the cache hit rate is very high and the suffixes are short, consider lowering --max-num-batched-tokens so that the first chunk covers more of the shared prefix.

# Configuration optimised for a high prefix cache hit rate
vllm serve my-model \
 --enable-prefix-caching \
 --enable-chunked-prefill \
 --max-num-batched-tokens 4096 # smaller chunks = prefix fits in chunk 1

3. FP8 KV cache — `--kv-cache-dtype fp8`

The KV cache takes up VRAM. The more VRAM it consumes, the fewer concurrent requests you can keep in flight. On a 24 GB RTX 4090, the Qwen2.5-14B model in BF16 already takes ~28 GB, so it does not fit. In Q4 it takes ~9 GB, leaving ~14 GB for the KV cache.

How many context tokens fit in 14 GB of BF16 KV cache for a 14B with GQA?

$$\text{KV size per token} = 2 \cdot n_{kv\_heads} \cdot d_{head} \cdot n_{layers} \cdot 2 \text{ bytes}$$

For Qwen2.5-14B: $n_{kv\_heads}=8$, $d_{head}=128$, $n_{layers}=40$, BF16 → $2 \cdot 8 \cdot 128 \cdot 40 \cdot 2 = 163{,}840$ bytes ≈ 160 KB/token.

14 GB / 160 KB ≈ 87,500 tokens of total context. With 8 users in parallel and 4,096 tokens of context each: 32,768 tokens used out of 87,500. There is headroom, but it is finite.

Moving to FP8 (1 byte instead of 2):

$$\text{KV FP8} = 80 \text{ KB/token} \implies 14 \text{ GB} / 80 \text{ KB} = 175{,}000 \text{ tokens}$$

Twice the context capacity with the same VRAM. That buys either more concurrency or longer contexts.

vllm serve my-model \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales # calibrates the scales dynamically; without this there is degradation

Warning for RTX 4090 and L40: Ada Lovelace has FP8 instructions at the CUDA level but without Hopper’s (H100’s) dedicated scaling hardware. The memory reduction is real; the compute speedup is smaller than on an H100. Do not expect the same speedup as in a Hopper datacenter. On the L40S (the variant with optimised FP8 tensor cores) the benefit is larger than on an RTX 4090.

4. Context budget — `--max-model-len`

--max-model-len defines the maximum number of tokens vLLM can handle in a single request (prompt plus generation). It is the hard limit that determines how much VRAM is reserved for the KV cache in the worst case.

On small hardware, lowering it frees VRAM for more concurrency:

# 7B model on RTX 4090, typical context of 4K but the model supports 128K
vllm serve my-model \
 --max-model-len 8192 \  # instead of 131072
 --gpu-memory-utilization 0.92

With the context cut to 8,192 tokens, vLLM does not reserve KV cache for 131,072 potential tokens and can fit more simultaneous requests. The risk is obvious: requests above 8,192 tokens fail with an error. Tune it to the P99 of your real length distribution.

Interaction between parameters

The four parameters are not independent. A common mistake is enabling prefix caching without tuning the block size, or raising max-num-batched-tokens without checking that max-num-seqs allows it to be filled:

max-num-batched-tokens = 8192
max-num-seqs = 4
average prompt = 512 tokens → 4 × 512 = 2048 prefill tokens < 8192
The 8192 budget is never filled because max-num-seqs caps it first.
Fix: raise max-num-seqs or lower max-num-batched-tokens.

A balanced configuration for RTX 4090 + a 7B model:

vllm serve Qwen/Qwen2.5-7B-Instruct \
 --gpu-memory-utilization 0.92 \
 --max-model-len 8192 \
 --enable-chunked-prefill \
 --max-num-batched-tokens 8192 \
 --max-num-seqs 64 \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales

Configuration for an L40 (48 GB) + a 14B model:

vllm serve Qwen/Qwen2.5-14B-Instruct-AWQ \
 --gpu-memory-utilization 0.90 \
 --max-model-len 16384 \
 --enable-chunked-prefill \
 --max-num-batched-tokens 16384 \
 --max-num-seqs 128 \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales

How to measure that it is working

The metrics that confirm prefill is optimised:

# In vLLM's Prometheus metrics (port 8000/metrics):
vllm:time_to_first_token_seconds_bucket → TTFT distribution
vllm:gpu_cache_usage_perc → KV cache utilisation
vllm:prefix_cache_hit_rate → APC hit rate (if enabled)
vllm:num_running_seqs → simultaneous in-flight requests

A prefix_cache_hit_rate below 30% on workloads with a fixed system prompt means something in the hash is not working (a system prompt that varies by timestamp, a date format in the prompt, and so on).

Implications for on-premise inference

Chunked prefill and prefix caching are zero cost: they are enabled with flags and require no extra hardware. FP8 KV cache requires the model to be compatible (almost every modern transformer is) and that you are on Ada Lovelace or newer.

For sovereign ENS deployments where the hardware is fixed and you cannot scale horizontally on demand, a well-configured prefill is the difference between needing 4 nodes and needing 2 for the same load.

The second article in this series covers the decode optimisations: speculative decoding, KV cache tuning to maximise concurrency and how to configure gpu-memory-utilization without vLLM running out of VRAM at midnight.

References

Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023
Sarathi-Serve: Chunked Prefill and Stall-Free Scheduling — Agrawal et al., 2024
vLLM Optimization and Tuning — documentación oficial
Quantized KV Cache en vLLM

Prefix cache: hit rate engineering to go from 15% to 75%

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

vLLM’s prefix cache stores the KV cache blocks of shared prefixes and reuses them in later requests. A hit avoids recomputing that prefix: TTFT drops to the cost of the variable suffix only. In enterprise workloads with fixed system prompts (RAG, domain chatbots, assistants with long instructions) the hit rate should be >70%. In practice it is 10-20% for entirely avoidable reasons. This article identifies them, fixes them, and gives the OTel queries to confirm the result.

The analogy

A simultaneous conference interpreter who has to translate the speeches of twenty speakers. All of them open with the same two-page protocol preamble: the conference declaration, the rules of conduct, the day’s programme. An interpreter with no memory rereads the two pages for each speaker before starting on their specific speech. An interpreter with good notes reads them once, files them, and when the second speaker starts goes straight to the speech.

The prefix cache is that file. The prefix hash is the reference that lets you skip to the new part. But if the preamble changes by even one word, because someone drops in the day’s date, the interpreter has to reread everything from the beginning.

How the prefix cache hash works

vLLM splits the KV cache into blocks of 16 tokens. Each block has a hash computed over its exact content. When a new request arrives, vLLM checks whether any initial block of the prompt is already in cache by comparing hashes.

The hash is computed over the byte-by-byte content of the tokens. Any difference, a space, a different character, one extra token, produces a completely different hash. There is no partial matching inside a block.

Direct consequence: if your system prompt has 512 tokens and token number 3 changes between requests (because you interpolate a date, an ID, a version number), no block hits even though 99% of the text is identical.

Block 0 (tokens 0-15): hash = a3f7... ← in cache?
Block 1 (tokens 16-31): hash = 9d2c... ← in cache?
...
Block 31 (tokens 496-511): hash = 7e1a... ← in cache?

If block 0 does not hit (because its content changed), blocks 1-31 are not even checked although they are identical. The prefix cache is sequential.

Audit: why your real hit rate is low

Before changing anything, you need to know what is breaking the hash. The most direct method: pull the last 1000 production prompts and work out what fraction of the prefix varies.

# audit_prefix_cache.py
import langfuse, hashlib, collections
from transformers import AutoTokenizer

client = langfuse.Langfuse()
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-14B-Instruct")

traces = client.fetch_traces(limit=1000).data
prompts = [t.input for t in traces if t.input]

# Tokenise and extract the first 512 tokens (the typical system prompt)
prefixes = []
for prompt in prompts:
 tokens = tokenizer.encode(prompt, add_special_tokens=False)
 prefix_tokens = tuple(tokens[:512])
 prefixes.append(prefix_tokens)

# How many unique prefixes are there?
unique = len(set(prefixes))
total = len(prefixes)
print(f"Unique prefixes: {unique}/{total} ({unique/total*100:.1f}%)")
print(f"Theoretical hit rate if they were all equal: {(1 - unique/total)*100:.1f}%")

# Find which token differs between the most common prefix and the rest
from collections import Counter
most_common_prefix = Counter(prefixes).most_common(1)[0][0]

divergence_positions = []
for prefix in prefixes:
 if prefix == most_common_prefix:
 continue
 for i, (a, b) in enumerate(zip(most_common_prefix, prefix)):
 if a != b:
 divergence_positions.append(i)
 break

if divergence_positions:
 pos = Counter(divergence_positions).most_common(1)[0][0]
 token_text = tokenizer.decode([most_common_prefix[pos]])
 print(f"\nMost frequent divergence at position {pos}: '{token_text}'")
 print("→ The token at that position varies between requests")

The most common culprits, in order of frequency:

1. Timestamps and dates:

# ❌ Breaks the hash on every request
system = f"Current date: {datetime.now().strftime('%Y-%m-%d %H:%M')}. You are an assistant..."

# ✅ Take the date out of the system prompt
system = "You are an assistant specialising in cloud infrastructure."
# Pass the date as part of the user message if it is needed

2. Session and user IDs:

# ❌
system = f"User ID: {user_id}. Preferences: {user_prefs}. You are an assistant..."

# ✅ Separate the static from the contextual
system = "You are a specialised assistant." # always the same
# Add the user context as the first message of the history

3. Interpolated prompt versions:

# ❌
system = f"[v{PROMPT_VERSION}] You are an assistant..." # changes with every deploy

# ✅ Do not version in the text, version in the prompt name in Langfuse
system = "You are an assistant..."

4. Dynamic few-shots:

# ❌ Examples retrieved at random from a pool
examples = random.sample(example_pool, k=3)
system = f"Examples:\n{format_examples(examples)}\n\nYou are an assistant..."

# ✅ Fixed few-shots always in the same order
FIXED_EXAMPLES = [example_pool[0], example_pool[1], example_pool[2]]
system = f"Examples:\n{format_examples(FIXED_EXAMPLES)}\n\nYou are an assistant..."

Template engineering: the structure that maximises hits

The principle is simple: everything static goes first, everything dynamic goes after. The prefix cache is sequential. Once a block fails to hit, the rest are not looked up either.

OPTIMAL STRUCTURE for maximising prefix cache:
┌──────────────────────────────────────────────┐
│ STATIC BLOCK (tokens 0-511) │ ← hit rate ~100%
│ Invariant system prompt │
│ Fixed instructions │
│ Few-shots always in the same order │
├──────────────────────────────────────────────┤
│ SEMI-STATIC BLOCK (tokens 512-1023) │ ← hit rate ~60-80%
│ RAG documents for this session │
│ Conversation history so far │
├──────────────────────────────────────────────┤
│ DYNAMIC BLOCK (tokens 1024+) │ ← hit rate ~0% (expected)
│ Current user message │
│ Context specific to this request │
└──────────────────────────────────────────────┘

For RAG specifically: if the retrieved documents are the same across a set of similar queries (very frequent in RAG over fixed corporate documents), ordering them always in the same order (by ID, by a fixed score, not by a variable score) multiplies the hit rate of the semi-static block.

Prefix-aware routing: the next level

With a single vLLM instance, the prefix cache works automatically. The problem shows up with multiple replicas: the load balancer distributes requests round-robin, and the prefix cached on replica A is no use at all when the request lands on replica B.

The solution is prefix-aware routing: send requests with the same prefix to the same node.

With Ray Serve (native integration):

# ray_serve_prefix_router.py
from ray import serve
from ray.serve.llm import LLMConfig, build_llm_deployment

@serve.deployment
class PrefixAwareRouter:
 def __init__(self, replicas):
 self.replicas = replicas # list of vLLM handles

 async def __call__(self, request):
 body = await request.json()
 messages = body.get("messages", [])

 # Compute the hash of the system prompt (static prefix)
 system_content = ""
 for msg in messages:
 if msg["role"] == "system":
 system_content = msg["content"]
 break

 prefix_hash = hash(system_content)
 # Deterministic routing: same hash → same node
 replica_idx = prefix_hash % len(self.replicas)
 return await self.replicas[replica_idx].remote(request)

With an L7 gateway (Nginx/Traefik):

# nginx.conf — routing by X-Prefix-Hash header
upstream vllm_backends {
 hash $http_x_prefix_hash consistent;
 server vllm-0:8000;
 server vllm-1:8000;
 server vllm-2:8000;
 server vllm-3:8000;
}

The client computes the hash of the static prefix and includes it as a header:

import hashlib, requests

def llm_request(messages, base_url):
 system_msg = next((m["content"] for m in messages if m["role"] == "system"), "")
 prefix_hash = hashlib.sha256(system_msg.encode()).hexdigest()[:16]

 return requests.post(
 f"{base_url}/v1/chat/completions",
 json={"messages": messages, "model": "my-model"},
 headers={"X-Prefix-Hash": prefix_hash}
 )

Measuring the impact with OTel

# Current hit rate (0.0 to 1.0) — target > 0.70 with enterprise workloads
vllm:gpu_prefix_cache_hit_rate

# TTFT by percentile — should fall when the hit rate rises
histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))
histogram_quantile(0.95, rate(vllm:time_to_first_token_seconds_bucket[5m]))

The inverse correlation between hit rate and TTFT is the proof that the cache is working. If the hit rate goes from 15% to 70% and the p50 TTFT does not change, there is a configuration problem: the cache may be disabled, or the routing may not be sending requests to the right node.

Correlation query in Grafana (two-axis panel):

# Left Y axis: hit rate
vllm:gpu_prefix_cache_hit_rate

# Right Y axis: TTFT p50 (inverted)
histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))

The inverse slope should be visible: when the hit rate drops (a spike of requests with new prompts), TTFT rises. When the hit rate stabilises (users repeating the same flow), TTFT falls.

The impact in numbers

For a system with 100 req/min, a 512-token system prompt and a before/after hit rate:

Metric	Hit rate 15%	Hit rate 75%	Difference
Prefill tokens per minute	5,100	12,800 — 50% cached → 6,400 effective	−37% load
TTFT p50 (512 prompt + 100 suffix)	~820 ms	~180 ms (suffix only)	−78%
Prefill capacity freed	—	+1,200 tok/min	available for more requests

The 75% hit rate in this example is equivalent to being able to serve 37% more requests on the same hardware, because the prefill work of 3 out of every 4 requests is already done.

When the prefix cache does not help

The prefix cache is ineffective in workloads where every request has a completely unique prompt: translating a different document every time, code analysis with always-different context, creative generation with no system prompt. In these cases the hit rate structurally cannot exceed 5-10% and the template engineering effort does not pay off.

The signal: if your p99 input length is greater than the p50, you have high prompt variance and the prefix cache contributes little. If the p50 and the p99 are similar (consistent prompts), the prefix cache is the cheapest lever available.

References

Function calling and tool-augmented retrieval: the detective who knows which file to ask for

Thu, 04 Jun 2026 10:00:00 +0200

This post goes deeper into the agentic reasoning mechanism that extends the RAG described in RAG with reranker and hybrid retrieval. The retriever invoked when the LLM chooses vector_search is exactly the pipeline in that article. The JSON Schema that defines each tool call is structured output applied to the tool interface. And the agent’s requests pass through the L7 inference gateway before reaching the model.

TL;DR

A naive RAG always queries the same source. Function calling gives the LLM the ability to decide which tool to invoke (vector store, SQL, web search) according to what the query actually needs. The ReAct pattern chains those invocations in a reasoned loop until there is enough evidence. A 3-iteration pipeline with Llama-3.1-70B on on-premise hardware takes ≈ 1.1 s against the ≈ 300 ms of a single-step RAG; the gain is not in speed but in queries that a naive RAG simply cannot answer. The critical eval metric is tool selection accuracy: the percentage of turns in which the model picks the right tool, measured over a synthetic eval set.

The analogy: the detective who knows which file to ask for

A detective in a novel does not go to the same filing cabinet regardless of the case that lands on his desk. When he takes a case, he reasons first: what kind of evidence do I need? If there are witnesses, he asks for the statements (vector search over unstructured documents). If there are financial transactions, he asks the bank for the accounting records (SQL over the structured database). If the suspect has recent activity that the company cannot possibly have indexed, he goes to the press archive (web search). He does not consult all three sources at once on every case: he picks the one the evidence requires, receives the result, reasons again if he needs more, and only writes the report when he has enough evidence.

A bad detective always goes to the same filing cabinet. A naive RAG is that bad detective: it vectorises the query, goes to the vector store, and returns whatever it finds even if the question was “how many contracts?”, something no PDF chunk can answer better than a COUNT(*) in SQL.

Function calling is giving the LLM the ability to reason about which source to ask for, and to invoke it in a structured way. The analogy has three edges worth keeping in mind:

The detective does not improvise the file he asks for: there is a catalogue of available sources with a description of what each one contains. The tool description in the system prompt does that job.
The detective can ask for several pieces of evidence at once if they are independent (parallel tool calling).
The detective knows when to stop: if after N rounds he reaches no conclusion, he declares that he does not have enough evidence. The agent has an iteration limit for the same reason.

What function calling is: the anatomy of a tool call

Function calling, also called tool use, is a mechanism by which the LLM, instead of generating free text as its answer, generates a structured JSON object representing a tool invocation. The system intercepts that JSON, executes the real tool, and returns the result as a message with the tool role in the conversation.

Defining tools in the system prompt

Each tool is defined through a JSON Schema that specifies name, description and parameters. This JSON Schema is exactly the same mechanism described in structured output, applied here to the tool interface:

{
 "tools": [
 {
 "name": "vector_search",
 "description": "Search internal company documents about policies, contracts and procedures. Use when the query requires unstructured text, document context or semantic similarity.",
 "parameters": {
 "type": "object",
 "properties": {
 "query": {"type": "string", "description": "Natural language search query"},
 "top_k": {"type": "integer", "default": 5}
 },
 "required": ["query"]
 }
 },
 {
 "name": "sql_query",
 "description": "Query the SQL database for structured metrics, counts, aggregations and financial data. Use when the query requires exact numbers, filters, sums or joins over structured records.",
 "parameters": {
 "type": "object",
 "properties": {
 "query": {"type": "string", "description": "Parameterized SQL query with $1, $2 placeholders"},
 "params": {"type": "array", "items": {}, "description": "Parameter values for the placeholders"}
 },
 "required": ["query"]
 }
 },
 {
 "name": "web_search",
 "description": "Search public web for real-time information, recent news or current prices. Use only when data is public and not covered by internal sources.",
 "parameters": {
 "type": "object",
 "properties": {
 "query": {"type": "string"}
 },
 "required": ["query"]
 }
 }
 ]
}

The life cycle of a tool call

When the LLM decides to invoke a tool, the message it generates in place of free text has this structure (OpenAI-compatible format, the same one vLLM supports):

{
 "role": "assistant",
 "tool_calls": [
 {
 "id": "call_01",
 "type": "function",
 "function": {
 "name": "sql_query",
 "arguments": "{\"query\": \"SELECT COUNT(*), SUM(amount) FROM contracts WHERE amount > $1 AND year = $2 AND provider_region = $3\", \"params\": [100000, 2025, \"EU\"]}"
 }
 }
 ]
}

The system executes the tool and returns:

{"role": "tool", "tool_call_id": "call_01", "content": "{\"count\": 47, \"total\": 8300000}"}

The LLM receives that message as a continuation of the conversation and decides whether it needs more information or can generate the final answer.

Support in OSS models

In 2026, native function calling support (not emulated via system prompt) is available in:

Llama 3.1 / 3.3+: native tool call format, supported in vLLM with --enable-auto-tool-choice --tool-call-parser llama3_json
Qwen 2.5+: native support with --tool-call-parser hermes
Mistral NeMo / Mistral 7B Instruct v0.3+: native format with --tool-call-parser mistral
Qwen3 (2025+): extended native support with parallel tool calling

Without --enable-auto-tool-choice, vLLM can force the tool call format via guided decoding (JSON Schema), but the ability to decide when to invoke a tool versus generating free text requires specific model training.

The ReAct pattern: Reasoning + Acting

ReAct (Yao et al., 2022) is the loop that turns an LLM with tools into an agent with reasoning. The central idea is that the model interleaves thoughts (explicit reasoning about what to do) with actions (tool invocations) and observations (results received). This loop repeats until the model reckons it has enough information to answer, or until the iteration limit is reached.

Thought → Action → Observation → Thought → Action → Observation → ... → Response

A full ReAct trace: a real production query

Query: “How many contracts above 100k€ did we sign in 2025 with EU providers, and which special conditions recur?”

This query requires two different sources: the exact numbers live in SQL, and the context of special conditions lives in the contract PDFs indexed in the vector store.

Iteration 1

Thought: The query asks for aggregated data (count and sum) with exact filters over structured fields (amount, year, region). This is a SQL query, not a semantic search.

Action: sql_query

SELECT COUNT(*) AS num_contratos, SUM(amount) AS total_eur
FROM contracts
WHERE amount > $1 AND year = $2 AND provider_region = $3

params: [100000, 2025, "EU"]

Observation: {"num_contratos": 47, "total_eur": 8300000}

Iteration 2

Thought: I have the numbers. Now I need the special conditions that recur in these contracts. That kind of information is in the free text of the PDFs, not in the structured table. I go to the vector store.

Action: vector_search

"EU provider contracts 2025 amount above 100k special conditions recurring clauses"

top_k: 5

Observation:

[chunk_1] Provider A contract — clause 8.3: late-delivery penalty of 2% per month on the invoiced amount...
[chunk_2] Provider C contract — clause 7.1: price revision linked to the euro area CPI capped at 4% per year...
[chunk_3] Provider B contract — clause 8.3: late-delivery penalty of 2% per month...

Thought: I have enough data from both sources. Clause 8.3 on late-delivery penalties appears in at least two of the retrieved contracts. I can synthesise the answer.

Response: “In 2025 you signed 47 contracts with EU providers above 100,000 €, for a cumulative total of 8.3 M€. The special condition that appears most often in the higher-value contracts is the late-delivery penalty clause (8.3 in the contract templates of Provider A and B): 2% per month on the invoiced amount. The price revision linked to the euro area CPI, capped at 4% per year (clause 7.1), also recurs.”

This trace has 2 iterations. A more complex query (comparing with the previous year, cross-referencing with payment status) would add further iterations. Common practice is to set a limit of 5–10 iterations maximum to avoid loops.

The three sources and when to use each one

The choice of source is neither arbitrary nor left solely to the LLM: the architecture defines which tools exist and how they are described. The table below summarises the selection criteria:

Characteristic	Vector store (Qdrant)	SQL / structured (PostgreSQL)	Web search
Data type	Free text, documents, PDFs	Tables with a fixed schema	Public pages, news
Structure	Unstructured	Highly structured	Semi-structured
Temporality	Static index (periodic refresh)	Real time (transactional)	Real time (crawl)
Typical latency	5–50 ms	10–200 ms	200–2,000 ms
Privacy	Internal data, full sovereignty	Internal data, full sovereignty	Public data only
Natural query	Yes (natural language → embedding)	No (parameterised SQL)	Yes
Exact aggregations	No	Yes (`COUNT`, `SUM`, `GROUP BY`)	No
When to use	Document context, semantics, PDFs	Metrics, counts, exact filters, joins	Data that does not exist internally and is public

The most important rule of thumb: if the question contains words like “how many”, “sum”, “total”, “more than X”, “in year Y” and the data is in a structured table, the right answer is sql_query. If the question asks for context, explanations, clauses, procedures or examples from documents, the answer is vector_search. If it asks for the current price of something or recent news about a third party, web_search, but only if no data sovereignty is involved.

Tool routing: how the LLM picks the right tool

The description of each tool in the system prompt is the most critical factor for routing accuracy. An LLM with good function calling capabilities can still choose badly if the descriptions are ambiguous or overlap.

Descriptions that work versus those that do not

Weak description (leads the LLM to use the wrong tool):

"search_docs" — Searches for information in the available sources.
"query_data" — Gets data from the system.

Strong description (delimits precisely when to use each one):

"vector_search" — Search internal company documents about policies, contracts and procedures.
Use when the query requires unstructured text, document context or semantic
similarity. NOT for counts, sums or exact filters.
"sql_query" — Query the SQL database for structured metrics, counts, aggregations and
financial data. Use when the query requires exact numbers, filters, sums or
joins over structured records. NOT for finding document context.

The difference lies in two elements: (1) positive use-case examples, and (2) explicit exclusions with NOT for. Both reduce the semantic overlap between tools and improve tool selection accuracy.

Parallel tool calling

When two tools are independent of each other, that is, when the result of one does not affect the query of the other, the LLM can invoke them simultaneously in the same turn:

{
 "tool_calls": [
 {"id": "call_01", "function": {"name": "sql_query", "arguments": "..."}},
 {"id": "call_02", "function": {"name": "vector_search", "arguments": "..."}}
 ]
}

The system executes both in parallel and returns both observations before the LLM’s next turn. This cuts total latency when the queries are independent: instead of 2 sequential iterations (2 × tool_latency), you pay 1 × max(sql_latency, vector_latency). For the detective example: if he needs both the accounting records and the witness statements to answer, he can ask for them at the same time.

Tool selection accuracy: the eval metric

Tool selection accuracy is the percentage of turns in which the LLM picks the right tool given a set of evaluated queries:

\[ \text{TSA} = \frac{\text{turns with correct tool chosen}}{\text{total turns with expected tool call}} \]

It is measured over a synthetic eval set built with (query, expected_tool, expected_args) triples. An example of a minimal eval set:

Query	Expected tool	Failure indicator
“How many orders in March?”	`sql_query`	LLM uses `vector_search`
“What does the returns policy say?”	`vector_search`	LLM uses `sql_query`
“What is the price of copper today?”	`web_search`	LLM uses `vector_search`
“Sum the Q3 contracts”	`sql_query`	LLM uses `vector_search`

A TSA < 0.85 in a production agent is a sign that the tool descriptions need revising before the model does. For more detail on how to build these evals, see LLM evals.

Error handling in tool calls

SQL injection via prompt

The most serious risk in tool-augmented retrieval is that the LLM generates malicious SQL, either because a user induced it via prompt injection, or because the model hallucinated a destructive query. This attack vector is covered in detail in LLM Guard, but the minimum rules on the tool side are:

Rule 1: parameterised queries always, never direct interpolation.

# NEVER this:
cursor.execute(f"SELECT * FROM contracts WHERE provider = '{llm_output}'")

# Always this:
cursor.execute("SELECT * FROM contracts WHERE provider = $1", [llm_output])

Rule 2: DB user with minimum permissions. The user the agent runs SQL as must have SELECT on the necessary tables and nothing else. No DROP, INSERT, UPDATE or DELETE.

Rule 3: table allowlist. The system validates that the query generated by the LLM only references tables on an allowlist before executing it.

Rule 4: per-query timeout. Queries that block for too long, potentially induced to DoS the database, are cancelled with a configured timeout.

Rate limits, timeouts and tool errors

When a tool fails, the error is returned to the LLM as an observation:

{"role": "tool", "tool_call_id": "call_01", "content": "{\"error\": \"timeout after 5s\", \"tool\": \"web_search\"}"}

The system prompt must instruct the LLM on what to do in this case:

If a tool returns an error or is unavailable, acknowledge the limitation in your response.
Do not retry more than once. If web_search is unavailable, state that real-time data
is not accessible at this moment and answer with available internal sources only.

This stops the agent from entering retry loops and handles graceful degradation: if web_search is unavailable, it answers with what it has in the internal sources.

Diagram of the ReAct loop with the three sources

Latency maths of the ReAct pipeline

Each iteration of the ReAct loop has three latency components:

\[ T_{\text{iter}} = \text{TTFT}_{\text{LLM}} + T_{\text{tool}} + \Delta_{\text{context}} \]

where:

TTFT_LLM: time to the LLM’s first token (dominated by the prefill of the accumulated context)
T_tool: tool execution time
Δ_context: overhead of the growing context window (each iteration adds the previous output to the context)

Reference values: Llama-3.1-70B on 4×H100 SXM (320 GB, NVLink)

With Llama-3.1-70B in FP8 on a node with 4×H100 SXM (320 GB HBM3, NVLink 900 GB/s), the typical values in production are:

Component	Value
TTFT with context < 4k tokens	≈ 150 ms
TTFT with context 8k tokens	≈ 220 ms
`sql_query` (simple query, indexed)	≈ 50 ms
`vector_search` (top-5, Qdrant in RAM)	≈ 20 ms
`web_search` (external API)	≈ 600 ms

Total latency by number of iterations

1-iteration pipeline (simple query, a single tool):

\[ T_1 = 150 + 50 = 200 \text{ ms} + \text{final synthesis} \approx 200 + 300 = 500 \text{ ms} \]

2-iteration pipeline (SQL + vector_search in sequence):

\[ T_2 = (150 + 50) + (180 + 20) + 400 = 800 \text{ ms} \]

The context in the second iteration already includes the result of the first, so TTFT rises slightly to ≈ 180 ms.

3-iteration pipeline (the most common case in complex queries):

\[ T_3 = (150 + 50) + (180 + 20) + (200 + 50) + 450 \approx 1,100 \text{ ms} \]

Parallel tool calling (SQL + vector_search in parallel, a single iteration):

\[ T_{\text{parallel}} = 150 + \max(50, 20) + 400 = 600 \text{ ms} \]

When the two queries are independent, parallel tool calling trims latency from ≈ 800 ms to ≈ 600 ms: a 25% improvement over the 2 sequential iterations case.

Comparison with a naive RAG

Configuration	Latency	Queries it can answer
Naive RAG (1 retriever, 1 step)	≈ 300 ms	Document context queries
ReAct 1 iteration (SQL)	≈ 500 ms	Structured aggregation queries
ReAct 2 iterations (SQL + vector)	≈ 800 ms	Hybrid numeric + context queries
ReAct 3 iterations	≈ 1,100 ms	Complex multi-source queries
ReAct with web_search	≈ 1,500 ms	Queries requiring real-time data

The practical conclusion: multi-hop ReAct is 3–5× slower than a single-step naive RAG. The gain is not in speed but in query coverage: a naive RAG cannot answer “how many contracts?” because that answer is in no text chunk at all. For applications with a strict latency SLO (< 500 ms), you have to work out whether the use case really needs ReAct or whether a well-configured RAG with hybrid retrieval covers 90% of the queries.

On-premise hardware for ReAct agents

A ReAct agent with Llama-3.1-70B in production has different requirements from a naive RAG, because the context grows with each iteration and prefill throughput is more critical.

Recommended configuration: 4×H100 SXM (320 GB HBM3, NVLink 900 GB/s)

Llama-3.1-70B in FP8: fits in 2×H100 (70B params × 1 byte/param ≈ 70 GB + KV cache). With 4×H100 you can serve it with tensor parallelism TP=4, cutting prefill TTFT by ≈ 2×.
Qdrant instance: it can sit on the same node (if the collection fits in RAM) or on a dedicated node. For collections < 50M vectors of 768 dims: ≈ 150 GB, which fits in the RAM of a dual-socket server.
PostgreSQL: separate node or managed instance. The agent adds no unusual load to SQL; the queries are simple and bounded by timeout.
vLLM with --enable-auto-tool-choice --tool-call-parser llama3_json --max-model-len 16384: a 16k token context comfortably covers the 5–10 turns of a ReAct pipeline.

Minimum viable configuration: 2×H100 SXM (160 GB)

Llama-3.1-70B in FP8 with TP=2. TTFT ≈ 250–300 ms for 4k token contexts (a 60–100% increase over TP=4).
Good enough for internal workloads with < 20 concurrent requests.
Not advisable for an SLO < 1 s with more than 5 concurrent users and a long context.

What we have not covered

Agentic retrieval loops with planning. ReAct is the simplest agent pattern. When a query requires decomposition into sub-tasks with dependencies, you need orchestration frameworks such as LangGraph (state graphs), smolagents (Hugging Face, agents with Python code as actions) or llama-index Agents (planning + retrieval pipeline). These introduce a planning step before the execution loop.

MCP (Model Context Protocol). Anthropic’s emerging standard, with OSS implementations, for defining tools portably across frameworks and hosts. Instead of defining each tool’s JSON Schema separately in every application, MCP centralises those definitions in an MCP server that any compatible client can discover and invoke. Adoption in 2025–2026 is fast among OSS frameworks (LangChain, smolagents, OpenWebUI).

Tool caching. If the same tool call (same arguments, same tool) is going to be invoked several times within the same context or in very similar contexts, the result can be cached. The mechanism is analogous to the semantic cache described for RAG: before executing the tool, the hash of the arguments (or its embedding, for semantic matching) is compared against a cache with a TTL. Especially valuable for sql_query with frequent queries and data that changes little.

Multi-agent. When an orchestrating agent delegates sub-tasks to specialised agents, one for SQL, one for document retrieval, one for code generation, you enter multi-agent system territory. Each sub-agent can have its own tool set and its own LLM (possibly smaller and more specialised). Coordination between agents introduces additional tracing and observability complexity.

References

Yao, S. et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629. https://arxiv.org/abs/2210.03629
vLLM documentation. Tool calling. https://docs.vllm.ai/en/stable/features/tool_calling.html
Qdrant documentation. Search. https://qdrant.tech/documentation/concepts/search/
OpenAI. Function calling. https://platform.openai.com/docs/guides/function-calling
Meta AI. Llama 3.1 Model Card. https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md
Qwen Team (Alibaba). Qwen2.5 Technical Report. arXiv:2412.15115. https://arxiv.org/abs/2412.15115
Anthropic. Model Context Protocol. https://modelcontextprotocol.io
OWASP. LLM Top 10 for Large Language Model Applications. LLM01: Prompt Injection. https://owasp.org/www-project-top-10-for-large-language-model-applications/

Embeddings in 2026: the three families (dense, sparse, multi-vector), the model zoo and the decision that matters in production

Wed, 03 Jun 2026 03:00:00 +0200

This post opens the data sub-saga inside the six-stage LLMOps pipeline by going into the piece that holds up three-layer retrieval: the embedder. If the librarian of curation decided what enters the index and the faculty committee decided what reaches the model’s face, this post looks at the one in the middle: the cartographer who draws the map you search over.

TL;DR

The conversation about embeddings has been simplified in production to the point where “which embedder do you use” always gets the same answer: OpenAI text-embedding-3-large in the demo, bge-m3 in the “sovereign-ready” version. That simplification hides the fact that an embedder is three different models at once, dense single-vector, learned sparse (SPLADE) and multi-vector late-interaction (ColBERT), and that in 2026 the leading models do not compete within the same family: gte-Qwen2-7B-instruct and NV-Embed-v2 break MTEB in dense single-vector, SPLADE-v3 and the sparse head of bge-m3 dominate the learned lexical descriptor, Jina-ColBERT-v2 and ColNomic-7B are the strongest in multilingual multi-vector, and Snowflake Arctic Embed L 2.0 has slipped in as the small multilingual favourite with decent Matryoshka. This post takes apart the three families with their maths (InfoNCE with τ, MaxSim, SPLADE’s FLOPS regularization, Matryoshka Representation Learning), goes through the open source model zoo with the dimension, licence and niche of each one, poses the specific problem of multilingual Spanish, which cuts the list of viable models to fewer than six, gives the real storage cost per million chunks with int8 / binary / TurboQuant, describes how they are served on-premise with TEI, Infinity and vLLM --task embed, sets the minimum hardware at an RTX 4090 and the good hardware at a 4×H100 cluster, lists the seven operational traps that sink quality without warning (corpus drift, forgotten normalisation, badly chosen Matryoshka dimension, missing hard negatives, chat template slipped into the embedder, tokenizer drift, MTEB overfit) and closes with a license-clean stack for sovereign production.

The analogy: three librarians cataloguing the same book

A technical library receives a new book and, before putting it on the shelves, has to generate a searchable identifier for it. Three librarians with three different trades live in the library, and all three catalogue the same book at once:

Librarian A, the thematic one. He reads the whole book and gives it a single rich RFID tag. That tag is a vector of 1,024 numbers where each coordinate encodes a latent semantic axis (one the librarian never verbalises: he has learned it by reading a hundred million previous books). Two books about Kubernetes in production will end up with RFID tags very close together in the space even if one talks about Linkerd and the other about Cilium, because they share thematic axes. To search, you compare the question’s tag with the book’s and return the nearest ones by cosine. It is fast, it scales to millions, and it loses fine nuance. This is the dense single-vector embedder: bge-m3 in dense mode, gte-Qwen2-7B-instruct, Snowflake Arctic Embed L 2.0, multilingual-e5-large-instruct.

Librarian B, the lexical one. He summarises nothing. What he does is write on a card the weighted list of relevant terms from the book, expanded with synonyms from the field. The Kubernetes book carries on its card “kubernetes 4.2, linkerd 3.8, cilium 3.7, service-mesh 4.1, sidecar 3.2, mtls 2.9, ebpf 2.6, k8s 4.0…”, each term with a weight. The trick is that the model does the expansion: if your question says “service mesh” and the original book only said “Linkerd”, librarian B’s card did record “service-mesh 4.1” because he understood the relationship. To search, you intersect the question’s card with the book’s the old-fashioned way: inverted index, posting lists. It is decisive when the reader writes few, very specific words (product names, errors, jargon). This is the learned sparse embedder: SPLADE-v3 or the sparse head of bge-m3. It is the modern successor to BM25, not its rival; we will see why.

Librarian C, the copyist. He gives up on summarising. He takes every word on every page of the book and gives it a 128-dimension mini-RFID. He ends up with a 30,000-token book turned into 30,000 mini-RFIDs. When you search for something, the librarian compares every word of your question with every word of the book and keeps the maximum for each word of the question, adding them up. It captures nuance the other two lose by construction (proper nouns, numbers, specific phrasings), but its filing system is an order of magnitude larger in disk space. This is ColBERT-v2 / Jina-ColBERT-v2 / ColNomic-7B: late interaction, MaxSim.

And then there is the all-rounder librarian who does all three jobs in a single pass: bge-m3. A single 568 M parameter model that simultaneously returns the thematic RFID tag (1,024-d), the weighted list of terms (sparse) and the set of per-token mini-RFIDs (128-d). That is why it has settled in as the standard embedder for multilingual on-premise RAG: a single model.encode(chunk) produces the three outputs that feed hybrid retrieval without orchestrating three different models.

The same chunk indexed by three librarians. The all-rounder bge-m3 runs all three in a single 568 M parameter pass.

What an embedding really is

A text embedding is a function f : text → ℝᵈ trained so that two “semantically similar” texts produce nearby vectors in that space. The important part is not “vector”, any fixed-length hash is one, but what closeness means. Closeness is defined implicitly by the loss the model is trained with.

Almost every modern embedder is trained with InfoNCE (also called Multiple-Negatives Ranking Loss in sentence-transformers):

\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp(\text{sim}(q, d^+)/\tau)}{\sum_{d \in \mathcal{B}} \exp(\text{sim}(q, d)/\tau)}

For each (query, doc⁺) pair the model has to assign the positive more similarity than all the other documents in the batch B, where the other documents act as free in-batch negatives. The temperature τ (typically 0.02–0.07, almost always 0.05) controls how sharp the distribution gets: lower τ means the model becomes more demanding about the positive but less stable. A large batch size |B| means many more negatives per gradient, so a better informed model. That is why embedders are trained with batch ≥ 1,024 on H100 clusters with AllGather across nodes to stack every negative in the cluster into a single effective batch.

On top of the in-batch negatives comes hard negatives mining: documents deliberately picked because they are “almost in the answer” (typically the next 10-100 neighbours from a previous BM25 / dense retrieval). Without hard negatives the model learns to discriminate the trivial and real quality on BEIR / MTEB drops 5-10 points.

This matters because the embedder family depends on what you feed into sim(·,·):

Family	`sim(q, d)`	Model output
Dense single-vector	dot product of two normalised 1024-d vectors	`f(q), f(d) ∈ ℝ^d`
Learned sparse	dot product of two 30,522-d vectors (≈80 non-zero each)	`f(q), f(d) ∈ ℝ^V`, `V =
Multi-vector	`Σᵢ maxⱼ ⟨qᵢ, dⱼ⟩` (MaxSim)	`f(q) ∈ ℝ^{

The loss is the same in all three cases, InfoNCE, but the geometry of the space changes, and with it quality, storage cost and search latency.

The three families in detail

Dense single-vector, the cartographer

The embedder reads the full chunk, passes it through an encoder transformer (XLM-RoBERTa, BERT, a Mistral decoder with a prompt) and aggregates the token representations into a single vector by:

CLS pooling: uses the embedding of the [CLS] token. Standard in BERT-base.
Mean pooling: plain average of the embeddings of all tokens. Standard in multilingual-e5-large-instruct, bge-m3 dense, Snowflake Arctic Embed L 2.0.
Last-token pooling: for decoder-LLM based embedders (e5-mistral-7b-instruct, gte-Qwen2-7B-instruct, NV-Embed-v2) that take the final token as the aggregate. It works because the model is causal and the last token “has seen” the whole context.
Latent-attention pooling: new in NV-Embed-v2. A learnable attention layer that weights tokens instead of averaging them. +2-3 MTEB points over mean pooling.

After pooling the vector is normalised to norm 1: v ← v / ‖v‖₂. With normalised vectors, cosine and dot product coincide:

\cos(q, d) = \frac{q \cdot d}{\|q\|\|d\|} = q \cdot d, \qquad \|q - d\|^2 = 2 - 2\,(q \cdot d)

That is why almost every vector DB indexes by inner product and leaves it to the user to normalise beforehand (Qdrant, Faiss IP, Milvus). If you forget to normalise the query vector but the corpus vectors are normalised, retrieval degrades silently: the query’s magnitude distorts the ranking. It is the number one bug in RAG in production.

A minimal numerical example, two normalised 4-d vectors:

q = [0.5, 0.5, 0.5, 0.5] ‖q‖ = 1
d₁ = [0.6, 0.4, 0.5, 0.5] ‖d₁‖ = 1.005, normalised [0.597, 0.398, 0.498, 0.498]
q · d₁ = 0.5·0.597 + 0.5·0.398 + 0.5·0.498 + 0.5·0.498 = 0.995

Closeness of nearly 1, as expected. If d₁ had not been normalised, q · d₁ = 1.0 and it would appear closer than any d that is perfectly aligned but with a norm < 1.005. Normalisation is not a detail: it is the contract of the space.

Learned sparse, the lexical descriptor

SPLADE-v3 (Naver, March 2024) has consolidated the modern version of the lexical librarian. Internally it is a small BERT (~110 M parameters, DistilBERT/BERT base) that produces, for each input token, a distribution over the whole vocabulary (30,522 dimensions in BERT WordPiece), and then does max-pool over the tokens:

w_j = \max_{i \in \text{seq}} \log\bigl(1 + \text{ReLU}(W_{ij})\bigr)

where Wᵢⱼ is the logit of input token i for vocabulary term j. The log(1+ReLU) saturates high logits (it stops a single word from dominating the vector) and the ReLU cuts the negatives. The result is a 30,522-dimension vector of which typically 50-200 entries stay non-zero.

The key part is the FLOPS regularization added to the loss during training:

\mathcal{L}_{\text{FLOPS}} = \lambda \cdot \sum_{j=1}^{V} \bar{w}_j^2, \qquad \bar{w}_j = \frac{1}{|B|}\sum_{i \in B} w_{ij}

It penalises the expected cost of the posting lists: if a vocabulary word appears on average in many documents, adding it to a new document costs double. The model learns to generate sparse vectors by construction.

What does this do to the text “Linkerd 3.8 brings mTLS by default”? The model does not only write the literal terms, it also writes, with a smaller but non-zero weight, “service-mesh”, “kubernetes”, “tls”, “sidecar”, “envoy”, “istio” (its competitor, also semantically related), “encryption”, “k8s”. That semantic expansion of the document is what sets SPLADE apart from BM25. BM25 only knows what was literally in the text; SPLADE knows what an expert would add as a descriptor.

In practice SPLADE-v3 beats BM25 by 3-6 MRR@10 points on MS MARCO and dominates BEIR zero-shot. The cost is ~2-4× the query latency of BM25 over the same inverted index, which static pruning can mitigate.

For the multilingual case, bge-m3 with its sparse head is the only maintainable option: SPLADE-v3 is trained in English and the multilingual ports are in an experimental state.

Multi-vector, the copyist

ColBERT-v2 (Stanford, NAACL 2022) introduced the late interaction paradigm. Instead of compressing the document to a single vector, it leaves it as a matrix (|d|, k) with a k-dimension vector per token. The similarity between query and document is computed token by token and aggregated with MaxSim:

s(q, d) = \sum_{i \in q} \max_{j \in d} \langle q_i, d_j \rangle

What is being computed: for each word of the query, find the document token that fits it best and add that similarity. The sum is over the query, not over the document. This lets a 30,000-token document compete fairly with a 200-token one, because the query always adds |q| terms.

Why does it give more quality than dense single-vector? Because compressing to one vector loses information about where each idea was. If the query is “which version brought mTLS by default in Linkerd”, the dense summary of the document only knows that the chunk is about “Linkerd and mTLS”; ColBERT’s copyist can match “which version” with “3.8” because it stores the embedding of the 3.8 token separately. On BEIR / out-of-domain, late interaction beats single-vector by between +2 and +6 nDCG@10 with the same backbone.

The price is storage. Per document, a 1024-dimension dense single-vector in fp16 takes 1024 × 2 = 2 KB. ColBERT-v2 with 128-dimension tokens for a 256-token chunk takes 256 × 128 × 2 = 65,536 B ≈ 64 KB: 32× more space. With ColBERT-v2’s nbits=2 residual compression it drops to ~16 KB (8×). Jina-ColBERT-v2 adds Matryoshka over the token dimensions (truncatable to 128 / 96 / 64), cutting another 50%.

For 1 million chunks:

Family	Per doc	Total
Dense fp32 (1024-d)	4,096 B	4.0 GB
Dense fp16 / halfvec	2,048 B	2.0 GB
Dense int8 (SQ)	1,024 B	1.0 GB
Dense binary (1 bit/d)	128 B	128 MB
SPLADE (≈ 80 terms × 8 B)	~640 B	~640 MB
ColBERT fp16 (256 tok × 128 d)	65,536 B	64 GB
ColBERT residual `nbits=2`	~16,000 B	~16 GB
Jina-ColBERT-v2 (MRL token 64)	~8,000 B	~8 GB

ColBERT in on-premise production is reserved for corpora of up to a few million chunks, or applied only as a reranker over the top-100 from the first committee (dense + sparse), as described in the reranker post.

Matryoshka, the truncatable dimension

An operational lever that changed the conversation about embeddings between 2024 and 2026 is Matryoshka Representation Learning (Kusupati et al., NeurIPS 2022). The trick: during training, on top of the loss over the full D-dimension vector, the same loss is computed over prefixes of the vector:

\mathcal{L}_{\text{MRL}} = \sum_{k \in \{64, 128, 256, 512, 1024\}} \alpha_k \cdot \mathcal{L}_{\text{InfoNCE}}\bigl(\text{emb}[:k]\bigr)

The first k dimensions of the embedding are trained to be, on their own, a valid embedding. At inference time, if you want a cheaper embedding, you truncate the vector: the first quarter is already a usable embedding. Without Matryoshka, truncating wrecks the geometry: the first 256 dimensions of an embedding trained only at 1024-d encode nothing coherent.

Typical degradation in MTEB nDCG@10 when truncating an MRL embedder:

Truncation	Average loss
1024 → 512	-1 to -2 points
1024 → 256	-3 to -5 points
1024 → 128	-5 to -8 points
1024 → 64	-8 to -12 points

Native MRL models in 2026 (the ones that let you pick the dimension at runtime without retraining):

jina-embeddings-v3 (1024 → 32, fine steps, CC-BY-NC-4.0)
jina-embeddings-v4 (2048 → 128, multimodal text+image, CC-BY-NC-4.0)
nomic-embed-text-v2-moe (768 → 256, Apache 2.0)
Snowflake-arctic-embed-l-v2.0 (1024 → 256, Apache 2.0)
mxbai-embed-large-v1 and mxbai-embed-2d-large-v1 (the latter also truncatable in layer depth)
Stella_en_1.5B_v5 (multiple steps 512 / 768 / 1024 / 2048 / 4096 / 6144 / 8192, MIT, English only)
text-embedding-3-large (OpenAI, 3072 → 256, API only)
voyage-3 family (1024 → 256 / 512 / 1024 / 2048, API only)

Production recommendation: always use an MRL model even if you do not truncate at the start, because it simplifies the future quantisation decision. And above all, evaluate truncation on your real corpus: the average MTEB degradation of “-3 points” becomes “-12 points” in a niche domain.

The 2026 open source model zoo

What follows is a spec sheet per model. Verified as of 2026-06: HuggingFace cards + reference papers + MTEB / MMTEB leaderboard.

Dense single-vector

Model	Params	Dim	Tokens	Languages	Licence	Distinctive
`BAAI/bge-m3` (dense)	568 M	1024	8192	100+	MIT	Tri-mode (dense + sparse + colbert) in one forward. De facto standard for multilingual on-prem.
`Snowflake/snowflake-arctic-embed-l-v2.0`	568 M	1024 (MRL → 256)	8192	~100	Apache 2.0	Trained for multilingual + English without degrading either. MIRACL 55.8.
`intfloat/multilingual-e5-large-instruct`	560 M	1024	512	~100	MIT	Veteran multilingual baseline. Short window.
`intfloat/e5-mistral-7b-instruct`	7.1 B	4096	4096	English	MIT	First decoder-as-embedder to break MTEB. English.
`Alibaba-NLP/gte-Qwen2-7B-instruct`	7 B	3584	32,768	100+	Apache 2.0	The only one with 32k context. Top MTEB-en, strong multilingual.
`nvidia/NV-Embed-v2`	7.85 B	4096	32,768	English	CC-BY-NC-4.0	Latent-attention pooling. Top quality. Licence blocker in prod.
`Linq-AI-Research/Linq-Embed-Mistral`	7 B	4096	4096	English	CC-BY-NC-4.0	Top MTEB retrieval Aug 2024. Non-commercial.
`NovaSearch/stella_en_1.5B_v5`	1.54 B	8192 (multiple MRL)	8192	English	MIT	Small + rich MRL. English.
`BAAI/bge-multilingual-gemma2`	9 B	3584	8192	100+	Gemma	High quality, the Gemma licence restricts redistribution.
`BAAI/bge-en-icl`	7 B	4096	8192	English	MIT-style	In-context learning from examples in the prompt.
`mixedbread-ai/mxbai-embed-large-v1`	335 M	1024 (MRL)	512	English	Apache 2.0	MRL + native binary. Short window.
`jinaai/jina-embeddings-v3`	570 M	1024 (MRL → 32)	8192	89	CC-BY-NC-4.0	Per-task LoRA. Non-commercial without a licence.
`nomic-ai/nomic-embed-text-v2-moe`	475 M / 305 M active	768 (MRL → 256)	512	~100	Apache 2.0	First general-purpose MoE in embeddings.

Learned sparse

Model	Params	Vocab	Tokens	Languages	Licence	Distinctive
`naver/splade-v3`	110 M	30,522	512	English	CC-BY-NC-SA-4.0	SOTA learned sparse. Non-commercial.
`BAAI/bge-m3` (sparse head)	568 M	XLM-R vocab	8192	100+	MIT	The only license-clean multilingual option.

Multi-vector (late interaction)

Model	Params	Dim/token	Tokens	Languages	Licence	Distinctive
`colbert-ir/colbertv2.0`	110 M	128	512	English	MIT	The original paper, the basis for everything.
`jinaai/jina-colbert-v2`	560 M	128 / 96 / 64 (MRL)	8192	89	Apache 2.0	The license-clean multilingual multi-vector.
`nomic-ai/colnomic-embed-multimodal-7b`	7 B	3584 (Qwen2-VL)	—	~100	Apache 2.0	Multimodal text+image multi-vector. Vidore-v2 open SOTA.

The leaderboard, with caution

MTEB / MMTEB (Massive Multilingual Text Embedding Benchmark, Enevoldsen et al., arxiv 2502.13595) is the standard thermometer. Top retrieval on MMTEB in mid-2026 is dominated by Qwen3-Embedding-8B (~70.6 multilingual avg) and Llama-Embed-Nemotron-8B. Below them, the 7B models (gte-Qwen2-7B, NV-Embed-v2) and the 568M ones (bge-m3, Snowflake-Arctic-L-2.0) compete task by task.

Trap: MTEB has started to saturate through dataset contamination. The higher the ranking, the more likely it is that the model saw subsets of the evaluation datasets during training. The rule in production: the leaderboard is for discarding bad models, not for picking the best one. The final decision is made on your own domain eval set, generated with the LLM-as-judge or evals recipe.

The specific problem of multilingual Spanish

For a Spanish customer serving corporate, legal or support documentation in Spanish (and often with Catalan / Portuguese / English mixed in), the embedder zoo narrows to fewer than six viable models. The operational exclusions:

English-only models: e5-mistral-7b-instruct, stella-en-1.5B-v5, Linq-Embed-Mistral, mxbai-embed-large-v1, NV-Embed-v2, SPLADE-v3. They cut Spanish performance below an acceptable level: translating the query into English before searching is one route, but it introduces latency, tokenisation drift and another model dependency.
Models with a non-commercial licence: jina-embeddings-v3, jina-embeddings-v4, NV-Embed-v2, Linq-Embed-Mistral. They work for a PoC, but commercial production needs an explicit agreement with the vendor. Unless you have the licence signed, they have to be excluded.
Models with a Gemma licence: bge-multilingual-gemma2. Allowed for internal use, awkward to redistribute weights to a customer.

The ones left, ordered by practical choice in sovereign production:

BAAI/bge-m3 — MIT, 568 M, 100+ languages (including Spanish and Catalan, explicitly trained), 8,192 tokens, tri-mode dense+sparse+colbert. A reasonable default. Fits in an RTX 4090. TEI and Infinity serve it natively.
Snowflake/snowflake-arctic-embed-l-v2.0 — Apache 2.0, same size, explicit Matryoshka, better MIRACL/CLEF than bge-m3 on some multilingual tasks, no sparse head. If the priority is pure MMTEB in Spanish.
intfloat/multilingual-e5-large-instruct — MIT, 560 M, veteran baseline. Its 512-token window is the big limitation: long documents have to be split first. If what you already have in production works, do not migrate out of fashion.
Alibaba-NLP/gte-Qwen2-7B-instruct — Apache 2.0, 32k context, high quality in Spanish (Qwen2 is well trained on Spanish). If the chunks are long (more than 4k tokens) and you have the GPU to serve it (it does not fit in a 4090; it does in an H100). It fits alongside an LLM on an H100 80GB with care.
nomic-ai/nomic-embed-text-v2-moe — Apache 2.0, 305 M active, MRL, ~100 languages. If latency and cost per token rule: the MoE gives it disproportionate throughput for its quality.
jinaai/jina-colbert-v2 — Apache 2.0, multilingual multi-vector, as a reranker or as the main retrieval on a small corpus (< 1 M chunks). The only license-clean multi-vector in Spanish.

The rule of thumb: bge-m3 as dense+sparse on the front line, jina-colbert-v2 as a third reranking layer when the use case warrants it, and Snowflake Arctic L 2.0 as the alternative if the corpus-specific eval prefers its geometry.

Serving embeddings on-premise

Three engines split the on-prem embedding serving landscape in 2026, with different profiles.

Text Embeddings Inference (TEI), the standard

huggingface/text-embeddings-inference is a server written in Rust with a Candle / ONNX backend, integrated FlashAttention and dynamic batching by tokens. It exposes an OpenAI-compatible /v1/embeddings API and supports the three bge-m3 modes simultaneously from version 1.5 onwards. For multilingual production it is the obvious default.

# values.yaml — TEI serving multilingual bge-m3 on an RTX 4090
image: ghcr.io/huggingface/text-embeddings-inference:1.5
args:
 - --model-id=BAAI/bge-m3
 - --pooling=cls
 - --max-batch-tokens=16384
 - --max-concurrent-requests=512
 - --dtype=float16
resources:
 limits:
 nvidia.com/gpu: 1

Indicative throughput with bge-m3, fp16, 512-token sequence, batch 32:

RTX 4090 (24 GB): ~8–15 k tokens/s
A100 80 GB: ~60 k tokens/s sustained
H100 80 GB: ~40–80 k tokens/s, with fp8 about 50% more

(The ranges are approximate and depend on the real batch, the average sequence length and compilation with FA2/FA3.)

Infinity, the flexible one

michaelfeil/infinity (MIT) is a multi-model FastAPI server able to load bge-m3, Snowflake Arctic, Jina-v3, Nomic, ColPali, CLAP and rerankers simultaneously behind the same OpenAI-style API. Backend PyTorch + Optimum (ONNX/TensorRT) or CTranslate2. Useful when you need to serve several different embedders (one for text, another for code, another for images) behind a single endpoint, or when the model does not yet have TEI support.

vLLM `--task embed`, for the 7B embedders

When the embedder is really an LLM decoder turned into an embedder (e5-mistral-7b-instruct, gte-Qwen2-7B-instruct, NV-Embed-v2, Stella-1.5B), the natural place to serve it is vLLM, which already has the PagedAttention and continuous batching stack in production:

vllm serve Alibaba-NLP/gte-Qwen2-7B-instruct \
 --task embed \
 --dtype bfloat16 \
 --max-model-len 32768 \
 --trust-remote-code

vLLM detects the correct pooling (last-token on the Qwen / Mistral based ones) and exposes an OpenAI-compatible /v1/embeddings. For inference clusters already running vLLM with an LLM on another port, it is the natural way to serve the embedder without standing up another stack.

fastembed, the lightweight one

qdrant/fastembed loads bge-small, MiniLM, ColBERT and BM25/SPLADE sparse in ONNX-CPU. It is not competitive on throughput against TEI/Infinity with a GPU, but it is the right option when you have to serve embeddings on a NUC node without a GPU (see mixed NVIDIA + Intel environments) or when the embedder is part of the client (a preview in a UI, prior scoring at the edge).

Storage, quantisation and the corpus arithmetic

The embedding does not stay in the embedder’s memory: it lives in the vector DB index and is materialised every time you ingest a new chunk. The storage cost calculation is what decides the final dimension, not MTEB quality. For 1 million chunks with a dense embedder at 1024-d:

fp32 : 1,024 dims × 4 B × 1 M = 4,096 MB ≈ 4.0 GB
fp16/halfvec: 1,024 dims × 2 B × 1 M = 2,048 MB ≈ 2.0 GB
int8 (SQ) : 1,024 dims × 1 B × 1 M = 1,024 MB ≈ 1.0 GB
binary : 128 B × 1 M ≈ 128 MB

The quantisation options in order of real use (mid-2026):

halfvec (fp16): the default in pgvector 0.7+ and in any serious vector DB. Zero MTEB loss, 2× compression. Always turn it on.
Scalar Quantization int8 (SQ): each vector component is mapped to int8 with a global min/max. Typical recall@10 loss: 0–1%. 4× compression. The Qdrant default, supported in Milvus and Weaviate.
Binary quantisation: bit = sign(v_i - μ_i). 32× raw compression. Cold loss 5–15%. Mitigated with a Hadamard / TurboQuant pre-rotation (Qdrant 1.18, Dec 2025): it pre-multiplies by a random orthogonal matrix that spreads the energy across dimensions before binarising. After TurboQuant the loss drops to 1–3%. It also combines with rescoring over the original fp16 for the top-100 candidates.
Product Quantization (PQ): the FAISS classic. Up to 64× compression, 2–5% loss. More complex to operate (it needs a trained codebook); in 2026 it has given ground to binary + rescoring.

A corpus of 100 million chunks (a real figure for a large corporate RAG) with dense bge-m3:

Format	Total
fp32	400 GB
fp16	200 GB
int8	100 GB
binary + Hadamard	12.5 GB

The difference between 200 GB and 12.5 GB is the difference between needing a dedicated vector DB node with 8 NVMe in RAID and fitting in the RAM of a single node. For large corpora, quantisation is no longer an optimisation: it is the only way to operate.

The vector DB integration

The vector DBs of 2026 have become hybrid DBs that index all three types at once. The quick map:

Vector DB	Native hybrid	Multi-vector / ColBERT	Quantisation
Qdrant ≥1.10	RRF/DBSF in `query_points` with dense + sparse + colbert in one collection	Yes, native (one-shot MaxSim)	SQ int8, binary, TurboQuant 1.18
Weaviate	`hybrid(alpha=0.75)` BM25 + dense, multi-target named vectors	Yes, as a multi-vector named vector	PQ, SQ, rotational 8-bit BBQ
Milvus ≥2.4	Multi-vector + sparse in the schema; 2.5 adds native BM25 full-text	Multi-vector field, MaxSim orchestrated from the client	SQ, PQ, CAGRA GPU
pgvector 0.7+/0.8	`halfvec`, `sparsevec`, `bit`; HNSW for all three	Not native (separate-table workaround)	`binary_quantize()`, halfvec, rescoring with exact `<#>`
Elasticsearch / OpenSearch	`sparse_vector` (ELSER, SPLADE) + `dense_vector` HNSW; RRF	OpenSearch 3.x yes	ES 9: `int8_hnsw` by default, BBQ binary quantization

For sovereign on-prem production in Spanish, the easiest combination to operate in 2026 is Qdrant + bge-m3: a single collection indexes the three modes of the same model, the hybrid query with RRF is done in one call, TurboQuant quantisation brings the corpus down to manageable levels, and the operator is a Go binary with simple backups to S3/MinIO. pgvector + bge-m3 is the other reasonable option when you already have Postgres with HA and do not want to add a second DB to the operational inventory; you lose native multi-vector, but you gain cross-cutting SQL over the chunks.

The HNSW parameters you absolutely have to touch:

M: connections per node in the graph. 16–32 typical. Higher means more recall, more RAM. For small corpora (<1 M) M=16; for medium corpora (10 M) M=24; for large corpora M=32+IVF-PQ or M=32+binary.
ef_construction: search width during construction. 100–400. Higher means a better graph, slower construction. Build with ef_construction=400 even if it is slow; you pay it once.
ef_search: width during the query. 50–200. The main knob for the recall/latency trade-off at runtime. Start at 64 and measure.

Implications for on-premise inference

The embedder does not share hardware with the LLM as comfortably as it might seem. The numbers:

bge-m3 (568 M) takes about 568 × 2 = 1,136 MB in fp16 for the weights, plus the batch KV cache, plus temporary activations. In practice it is served comfortably in 6–8 GB of VRAM even at a high batch. It fits alongside a 7B-Q4 LLM on an RTX 4090.
gte-Qwen2-7B-instruct needs ~14 GB fp16 for weights alone. It does not fit next to a 7B LLM on a 4090; on an H100 80 GB it does, with care over simultaneous batching.
jina-colbert-v2 (560 M) takes ~1.1 GB of weights, but the storage of the multi-vector index is the real cost: 8 GB per million chunks even with Matryoshka and compression.

On the RTX 4090 (24 GB)

A realistic minimum stack for a Spanish RAG with a corpus under 1 M chunks:

GPU 24 GB ┐
 ├─ TEI bge-m3 (dense + sparse + colbert) │ ~6 GB VRAM, ~12 k tok/s
 └─ vLLM Qwen2.5-7B-Instruct AWQ Q4 │ ~8 GB VRAM, ~80 tok/s
CPU/RAM ┐
 ├─ Qdrant with bge-m3 dense + sparse + colbert │ ~3 GB RAM per M chunks
 └─ FastAPI gateway (LiteLLM)

It serves a few tens of RAG QPS with decent multilingual quality. It is the PoC configuration and the deployment for a small site.

On the 4×H100 80 GB cluster

For the production case with several million chunks and an SLO of p99 < 500 ms:

H100 #1 (80 GB) ── vLLM Qwen3-72B-Instruct AWQ + Qwen2.5-7B speculative ┐
H100 #2 (80 GB) ── vLLM gte-Qwen2-7B-instruct (embedding 32k ctx) │ LLM + large embed
H100 #3 (80 GB) ── TEI bge-m3 multi-tenant + jina-colbert-v2 reranker │ medium embed
H100 #4 (80 GB) ── Hold-out for canary / shadow │ see canary post
 ┘
Qdrant cluster (3 CPU nodes + NVMe) ── 100 M chunks indexed (binary + TurboQuant + rescoring)

This configuration separates the large LLM from the large embedder (they share the Qwen2 architecture but compete for VRAM if put on the same GPU) and leaves a whole H100 for variants in canary. bge-m3 fits with room to spare alongside the reranker on a single H100, serving tens of thousands of requests/min.

The seven operational traps of the embedder

Not normalising the query vector. Cosine and dot product coincide only when both vectors are unit vectors. If you forget v ← v / ‖v‖₂ in the client, the results are “almost right”, the top-1 is still correct on trivial queries, the top-10 no longer is, and nobody notices until RAG quality drops 8 points. Fix: bake normalisation into the embedder adapter, not into the client.
Chat template slipped into the embedder. Some LLM-based embedders (e5-mistral-7b-instruct, gte-Qwen2-7B) expect a specific instruction prompt before the text to embed ("Instruct: Retrieve relevant passages\nQuery: ..."). Forgetting it leaves performance ~5 MTEB points lower. Fix: read the usage_template in the model card and put it into the embedder wrapper.
Badly chosen Matryoshka dimension. The default in many Qdrant / pgvector clients is dim=768. If your embedder is natively MRL at 1024 → 768, fine. If it is 1024 without MRL, truncating to 768 wrecks the space (typical loss -8 MTEB points). Fix: use the model’s native dim and truncate only when storage demands it, and only on MRL models.
Missing hard negatives in fine-tuning. When the embedder is fine-tuned with your own data (which should be standard practice for corporate RAG), if the mini-batch only carries positives and in-batch negatives from the same domain, the model learns that anything outside the domain is negative, but inside the domain it does not discriminate. Fix: mine hard negatives with BM25 / dense from your own corpus before fine-tuning.
Corpus drift without reindexing. When you retrain or replace the embedder but only apply the new model to new chunks, you end up with the index mixing two incompatible geometries. Chunks from the old and the new model are not comparable by cosine. Fix: every change of embedder is a full reindexing of the corpus, planned as an operational retrain.
Tokenizer drift between client and model. The Python client that prepares the queries uses its own tokenizer (sometimes tiktoken by default) and truncates at 8,192 tokens. The embedder uses XLM-R with sentencepiece and truncates at 8,192 of its own tokenizer. Long queries are truncated differently; the corpus embeddings are consistent but the query ones are not. Fix: use the model’s tokenizer in the client or in the wrapper.
MTEB overfit as a selection guide. The MTEB leaderboard has become a contaminated metric: there is evidence that leading models saw subsets of the evaluation datasets during training. The model that is +0.5 points above the second is not necessarily better for your domain. Fix: your own domain eval set (100-300 labelled query-doc pairs) run with the evals recipe decides.

License-clean stack for sovereign production

Let us write down the recommended ending. For a Spanish organisation serving corporate RAG on-prem under ENS / ISO 42001 / EU AI Act, with a corpus of 1-50 M chunks in Spanish + English + Catalan:

Layer	Component	Licence	Rationale
Dense embedder	`BAAI/bge-m3`	MIT	Robust multilingual, 8k tokens, license-clean, served by TEI
Sparse embedder	`bge-m3` sparse head	MIT	Same pass as the dense one, no second model needed
Layer 2 reranker	`BAAI/bge-reranker-v2-m3`	MIT	Multilingual cross-encoder from the same team
Layer 3 reranker (optional)	`jinaai/jina-colbert-v2`	Apache 2.0	License-clean multilingual multi-vector
Embed server	`TEI` + `Infinity` for multi-model	Apache 2.0 / MIT	Supported stack
Vector DB	`Qdrant` (preferred) or `pgvector 0.8`	Apache 2.0 / PostgreSQL	Native hybrid + quantisation
Quantisation	`int8 SQ` + `binary + TurboQuant + rescoring`	Apache 2.0	Cuts the corpus 16×–32× with < 3% loss
Minimum hardware	RTX 4090 24 GB	—	For PoC and small sites
Production hardware	4×H100 80 GB cluster	—	For RAG with an SLO of p99 < 500 ms

The alternative stack, if explicit MMTEB in Spanish weighs more than the tri-modality of bge-m3: replace bge-m3 with Snowflake/snowflake-arctic-embed-l-v2.0 (Apache 2.0, MRL → 256) and explicitly add SPLADE-v3 or plain BM25 for the sparse layer. It loses the elegance of the single forward and gains 1-2 points on Spanish MIRACL.

Conclusion

The embedder is the easiest piece of a RAG to oversimplify and the one that decides real quality the most once everything else is in place. The three families (dense, sparse, multi-vector) are not three options to choose between but three trades that bge-m3 performs in a single pass and that hybrid retrieval consumes in parallel. The maths that matters is modest, InfoNCE with τ, MaxSim, FLOPS regularization, MRL, but the operational traps are many and quiet: forgotten normalisation, missing chat template, badly chosen Matryoshka dimension, tokenizer drift. For sovereign production in Spanish the list of viable models fits in fewer than ten, and the real decision comes down to “bge-m3 or Snowflake Arctic L 2.0”, with jina-colbert-v2 added as layer three when the fine-grained quality justifies the cost. The license-clean stack fits on an RTX 4090 for a PoC and scales to a 4×H100 cluster for real production.

References

Chen et al. M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv:2402.03216. https://arxiv.org/abs/2402.03216
Sturua et al. Jina Embeddings v3: Multilingual Embeddings With Task LoRA. arXiv:2409.10173. https://arxiv.org/abs/2409.10173
Günther et al. Jina Embeddings v4: Universal Embeddings for Multimodal Multilingual Retrieval. arXiv:2506.18902. https://arxiv.org/abs/2506.18902
Nussbaum et al. Nomic Embed v2: Multilingual Mixture of Experts. arXiv:2502.07972. https://arxiv.org/abs/2502.07972
Wang et al. Improving Text Embeddings with Large Language Models (E5-Mistral). arXiv:2401.00368.
Yu et al. Arctic-Embed 2.0: Multilingual Retrieval Without Compromise. Snowflake, 2024-12. https://www.snowflake.com/blog/arctic-embed-2-multilingual/
Lee et al. NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models. arXiv:2405.17428.
Khattab y Zaharia. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. SIGIR 2020.
Santhanam et al. ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. arXiv:2112.01488.
Jha et al. Jina-ColBERT-v2: A General-Purpose Multilingual Late Interaction Retriever. arXiv:2408.16672.
Lassance et al. SPLADE-v3. arXiv:2403.06789.
Kusupati et al. Matryoshka Representation Learning. NeurIPS 2022, arXiv:2205.13147.
Enevoldsen et al. MMTEB: Massive Multilingual Text Embedding Benchmark. arXiv:2502.13595.
Hugging Face Text Embeddings Inference. https://github.com/huggingface/text-embeddings-inference
Michael Feil. Infinity. https://github.com/michaelfeil/infinity
Qdrant. TurboQuant 1.18 release notes. https://qdrant.tech/articles/turboquant-quantization/
pgvector. Release notes 0.7 / 0.8. https://github.com/pgvector/pgvector
Hugging Face. Embedding Quantization. https://huggingface.co/blog/embedding-quantization

Anatomy of the twelve DCGM and five vLLM metrics: analogies, documented anomalies and real cases 2024-2026

Tue, 02 Jun 2026 04:00:00 +0200

This post goes deeper into the list of metrics presented in GPU observability for LLM inference. There each metric got its G/A/R threshold and PromQL query; here each one gets its explanatory analogy and the anomaly documented in production with a referenced public case. It is the post worth having open when an alert fires and you still do not know what to do with it; the next post on runbooks translates each anomaly into concrete action.

TL;DR

The twelve DCGM metrics (compute, memory, thermal-power, health) and the five vLLM engine metrics (concurrency, KV pool, SLO latencies) covered in the previous post paint the cluster’s cockpit, but a list without context does not teach you to diagnose. Each metric has a recurring anomalous pattern documented in the public literature (academic papers, GitHub issues, OEM KBs, operator blogs) that the veteran operator recognises instantly and the junior one does not. This post develops each metric with an analogy of its own that pins down which question it answers, and with the statistically relevant anomaly backed by figures from documented incidents. Three examples of the calibre involved: Meta published that during the training of Llama 3 405B on 16,384 H100s there were 419 unplanned failures in 54 days, one every 3 hours, with GPU + HBM3 accounting for 47 % of the total; the paper Story of Two GPUs (arXiv 2503.11901) quantifies that the H100 has 3.2× worse MTBE from uncorrectable ECC than the A100, attributable to the higher density of HBM3; issue vllm#16300 documents that on a cluster of 8×A100 80 GB TP=8 delivers worse throughput than TP=4 because NVLink saturation kills the partition speedup. The KBs Dell 000220508 and Lenovo HT514380 formalise the recurring HW Power Brake case in H100 racks overcommitted at PDU level. Issue vllm#25677 showed chunked prefill 10× slower than without it on Qwen3-30B-A3B (bad calibration of max_num_batched_tokens). Issue vllm#11912 documents a TPOT regression from 15.7 ms to 25.7 ms crossing version 0.6.4. Each case includes a verifiable URL. The operational rule: when an alert arrives, look first at the anomalous pattern associated with the metric that fired, before opening the request’s trace; 80 % of degradations match one of the documented patterns.

You are here: OBSERVE — the diagnosis layer

Family 1 — Compute

`DCGM_FI_PROF_SM_OCCUPANCY` — is there parallel work in the engines?

The analogy. An industrial kitchen with 32 hobs and a single chef. The metric answers “how many hobs have a pan on them right now?”. If half are empty, the kitchen is under-used: the orders come out one after another because the chef does not open up parallelism. If all of them are occupied but the chef is standing still watching a stopwatch, the hobs are on but nothing is being cooked (a pathological kernel saturating SMs without doing useful work).

The documented anomaly. The best-known trap: high SM occupancy does not imply real throughput. The article “GPU Utilization Is a Counter, Not a Cause” (Ingero, May 2026) put it in one exact sentence: “a kernel running at 5 % of peak FLOPS for 100 ms still reads 100 % on SM_ACTIVE”. In MoE workloads the effect turns pathological: overloaded experts produce the Straggler Effect (paper arXiv 2503.05066). The SMs look busy while waiting for the saturated expert, and the utilisation dashboard paints green while latency goes through the roof.

Operational implication. Do not trust sizing or autoscaling to SM occupancy alone. Always combine it with PIPE_TENSOR_ACTIVE (is there useful compute?) and DRAM_ACTIVE (is memory the bottleneck?). The normal LLM decode regime is 30–55 %, not 99 %; seeing a sustained 99 % with high TPOT is a symptom of a kernel bug or an MoE straggler.

`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE` — are the tensor cores producing?

The analogy. A factory with two lines: the manual one (CUDA cores) and the automated one (tensor cores). The metric answers “what percentage of the time is the automated line active?”. If you buy an H100 for its tensor cores and the automated line is at 5 %, you have paid for a Ferrari to do bicycle courier work.

The documented anomaly. Issue vllm#20783 (July 2025) was titled literally “Performance Anomaly: compressed-tensors shows no speedup over BF16 on H100”. The operator expected 1.5–2× with FP8 quantisation and got parity with BF16; the PIPE_TENSOR_ACTIVE metric revealed that the FP8 path was not running on the HMMA units (the tensor unit for FP16/BF16/FP8) and was falling back to CUDA cores. Issue vllm#31475 documented the parallel case on MI300X: FP8 slower than BF16 because of a regression in the ROCm path. DCGM exposes separate counters per unit (HMMA for FP16/BF16/FP8, IMMA for INT8, DMMA for TF32/FP32); if HMMA is low even though the model is BF16, the engine is not using tensor cores.

Operational implication. Check PIPE_TENSOR_ACTIVE after every change of quantisation or engine version; a supposedly neutral change may have disabled the optimised path. For prefill expect 50–80 %; for decode 15–30 % is normal (decode is memory-bound, not compute-bound). A figure below 5 % in prefill means the engine is not using tensor cores.

`DCGM_FI_PROF_DRAM_ACTIVE` — is the HBM saturated?

The analogy. A motorway with N lanes. The metric answers “what percentage of the time are all the lanes occupied moving cars?”. When the tensor cores ask for data faster than the HBM delivers it, the motorway is at 95 % and the engines wait. In decode, this is the normal regime: you walk the model weights and the KV cache through for every token.

The documented anomaly. The paper “Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference” (arXiv 2503.08311) quantifies that at contexts ≥ 128k, reading the KV cache dominates the total decode time and saturates the HBM3 (3.35 TB/s on the H100). The distinctive pattern: DRAM_ACTIVE > 80 % with PIPE_TENSOR_ACTIVE around 10–20 %. Raising the batch no longer helps: the bottleneck is not FLOPS, it is bandwidth. The useful lever is compressing the KV, see Quantisation for --kv-cache-dtype=fp8, which cuts the KV footprint by roughly 50 %.

Operational implication. If DRAM_ACTIVE > 95 % is sustained and gpu_cache_usage_perc < 70 %, something other than your engine is asking for HBM (a leak in a library, another process sharing the GPU without MIG). Investigate immediately with nvidia-smi and fuser /dev/nvidia*.

Family 2 — Memory

`DCGM_FI_DEV_FB_USED` — how much VRAM has been consumed?

The analogy. The fuel level in an aircraft’s tank in flight: the pilot needs to know how much is left and at what rate it is being consumed, not just the instantaneous figure. An H100 at 88 % FB used and stable can operate calmly; the same figure rising 2 %/min announces OOM in 7 minutes.

The documented anomaly. Issue dcgm-exporter#512 documents a surprise relevant to MIG clusters: DCGM_FI_DEV_FB_USED and DCGM_FI_DEV_FB_FREE are absent on H100 GPU instances with MIG enabled, present on A100 and B200, but an exporter bug hides them on H100-MIG. Operators who assume the dashboard covers everything discover the blind spot on the day of the first OOM. Issue dcgm-exporter#271 documents another detail: FB_USED + FB_FREE does not always add up to a constant because there is driver-reserved overhead that shows up in the delta. The original PagedAttention/vLLM paper estimated that pre-PagedAttention serving frameworks wasted 60–80 % of the KV cache to fragmentation; PagedAttention brought it down to under 4 %.

Operational implication. On H100 MIG clusters, check that DCGM_FI_DEV_FB_USED appears per instance before trusting alerts; if it is absent, monitor through nvidia-smi --query-gpu=memory.used directly. Operational rule: alert on the delta (a sustained rise), not just an absolute threshold.

`DCGM_FI_DEV_FB_FREE` — the absolute complement

The analogy. The “range remaining” indicator in a modern car: it complements the percentage with an absolute figure that is directly actionable.

The documented anomaly. When an aggressive PagedAttention pool leaves FB_FREE at small absolute values (< 2 GiB), any normal allocation of transient buffers (the activations of a large prefill) can push you into OOM. The classic pattern: a “green” percentage (87 %) but a “red” absolute (< 4 GiB free on an 80 GB H100).

Operational implication. A complementary alert with an absolute threshold: DCGM_FI_DEV_FB_FREE < 4096 (MiB). It is the safety net for the cases where the percentage misleads because the engine is configured with a very high gpu_memory_utilization.

`DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL` — does the internal bus hold up?

The analogy. An interstate motorway between four cities. Every car that crosses to do a tensor parallel all-reduce pays a toll and consumes width. When there are more cars than the motorway supports, the latency to reach the destination shoots up, even if each individual car is fast.

The documented anomaly. Issue vllm#16300 (April 2025) was titled “Performance degradation with tp=8 compared to tp=4 on 8×A100(80G)” and documented TP=8 delivering worse throughput than TP=4 on the same cluster, same model, same quantisation. Root cause: tensor parallelism requires an all-reduce after each attention and MLP block; at TP=8, the communication cost between 8 GPUs (even over NVSwitch) grows faster than the speedup of partitioning the compute. The practical rule that emerges: TP=4 + 2 replicas usually delivers better latency/throughput than TP=8 + 1 replica, except for extremely long contexts (≥128k) where you need the aggregate VRAM. Theoretical NVLink 4.0 capacity on the H100 SXM: around 450 GB/s per GPU; typical sustained TP=4 regime: 50–150 GB/s.

Operational implication. If NVLINK_BANDWIDTH_TOTAL > 90 % of capacity is sustained, this is not a problem solvable by raising parallelism. The opposite: lower TP. The metric is orthogonal to the sizing from capacity planning: the ceiling is not only VRAM/time, it is also the bus.

Family 3 — Thermal and power

`DCGM_FI_DEV_GPU_TEMP` — is the GPU breathing?

The analogy. The body temperature of an elite athlete under full effort. 36–37 °C is normal; 38 °C is sustainable stress; above 39 °C the body activates protective mechanisms (sweating, slowing down) that degrade performance. The GPU does the same: above a thermal threshold, it reduces its clock automatically. If it did not, it would break.

The documented anomaly. The H100 SXM5 with its 700 W TDP has thermal thresholds that are not entirely public (NVIDIA does not publish them exhaustively in the datasheet), but the behaviour is well known: above roughly 85 °C edge or roughly 95 °C HBM, the bit 0x40 HW_THERMAL appears in the clock throttle reasons. Operators on the NVIDIA developer forum report that with a rack inlet temperature above 27 °C, throttling is routine. The NVIDIA HGX Platform paper states that the minimum recommended air flow is > 1000 CFM/kW; densities above 30 kW/rack at 700 W TDP demand mandatory liquid cooling because forced air does not get there.

Operational implication. If GPU_TEMP > 83 °C is sustained, look first at CLOCK_THROTTLE_REASONS (bit 0x40) and at the rack inlet temperature: this is not an engine problem, it is an air flow one. For legacy air-cooled racks, consider redistributing thermal load or installing a rear-door HX.

`DCGM_FI_DEV_POWER_USAGE` — how much is it asking of the socket?

The analogy. The instantaneous draw of an industrial appliance plugged into a three-phase outlet with a sized breaker. If the washing machine starts at 9 kW and the breaker is 10 kW, you live on the edge; if the washing machine “gets along” with the breaker it is because somebody sized it consciously.

The documented anomaly. A published real measurement: an H100 SXM5 with vLLM running Llama 3.1 405B at batch=4 draws around 697 W at-wall sustained (NVIDIA TDP 700 W). Now the interesting operational lever: lowering nvidia-smi -pl from 700 W to 500 W delivers around 30 % energy savings with only around 20 % loss of throughput. A cluster of 4 nodes × 8 H100 at 700 W equals around 22 kW of GPU alone; at 500 W, around 16 kW. The difference pays an entire quarter’s electricity bill in clusters operated around the clock. A three-phase 415 VAC PDU branch at 60–80 A supports around 32 kW, roughly 4 DGX H100. Legacy 208 V does not support H100 density; reference: NVIDIA DGX SuperPOD Electrical Specifications.

Operational implication. A useful metric for three things: (1) detecting anomalously low workloads (unexpected idle), (2) computing per-tenant showback of real energy cost (not estimates), (3) alerting if the draw approaches the PDU branch limit. Keep GPU → PDU branch → breaker mapped in the CMDB.

`DCGM_FI_DEV_CLOCK_THROTTLE_REASONS` — who is stepping on the brake?

The analogy. The “limited mode” warning light on the dashboard of a modern car. When it lights up, the car reduces its performance automatically, but it does not tell you why unless you know how to read the combination of letters. The bits of the bitmap are those letters.

The documented anomaly. A public case formally acknowledged by two different OEMs: Dell KB 000220508 and Lenovo HT514380 address the same phenomenon: HW Power Brake Slowdown active (bit 0x80) on the H100 SXM. The cause is not the GPU: it is the chassis PDU sending an electrical power-brake signal because the rack branch is close to the breaker limit. The operator sees throughput down 30–50 % with no XID and no ECC, and the inference engine “is healthy”; the problem is in the electrics. The NVIDIA developer forum thread on “HW Power Brake Slowdown” corroborates the pattern. Bit 0x40 HW_THERMAL appears in badly ventilated racks; bit 0x04 SW_POWER_CAP appears if somebody left nvidia-smi -pl 500 set and nobody reverts it.

Operational implication. Any bit other than 0 or Idle (bit 0x01) sustained is an immediate alert. The recommended decoding: record the complete bitmap value in the log plus the attribute throttle.reasons.decoded=["HW_THERMAL", "HW_POWER_BRAKE"] on the OTel span. Without that, incident response does not know what to do.

Family 4 — Health (the catastrophic reports)

`DCGM_FI_DEV_XID_ERRORS` — the driver’s red codes

The analogy. The critical alarm lights in a nuclear control room. They do not climb gradually; they either appear or they do not. Each XID is a predefined code (XID 13 = graphics engine exception; XID 31 = MMU fault; XID 43 = stopped channel; XID 79 = GPU fallen off the bus; XID 95 = uncontained ECC), and each one has its documented procedure.

The documented anomaly. The most studied public case: Meta published that during the training of Llama 3 405B on 16,384 H100s over 54 days there were 419 unplanned failures, one every 3 hours at cluster scale. GPUs accounted for 148 (35 %) plus HBM3 for 72 (17 %), almost half of all failures. The paper “Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100” (arXiv 2503.11901) quantifies with a different dataset (2.1M GPU-hours) that the H100 has 3.2× worse MTBE for uncorrectable ECC than the A100. The ByteDance MegaScale paper reports that XID 79 (“GPU fallen off the bus”) co-occurs with PCIe errors in 43 % of cases. The NVIDIA developer forum documents persistent cases of XID 31 (MMU fault) that follow the GPU when it is moved to a different PCIe slot, a hardware bug in the module, not in the backplane.

Operational implication. Any increment of the counter is an immediate alert: many XIDs demand a node reset or an RMA of the GPU. The XID-by-XID distinction matters: XID 13/43 is usually a software bug if it coincides with a recent change; XID 31/48/79/94/95 is usually hardware. Keep a canonical xid → procedure table. See the runbooks for the translation into concrete action.

`DCGM_FI_DEV_ECC_DBE_VOL_TOTAL` — the errors that corrupt data

The analogy. An accounting ledger where sometimes someone erases an entry and rewrites it (corrected single-bit ECC: note a change in the margin and carry on) and sometimes someone burns two pages at once (double-bit: the information is lost, the audit has to stop).

The documented anomaly. The paper “Characterizing GPU Resilience” quantifies for the H100 that when XID 48 (DBE) appears, the job in flight dies with 100 % probability (5 out of 5 in the dataset studied). The documented recovery: drain the node, reset, complete the row remap, equalling around 19 hours of node downtime. HBM3 density explains the worse MTBE against HBM2e: there are more cells per unit area, and therefore a higher statistical probability of degradation. In Llama 3, HBM3 caused 72 of the 419 interruptions (17 %).

Operational implication. Any value > 0 is a critical alert. The GPU must be drained immediately, removed from the scheduler, fully reset, validated for row remap with nvidia-smi -q -d ROW_REMAPPER (Pending: No), and given an extensive smoke test before returning to the pool. If the row remap uses more than 4–8 spare pages on one GPU, plan a replacement in the next window: the degradation is progressive.

`DCGM_FI_DEV_RETIRED_DBE` — the pages marked for retirement

The analogy. The tiles a museum restorer marks with yellow tape because they are damaged. They pose no immediate danger (the room stays open), but the accumulation says the floor is degrading structurally and the full replacement has to be planned.

The documented anomaly. NVIDIA documents up to 512 spare pages per HBM bank on the H100; the RETIRED_DBE counter indicates how many have been used. Operators on NVIDIA forums report that above 4–8 retired pages on a specific GPU, the frequency of XID 48 rises. The pattern: a GPU with 6 retired pages today becomes 12 in a month, a first XID 48 two months later, and a forced drain.

Operational implication. A trend metric, not an immediate alert. Record the value per GPU and review it monthly; GPUs with rising values go into the proactive replacement plan before the catastrophic failure.

The five vLLM engine metrics

`vllm:num_requests_running` — how many requests fit in the batch?

The analogy. The number of cars a toll booth lets through at once. If the barrier opens for N at a time, car N+1 waits in the queue. Saturation shows up because the line does not get shorter.

The documented anomaly. Reaching the configured --max-num-seqs and staying there is the typical symptom of a cluster below its sizing; the engine admits up to the ceiling and no further. The query vllm:num_requests_running == max_num_seqs for more than 5 minutes indicates firm saturation.

Operational implication. Combine it with num_requests_waiting: if running is at the ceiling AND waiting > 0, you have to scale. If running is at the ceiling and waiting is 0, you are in the optimal regime (cluster used to the maximum with no queue).

`vllm:num_requests_waiting` — the primary saturation indicator

The analogy. The visible queue in front of the toll booth. As long as it is empty, the system flows; the moment a sustained queue forms, drivers start arriving late at their destination, and TTFT goes through the roof.

The documented anomaly. A public case in “11-Second Time to First Token on a Healthy vLLM Server” (Medium, Ingero, 2026): a server with no XIDs, no preemption, DCGM metrics all green, but num_requests_waiting sustained above 0 and a TTFT of 11 seconds. Issue vllm#16985 documents progressive degradation in long sessions: the queue grows slowly over hours without any other indicator moving. The root cause is not hardware, it is admission control: the arrival rate exceeds the completion rate and the system does not enqueue any more, it leaves requests in WAITING. Red Hat designates it as the primary saturation metric in its tutorial “5 steps to triage vLLM performance”.

Operational implication. The primary HPA metric in KEDA, see Autoscaling LLM on Kubernetes. Typical threshold: alert if avg_over_time(vllm:num_requests_waiting[5m]) > 5. For canaries: if the queue only forms in the v2 pool, it is a regression of the new model, not cluster load.

`vllm:gpu_cache_usage_perc` — the KV cache pool

The analogy. The capacity of an events hall where each guest takes up a variable amount of space. The maître d’ admits guests up to capacity; when a new guest arrives and there is no room, he throws out the guest who has been there longest to make space for the newcomer. That is vLLM’s preempt-on-OOM.

The documented anomaly. Issue vllm#5051 “Add num_requests_preempted metric” was born exactly out of operators observing degradation with no direct metric telling them how many requests were being thrown out. The official vLLM documentation confirms: “sustained gpu_cache_usage_perc above 90 % indicates the server is approaching its KV cache limit and will begin preempting sequences” (oldest-first). The distinctive visual pattern: a sawtooth near 100 % with preemption spikes. In swap mode, the latency of the preempted request explodes because there is a host↔device PCIe copy; in recompute mode (the default in V1), the preempted request redoes its prefill from scratch, which pushes its TTFT to double or triple.

Operational implication. If gpu_cache_usage_perc > 92 % is sustained, there are two levers: lower max_num_seqs (admit less concurrency, but none of it gets preempted) or raise gpu_memory_utilization (more pool, less VRAM for transient activations, a different risk). The choice depends on the workload. The metric that is directly missing, a preempted counter, is exported from vLLM v1.0 onwards as vllm:num_preemptions_total (see issue #5051).

`vllm:time_to_first_token_seconds` — the latency the client sees

The analogy. The time from a customer walking into a restaurant to receiving the first piece of bread at the table. Too long and the customer thinks they have been forgotten, even if the main course is going to arrive perfect.

The documented anomaly. Three documented patterns of recurring TTFT spikes:

Badly calibrated chunked prefill. Issue vllm#25677 (Qwen3-30B-A3B) reported prefill 10–11× slower with chunked prefill enabled than without it. Cause: a very low max_num_batched_tokens forces small chunks that do not fill the kernels. Issue vllm#7604 documents an equivalent regression on Llama-3-70B v0.5.4. The lever: raise max_num_batched_tokens to 4096–8192 for typical prompts above 2k.
Regression between engine versions. Issue vllm#8819 documents a regression of vllm:time_to_first_token_seconds_sum between minor versions. Issue vllm#11912 reports that with a prompt of around 8000 tokens, TPOT rose from 15.7 ms → 25.7 ms from v0.6.4.post1 onwards with no config change, a regression confirmed and trackable only with the metric.
Long-context prefill blocking decodes. The “11s TTFT on healthy server” case cited above: a 30k-token prefill monopolises the GPU for several seconds and the active decodes freeze. Fix: well-calibrated chunked prefill, or disaggregated serving (see Disaggregated serving).

Operational implication. Do not alert only on the absolute P95; alert also on the v2/v1 ratio when there is a canary (histogram_quantile(0.95, ..., version="v2") / histogram_quantile(0.95, ..., version="v1") > 1.10). If TTFT grows and the queue is stable, the bottleneck is prefill: not solvable by adding replicas, but there are levers in quantisation or chunked prefill.

`vllm:time_per_output_token_seconds` — streaming smoothness

The analogy. The speed at which the waiter brings the courses one after another after the first. If the next one takes a while, the diner notices something is wrong even though the first course arrived on time.

The documented anomaly. The distinctive pattern is the abrupt step when gpu_cache_usage_perc crosses roughly 85 %: TPOT jumps from 35 ms to 80 ms in a few seconds because the engine starts competing for the HBM with its own evictions. Issue vllm#35387 documents another anomalous case: MTP (speculative decoding) causing a 76 % latency regression on Qwen3-Next-80B-A3B-Instruct-FP8. The TPOT metric caught it before any client complaints were filed.

Operational implication. The difference from TTFT: if TTFT grows and Queue Time is stable, you are prefill bound; if TPOT grows at a steady rate, there is pressure on the HBM (KV cache pool or swap enabled). A secondary alert on the TPOT SLO, but also watch the derivative: TPOT rising 1 ms every 10 minutes is a latent regression that does not break the SLO yet but will.

The operational rule: read metrics by family, not in isolation

Three anti-patterns of the novice operator

Anti-pattern 1 — alerting only on absolute thresholds. An H100 at 87 % FB is not necessarily an alarm; an H100 at 87 % rising 2 %/min is. Alerts that fire on a fixed threshold without looking at the derivative produce twice the noise and half the value. Rule: for metrics with known dynamics (KV cache, FB, queue), alert on a sustained delta, not just on the level.

Anti-pattern 2 — confusing SBE with DBE. The DCGM_FI_DEV_ECC_SBE_VOL_TOTAL counter (single-bit, correctable) grows continuously on any HBM under load; it is not an alarm, it is physics. The one that matters is DCGM_FI_DEV_ECC_DBE_VOL_TOTAL (double-bit, uncorrectable). Confusing them yields false negatives (not alerting on a real DBE) or false positives (alerting on a harmless SBE).

Anti-pattern 3 — treating SM_OCCUPANCY at 99 % as “saturated”. The LLM decode regime is memory-bound, not compute-bound; high SM occupancy with low TENSOR_ACTIVE and high DRAM_ACTIVE is normal. Sizing for “GPU at 60 %” and asking for more hardware when the cluster is saturated in HBM (not in SMs) means buying twice the GPUs without gaining throughput. Rule: always read SM_OCCUPANCY together with TENSOR_ACTIVE and DRAM_ACTIVE; on its own it means nothing.

Applied to typical on-premise hardware

For a generic cluster of 4 nodes × 4×H100 SXM 80 GB with intra-node NVLink:

DCGM Exporter per node (a GPU Operator DaemonSet) emitting every 15 s; cardinality per GPU is around 80 series. A 16-GPU cluster gives roughly 1.3k base series, around 85k samples/min with a 15 s scrape.
vLLM /metrics per inference pod; each replica emits around 50 base series. For 16 replicas, around 800 additional series, around 3k samples/min.
Prometheus retention: 30 days at high resolution plus 1 year downsampled through a Thanos sidecar or Mimir. Estimated volume: 25–35 GB/day.
Alertmanager: the 6 critical alerts from the previous post plus derived alerts (delta, v2/v1 ratio, decoded throttle bitmap).

Each metric is worth exposing also as an OTel attribute on the spans of the GenAI tracing: gpu.fb_used_pct, gpu.dram_active, gpu.throttle_reasons.decoded. That lets you correlate a slow request with the state of the GPU at that instant, without jumping between dashboards.

What we have not covered (upcoming posts)

Per-alert runbooks — the translation of each anomalous metric into concrete action (drain, reset, RMA, scaling, rollback) in the next post: Incident response runbooks.
Tail sampling for metric ↔ trace correlation — what is preserved when an alert fires, for post-mortem investigation.
Per-tenant showback combining vllm:request_success_total × gen_ai.usage.* × DCGM_FI_DEV_POWER_USAGE to bill real energy cost.
Multi-tenant fairness metrics — when a tenant hogs the KV cache pool and how to detect it.

References

Meta — Faulty Nvidia H100 GPUs and HBM3 memory caused half of failures during Llama 3 training (Tom’s Hardware, 2024). tomshardware.com
Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100. arXiv 2503.11901. https://arxiv.org/html/2503.11901v3
ByteDance — Robust LLM Training Infrastructure at ByteDance. arXiv 2509.16293. https://arxiv.org/pdf/2509.16293
Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference. arXiv 2503.08311.
Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts. arXiv 2503.05066.
NVIDIA — Analyzing Xid Errors with the Xid Catalog and Memory Error Management (docs.nvidia.com/deploy).
Dell — PowerEdge XE8640 with H100 - GPU Performance Issue HW Power Brake Slowdown - Active (KB 000220508).
Lenovo — Power brake reporting on H100 GPU (HT514380).
vLLM project — issues #5051 (preempted metric), #7604 and #25677 (chunked prefill regression), #11912 (long-prompt regression), #16300 (TP=8 worse than TP=4), #16985 (long-running degradation), #20783 (compressed-tensors no speedup), #35387 (MTP regression).
Red Hat — 5 steps to triage vLLM performance. https://developers.redhat.com/articles/2026/03/09/5-steps-triage-vllm-performance
AI21 — Go big or go OOM: the art of scaling vLLM. https://www.ai21.com/blog/scaling-vllm-without-oom/
11-Second Time to First Token on a Healthy vLLM Server (Medium, Ingero, 2026).
NVIDIA — DGX SuperPOD Electrical Specifications (docs.nvidia.com/dgx-superpod).

Sources: the full URLs are linked inline on each reference.

Canary, blue-green and shadow for LLM models: how to deploy a new version without breaking the SLO

Mon, 01 Jun 2026 16:30:00 +0200

This post complements those on LLM autoscaling on Kubernetes (the autoscaler coexists with the rollout and must respect it), GPU observability for LLM inference (the metrics that act as gates come from there), Evals for LLMs (the eval that decides whether the new model is ready), LLM-as-judge (the technique that puts the “quality” into the canary gate) and Retrain: closing the loop (the previous step that the new model comes out of).

TL;DR

Promoting a new version of an LLM model to the production cluster without cutting traffic or breaking the SLO demands progressive deployment. The three canonical strategies, blue-green, canary and shadow, answer different questions and have different costs. Blue-green: a complete new pool brought up in parallel, atomic load balancer switch. Instant rollback (point back at the old pool); it demands twice the GPUs during the window. Canary: traffic is shared progressively between the old version and the new one (1 % → 5 % → 25 % → 100 %), measuring regression gates at each step; it consumes incrementally less hardware but exposes real users to the new model from the first percentage point. Shadow / mirror: the old model serves 100 % of the real traffic to the client and, in parallel, a copy of every request goes to the new model without returning its response to the user; it isolates you from quality risk but spends the new model’s GPU on responses nobody consumes, and it does not work well with long SSE streaming. The choice depends on three factors: available GPU budget, service criticality and the availability of a fast automatic eval. The five regression metrics any LLM canary should gate on are: TTFT P95, error rate (HTTP 5xx + premature finish_reason="length"), quality score with LLM-as-judge over a golden set, statistical drift of output embeddings (Wasserstein or KL against the baseline distribution) and cost per request (tokens/s and kW/request). On Kubernetes, Argo Rollouts manages traffic and the AnalysisTemplates as automatic gates; Flagger is the more opinionated alternative. vLLM v1 does not support hot model swap robustly as of May 2026, so the rollout unit is the whole replica (deployment v2 alongside deployment v1). The three specific pitfalls: LB sticky sessions break the statistical comparability of the canary (a client A always lands on the new one, B on the old one, so the populations are not equivalent); semantic eval with LLM-as-judge takes 2–8 seconds per sample and is no use as a real-time gate (it is used in post-analysis or offline before promotion); SSE streaming complicates shadowing because the new model’s response has to be discarded without affecting the old one’s. This post includes a minimal Argo Rollouts manifest applicable to a generic cluster with the NVIDIA GPU Operator.

You are here: DEPLOY (and the transition to RETRAIN)

A new model does not appear in the cluster by magic: it comes from the retrain loop or from a weights provider update. The step between “I have an artifact that passed offline eval” and “it is serving 100 % of the traffic” is exactly this post.

The analogy: opening night of a play

A theatre company is about to premiere a new version of a play that has been running successfully for a year. The company knows several hard facts: the current audience pays for a consistent experience; a bad opening damages the business for months; but premiering nothing leaves the company obsolete against the competition.

The three opening routes the management can choose are the same three of the LLM rollout.

Dress rehearsal behind closed doors (shadow / mirror). The new actors perform the whole play to an empty theatre. There is no audience; nobody buys a ticket. Three full run-throughs are enough to check continuity, timing and the chemistry of the cast. It is expensive because there are salaries and theatre rent, but it does not expose the audience to risk. Useful when the new cast is untested and the director wants to see how it holds up over a complete performance before selling it. In LLM terms: the new model processes every real request in parallel with the old one but its responses are discarded; you spend the new model’s GPU on responses nobody sees.

Cast by performance, alternating (canary). Instead of changing the whole cast at once, Thursday performances belong to the new cast, Friday ones to the old one, Saturday ones half and half. The management reads the visitors’ book comments and the audience numbers performance by performance, deciding after two weeks whether to promote the new cast to permanent or withdraw it. Cheaper than the dress rehearsal because the performances still sell tickets, but it exposes a real audience to risk from the first Thursday. In LLM terms: traffic is shared progressively between the old version and the new one, measuring gates at each step.

Double company with an atomic switch (blue-green). The company hires the new cast, prepares it for a month behind closed doors, and one Saturday announces: “from the next opening night every performance is with the new cast”. If the first performance goes badly, they go back to the old cast in 24 hours, but during that month of preparation double salaries are paid to both companies. In LLM terms: two complete pools of the same size, instant LB switch from one to the other, rollback in seconds if the metrics break.

The analogy also supports the decision: the choice depends on how critical the play is for the business (criticality of the LLM service), how much budget there is to sustain two casts at once (GPU budget), and how much confidence there is in the new cast from the chamber rehearsals (offline eval prior to the canary).

The three strategies in detail

Blue-green

The operator keeps two replica pools identical in size: the blue one (production version v1) and the green one (candidate version v2). When v2 is validated offline (eval passed, smoke tests), the LoadBalancer switch redirects 100 % of the traffic from blue to green in a single step. If the SLO metrics break, the switch goes back in seconds.

Cost: 2× GPUs for the whole window (preparing v2 + the post-switch observation window). For a 16-GPU cluster serving Llama 70B with TP=4 (4 replicas), preparing the blue-green needs 16 extra GPUs for 1–3 days.

Risk: the switch is atomic. If v2 has a problem that did not show up in offline eval but does show up at scale (for example, edge cases that only appear at 200 RPS), 100 % of users notice it at once. Rollback is instant, but the requests of the first minute after the switch were already affected. So blue-green is preferable when you have high confidence in v2 (a minor change: same architecture, same format, just a new version of the weights) and you prioritise immediate rollback over gradual exposure.

Canary

The operator deploys v2 with a small number of replicas (typically 1) next to the v1 pool. The LoadBalancer shares the traffic out progressively on a schedule: 1 % for 30 minutes → 5 % for 1 hour → 25 % for 2 hours → 50 % for 4 hours → 100 %. Between each step, an analysis gate evaluates regression metrics over the traffic that is already landing on v2. If the gate fails, the rollback withdraws traffic from v2 automatically and leaves v1 serving everything.

Cost: incremental. At the start (1 % of traffic) one v2 replica is enough; at 50 % you need half as many v2 replicas as v1 has in total. Peak extra GPU during the canary: about 30–50 % above baseline.

Risk: real users are seeing v2 from the first 1 %. If v2 produces responses with degraded quality but normal TTFT and error rate, the affected users perceive the degradation without the gate detecting it (unless the gate includes quality drift, which is slow). So canary is preferable when you have medium confidence in v2 (a significant change: different architecture or training) and you accept that a low percentage of users will be guinea pigs.

Shadow / mirror

The LoadBalancer sends 100 % of the real traffic to v1 (which answers the client) and duplicates every request towards v2 (whose response is discarded or stored for offline analysis). The client never sees v2; it is never exposed to the risk.

Cost: 100 % extra compute for v2 with no direct user value for the whole shadow window. For a 16-GPU cluster serving Llama 70B with TP=4 (4 replicas), a shadow of the same size consumes 16 extra GPUs full time.

Risk: shadow is the safest for the user. But it has two serious limitations: (a) if v2 has a bottleneck that makes the copied request to the shadow take a long time, the shadowing proxy can consume LB connections; it must be out-of-band (asynchronous); (b) long SSE streaming complicates mirroring because you have to keep two parallel streams and discard one while the other flows to the client. Common pattern: shadow only non-streaming requests (short completions, classification), manual offline eval of the streaming requests.

The five regression metrics that act as a gate

Without automatic gates, the “canary” is just a nice name for “manual rollout with a variable percentage”. The gates are the piece that turns the canary into a defensible operation.

Metric 1 — TTFT P95. Comparison of the new model’s P95 against the baseline (v1) P95 in 5-minute windows. Gate: ttft_p95(v2) / ttft_p95(v1) < 1.10. It detects prefill latency regressions (a slower new model) or engine problems (suboptimal config). Source: vllm:time_to_first_token_seconds_bucket (see GPU observability for LLM inference).

Metric 2 — Error rate. Sum of HTTP 5xx + unexpected 4xx + rate of premature finish_reason="length" (responses cut short because the new model does not generate EOS). Gate: error_rate(v2) - error_rate(v1) < 0.01 (1 percentage point). It detects engine crashes, a broken tokenizer, generation problems. Source: vllm:request_success_total{status=...}.

Metric 3 — Quality score (LLM-as-judge). Over a golden set of 200–1,000 representative prompts, v1 and v2 are run offline and a judge model (typically larger: GPT-4 class, Claude, local Llama 405B) scores each pair. Typical gate: mean_score(v2) >= mean_score(v1) - 0.05. This metric is not measured in real time during the canary: judge inference takes 2–8 seconds per sample and does not scale as an inline gate. It is used as an offline pre-promotion gate (before starting the canary) and as a post-mortem over a sample of real traffic captured during the canary. See LLM-as-judge for the mechanics.

Metric 4 — Statistical output drift. For every request that lands on v2 during the canary, embed the response with a lightweight embedding model (e5, BGE) and compare the distribution of v2 embeddings against the distribution of the v1 baseline over the same window. Usable metrics: Wasserstein distance, KL divergence, or more simply, comparing means and variances per dimension. Gate: normalised distance < a calibrated threshold (typically Wasserstein < 0.15). It detects subtle changes in style, length and vocabulary that LLM-as-judge does not capture without going through it too. It is fast: the lightweight embedding takes around 50 ms per response.

Metric 5 — Cost per request. Output tokens per request and kW per request. Gate: cost_per_request(v2) / cost_per_request(v1) < 1.20. It detects new models that generate significantly longer responses or that consume more energy for the same load (quantisation degradation, optimisations failing). Without this gate, an “update” can silently double the bill.

Metric	Type	Measurement latency	Typical gate	Detection
TTFT P95	Quantitative	5 min	`< 110% baseline`	Latency regression
Error rate	Quantitative	1 min	`< 1pp over baseline`	Crashes, generation broken
Quality (LLM-judge)	Offline semantic	hours, over golden set	`> baseline − 0.05`	Functional quality
Statistical drift	Statistical	~5 min	Wasserstein < 0.15	Style, length, vocabulary
Cost per request	Quantitative	5 min	`< 120% baseline`	Economic/energy efficiency

The mechanics on Kubernetes: Argo Rollouts

Argo Rollouts extends the standard Kubernetes Deployment with a new Rollout resource that orchestrates traffic progression and automatic analyses. It integrates with any service mesh (Istio, Linkerd) or ingress controller that supports traffic splitting (NGINX, Traefik, Gateway API).

A minimal example of a 1 → 5 → 25 → 100 % canary with TTFT and error rate gates:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: vllm-llama70b }
spec:
 replicas: 4
 strategy:
 canary:
 canaryService: vllm-llama70b-canary
 stableService: vllm-llama70b-stable
 trafficRouting:
 nginx:
 stableIngress: vllm-llama70b-ingress
 steps:
 - setWeight: 1
 - pause: { duration: 30m }
 - analysis: { templates: [{ templateName: ttft-error-gate }] }
 - setWeight: 5
 - pause: { duration: 1h }
 - analysis: { templates: [{ templateName: ttft-error-gate }] }
 - setWeight: 25
 - pause: { duration: 2h }
 - analysis: { templates: [{ templateName: ttft-error-gate }, { templateName: drift-gate }] }
 - setWeight: 50
 - pause: { duration: 4h }
 - analysis: { templates: [{ templateName: ttft-error-gate }, { templateName: drift-gate }] }
 - setWeight: 100
 selector: { matchLabels: { app: vllm-llama70b } }
 template:
 metadata: { labels: { app: vllm-llama70b } }
 spec:
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.10.0
 args: [ --model=/models/llama-70b-fp8-v2 ]  # new version
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: ttft-error-gate }
spec:
 metrics:
 - name: ttft-p95-ratio
 interval: 1m
 count: 5
 failureLimit: 1
 successCondition: result < 1.10
 provider:
 prometheus:
 address: http://prometheus.observability.svc:9090
 query: |
 histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{version="v2"}[5m])))
 /
 histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{version="v1"}[5m])))
 - name: error-rate-diff
 interval: 1m
 count: 5
 failureLimit: 1
 successCondition: result < 0.01
 provider:
 prometheus:
 address: http://prometheus.observability.svc:9090
 query: |
 sum(rate(vllm:request_total{version="v2",status=~"5.."}[5m])) / sum(rate(vllm:request_total{version="v2"}[5m]))
 -
 sum(rate(vllm:request_total{version="v1",status=~"5.."}[5m])) / sum(rate(vllm:request_total{version="v1"}[5m]))

If any of the AnalysisTemplates fails, Argo Rollouts backs out automatically: it sets weight=0 on the canary, alerts the operator and keeps v1 serving 100 %. Human operation is reduced to investigating the failure and deciding whether to relaunch or abort.

Flagger offers a more opinionated alternative: weight progression is automatic as a function of metric success instead of a fixed pause; the operator defines a target (maxWeight: 100, stepWeight: 10, metrics: [...]) and Flagger raises or lowers it according to behaviour. Both are mature as of May 2026; the choice is usually dictated by which service mesh is already in the cluster.

The vLLM detail: why there is no “hot swap” of the model

As of May 2026, vLLM v1 does not support hot swapping the model inside the same replica without restarting the engine. The --model argument is evaluated at start-up; changing it requires re-instantiating the LLMEngine, which restarts connections and discards the KV cache. So the rollout unit is the whole replica: you do not do “v1 loads the new model on one of its GPUs” but “a v2 replica is brought up next to a v1 replica and traffic is shared out via the LB”.

TensorRT-LLM with Triton has a similar mechanism: changing the model requires a reload of the Triton backend. SGLang does not support robust hot swap either. The operational consequence: LLM rollout is always going to cost extra GPUs during the window, and the choice between blue-green, canary and shadow is exactly the question of how many extra and for how long.

The three pitfalls specific to LLM rollout

Pitfall 1 — sticky sessions break canary comparability. If the LoadBalancer does session affinity by client IP (common in NGINX and Traefik with loadbalancer.kubernetes.io/session-affinity: ClientIP), a user A always lands on v2 while B always lands on v1. Load distributions, prompt profiles and client behaviour are not random between the two pools, which statistically invalidates any gate comparison. Solution: for a canary, disable session affinity (sessionAffinity: None) or use affinity by random request-id. If the client app requires stickiness for functional reasons (conversational memory persisted in cache), canary is not the right strategy; use blue-green or shadow.

Pitfall 2 — LLM-as-judge is not an inline real-time gate. The temptation to use quality score as a live gate is strong, but the judge’s latency (2–8 s per sample) makes it unfeasible to evaluate more than a 1–2 % sampling of the traffic, and the results arrive minutes late. Operational solutions: (a) offline pre-canary eval over a golden set as a prerequisite for starting (if it fails, the canary does not even begin); (b) during the canary, capture requests + responses from v2 in real time and run the judge asynchronously in a batch job that finishes before the next step; (c) use statistical embedding drift as a fast proxy for inline quality, and reserve the judge for intermediate gates between steps.

Pitfall 3 — SSE streaming complicates shadowing. Classic traffic mirroring (NGINX mirror, Istio MirrorPolicy) is designed for request/response HTTP: it copies the request, lets the primary server answer the client, and duplicates the request to the secondary, discarding the response. With SSE, the secondary’s response is a continuous stream of several seconds, and keeping two streams in parallel doubles the load on the proxy. Solutions: (a) shadow only non-streaming requests (chat without streaming, embeddings, classification, batch eval), (b) shadow the streaming traffic but with a short timeout on the secondary (discard the shadow if it takes longer than 30 s), (c) replace the shadow with a small-weight canary (1 %), which does handle streaming well.

Applied to typical on-premise hardware

For a generic cluster of 4 nodes × 4×H100 SXM 80 GB = 16 GPUs, serving Llama 70B FP8 with TP=4 (4 possible replicas, one per node):

Blue-green: impossible to keep two complete pools of 4 replicas without extra GPUs. Practical solution: blue-green with reduced pools (2 v1 replicas + 2 v2 replicas) during the window, with accepted capacity degradation (half the sustained RPS SLO), or having a parallel cluster (another node) reserved for rollouts.
Canary: feasible. Start with 3 v1 replicas + 1 v2 replica (25 % nominal weight, though the traffic weight also varies). Move to 2 v1 + 2 v2 at 50 %, then 1 v1 + 3 v2, finally 0 v1 + 4 v2.
Shadow: awkward because of the GPU cost. Reserve it for pre-canary validation of major changes, during a short window (4–8 hours) with shadowed traffic limited to a sample (10–20 % of requests, not 100 %).

For clusters of 8 GPU nodes, the three patterns are all sustainable. The operational rule: the rollout budget is typically 25–30 % of the cluster’s sustained capacity, and buying for the peak plus that head-room makes the capacity planning numbers add up.

What we have not covered (upcoming articles)

Multi-region rollouts: how to coordinate a canary when the cluster is geographically distributed.
A/B testing of prompts (not of models): the same model with two different system prompts, measuring conversion.
Embedding rollback: changing the embedding model of a RAG system means re-embedding the whole corpus, so the canary mechanics are different. See RAG corpus curation.
Feature flags for LLM: granularity per tenant or per feature within the same model.
End-to-end continuous deployment: integration with the retrain pipeline so a new adapter is promoted automatically after passing evals.

References

Argo Rollouts project — argoproj.io/argo-rollouts (Rollout and AnalysisTemplate CRDs).
Flagger project — fluxcd.io/flagger (alternative with automatic progression).
Istio — Traffic Mirroring (mirror configurable at VirtualService level).
NGINX Ingress — Canary annotations (nginx.ingress.kubernetes.io/canary-*).
vLLM project — issue tracker on hot model swap (status as of May 2026: in design, not production-ready).
Hou et al. — DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024) — reference on goodput metrics applicable to canary gates.
Bürkner et al. — Statistical methods for detecting model drift in production (various articles on Wasserstein and KL in ML monitoring).

Autoscaling LLM inference on Kubernetes: HPA with custom metrics and KEDA for vLLM

Mon, 01 Jun 2026 16:00:00 +0200

This post complements those on GPU observability for LLM inference (where the metrics that feed the HPA come from), Capacity planning (what ceiling and what head-room the autoscaler assumes) and Continuous batching (which explains why num_requests_waiting is the primary metric).

TL;DR

Classic Kubernetes autoscaling, HPA on cpu or memory, is no use for LLM inference. The reason: the vLLM pod consumes little CPU (the GPU does the work) and the process RSS is flat; both metrics can sit at 30 % while the GPU is saturated and the request queue grows unchecked. The four viable signals that do respond to the real load are: vllm:num_requests_waiting (the queue, the primary metric), vllm:gpu_cache_usage_perc (pressure on the KV cache pool), TTFT P95 via the vllm:time_to_first_token_seconds_bucket histogram (the SLO guarantee) and the batch fill ratio num_requests_running / max_num_seqs (utilisation of the concurrency ceiling). For an HPA to consume Prometheus metrics you need an adapter; as of May 2026 there are two mature options: prometheus-adapter (cluster-wide sigma, static configuration, external.metrics.k8s.io output) and KEDA (ScaledObject with a Prometheus trigger, configurable polling, optional scale to zero, cron integration). KEDA is the dominant option for LLM on a generic cluster because it solves the “warm pool + cron + engine metric” pattern in a single CRD. The dominant operational challenge is not the scaling logic but the cold start: a vLLM pod with Llama 70B BF16 (140 GB) takes between 90 seconds (model pre-cached on a local PV) and 6 minutes (image pull + model download from object store) to serve the first token. The five levers that cut it down are an image pre-pulled via DaemonSet, the model cached on a regional PV or tmpfs, a warm pool with minReplicaCount > 0, predictive scaling via KEDA cron when the traffic pattern is predictable (offices 9–18 h), and parallel model download. The three specific LLM scale-down pitfalls: cutting streaming SSE connections halfway through a response (graceful drain with terminationGracePeriodSeconds ≥ 60 s), scale-out/in oscillation from a badly calibrated stabilization window, and forgetting that the HPA only scales pods; GPU nodes are scaled with cluster-autoscaler over labelled nodepools. This post includes the minimal YAML manifests.

You are here: DEPLOY

The analogy: the bakery with wood-fired ovens

An artisan bakery has three wood-fired ovens. Each oven takes 25 minutes to reach temperature from cold. Once hot, it bakes bread continuously with a run of 18 minutes per batch. The manager wants to maximise bread sold per day without burning wood for nothing, and she knows three things: that there is a demand peak at 7:30 every morning, that Mondays sell almost nothing, and that when the bread on the counter runs out the customers go to the supermarket next door.

The cheap strategy, lighting ovens when there is a queue in the shop, does not work. By the time the queue grows and the manager lights the second oven, that oven will not be ready until 25 minutes later; the customers in that window are lost. The “queue at the counter” signal arrives late.

The smart strategy: light the second oven at 6:55, ahead of the predictable 7:30 peak, and leave it running until 10:00 even though the queue drops at 8:15. Keep the third oven off from Monday to Wednesday because demand does not arrive; light it proactively on Thursdays at 12:00 because historically it goes up. Keep a stock of pre-proved raw dough in the chiller so that when the oven is ready the bread goes in within 30 seconds and there is no two-hour proving wait.

Autoscaling an LLM inference cluster works the same way:

Lighting ovens from cold = reactive scale-out when the queue grows (slow, loses customers).
Proactive cron = predictive scaling when the pattern is known (working hours, expected peaks).
Pre-proved dough = warm pool of replicas with the model loaded but under 0 load.
Switching off ovens with no bread inside = scale-down respecting the active streams (you do not shut the oven with bread in it).

The key metric, “how many customers are in the queue”, is called num_requests_waiting. The metric that says “the oven is about to run out of dough for new loaves” is called gpu_cache_usage_perc. And the quality-of-service metric, “how long the first loaf takes to come out when a new customer walks in”, is called TTFT.

Why HPA on CPU is no use

The classic Kubernetes HPA looks at the pod’s resource.cpu. For a conventional HTTP service, Node.js or a REST API, CPU moves linearly with traffic and the HPA scales with reasonable accuracy. For a vLLM or SGLang pod on GPU, the pod’s CPU typically lives between 5 % and 15 % regardless of whether the GPU is at 30 % or 99 % load: the real work is done by the device, not the process. Result: the CPU-based HPA never triggers scale-out even though the GPU is bursting, and clients pile up in the queue until TTFT P95 crosses the SLO. The operator discovers the problem through the TTFT alert, not through the HPA.

memory is no use either: the vLLM process RSS is flat after startup (model plus buffers loaded in one go); it does not reflect the real pressure on the GPU. The only things that rise and fall with the useful inference load are metrics the engine publishes explicitly: request queue, KV cache pool, SLO latencies. Without an adapter that makes them visible to the HPA, autoscaling is blind.

The four viable signals

Signal 1, vllm:num_requests_waiting (queue). It is the most direct metric: how many requests are waiting to enter the batch. It reacts the instant the target concurrency saturates. It is robust against model changes (the number of requests is the same concept whether it is Llama 7B or 70B). It is the primary metric of the LLM HPA. Typical threshold: target = 5 requests waiting on average; if the queue grows above 5 sustained for 2 minutes, scale out.

Signal 2, vllm:gpu_cache_usage_perc (KV pool). It moves before the queue does: the KV pool fills up while the batch slots are still free, until the engine starts rejecting new requests for OOM prevention and the queue forms. It is therefore predictive: it triggers scale-out before the client notices degradation. Typical threshold: target = 0.85 (85 % of pool used).

Signal 3, TTFT P95. The contractual guarantee. If TTFT P95 leaves the SLO, scale out even though the queue and the KV pool look reasonable (there may be a spike of long prompts). It is reactive, it leaves the SLO before your HPA reacts, but it works as a final guardrail.

Signal 4, batch fill ratio. The num_requests_running / max_num_seqs quotient (the latter is engine config, not a metric). Useful for scale-down: if the ratio stays below 0.4 for 10 minutes, there is spare capacity and replicas can be reduced safely.

The recommended policy combines all four: the queue and the KV pool trigger scale-out (whichever arrives first), TTFT confirms it as a guardrail, and the batch fill ratio manages scale-down. Implementing that in a single HPA demands external metrics; KEDA makes this manageable.

The wiring: KEDA as the Prometheus adapter

KEDA introduces two main CRDs: TriggerAuthentication (how to authenticate against the source) and ScaledObject (which deployment to scale with which triggers). For a vLLM deployment with Prometheus as the source:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
 name: vllm-llama70b-scaler
 namespace: inference
spec:
 scaleTargetRef:
 name: vllm-llama70b
 minReplicaCount: 2 # warm pool
 maxReplicaCount: 20
 pollingInterval: 15
 cooldownPeriod: 300 # 5 min before scale-down
 advanced:
 horizontalPodAutoscalerConfig:
 behavior:
 scaleDown:
 stabilizationWindowSeconds: 600 # large window to avoid oscillation
 policies:
 - type: Pods
 value: 1
 periodSeconds: 120
 scaleUp:
 stabilizationWindowSeconds: 30
 policies:
 - type: Pods
 value: 2
 periodSeconds: 60
 triggers:
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_queue_depth
 threshold: "5"
 query: |
 avg(vllm:num_requests_waiting{deployment="vllm-llama70b"})
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_kv_cache
 threshold: "0.85"
 query: |
 avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama70b"})
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_ttft_p95
 threshold: "1.5"
 query: |
 histogram_quantile(0.95,
 sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{deployment="vllm-llama70b"}[5m])))

Three non-obvious operational details:

minReplicaCount: 2. This is the warm pool. Keeping at least two replicas guarantees availability if a node is lost and absorbs spikes without waiting for the cold start of the first scale-up. Dropping it to 0 saves GPU off-peak but introduces 90 s–6 min of latency for the first new client.

stabilizationWindowSeconds: 600 on scale-down. Ten minutes. Models are not nginx: if a replica closes prematurely and two minutes later there is another peak, the cold start of a new pod is what the client waits. Better to keep extra replicas around for twice as long as you would for a normal web service.

scaleUp: stabilizationWindowSeconds: 30. Thirty seconds. Scale-out has to be fast: the new pod’s cold start adds its own delay, and if on top of that the HPA waits several more minutes before triggering, the SLO is already broken.

The big operational problem: cold start

A vLLM pod loading Llama 70B goes through these phases before serving the first token:

Phase	Typical time	Can be accelerated with
Image pull (4–6 GB)	30–90 s	DaemonSet pre-pull
Model download (140 GB BF16)	60–300 s	Cached regional PV, S3 + multi-thread
Loading the model into HBM	30–90 s	tmpfs or local NVMe
CUDA graph capture	20–60 s	`--enforce-eager` (slower at runtime but fast startup)
PagedAttention warmup	5–15 s	—
Health check ready	10–30 s	probe tuning

Total without optimisation: 4–10 minutes. That is how long a new replica takes to absorb traffic. With all the levers combined: 45–90 seconds. The difference between those two numbers is the main platform work for LLM autoscaling.

The five levers

Lever 1, pre-pulled image. A trivial DaemonSet runs ctr image pull (or crictl pull) on the GPU nodes as soon as they join the cluster. The inference engine image stays on disk; new pods skip the 30–90 s of pull. Cost: around 6 GB of disk per node.

apiVersion: apps/v1
kind: DaemonSet
metadata: { name: vllm-image-warmer }
spec:
 selector: { matchLabels: { app: vllm-warmer } }
 template:
 metadata: { labels: { app: vllm-warmer } }
 spec:
 nodeSelector: { workload: gpu }
 initContainers:
 - name: pull
 image: vllm/vllm-openai:v0.10.0
 command: ["/bin/true"]
 containers:
 - name: pause
 image: registry.k8s.io/pause:3.10

Lever 2, model on a regional PV. The model download (140 GB BF16 or 35 GB FP8) from central object storage is the dominant component of the cold start. Caching the model on a zone or rack PV, Rook-Ceph RBD or local NVMe provisioned by the operator, cuts 60–300 s down to 5–15 s. The antipattern: downloading the model on every startup from external S3.

volumeMounts:
 - name: model-cache
 mountPath: /models
 readOnly: true
volumes:
 - name: model-cache
 persistentVolumeClaim:
 claimName: llama70b-fp8-pvc  # RWX shared, filled offline

Lever 3, warm pool. minReplicaCount > 0 keeps pre-loaded replicas idle. The cost is idle GPU; the benefit is 0 s of cold start for the first client of a peak. For production clusters with continuous traffic: a warm pool of 2–3 replicas. For nightly batch clusters with 0 traffic: warm pool 0 and accept the cold start, or KEDA with a cron that switches on 10 minutes earlier.

Lever 4, predictive scaling with cron. When the pattern is predictable (offices 9–18 h):

triggers:
 - type: cron
 metadata:
 timezone: Europe/Madrid
 start: "30 8 * * 1-5" # 8:30 Monday–Friday
 end: "0 19 * * 1-5" # 19:00
 desiredReplicas: "6"

Combined with reactive triggers. The HPA scales according to the maximum of the signals: if the cron asks for 6 and the queue asks for 10, the result is 10.

Lever 5, parallel download and an efficient format. For PVs that are not pre-loaded, tools such as nvidia-modelmanager, s5cmd or aria2c parallelise the model download. Going from serial download (~150 MB/s) to parallel with 8 threads (~1.2 GB/s) divides the time by 8. And formats such as safetensors load into HBM faster than the original PyTorch pickle.

When to scale nodes, not just pods

The HPA scales pods. If the cluster has no free GPU nodes, the new pod stays Pending for lack of resources. To scale nodes, you need cluster-autoscaler with a specific GPU nodepool, labelled:

# nodepool config (Karpenter or cluster-autoscaler equivalent)
labels:
 workload: gpu
 gpu-model: h100-sxm-80gb
taints:
 - key: nvidia.com/gpu
 effect: NoSchedule
limits:
 min: 2 nodes
 max: 8 nodes

Without this, the HPA can ask for 10 replicas but the cluster only delivers those that fit on nodes already up. The cold start of a new node (bare metal or cloud provisioning, PXE, OS boot, NVIDIA drivers, cluster join) is much longer than the cold start of a pod: typically 5–15 minutes on preconfigured bare metal, 30–60 minutes on real provisioning. For on-premise clusters, the nodepool must be always sized to the expected maximum, and the “scaling” is only on the pod side. The concept of reactive node scale-out only applies to clouds; on-premise you have to buy for the peak.

Three pitfalls specific to LLM scale-down

Pitfall 1, cutting streaming SSE connections. When a replica enters Terminating, Kubernetes sends SIGTERM to the pod and, by default, kills it 30 seconds later. For vLLM that means cutting streaming SSE connections halfway through the response. The client gets a 502 error with the partial output lost. Solution: terminationGracePeriodSeconds: 120 plus a preStop hook that tells the engine not to accept new requests but to finish those in flight:

spec:
 terminationGracePeriodSeconds: 120
 containers:
 - name: vllm
 lifecycle:
 preStop:
 httpGet:
 path: /shutdown
 port: 8000

This requires the engine to expose a graceful shutdown endpoint; vLLM v1 supports it via --enable-graceful-shutdown. Without it, scale-down breaks the SLO even though the metrics do not capture it (cut requests never enter the TTFT histogram).

Pitfall 2, scale-up/scale-down oscillation. If the scale-down stabilizationWindowSeconds is short (~60 s default), the next drop in the queue triggers scale-down, and two minutes later the next peak triggers scale-up. The system oscillates, pays repeated cold starts, and never reaches a stable regime. Solution: scale-down with a window of 10 minutes minimum and conservative policies (type: Pods, value: 1, periodSeconds: 120, at most one replica fewer every 2 minutes).

Pitfall 3, vllm:num_requests_waiting with avg when there is rebalancing. If two replicas are unbalanced (one with queue 20, the other with queue 0), avg gives 10 and the HPA triggers scale-out when the right move would be to rebalance via the load balancer. To detect it: add an alert on stddev(vllm:num_requests_waiting) per deployment. If the dispersion is high, the problem is not capacity but routing.

Full example manifest

For a vLLM deployment with Llama 70B FP8 on 4×H100 SXM per replica, KEDA with warm pool 2:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: vllm-llama70b
 namespace: inference
spec:
 replicas: 2 # managed by KEDA afterwards
 selector: { matchLabels: { app: vllm-llama70b } }
 template:
 metadata:
 labels: { app: vllm-llama70b, deployment: vllm-llama70b }
 spec:
 terminationGracePeriodSeconds: 120
 nodeSelector: { workload: gpu, gpu-model: h100-sxm-80gb }
 tolerations:
 - key: nvidia.com/gpu
 operator: Exists
 effect: NoSchedule
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.10.0
 args:
 - --model=/models/llama-3.3-70b-fp8
 - --tensor-parallel-size=4
 - --max-num-seqs=64
 - --enable-prefix-caching
 - --enable-graceful-shutdown
 ports:
 - { name: http, containerPort: 8000 }
 - { name: metrics, containerPort: 8000 }
 resources:
 limits:
 nvidia.com/gpu: "4"
 memory: 200Gi
 readinessProbe:
 httpGet: { path: /health, port: 8000 }
 initialDelaySeconds: 60
 periodSeconds: 10
 failureThreshold: 30 # tolerates the warmup
 lifecycle:
 preStop:
 httpGet: { path: /shutdown, port: 8000 }
 volumeMounts:
 - { name: model-cache, mountPath: /models, readOnly: true }
 - { name: dshm, mountPath: /dev/shm }
 volumes:
 - name: model-cache
 persistentVolumeClaim: { claimName: llama70b-fp8-pvc }
 - name: dshm
 emptyDir: { medium: Memory, sizeLimit: 16Gi }
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata: { name: vllm-llama70b-metrics, namespace: inference }
spec:
 selector: { matchLabels: { app: vllm-llama70b } }
 podMetricsEndpoints:
 - port: metrics
 path: /metrics
 interval: 15s
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: vllm-llama70b-scaler, namespace: inference }
spec:
 scaleTargetRef: { name: vllm-llama70b }
 minReplicaCount: 2
 maxReplicaCount: 20
 pollingInterval: 15
 cooldownPeriod: 300
 advanced:
 horizontalPodAutoscalerConfig:
 behavior:
 scaleDown:
 stabilizationWindowSeconds: 600
 policies:
 - { type: Pods, value: 1, periodSeconds: 120 }
 scaleUp:
 stabilizationWindowSeconds: 30
 policies:
 - { type: Pods, value: 2, periodSeconds: 60 }
 triggers:
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_queue
 threshold: "5"
 query: avg(vllm:num_requests_waiting{deployment="vllm-llama70b"})
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_kv
 threshold: "0.85"
 query: avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama70b"})
 - type: cron
 metadata:
 timezone: Europe/Madrid
 start: "30 8 * * 1-5"
 end: "0 19 * * 1-5"
 desiredReplicas: "6"

This set is the minimum viable one for LLM autoscaling on a generic cluster with the NVIDIA GPU Operator. Each team adapts it to its own SLO.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB per node, 4 GPU nodes:

Each node hosts one vLLM replica with TP=4 and Llama 70B FP8 (one model per node, they are not shared).
Warm pool of 2 replicas off-peak; the KEDA cron raises it to 4 during working hours.
Cluster-autoscaler does not apply (4 physical nodes bought; scaling is pods only). The number of concurrent replicas is at most the number of available nodes (if each replica uses all 4 GPUs of the whole node).
If the sizing requires more simultaneous replicas than nodes, there are two routes: (a) lower the TP of each replica so two fit per node, (b) expand the physical nodepool. The decision is dictated by capacity planning, see Capacity planning for on-premise LLM inference.

KEDA event volume: around 5 evaluations/min per ScaledObject. For 10 models served in parallel, 3,000 evaluations/h. Manageable with one KEDA operator per cluster.

What we have not covered (upcoming articles)

Cluster-autoscaler for on-premise GPU nodes: how to orchestrate bare metal provisioning (Tinkerbell, Metal³) as a function of demand.
Multi-cluster autoscaling: scaling across clusters in different DCs for geographic resilience.
Cost-aware autoscaling: prioritising nodes by hourly energy cost (on clusters with indexed tariffs).
Predictive ML-based scaling: instead of a static cron, training a model that predicts demand 30 minutes ahead.
Multi-tenant quotas and fairness: KEDA with namespace quotas so that one tenant does not monopolise the HPA.

References

KEDA project — keda.sh (documentación oficial de triggers Prometheus y cron).
Kubernetes — Horizontal Pod Autoscaler walkthrough (kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale).
NVIDIA — GPU Operator on Kubernetes (Helm chart oficial con DaemonSet de drivers y DCGM).
vLLM project — production_monitoring/ (métricas Prometheus expuestas por el servidor).
Karpenter — NodePool spec (etiquetado y taints para nodepools GPU).
Cluster Autoscaler — Scaling GPU nodes (caveats de descubrimiento de recursos GPU).
Kubernetes — Pod lifecycle and termination (preStop, terminationGracePeriodSeconds).

GPU observability for LLM inference: the twelve DCGM and vLLM metrics that dictate the health of your production

Mon, 01 Jun 2026 15:30:00 +0200

This post complements those on LLM tracing with OpenTelemetry GenAI (the tracing layer above the metrics), Capacity planning (what was sized and what must be watched) and Continuous batching (the mechanism that explains several of the engine metrics).

TL;DR

Observability for an LLM inference cluster is built on two complementary sources: the GPU hardware metrics exposed by the DCGM (Data Center GPU Manager) Exporter, part of the NVIDIA GPU Operator, and the metrics from the inference engine (vLLM, SGLang, TensorRT-LLM) exposed on a Prometheus-compatible /metrics. Neither is enough on its own. The classic nvidia-smi metric called GPU utilization is misleading for LLMs: it reads high whenever any kernel is running, without distinguishing tensor cores burning from SMs waiting on HBM. The full cockpit has twelve DCGM metrics in four families (compute: DCGM_FI_PROF_SM_OCCUPANCY, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, DCGM_FI_PROF_DRAM_ACTIVE; memory: DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_FB_FREE, DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL; thermal-power: DCGM_FI_DEV_GPU_TEMP, DCGM_FI_DEV_POWER_USAGE, DCGM_FI_DEV_CLOCK_THROTTLE_REASONS; health: DCGM_FI_DEV_XID_ERRORS, DCGM_FI_DEV_ECC_DBE_VOL_TOTAL, DCGM_FI_DEV_RETIRED_DBE) and five vLLM engine metrics (vllm:num_requests_running, vllm:num_requests_waiting, vllm:gpu_cache_usage_perc, vllm:time_to_first_token_seconds, vllm:time_per_output_token_seconds). Each one has a defensible green/amber/red threshold, a PromQL query for alerting, and at least one common false reading that confuses the junior operator. The six critical alerts that any production cluster must fire are: HBM > 92 %, thermal or power throttle, XID error, ECC double-bit, KV cache pool > 95 %, and TTFT P95 outside SLO for 5 minutes. The point of having this panel: that the operator on shift diagnoses the origin of a degradation in under five minutes, without opening an SSH console to the GPUs. When that holds, the cluster has moved to professional operation; until then, it is operated on intuition.

You are here: OBSERVE (the other half of tracing)

Tracing, already covered in LLM tracing with OpenTelemetry GenAI, answers what happened in this specific request. Metrics answer what is happening across the cluster in aggregate. They are complementary: an alert on the metrics side tells you “the cluster is degrading”, tracing tells you “and here is the specific trace that proves it”. A cluster without tracing but with metrics operates; a cluster without metrics but with tracing does not operate, it debugs.

The analogy: the cockpit of a modern aircraft

In a modern commercial aircraft, the pilot’s instrument panel has more than 70 active indicators. If there were only one, the altimeter, say, the aircraft would fly into the ground at the first moment of low visibility. You need the altimeter and the attitude indicator, and the airspeed indicator, and the turn indicator, and the fuel gauge, and the oil pressure gauges for each engine, and the turbine exit temperatures. Each answers a different question. And together they cover the operational question: is the aircraft healthy, is it where it should be, and is it going where we want?

Observability for an LLM inference cluster works the same way. A single metric, “GPU utilization 99 %”, answers nothing useful. It is like looking only at the car’s odometer to diagnose why the engine is making a noise. The full cockpit is twelve instruments on the hardware side plus five on the inference engine side, organised into families that answer different questions:

Compute and efficiency: are the tensor cores doing the work we expect, or are they waiting?
Memory: is there VRAM left for new requests, or are we on the edge of OOM?
Thermal and power: is the hardware healthy, or is it silently limiting throughput?
Health and errors: is there hardware degradation under way (ECC, XID, NVLink)?
Inference engine: is the queue growing, is the KV pool saturated, is the SLO being met?

The first four answer “is the GPU fine?”. The fifth answers “is it delivering the service we promised?”. The two questions are different and both must have an answer at a glance.

Why `nvidia-smi` `GPU-Util` misleads with LLMs

The classic metric that appears in nvidia-smi as GPU-Util corresponds to DCGM_FI_DEV_GPU_UTIL. Its official definition: “percentage of time during which one or more kernels were executing on the GPU”. The problem with LLMs: the decode phase is memory-bound, not compute-bound. When the inference engine decodes token by token, the GPU spends 90 % of the time waiting for the HBM to finish delivering the model weights and the KV cache. There is a kernel running (the HBM read), so GPU-Util reports values close to 100 %. But the tensor cores are idle: the bottleneck is memory, not compute.

Practical result: the operator sees “GPU-Util 99 %” in Grafana and assumes “GPU saturated, no more load can go in”. But the reality may be “compute at 25 %, HBM saturated at 95 %”, which changes the operational decisions (quantisation, batch size, parallelism). The classic metric lies by oversimplifying.

The right move is to look at the three DCGM profiling metrics of the _FI_PROF_* subsystem:

DCGM_FI_PROF_SM_OCCUPANCY: ratio of active warps over the maximum per SM. Is there parallel work?
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE: % of cycles with tensor cores effectively active. Is compute working?
DCGM_FI_PROF_DRAM_ACTIVE: % of cycles with HBM transferring. Is memory saturated?

A typical decode-bound GPU running Llama 70B on an H100 shows: SM occupancy 35–55 %, tensor active 15–30 %, DRAM active 80–95 %. That is the real “saturated GPU” for LLMs. The three together tell the regimes apart; none of them alone says anything actionable.

How they are wired up in production

The platform side is covered in Five maturity levels (level 4, the GPU plane) and Seven deployment phases (phase F5). For the observer, the key pieces are:

NVIDIA GPU Operator. Helm manifests that deploy on each GPU node: drivers, container toolkit, MIG manager and DCGM Exporter. The last one exposes /metrics in Prometheus format with all the DCGM_FI_* listed above. It is scraped from the cluster’s internal Prometheus.

Inference engine. vLLM exposes /metrics on port 8000 (default) with vllm:* metrics. SGLang exposes it too, with the sglang: prefix. TensorRT-LLM exposes it through Triton Inference Server with the nv_inference: prefix. The basic naming convention is similar across the three engines; the thresholds and queries in this post assume vLLM, but they translate.

ServiceMonitor / PodMonitor. The Prometheus operator resource that says what to scrape. Minimal example:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
 name: vllm-inference
spec:
 selector:
 matchLabels: { app: vllm }
 podMetricsEndpoints:
 - port: metrics
 interval: 15s

Dashboards. The NVIDIA operator publishes reference Grafana dashboards for DCGM in nvidia/dcgm-exporter (the official repo). vLLM publishes one in vllm-project/vllm (the examples/ folder). Both work as a base; each team adds the panels specific to its own SLO.

The twelve DCGM metrics organised by family

Family 1 — Compute

DCGM_FI_PROF_SM_OCCUPANCY: ratio of active warps per SM over the maximum possible. Value between 0 and 1.

Green: 0.30–0.70 (typical LLM decode regime).
Amber: < 0.20 sustained (batch too small, GPU under-used in parallelism).
Red: 0.95 sustained with low DRAM_ACTIVE (pathological kernel saturating the SMs).

DCGM_FI_PROF_PIPE_TENSOR_ACTIVE: % of cycles with tensor cores executing. The key metric for “is compute producing?”.

Green in prefill: 50–80 %.
Green in decode: 15–30 % (decode is memory-bound, this is not a symptom of a problem).
Red: < 5 % sustained in prefill, or the engine is not using the tensor cores at all (bad config, incompatible format).

DCGM_FI_PROF_DRAM_ACTIVE: % of cycles with HBM transferring data. The key metric for detecting memory saturation.

Green in decode: 60–85 %.
Amber: > 90 % sustained (HBM is a firm bottleneck, which explains high TPOT).
Red: > 95 % sustained with KV cache pool < 70 % (something other than the engine is asking for HBM; investigate leaks).

Family 2 — Memory

DCGM_FI_DEV_FB_USED: Frame Buffer (HBM) used, in MiB.

Green: 70–85 % of total.
Amber: 86–92 %.
Red: > 92 % (risk of OOM on the next paged-attention allocation).

PromQL for the cluster-wide percentage: 100 * sum(DCGM_FI_DEV_FB_USED) / sum(DCGM_FI_DEV_FB_TOTAL).

DCGM_FI_DEV_FB_FREE: free Frame Buffer. Complementary to the previous one; useful for absolute alerts (< 4096 MiB free).

DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL: aggregate NVLink bandwidth in MB/s. For TP (tensor parallel) topologies that cross GPUs over NVLink, this metric reveals whether the parallelism split is saturating the bus.

Green: varies with topology. On 4×H100 SXM with NVLink 4.0, theoretical capacity is 450 GB/s per GPU. A typical TP=4 regime: 50–150 GB/s sustained.
Red: > 90 % of capacity sustained (check whether the model would fit with lower TP or with pipeline parallel).

Family 3 — Thermal and power

DCGM_FI_DEV_GPU_TEMP: die temperature in °C.

Green: < 75 °C.
Amber: 75–82 °C.
Red: > 83 °C (close to the H100’s automatic thermal throttle; check ventilation, air flow, rack inlet temperature).

DCGM_FI_DEV_POWER_USAGE: draw in watts. For the H100 SXM, nominal TDP is 700 W. Useful for three things: spotting an unusually low workload (suspect idle or a stall), billing real energy cost, and firing alerts if the draw approaches the PDU limit.

DCGM_FI_DEV_CLOCK_THROTTLE_REASONS: encoded bitmap with the active throttle reasons. This is the metric that silently explains TPOT degradations.

Relevant bits:

0x0000000000000001: Idle (not a problem).
0x0000000000000002: App clocks setting.
0x0000000000000004: SW Power Cap (software limit, e.g. from nvidia-smi -pl).
0x0000000000000008: HW Slowdown.
0x0000000000000010: Sync Boost (NVIDIA Sync).
0x0000000000000020: SW Thermal Slowdown (software thermal limit).
0x0000000000000040: HW Thermal Slowdown (hardware thermal limit, an emergency).
0x0000000000000080: HW Power Brake Slowdown (PSU voltage drop).
0x0000000000000100: Display Clock Setting.

Any throttle other than Idle with a sustained value > 0 is an alert. TPOT degradation with DRAM_ACTIVE already high and thermal throttle active is the classic “the rack is badly ventilated, it is not the engine’s fault”.

Family 4 — Health

DCGM_FI_DEV_XID_ERRORS: cumulative counter of driver XID errors. XIDs are critical event codes that NVIDIA documents exhaustively (XID 13: graphics engine exception; XID 31: GPU memory page fault; XID 43: reset channel verif error; XID 79: GPU has fallen off the bus; XID 95: uncontained ECC error; and so on). Any increment is an immediate alert: many XIDs require a node reset or an RMA of the GPU.

DCGM_FI_DEV_ECC_DBE_VOL_TOTAL: volatile double-bit ECC errors (uncorrectable). Unlike single-bit ones (which ECC corrects silently and which are counted in DCGM_FI_DEV_ECC_SBE_*), double-bit errors corrupt data. Any value > 0 is a critical alert: the GPU must be drained and inspected.

DCGM_FI_DEV_RETIRED_DBE: physical HBM pages retired because of accumulated double-bit errors. NVIDIA retires faulty pages automatically to prevent future corruption. More than 4–8 retired pages on one GPU suggests silicon degradation: document it and plan a replacement in the next maintenance window.

The five inference engine metrics (vLLM)

The DCGM metrics answer “is the GPU healthy?”. The engine metrics answer “is the service meeting the SLO?”. Without them, you know the hardware works but you do not know whether the clients are happy.

vllm:num_requests_running: requests currently in the batch. If it reaches the configured --max-num-seqs and does not come down, the engine is saturated on concurrency (check VRAM and rebalance through the autoscaler, see Autoscaling LLM on Kubernetes).

vllm:num_requests_waiting: requests queued, not yet in the batch. Any value > 0 sustained over minutes means the cluster is not scaling with the load. This is the primary metric for HPA.

vllm:gpu_cache_usage_perc: % of the KV cache pool in use.

Green: 50–80 %.
Amber: 80–92 %.
Red: > 92 % (risk of preempt-on-OOM: vLLM will drop requests to free memory, which raises TTFT visibly).

vllm:time_to_first_token_seconds: histogram of TTFT per request. Consumed as histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket[5m]))). Compared against the TTFT P95 SLO, it fires the primary service alert.

vllm:time_per_output_token_seconds: histogram of TPOT. Equivalent to the previous one but for streaming smoothness. Compared against the TPOT P95 SLO, it fires the secondary alert.

The six alerts that must page in production

Any serious production cluster fires these six alerts to a channel with an on-call rotation. Without them, the SLO is met by luck, not by process.

groups:
 - name: gpu-llm-critical
 rules:
 - alert: GpuHbmNearOom
 expr: 100 * (DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL) > 92
 for: 2m
 labels: { severity: critical }
 annotations:
 summary: "HBM on {{ $labels.gpu }} at {{ $value }}% — OOM risk"

 - alert: GpuThermalOrPowerThrottle
 expr: (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS != 0) and ignoring(reason) (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS != 1)
 for: 1m
 labels: { severity: warning }
 annotations:
 summary: "GPU {{ $labels.gpu }} throttling (reasons={{ $value }})"

 - alert: GpuXidErrorDetected
 expr: increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0
 labels: { severity: critical }
 annotations:
 summary: "XID error on GPU {{ $labels.gpu }} — investigate immediately"

 - alert: GpuEccDoubleBit
 expr: DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0
 labels: { severity: critical }
 annotations:
 summary: "ECC double-bit on GPU {{ $labels.gpu }} — drain node"

 - alert: VllmKvCachePoolNearFull
 expr: vllm:gpu_cache_usage_perc > 0.95
 for: 3m
 labels: { severity: warning }
 annotations:
 summary: "KV cache pool > 95% on {{ $labels.instance }}"

 - alert: VllmTtftP95OutOfSlo
 expr: histogram_quantile(0.95, sum by(le, instance)(rate(vllm:time_to_first_token_seconds_bucket[5m]))) > 1.5
 for: 5m
 labels: { severity: warning }
 annotations:
 summary: "TTFT P95 over SLO ({{ $value }}s > 1.5s)"

These six cover 80 % of the incidents that affect the SLO. The remaining 20 % demand investigation with tracing (see LLM tracing with OpenTelemetry GenAI).

Master table: thresholds and queries

Metric	Green	Amber	Red	Base query (PromQL)
SM occupancy	0.30–0.70	0.15–0.30	< 0.10 sustained	`DCGM_FI_PROF_SM_OCCUPANCY`
Tensor active (decode)	15–30 %	< 10 %	< 3 %	`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`
DRAM active	60–85 %	85–95 %	> 95 % with low KV	`DCGM_FI_PROF_DRAM_ACTIVE`
FB used	70–85 %	86–92 %	> 92 %	`100 * DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL`
NVLink BW	< 70 % cap	70–90 % cap	> 90 % cap	`DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL`
GPU temp	< 75 °C	75–82 °C	> 83 °C	`DCGM_FI_DEV_GPU_TEMP`
Power usage	< 90 % TDP	90–98 % TDP	> 98 % TDP	`DCGM_FI_DEV_POWER_USAGE`
Throttle reasons	0 or Idle	App/SW	HW Therm/Power	`DCGM_FI_DEV_CLOCK_THROTTLE_REASONS`
XID errors	no change	—	any delta	`increase(DCGM_FI_DEV_XID_ERRORS[5m])`
ECC DBE	0	—	> 0	`DCGM_FI_DEV_ECC_DBE_VOL_TOTAL`
Retired pages	< 4	4–8	> 8	`DCGM_FI_DEV_RETIRED_DBE`
KV cache used	50–80 %	80–92 %	> 92 %	`vllm:gpu_cache_usage_perc`
Requests waiting	0	1–5 sustained	> 10 sustained	`vllm:num_requests_waiting`
TTFT P95	< SLO	80–100 % SLO	> SLO	see alert query above
TPOT P95	< SLO	80–100 % SLO	> SLO	`histogram_quantile(0.95, sum by(le)(rate(vllm:time_per_output_token_seconds_bucket[5m])))`

Three pitfalls that confuse the junior operator

Pitfall 1 — “GPU-Util at 99 % = saturated”. As explained at the start, DCGM_FI_DEV_GPU_UTIL lights up with any kernel. The right move is to look at the three _PROF_* metrics (SM occupancy, tensor active, DRAM active) together. GPU util 99 % + tensor active 8 % + DRAM active 92 % means “saturated by memory, not compute”; GPU util 99 % + tensor active 75 % + DRAM active 50 % means “saturated by compute, prefill heavy”. The two situations call for different levers.

Pitfall 2 — confusing single-bit ECC (SBE) with double-bit (DBE). Single-bit errors are corrected silently and are unavoidable in any HBM under load (cosmic radiation, voltage fluctuations). An SBE counter growing slowly is not an alert, it is physics. DBE is: it corrupts data. Telling the two metrics apart avoids false alarms and false negatives in equal measure.

Pitfall 3 — alerting on num_requests_waiting > 0 without context. An instantaneous value of 1 or 2 during a spike is normal. What matters is a sustained queue: use for: 5m with a threshold of 3–5. Without that window, the system floods the alert channel with noise.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB with intra-node NVLink:

DCGM Exporter deployed through the NVIDIA GPU Operator, one DaemonSet per GPU node.
Internal Prometheus with 30-day retention for high-frequency metrics, 1 year for downsampled ones (Thanos/Mimir if the volume justifies it).
Grafana with three standard dashboards: GPU hardware (DCGM), engine (vLLM), SLO (TTFT/TPOT/RPS against written targets).
Alertmanager with an on-call rotation and rate limiting through silences grouped by node.
Cardinality kept under control: gpu (local id), node, pod, model. Do not add request_id or other high-cardinality labels to metrics; that is tracing’s job.

Estimated volume for a 16-GPU cluster scraped every 15 s: around 2 million samples/min, around 25 GB/day of raw Prometheus. Manageable with one Prometheus per cluster plus retention; if the team scales beyond 64 GPUs, consider a Thanos sidecar or VictoriaMetrics. See OSS LLMOps tooling catalogue for equivalent alternatives.

What we have not covered (upcoming articles)

Tracing of LLM workloads: already covered in LLM tracing with OpenTelemetry GenAI.
Autoscaling based on these metrics: see Autoscaling LLM on Kubernetes.
Incident response runbooks: how each of these alerts translates into concrete action (drain, restart, RMA, scaling, rollback).
Cost accounting: using DCGM_FI_DEV_POWER_USAGE and vllm:request_success_total for per-tenant cost showback.
Multi-tenant fairness monitoring: when several tenants share a cluster, which metrics detect that one of them is hogging the KV cache.

References

NVIDIA — DCGM Exporter (repo nvidia/dcgm-exporter, documented metrics and units).
NVIDIA — DCGM Field Identifiers reference (complete list of DCGM_FI_*).
NVIDIA — XID Errors documentation (catalogue of XID codes and remediation procedures).
NVIDIA — NVIDIA GPU Operator (official Helm chart).
vLLM project — examples/production_monitoring/ (reference PromQL and Grafana dashboards).
Prometheus — Histogram and summary best practices (for building defensible percentile queries).
NVIDIA — H100 Tensor Core GPU datasheet (TDP, HBM bandwidth, NVLink capacities).

Capacity planning for on-premise LLM inference: how to size GPUs from an SLO

Mon, 01 Jun 2026 15:00:00 +0200

This post complements the ones on KV cache (the piece that dominates the VRAM budget), Continuous batching (what defines the effective utilisation of the compute) and Seven layers of the stack (the pieces the sizing takes for granted). Before reading this one, make sure your team has written down the SLOs it is going to chase; without that input the calculation is not defensible.

TL;DR

LLM inference capacity planning does not answer “how many tokens per second does a GPU give” — that question has no universal answer, because throughput depends on concurrency, the prefill/decode split, context length, the inference engine and quantisation. The right question has three inputs (SLO: TTFT P95, TPOT P95, sustained RPS), a hardware reference (GPU model, VRAM, HBM bandwidth, effective FLOPs) and a model (parameters, GQA/MHA/MoE architecture, weight format). The calculation resolves into two coupled budgets that cross each other. VRAM budget: from the GPU’s total you subtract model weights and activations, what remains is the KV cache budget, and from there you derive the maximum concurrency possible at the average context you expect. Time budget: the engine (vLLM, SGLang, TensorRT-LLM) has a tokens/s ceiling in decode given by HBM bandwidth and another in prefill given by useful FLOPs; from there you derive the expected TPOT and, dividing prefill_tokens by prefill throughput, the expected TTFT. Both budgets must meet the SLO simultaneously: whichever is tighter dictates the sizing. On the Llama 70B BF16 example with tensor parallel 4 on 4×H100 SXM, a single replica saturates at ~28 concurrent requests and delivers ~3,200 tokens/s of aggregate decode with a median TPOT of 35 ms; for 200 sustained RPS at a profile of 800 prompt tokens + 250 of output, you need between 4 and 5 replicas with a 25 % cushion over the observed peak. Quantisation (FP8 → INT4) divides VRAM cost and decode time by between 1.5 and 4×, but it degrades quality measurably — it is not assumed free, it is validated with evals. The five habitual traps: confusing the mean with P95, ignoring the prefill/decode split of the real workload, sizing without head-room for retraining or rollback, forgetting that a GPU at 100 % SM util means nothing if the HBM is saturated, and not documenting the assumptions behind the calculation (a sizing with no written assumptions is a throwaway calculation).

You are here: DEPLOY (with one foot in OBSERVE)

Capacity planning is a piece with dual residency. It lives in DEPLOY because without a valid sizing you do not buy hardware or configure the inference engine. But its input is real observations: the distribution of prompt and output lengths, the prefill/decode mix of the workload, the real P95s already being seen in pre-production. Without that data the calculation is a napkin sketch, defensible only until the first customer arrives who does not fit the assumed average.

The analogy: the hotel with variable-sized rooms

Imagine a hotel where the rooms have no fixed size: each guest pays for the square metres they need, and the building’s floor plan reorganises itself dynamically to accommodate whoever arrives. Management wants to maximise occupancy, but it has two real constraints and one quality metric.

Constraint 1 — physical space. The floor has 1,000 m² in total. If a family comes in needing 200 m², that family occupies that surface and it cannot be handed to the next guest. The largest room limits how many simultaneous guests fit.

Constraint 2 — service staff. There are 10 receptionists. Each can handle the check-in of one guest every two minutes. When 60 guests arrive within an hour, the last ones wait in a queue; the time from walking into reception to receiving their key depends on how many are ahead of them.

Quality metric — a promise about time. The brochure says “check-in in under 15 minutes”. If too many guests arrive at once, that promise breaks even if there is physical space free.

The physical space is the GPU’s VRAM. Each room is a request with its KV cache (larger the longer the conversation). The receptionists are the compute units (Streaming Multiprocessors + Tensor Cores). The check-in is the prefill phase; the nights the guest spends afterwards are the decode steps. The 15-minute promise is the TTFT P95 SLO.

The hotel’s capacity planning is exactly this: given the expected guest profile (how many arrive per hour, how much space they ask for on average, how many minutes of waiting they tolerate), work out how many floors and how many receptionists are needed. It is not done by estimating “rooms per hour” in the abstract; it is done by crossing the two budgets with the promise about time. The analogy holds up the calculation all the way to the end.

The three SLO inputs

Before putting a single number on the sheet, you have to write down the three dimensions of the SLO. Without this, the calculation is aesthetics, not engineering.

TTFT P95 (Time-To-First-Token). The time from the client sending the request until it receives the first token. It is dominated by the prefill phase (processing the whole prompt in one go) plus the scheduler queue. For conversational chat, a reasonable target sits between 0.5 and 2 seconds P95. For coding assistants with large prompts (5–10 K tokens of context), between 2 and 4 s P95. Below 500 ms you enter UX territory for voice-style conversations, but that demands serious architectural compromises.

TPOT P95 (Time-Per-Output-Token). The time between consecutive tokens during decode. It dominates the “perceived fluency” of streaming. Above 80 ms/token the human reader perceives pauses; below 30 ms/token the output flows faster than it can be read. The usual industry target: 40–60 ms P95.

Sustained RPS meeting the SLO. The throughput the system must support without violating TTFT or TPOT. This is DistServe’s key metric, called goodput (see Continuous batching). “200 RPS peak” is not the same as “200 RPS with TTFT P95 ≤ 1.5 s”. Without the SLO condition, the RPS number means nothing.

These three dimensions come with a workload profile: the distribution of prompt and output lengths. Medians are not enough; you need P50, P95, P99. A badly measured profile is the main reason sizings fail.

The central formula: two budgets that cross

The calculation resolves into two independent sums that are then crossed. The smaller of the two rules.

VRAM budget

For a GPU with total VRAM $V$, the space available for KV cache is:

$$V_{\text{kv}} = V - V_{\text{model}} - V_{\text{activations}} - V_{\text{overhead}}$$

where:

$V_{\text{model}}$ is the size of the weights: for a model of $P$ parameters in a format of $b$ bytes/parameter, $V_{\text{model}} = P \cdot b$. Llama 70B BF16 = $70 \times 10^9 \times 2 = 140$ GB. Under tensor parallel TP=4, each GPU carries $140 / 4 = 35$ GB.
$V_{\text{activations}}$ are the intermediate buffers of the forward pass. For vLLM with a reasonable batch, between 2 and 6 GB per GPU depending on batch size and maximum length.
$V_{\text{overhead}}$ is the CUDA context, NCCL buffers, the PagedAttention pool, reserved paged blocks. 2–4 GB typical.

The KV cache budget per GPU is what is left over. For an H100 SXM 80 GB with Llama 70B TP=4 BF16:

$$V_{\text{kv}} = 80 - 35 - 4 - 3 = 38 \text{ GB per GPU} = 152 \text{ GB aggregated over TP=4}$$

The KV cache cost per token for a model with $L$ layers, $H_{\text{kv}}$ KV heads (GQA), per-head dimension $d_h$, in a format of $b$ bytes:

$$\text{kv per token} = 2 \cdot L \cdot H_{\text{kv}} \cdot d_h \cdot b$$

The factor of 2 is because both K and V are stored. For Llama 70B (L=80, $H_{\text{kv}}$=8 with GQA, $d_h$=128, BF16 = 2 bytes):

$$\text{kv per token} = 2 \cdot 80 \cdot 8 \cdot 128 \cdot 2 = 327,680 \text{ bytes} = 320 \text{ KB/token}$$

And the maximum concurrency at average context $C$:

$$N_{\text{max}} = \frac{V_{\text{kv}}}{C \cdot \text{kv per token}}$$

With an aggregated $V_{\text{kv}}$ of 152 GB and an average context of 1,500 tokens (800 prompt + 700 generated at the worst moment of the conversation):

$$N_{\text{max}} = \frac{152 \times 10^9}{1,500 \cdot 320 \times 10^3} \approx 316 \text{ concurrent requests}$$

This is the physical ceiling of concurrency for that replica. It is not what you are going to use; it is what you cannot exceed without OOM. The operational number sits well below it (head-room for spikes).

Time budget

Two sub-calculations come in here: decode (memory-bound) and prefill (compute-bound).

Decode TPOT. For every token generated, the model weights (the ones relevant to that request) have to be walked through and the accumulated KV cache has to be read. The bottleneck is HBM bandwidth. For a GPU with bandwidth $B$ GB/s and a model of $V_{\text{model per gpu}}$ GB of weights:

$$\text{tpot}_{\text{theoretical}} \approx \frac{V_{\text{model per gpu}}}{B}$$

For an H100 SXM with HBM3 at 3.35 TB/s and Llama 70B TP=4 BF16 (35 GB/GPU):

$$\text{tpot}_{\text{theoretical}} \approx \frac{35}{3,350} \approx 10.4 \text{ ms/token}$$

This is the best theoretical case with batch=1 and 100 % HBM efficiency. In practice vLLM on H100 with Llama 70B TP=4 reaches 12–18 ms/token at low batch and 30–45 ms/token at high batch (at concurrency 32, tokens compete for the shared HBM). The defensible operational number: 35 ms/token at concurrency 24–32.

Prefill throughput. Prefill processes N prompt tokens in a single forward pass. It is compute-bound: the bottleneck is FLOPs. For an H100 SXM with 989 sustained BF16 TFLOPs and Llama 70B (each forward pass costs roughly $2 \cdot P \cdot N$ FLOPs per sequence of length N):

$$\text{prefill tps} = \frac{4 \cdot \text{TFLOPs} \cdot \eta}{2 \cdot P} = \frac{4 \cdot 989 \times 10^{12} \cdot 0.5}{2 \cdot 70 \times 10^9} \approx 14,000 \text{ tokens/s}$$

(the factor of 4 is the GPUs in TP, $\eta$ is real efficiency between 0.4 and 0.6 on H100). An 800-token prompt takes this long in prefill:

$$\text{prefill time} = \frac{800}{14,000} \approx 57 \text{ ms}$$

Adding a typical queue of 100–300 ms at high concurrency, TTFT P95 ≈ 350–500 ms for that profile. Well below the 1.5 s target; there is margin.

The crossing

The real operational concurrency $N_{\text{op}}$ is the minimum of the VRAM ceiling, the concurrency at which TPOT starts to degrade past the SLO, and the concurrency at which TTFT starts to degrade past the SLO (prefill queue). For the example:

VRAM ceiling: 316.
TPOT degrades to 80 ms (SLO) at around concurrency ~80–100 (measured empirically with a benchmark, not a closed formula).
TTFT degrades to 1.5 s at around concurrency ~40–60 because of the prefill queue.

The replica’s operational concurrency is ~50. Applying 25 % head-room for spikes and rebalancing, target concurrency per replica ≈ 35–40.

Step-by-step worksheet: Llama 70B BF16 on 4×H100 SXM

The exercise’s input:

SLO: TTFT P95 ≤ 1.5 s; TPOT P95 ≤ 60 ms; 200 sustained RPS.
Workload: prompt P50=600, P95=1,200, P99=2,500; output P50=180, P95=500, P99=900. Average prompt 800, output 250.
Generic hardware: 4×H100 SXM 80 GB with NVLink, vLLM v1 engine, tensor parallel 4, BF16.

Step 1 — VRAM per GPU. Weights 35 GB, activations 4 GB, overhead 3 GB → KV budget 38 GB/GPU = 152 GB aggregated. KV/token for Llama 70B GQA = 320 KB. Ceiling of live tokens in cache: $152 \times 10^9 / 320 \times 10^3 \approx 475,000$ tokens. At the operational average context (800 prompt + 200 already generated = 1,000 live tokens per request), concurrency ceiling $\approx 475$.

Step 2 — average duration of a request. Prefill 800 tokens / 14,000 tps = 57 ms. Decode 250 tokens × 35 ms/token = 8,750 ms. Total $\approx 8.8$ s per request.

Step 3 — replica throughput. If the replica sustains operational concurrency 40 and each request lasts 8.8 s, the replica delivers roughly $40 / 8.8 \approx 4.5$ requests/s in steady state.

Step 4 — number of replicas. For a 200 RPS target: $200 / 4.5 \approx 45$ replicas. That is 45 × 4 = 180 GPUs. Too much: this sizing does not work because the cost per request is high.

Step 5 — review the levers. Before buying more hardware, there are three levers to explore, in this order:

Quantisation. Dropping to FP8 reduces weights to 17.5 GB/GPU (leaving more VRAM for KV cache → more concurrency), roughly doubles decode tokens/s (HBM saturated by half), and degrades MMLU quality typically by 0.5–1.5 points on models like Llama 70B. Rewriting the calculation in FP8: TPOT drops to ~18 ms, total time per request to 4.7 s, RPS per replica rises to ~8.5, replicas needed ≈ 24, equivalent to 96 GPUs.
Speculative decoding. With a small drafter and 60–70 % acceptance, effective TPOT falls 30–40 %. RPS per replica rises to ~12, replicas ≈ 17 = 68 GPUs.
Disaggregated serving. Separating prefill workers and decode workers lets each be scaled to the real mix of the workload (see Disaggregated serving). It usually shaves off another 20–40 % under asymmetric workloads.

Step 6 — recommended sizing. For the example, with FP8 + speculative decoding and 25 % head-room: 20 vLLM replicas at TP=4 over 80 H100 SXM. If the team does not want to depend on aggressive quantisation (pure BF16 for maximum fidelity), the calculation rises to 30 replicas = 120 GPUs and forces a renegotiation of the SLO or the budget.

Step 7 — write down the assumptions. This is the part no valid sizing skips. In the team’s repo, next to the calculation:

# sizing/llama70b-prod.yaml
fecha: 2026-06-01
slo:
 ttft_p95_ms: 1500
 tpot_p95_ms: 60
 rps_target: 200
workload:
 prompt_tokens_p50: 600
 prompt_tokens_p95: 1200
 output_tokens_p50: 180
 output_tokens_p95: 500
 asunto: chat productivo con RAG ligero
modelo:
 arquitectura: llama-70b-instruct
 formato_pesos: fp8
 motor: vllm-v1
hardware:
 gpu: H100-SXM-80GB
 topologia: TP=4 con NVLink intra-nodo
 red_inter_replica: 25 GbE
optimizaciones:
 - paged_attention
 - chunked_prefill
 - speculative_decoding (drafter llama-1.1b, aceptación esperada 65%)
asunciones_criticas:
 - utilizacion_hbm_eficiente: 0.55
 - head_room_pico_sobre_p95: 0.25
 - aceptacion_speculative_min: 0.55
plan_validacion:
 - benchmark vllm bench serve antes de procurement
 - canary 10% durante 7 días post-deploy

Without this YAML, the calculation is not reproducible a month later.

MoE case: Mixtral 8×22B (~141 B total, 39 B active)

MoEs change the calculation along one key dimension: the total weights are large but the active weights per token are small. For Mixtral 8×22B with top-2 routing:

Weight VRAM: $141 \times 2 = 282$ GB BF16. With TP=4 → 70 GB/GPU. It does not fit in an H100 80 GB alongside KV cache + activations. You need TP=8 (~35 GB/GPU) or FP8 with TP=4 (~35 GB/GPU).
Decode TPOT: dominated by the weights read per token, which are $\sim 39 / 8 \cdot 2 \approx 9.75$ GB/GPU with TP=4 (one top-2 expert per token, divided across 4 GPUs). On an H100 with HBM at 3.35 TB/s, theoretical TPOT ≈ 3 ms/token. In practice, 10–20 ms at reasonable concurrency.
Prefill: similar to the dense model of the active weights, ~39 B FLOPs/token.

MoE sizing usually delivers more RPS per GPU than an equivalent dense model: the low cost per token compensates for the extra VRAM. See MoE inference for the detail of routing and why a high batch is decisive so that each expert sees enough tokens.

Sensitivity table: context and quantisation

For Llama 70B on 4×H100 SXM (TP=4), operational concurrency per replica with an SLO of TTFT 1.5 s / TPOT 60 ms:

Average context	BF16	FP8	INT4 (AWQ)
500 tokens	55	110	180
1,000 tokens	40	80	130
2,000 tokens	24	50	85
4,000 tokens	12	26	48
8,000 tokens	6	13	25

Approximate numbers from public vLLM benchmarks as of June 2026, with ±20 % variation depending on engine version and the headroom adopted. To validate on your own hardware: vllm bench serve with your profile of real prompts.

The five habitual traps

Trap 1 — confusing the mean with P95. Average throughput over an hour may be 50 RPS while the 5-minute peak reaches 180 RPS. Sizing against the mean guarantees breaking the SLO at every peak. Rule: size against the hourly P95, with 20–30 % head-room over P95.

Trap 2 — not measuring the real prefill/decode split. A “RAG with short answers” workload spends 70–80 % of GPU time in prefill; a “writing assistant generating essays” spends 80 % in decode. The useful optimisations (chunked prefill vs speculative decoding) change radically. Without measuring it, you buy badly balanced hardware.

Trap 3 — sizing with no head-room for retraining or rollback. The production cluster is not just the inference engine: there is a re-embedding batch when the embeddings model changes, continuous canary evals (see Canary, blue-green and shadow), light fine-tuning, hot stand-by for rollback. Reserve 15–25 % of capacity for those non-negotiable workloads.

Trap 4 — “GPU at 100 % SM utilization” as a target. SM occupancy of 95 % with saturated HBM produces the same throughput as SM at 60 % with saturated HBM. The bottleneck in decode is the HBM. Optimising for “GPU usage 100 %” without looking at HBM utilisation and arithmetic intensity makes you spend more on GPU without gaining throughput. See GPU observability for LLM inference for which metrics to actually watch.

Trap 5 — not documenting the assumptions. A sizing with no reproducible YAML (workload, model, engine, head-room, critical assumptions) leaves the team with no way of knowing what changed when the cluster stops meeting the SLO six months later. Documenting is cheap; losing a quarter to debugging is not.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB with intra-node NVLink and 25 GbE between nodes, the recurring configurations in May 2026 are:

Model	Format	TP	Replicas that fit	Typical RPS per node (ctx 1K)
Llama 8B	BF16	1	4 (one per GPU)	240–320
Llama 8B	FP8	1	4	450–600
Llama 70B	BF16	4	1	30–45
Llama 70B	FP8	4	1	60–90
Llama 70B	INT4 AWQ	2	2	90–130
Mixtral 8×22B	FP8	4	1	90–140
Qwen 72B	BF16	4	1	28–42

These numbers are orders of magnitude to start the conversation, not commitments. The definitive sizing is validated with vllm bench serve or genai-perf (NVIDIA) using the customer’s real prompt/output profile. The prefill/decode asymmetry of each case’s workload can move these numbers 30–50 % up or down.

For clusters of 8×H100 SXM (typical of DGX servers or equivalent replicas), the options open up to TP=8 for 405B-class models or multi-replica TP=2 for 70B models with greater density. The metric that decides is always the same: tokens meeting SLO per kW and per euro of amortised hardware.

How the sizing is validated before buying

The spreadsheet sizing is the first half. The second is the validation benchmark.

Stage 1 — back-of-the-envelope sizing. This post’s formulas applied to the SLO and the expected workload. Output: an approximate number of replicas and a topology.

Stage 2 — synthetic micro-benchmark. On a borrowed GPU or one rented by the day, bring up the engine with the chosen model and run vllm bench serve with prompts of representative lengths. Validate TPOT, prefill TPS and the concurrency ceiling. Calibrate the HBM efficiency factor ($\eta$) used in the formulas.

Stage 3 — load test with realistic traffic. Generate traffic following the real distribution of the customer’s workload (not Poisson, not constant: the real trace). Measure P50/P95/P99 of TTFT, TPOT, throughput. Confirm the head-room.

Stage 4 — canary in production. With the cluster sized, route 5–10 % of real traffic for 7–14 days before closing the procurement of additional hardware. See Canary, blue-green and shadow for the mechanics.

Jumping from Stage 1 to full procurement is the most frequent cause of a cluster that is 40 % oversized and 60 % undersized at the same time, in different regions of the workload. Four weeks of validation done well save four months of refactoring.

What we have not covered (upcoming articles)

The observability metrics that close the sizing loop in production, see GPU observability for LLM inference.
Autoscaling that adjusts replicas to the real traffic curve, see Autoscaling LLM on Kubernetes.
Detailed cost accounting per tenant (showback / chargeback) over the sized hardware.
Sizing for continuous fine-tuning (PEFT and light training) that shares a cluster with inference.

References

Kwon et al. — vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (SOSP 2023).
Zhong et al. — DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024).
Agrawal et al. — Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI 2024).
NVIDIA — H100 Tensor Core GPU Architecture Whitepaper (HBM3 memory, bandwidth, sustained FLOPs).
vLLM project — vllm bench serve reference (benchmarking CLI included in the repo).
NVIDIA — genai-perf (official tool for benchmarking LLM services).

LLM Guard: the sworn translator with a notebook of equivalences — anatomy, scanners and its integration with Langfuse, vLLM and LiteLLM

Mon, 01 Jun 2026 05:00:00 +0200

This post is a deep dive into a single piece inside the layer covered in the post on guardrails and LLM safety. That one maps the four lines of defence (input, retrieval, tool, output) and the 2026 OSS catalogue from a bird’s-eye view; this one gets down to ground level on LLM Guard, because its Anonymize/Deanonymize pattern, its composable scanner model and its four deployment modes deserve their own treatment. The analogies built up there (HACCP kitchen, four CCPs) still hold: this post zooms in on the tool that occupies the PII belt and the individual scanners within that architecture.

TL;DR

LLM Guard is the OSS tool (MIT, Protect AI) that materialises the LLM guardrail layer with a model radically different from that of NeMo Guardrails and Guardrails AI: instead of a declarative DSL (Colang) or a validator framework with external LLM-as-judge calls, it offers a catalogue of specialised compact detectors, 15 input scanners and 21 output scanners, composable as a Python pipeline, with one distinctive mechanism: the Anonymize → LLM → Deanonymize with Vault pattern. The Vault is a centralised store of the mapping between real entities (John Doe, 12345678X) and placeholders ([REDACTED_PERSON_1], [REDACTED_DNI_1]); on input, entities are redacted and the mapping is saved; the LLM never sees real personal data; on output, the Deanonymize scanner restores the originals before returning the answer to the user. This post takes apart: the internal anatomy (Vault + scanners + orchestrator with fail_fast and TTL cache), the four deployment patterns with their mathematics (in-process library, FastAPI API, OTel sidecar over vLLM, AI Gateway plugin — LiteLLM, Envoy AI Gateway, Kong AI Gateway), the integration diagrams with Langfuse (via LLM Guard’s OTel HTTP exporter + langfuse.score() from the AI Gateway), the mathematics with the project’s benchmarks (Anonymize at 177 ms CPU → 128 ms ONNX-CPU → 125 ms GPU FP16 → 38 ms GPU+ONNX, scaling ×4.6 when you combine ONNX + GPU), the ONNX pattern as the default acceleration without a dedicated GPU, the comparison with NeMo Guardrails (declarative Colang DSL oriented to conversational flow) and Guardrails AI (JSON contract-style validators with external judges), the application to on-premise hardware (which scanners hold up on CPU, which need a shared GPU) and the seven operational pitfalls specific to the tool.

The analogy: the sworn translator with a notebook of equivalences

A serious sworn translator working with sensitive documents, an employment contract, a medical record, a tax return, does not send the raw text to the machine translation service in the cloud. They keep a notebook of equivalences open on the desk. When the original document arrives, they open the notebook and start writing down: “Marta García” → [PERSON-1], “12345678X” → [DNI-1], “ES91 2100 0418…” → [IBAN-1]. They substitute each occurrence in the text with its label and pass the anonymised text to the translation service. The service returns a translation that still contains the labels. The translator opens the notebook again, restores each label with its original value, and hands the client the final translation with the PII intact. As far as the translation service is concerned, those personal data never existed: it only saw placeholders.

This is the exact operation that defines LLM Guard’s character against the rest of the ecosystem. NeMo Guardrails solves safety with a declarative graph of rules in Colang; Guardrails AI with validators that invoke an LLM-as-judge to verify contracts; LLM Guard with a catalogue of specialised compact detectors + the Vault pattern. All three are valid in different scenarios. The choice is not a matter of taste: it is structural, according to how the system is built and where the bottleneck is.

The translator also checks, of course, that the text contains no other problems besides PII: insults, instructions to reprogram itself, links to hostile pages, code that should not be there. That is what the rest of the scanner catalogue is for. But the house signature, what sets it apart, is that notebook.

Internal anatomy of LLM Guard

The three structural pieces are:

1. The orchestrator (scan_prompt, scan_output). It receives an ordered list of scanners and runs them sequentially over the text. It returns the triple (sanitized_text, results_valid, results_score) where:

sanitized_text is the text transformed by the scanners that mutate it (Anonymize, BanSubstrings with redaction).
results_valid is a {scanner_name: bool} dict indicating which scanners passed.
results_score is a {scanner_name: float} dict with the reported risk score (0 clean, 1 maximum violation).

It supports fail_fast=True to stop after the first failure. It supports a per-scanner timeout so it does not block on a slow detector. When exposed as a FastAPI API, it supports a TTL cache to avoid rescanning repeated prompts (the case of bots with identical questions).

2. The scanner catalogue. Fifteen input scanners and twenty-one output scanners, each with its own backend model and its configurable threshold:

Family	Input	Output	Dominant backend
PII	Anonymize	Deanonymize, Sensitive	Presidio + BERT-NER
Injection and jailbreak	PromptInjection	—	Fine-tuned DeBERTa (Protect AI’s own)
Toxicity and bias	Toxicity, Sentiment	Toxicity, Bias, Sentiment	Fine-tuned RoBERTa / BERT
Banned topics	BanTopics, BanCompetitors	BanTopics, BanCompetitors	Zero-shot BART-MNLI classifier
Substrings and regex	BanSubstrings, Regex	BanSubstrings, Regex	string matching + regex
Secrets	Secrets	—	detect-secrets (Yelp) + regex
Structure	TokenLimit, Language, InvisibleText, Gibberish	JSON, Language, LanguageSame, Gibberish, ReadingTime	tokeniser, lang-detect, JSON schema
Code	BanCode, Code	BanCode, Code	language classifier + regex
URLs	—	MaliciousURLs, URLReachability	block list + DNS lookup
Answer quality	—	NoRefusal, Relevance, FactualConsistency	NLI cross-encoder + cosine similarity

Each scanner is imported and instantiated individually, with its own threshold:

from llm_guard.input_scanners import Anonymize, PromptInjection, Toxicity, Secrets
from llm_guard.vault import Vault

vault = Vault()
scanners = [
 Anonymize(vault, threshold=0.5),
 PromptInjection(threshold=0.85),
 Toxicity(threshold=0.7),
 Secrets(),
]

3. The Vault. A unique piece not found in NeMo Guardrails or Guardrails AI with the same model. It is an in-memory dictionary per session or request that stores the placeholder → original_value mapping. The Anonymize scanner writes it on input and the Deanonymize scanner reads it on output. If the Vault is shared across multiple requests from the same user, the mapping persists (useful for multi-turn conversations). If it is per request, it is discarded after the answer.

The basic Vault is a Python dict; for distributed environments with multiple pods, it is replaced by a sticky Redis (same user → same pod) or by a custom Vault that reads from and writes to an external Redis, discarded after a TTL. This is operational, not part of the core library.

The Anonymize → LLM → Deanonymize flow in detail

The canonical usage pattern of LLM Guard breaks down into six exact steps:

1. Receive the user's prompt:
"My name is Marta García and my IBAN is ES9121000418450200051332,
can you review the charge from 14 March?"
2. scan_prompt() with [Anonymize(vault), PromptInjection(), Toxicity()]
→ Anonymize redacts entities and stores them in the vault:
vault["[REDACTED_PERSON_1]"] = "Marta García"
vault["[REDACTED_IBAN_1]"] = "ES9121000418450200051332"
→ PromptInjection checks there is no jailbreak (there is none)
→ Toxicity checks there are no insults (there are none)
→ results_valid = {Anonymize: True, PromptInjection: True, Toxicity: True}
→ sanitized_prompt:
"My name is [REDACTED_PERSON_1] and my IBAN is [REDACTED_IBAN_1],
can you review the charge from 14 March?"
3. Call the LLM with sanitized_prompt:
→ vLLM receives the prompt with no real PII
→ generates the answer:
"Yes, [REDACTED_PERSON_1], I am going to review the charge on account
[REDACTED_IBAN_1]. Can you confirm the amount?"
4. scan_output() with [Deanonymize(vault), Toxicity(), Relevance(), Sensitive()]
→ Deanonymize replaces placeholders with values from the vault:
[REDACTED_PERSON_1] → "Marta García"
[REDACTED_IBAN_1] → "ES9121000418450200051332"
→ Toxicity checks the answer is not offensive
→ Relevance checks it answers the prompt
→ Sensitive checks no unauthorised PII appears
(in this case, the restored PII is authorised because the user
brought it themselves and the Vault signs it → the rule applies
only to new PII invented by the LLM)
→ sanitized_response:
"Yes, Marta García, I am going to review the charge on account
ES9121000418450200051332. Can you confirm the amount?"
5. Return sanitized_response to the user.
6. If the session continues, the vault persists and the next turns reuse
the same placeholders. When the session ends, the vault is discarded.

Three details that matter operationally:

Persistent entities ([REDACTED_PERSON_1] for “Marta García”) stay constant throughout the session. If the user mentions another person (“I spoke to Juan Pérez”), Anonymize will assign [REDACTED_PERSON_2]. Cross-turn coherence is guaranteed by the Vault.
The LLM never sees the original data during the session. This is the key property for cases where the LLM is served from a cloud model or when the prompt is logged (Langfuse, OTel) without confidential access.
LLM Guard’s logging records the placeholders, not the original values. For audit with original values, an additional layer is needed (access to the Vault with privileged permissions), and this is by design, not by default.

Four deployment modes

Mode 1 — In-process Python library

The simplest: pip install llm-guard, import the scanners in the application code, call scan_prompt/scan_output directly. The models load into the process. The advantage is minimum latency; the disadvantage is that each application replica loads its own models into memory.

# on the app server
from llm_guard import scan_prompt, scan_output
from llm_guard.input_scanners import Anonymize, PromptInjection, Toxicity
from llm_guard.output_scanners import Deanonymize, Toxicity as OutToxicity, Relevance
from llm_guard.vault import Vault

vault = Vault()
input_scanners = [Anonymize(vault), PromptInjection(), Toxicity()]
output_scanners = [Deanonymize(vault), OutToxicity(), Relevance()]

# in the request handler
sanitized_prompt, valid_in, score_in = scan_prompt(input_scanners, user_prompt)
if not all(valid_in.values()):
 return error_response(score_in)

response = vllm_client.complete(sanitized_prompt)

sanitized_resp, valid_out, score_out = scan_output(output_scanners, sanitized_prompt, response)
if not all(valid_out.values()):
 return error_response(score_out)

return sanitized_resp

It fits pattern A (sidecar) from the guardrails post when the app and the sidecar share a process. And pattern C (in-process) if the app is itself the inference layer.

Mode 2 — Its own FastAPI API

The project includes a ready-made FastAPI server (llm-guard-api) that exposes the scanners behind two REST endpoints:

POST /analyze/prompt
body: {"prompt": "...", "scanners": [...] (optional)}
response: {"sanitized_prompt": "...", "is_valid": bool, "scanners": {scanner: {is_valid, risk_score}}}
POST /analyze/output
body: {"prompt": "...", "output": "...", "scanners": [...]}
response: analogous

Configuration through config/scanners.yml with environment variables (SCAN_FAIL_FAST, CACHE_MAX_SIZE, CACHE_TTL, SCAN_PROMPT_TIMEOUT…). It carries Prometheus metrics at /metrics and OTel HTTP exporter traces by default.

It fits pattern B (centralised service behind an AI gateway) from the guardrails post.

Mode 3 — OTel sidecar on the inference engine pod

For vLLM deployments on Kubernetes, a variant of mode 2 is to deploy the LLM Guard API as a sidecar container in the same vLLM pod, talking over localhost. The AI gateway in front invokes the sidecar before and after inference. The node’s OTel collector aggregates vLLM’s spans with LLM Guard’s gen_ai.guardrail.* spans automatically because they share a trace_id propagated via HTTP baggage.

This fits pattern A (sidecar) from the guardrails post, but with the discipline of a REST API so as not to couple languages (the AI gateway can be Envoy in C++, LLM Guard in Python).

Mode 4 — Plugin inside an AI gateway

Three AI gateways support LLM Guard as a native plugin in 2026:

LiteLLM Proxy (MIT, BerriAI) — llm_guard plugin, enabled in config with guardrails: ["llm_guard"]. It calls the API internally.
Envoy AI Gateway (CNCF, Apache 2.0) — ai-guardrails filter with a pluggable backend pointing at the LLM Guard service.
Kong AI Gateway (Apache 2.0) — ai-proxy plugin with a post-processor that invokes LLM Guard.

In all three cases, the AI gateway is the single entry point from the client app to the LLM; the gateway calls LLM Guard before and after passing to the inference engine. Advantage: zero lock-in in the application code; switching from LLM Guard to NeMo Guardrails means changing the gateway plugin, not rewriting the app. Disadvantage: the extra hop adds latency (typically 5-15 ms intra-cluster).

Graphical integration with Langfuse, vLLM and the OTel stack

The three integration routes with Langfuse that matter operationally:

Route A — LLM Guard’s OTel HTTP exporter. LLM Guard has a native OTel HTTP exporter. By setting OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://langfuse.cluster/api/public/otel, the gen_ai.guardrail.* spans emitted by each scanner arrive directly at Langfuse and appear as child spans of the main LLM span (provided the trace_id is propagated via HTTP baggage from the AI gateway). This is the canonical route in 2026.

Route B — Langfuse scoring API from the AI gateway. On receiving LLM Guard’s response with the per-scanner risk_score, the AI gateway (LiteLLM, Envoy AI, Kong AI) issues a langfuse.score(trace_id, name="guardrail.PromptInjection", value=0.87, comment="blocked") call for each scanner. In Langfuse it appears as scores attached to the same trace as the inference. It enables “blocks per category” dashboards and per-scanner time series. It is complementary to route A: A brings the spans, B brings the numeric score that is easy to aggregate in SQL.

Route C — Langfuse sessions + Vault metadata. In conversational mode, the AI gateway propagates langfuse_session_id to the Vault as its key. When a user has a multi-turn session, Langfuse shows the complete trace of the session, with the placeholders reused turn after turn. The original PII still never travels to Langfuse, only the placeholders and their categories.

The node’s OTel Collector is the glue: it receives spans from vLLM (via OpenLLMetry or native instrumentation), from LLM Guard (via its OTel exporter) and from the AI gateway (standard HTTP instrumentation), joins them by trace_id, and sends them in parallel to Langfuse (via OTLP HTTP) and to Tempo/Jaeger. LLM Guard’s Prometheus metrics go to VictoriaMetrics via normal scraping. Grafana offers the unified view for cross-trace investigation; Langfuse offers the LLM-centric view with sessions and scores. The OTel GenAI tracing post details the complete mechanics of the Collector.

The mathematics that matter

Latency per scanner — the real numbers

The project publishes reproducible benchmarks. For the Anonymize scanner (input length 317 chars, batch 5), the reference data are:

Platform	Backend	Avg latency	p99	QPS
AWS m5.xlarge (CPU)	Transformers	177 ms	326 ms	1,789
AWS m5.xlarge (CPU)	ONNX runtime	128 ms	180 ms	2,464
AWS r6a.xlarge (AMD CPU)	Transformers	244 ms	284 ms	1,298
AWS g5.xlarge (NVIDIA A10G)	Transformers FP16	125 ms	498 ms	2,532
AWS g5.xlarge (A10G)	ONNX + GPU	38 ms	99 ms	8,317

Three operational observations:

ONNX always wins. Even on CPU, ONNX brings the average down from 177 to 128 ms (a factor of 1.4×). On GPU with ONNX, it drops from 177 to 38 ms (a factor of 4.6×). The practical rule: always export the scanner’s model to ONNX before production. The official SaaS preview uses it by default.
A GPU without ONNX does not perform as well as you might expect. An A10G without ONNX (125 ms) is comparable to an m5.xlarge with ONNX (128 ms). The GPU alone does not compensate if the inference graph is not optimised. The relevant pairing is ONNX + GPU.
p99 latency without ONNX explodes. On GPU without ONNX, the p99 of 498 ms triples the average of 125 ms — queues and batching produce high tail latencies. With ONNX, the p99/avg ratio drops to 2.6× (99/38), far more predictable.

For a guardrail layer with five scanners run sequentially (Anonymize, PromptInjection, Toxicity, Secrets, BanTopics), the sum of the p99s is what determines the budget for line 1 (input). Five scanners at ~100 ms p99 each = 500 ms accumulated p99, outside the budget for interactive chat. With ONNX we get down to ~50 ms each = 250 ms p99, which is manageable. With fail_fast=True, the expected time is lower (the most likely case is that the cheap ones pass and the expensive ones only fail if they run at all).

For a finer calculation, the expected latency of the pipeline with fail_fast is:

\[ \mathbb{E}[L] = \sum_{i=1}^{N} L_i \cdot \prod_{j=1}^{i-1} p_j \]

where $L_i$ is the latency of scanner $i$ and $p_j$ the probability that scanner $j$ returns valid. On well-behaved traffic (most prompts pass every scanner), $\prod p_j \approx 1$ and the formula collapses to the direct sum. On adversarial traffic, the faster scanners at the start of the pipeline cut in earlier and the expected latency drops sharply.

Computational cost per scanner

The size of the backend model determines the cost and the possibility of running on CPU vs requiring a GPU:

Scanner	Typical backend model	Parameters	VRAM FP16 / ONNX-INT8	CPU viable
Anonymize (BERT-NER)	dslim/bert-base-NER	110 M	220 MB / 55 MB	Yes (with ONNX)
Anonymize (BERT-large)	dslim/bert-large-NER	335 M	670 MB / 170 MB	Yes but slow (~500 ms CPU)
PromptInjection	Fine-tuned DeBERTa-v3-base	184 M	370 MB / 90 MB	Yes (with ONNX)
Toxicity	unitary/toxic-bert	110 M	220 MB / 55 MB	Yes
Sentiment	distilbert-sst2	67 M	130 MB / 35 MB	Yes
Gibberish	small distilbert	67 M	130 MB / 35 MB	Yes
BanTopics	BART-MNLI zero-shot	407 M	815 MB / 200 MB	Slow on CPU (~400 ms)
Bias (output)	RoBERTa-bias	125 M	250 MB / 65 MB	Yes
FactualConsistency	cross-encoder/nli-deberta	184 M	370 MB / 90 MB	Yes
Relevance	sentence-transformers	110 M	220 MB / 55 MB	Yes
TokenLimit, Regex, JSON, BanSubstrings, Secrets	(no model)	—	0	Trivial

Sensible on-premise pattern: model-free scanners (TokenLimit, Regex, BanSubstrings, Secrets) run on CPU without blinking. Anonymize, PromptInjection, Toxicity, Sentiment and Relevance run comfortably on CPU with ONNX-INT8 at ~50-150 ms p99. BanTopics and the ones based on large cross-encoders (FactualConsistency) are the candidates to live on a shared GPU if you want p99 < 100 ms.

API throughput on a cluster

One instance of the FastAPI API with 4 Uvicorn workers on a node with 8 vCPUs reaches ~600-1,200 RPS over a typical 5-scanner pipeline on CPU + ONNX. To scale:

Horizontally: replicate pods behind a ClusterIP Service — linear scaling, because the scanners are stateless (except the Vault, which is per session and is externalised to Redis if you want it sticky or shared).
Vertically with GPU: 1 H100 serves ~5,000-10,000 RPS with all the scanners in ONNX-GPU. It is overkill for most deployments except multi-tenant ones with thousands of sustained QPS.

The practical rule from the guardrails post (1 guardrail GPU per 4-8 LLM GPUs) holds here: with a 4×H100 SXM cluster serving Llama 70B at TP=4, an L4 or RTX 4090 dedicated to the LLM Guard service covers the load.

Comparison with NeMo Guardrails and Guardrails AI

The three tools solve the same problem from three different architectural models. The choice between them is not about quality, since all three are mature, it is about fit with the rest of the stack:

Dimension	LLM Guard	NeMo Guardrails	Guardrails AI
Conceptual model	Pipeline of compact scanners	Declarative Colang graph (conversational flow)	JSON contract-style validators
Dominant detection	Specialised ML models (BERT, DeBERTa) per category	Rules + LLM-as-judge	Heuristic validators + external LLM-as-judge
PII workflow	Anonymize + Vault + Deanonymize	Via integrated Presidio, no built-in Vault	PII validators, no automatic restitution
Licence	MIT	Apache 2.0	Apache 2.0 (+ paid Hub)
Language	Python	Python + Colang DSL	Python
API maturity	Built-in FastAPI API, built-in OTel	Built-in FastAPI server, partial OTel	External API server
Cluster deployment	Lib + API + sidecar + gateway plugins	Lib + server	Lib + server + SaaS Hub
Typical latency (5 ONNX-GPU scanners)	50-200 ms	100-500 ms (more if there is an LLM judge)	100-300 ms (depends on the validator)
When it shines	Apps with heavy PII, multi-tenant with sessions, GDPR/HIPAA requirements	Conversational systems with defined flows, agents with a dialogue policy	Apps with strict JSON contracts, structured output with additional validation
When it does not fit	If you need a declarative dialogue policy	If you want compact detectors with no LLM judge	If you want an automatic Vault and Deanonymize

The three are complementary in large deployments. A mature pattern in 2026 is:

NeMo Guardrails orchestrates the dialogue flow (which tools the agent can invoke, in what order, with what cooldowns).
LLM Guard occupies the PII line + compact scanners on input and output, with its Vault doing the dirty work of anonymisation.
Guardrails AI validates structured outputs (JSON Schema, function calling) with its validators.

The separation of responsibilities avoids overlap and allows pieces to be swapped without rewriting everything. All three expose a FastAPI API and emit OTel spans; the AI gateway orchestrates them sequentially.

Applied to on-premise hardware

On the RTX 4090 (24 GB)

A 4090 dedicated to the LLM Guard service pod comfortably serves the complete pipeline in medium-scale production:

Anonymize (BERT-NER ONNX-INT8): ~50 MB VRAM.
PromptInjection (DeBERTa ONNX-INT8): ~90 MB.
Toxicity, Sentiment, Gibberish: ~150 MB total.
BanTopics (BART-MNLI ONNX-INT8): ~200 MB.
Bias, Relevance, FactualConsistency (output): ~250 MB total.

Total ~750 MB. The rest of the VRAM is idle or usable for aggressive batching. Sustained throughput of 3,000-6,000 RPS over the complete pipeline. For deployments with < 500 sustained RPS, the 4090 is under-used and can be shared with another workload (RAG embeddings, BGE reranker).

On the 4×H100 SXM cluster (320 GB total, NVLink)

There is capacity to spare by an order of magnitude. Sensible pattern:

3 H100s serving the main LLM at TP=3 (Llama 70B FP8).
1 H100 split into MIG instances (1g.10gb or similar) — one slice for LLM Guard (~10 GB MIG is more than enough), another for the reranker, another for embeddings.

Aggregate throughput for LLM Guard at that scale: 15,000-30,000 RPS. Plenty for a large multi-tenant setup with long sessions.

The specific operational pitfalls

Pitfall 1 — Vault with no TTL. The Vault grows without limit if it is not cleaned. In in-process lib mode per request there is no problem (the object is destroyed). In centralised service mode with Redis, the TTL is missing and Redis fills up. A silent trap discovered when the Redis pod gets OOM-killed in production after six weeks.

Pitfall 2 — Vault not shared between pods + AI gateway with no sticky session. If the AI gateway distributes round-robin across multiple LLM Guard pods, one pod’s local Vault knows nothing of the mapping created by another. Result: on turn 2 of a session, Deanonymize does not find the placeholders from turn 1 and leaves a literal [REDACTED_PERSON_1] in the answer. Solution: a shared Redis Vault or a sticky session by user_id.

Pitfall 3 — Models not exported to ONNX in production. It is deployed with the default config (Transformers) and latency is 3-5× worse than the benchmarks report. The team assumes LLM Guard “is slow”. The solution is to export to ONNX (built into the project) and configure recognizer_conf with the path to the model’s .onnx.

Pitfall 4 — fail_fast=False with many scanners. Without fail_fast, every scanner always runs, even if the first one already blocked. Latency is 3-5× worse on adversarial traffic. For production, barring an explicit reason (wanting complete per-scanner metrics even when blocking), fail_fast=True is the reasonable default.

Pitfall 5 — Infinite cache_ttl + prompts with variable PII. If the API’s cache stores the sanitized_prompt indefinitely, two different sessions with the same prompt structure but different PII can collide if the cache key does not include the Vault hash. You have to verify that the cache key includes either the complete content (without PII) or a hash of the original prompt.

Pitfall 6 — Structured logs with original PII. LLM Guard’s JSON stdout logs record only placeholders by default. But if custom hooks are added for debugging, it is easy to leak the original PII to the log. A regulatory audit (GDPR, ENS) detects this and it is a breach. Discipline: never add hooks that read from the Vault without explicit permission.

Pitfall 7 — scan_output without the original prompt. The scan_output method expects (prompt, output) for validators that compare both (Relevance, LanguageSame, FactualConsistency). If only the output is passed, those scanners fail silently or return is_valid=True by default. You have to keep the sanitized_prompt in the AI gateway and pass it to scan_output.

When to choose LLM Guard (and when not to)

Choose LLM Guard when:

The requirement for PII anonymisation with automatic restitution is on the list. It is the number one reason to use it. Banking, healthcare, legal advice, HR — any case with heavy PII that must not reach the LLM even when it is local.
You want a Pythonic pipeline with no new DSL. If the team is pure Python and prefers composing scanners as objects to learning Colang.
The stack already has an AI gateway (LiteLLM, Envoy AI, Kong AI) and it integrates as a plugin without touching the app.
You need built-in OTel and Prometheus with no additional instrumentation.

Do not choose LLM Guard when:

The system is a conversational agent with complex dialogue flows (policies, fallbacks, escalation to a human). There NeMo Guardrails with Colang is structurally better.
The safety layer reduces to validating structured outputs (JSON, function calling). Guardrails AI with its validators is more natural.
Your latency budget is ultra-aggressive (< 30 ms for the whole layer). You will have to cut scanners and accept lower coverage; perhaps a single PromptGuard 2 + Presidio in a sidecar (the pattern from the guardrails post) is simpler.
You do not want to carry the operational weight of a distributed Vault (Redis, TTL, sticky session). For small systems with no strong PII requirement, it is over-engineered.

What we have not covered (upcoming posts)

Custom scanners: how to write your own scanner when nothing in the catalogue fits (a complex domain regex, your own fine-tuned classifier). The project supports custom scanners by inheriting from InputScanner / OutputScanner with three methods.
Integration with SLSA / supply chain: how to sign the LLM Guard container with cosign, SLSA attestations, and verification in the cluster before admitting it. An operational supply chain security topic (OWASP LLM03).
Red teaming against LLM Guard: known techniques that evade detectors (homoglyphs, Unicode confusables, base64 encoding inside the prompt). The project publishes an adversarial test suite for doing your own benchmarking. How it is set up as a continuous gate in CI.
Comparative benchmark against Bedrock Guardrails and Azure AI Content Safety: F1 per category over real traffic across three different deployments. The OSS vs hyperscalers post has the strategic comparison; the technical detection comparison is still missing.

References

LLM Guard (Protect AI): https://llm-guard.com — official documentation, scanner list, benchmarks.
Repository: https://github.com/protectai/llm-guard.
LLM Guard API: https://github.com/protectai/llm-guard/tree/main/llm_guard_api.
Presidio (Microsoft): https://microsoft.github.io/presidio/ — basis of the Anonymize scanner.
detect-secrets (Yelp): https://github.com/Yelp/detect-secrets — basis of the Secrets scanner.
Langfuse OTel ingestion: https://langfuse.com/docs/opentelemetry/get-started.
LiteLLM guardrails: https://docs.litellm.ai/docs/proxy/guardrails.
Envoy AI Gateway: https://aigateway.envoyproxy.io.
Kong AI Gateway: https://docs.konghq.com/hub/kong-inc/ai-prompt-guard/.
OWASP Top 10 for LLM Applications 2025: https://owasp.org/www-project-top-10-for-large-language-model-applications/.
ONNX Runtime: https://onnxruntime.ai — exporting HF models to ONNX for acceleration.

Structured output: the form with dropdowns that strikes out invalid answers before the model picks one — Outlines, XGrammar, LLGuidance and the maths of the bitmask

Sat, 30 May 2026 16:30:00 +0200

This post complements those on Continuous batching (where the scheduler lives) and Speculative decoding (another technique operating on the last mile of the sampler). Structured output is the LLM’s output contract towards the code that consumes it; without it, the integration between an LLM and applications is fragile by default.

You are here: DEPLOY

TL;DR

An LLM produces free text, but many applications (function calling, entity extraction, routing, text-to-SQL, config generation) need to parse it as JSON, as a tool call with typed args, as a valid SQL statement, or as one option from an enum. The naive solutions fail: prompt engineering (“answer in JSON”) leaves 25 % of outputs unparseable in many models; post-hoc validation plus retry costs latency and does not guarantee termination; json-repair libraries are heuristic patches. Constrained decoding guarantees conformance 100 %, by construction: at every generation step, before sampling from the softmax over the V vocabulary tokens, the tokens that would break the target structure are masked to -∞. The output is valid by mathematical contract, not by luck. The four dominant families in May 2026 are Outlines (Willard & Louf, 2023; FSM + token trie precomputed from regex/JSON Schema/CFG), XGrammar (Dong et al., 2024-25, CMU+MLC; byte-level pushdown automaton with an adaptive cache of context-independent tokens, the default in vLLM v1, SGLang, TensorRT-LLM, NIM and MLC-LLM), LLGuidance (Microsoft Research; Earley parser + regex derivatives, ~50 µs of CPU per token, underneath OpenAI Structured Outputs) and LM Format Enforcer (noamgat; aimed at JSON Schema, integrated in many engines as a fallback). XGrammar-2 (May 2026) introduces Structural Tag and a cross-grammar cache for dynamic agentic tool calling. The real cost when well integrated: <5 % in TPOT and ~40 µs of CPU per token of mask computation, partially overlappable with the model’s forward pass. The open operational question: does it degrade reasoning? The paper Let me Speak Freely? (Tam et al., EMNLP 2024) reported significant degradation in reasoning under format constraints; the dottxt rebuttal showed that the effect came from prompts that were not equivalent between experiments. The emerging consensus in May 2026: use two-pass (free reasoning with CoT in text plus structured output in a second call) for tasks that require multi-step reasoning; use single-pass constrained for extraction, classification and function calling, where forced structure improves exactness and reduces hallucinations. This post takes apart the mechanism, the maths (bitmask size = V/8 bytes, per-step latency), the comparison table of backends, the pitfalls (compile time, tokenizer-specific FSM, streaming SSE) and the production deployment pattern.

The analogy: the form with dropdowns instead of free-text fields

Imagine two ways of asking someone to fill in a form.

The first way is to hand them the paper with blank fields and say “fill it in with this exact format: the name goes here, then the ID number without spaces, then the date as YYYY-MM-DD, then mark one of these five possible reasons separated by semicolons”. You explain the format in exhaustive detail, but the person is still free to write whatever they like in each field. If they are in a hurry they may drop a zero from the ID number, put the date in the American format, tick two reasons when only one was asked for. When you receive the form, you often have to send it back: field X badly formatted, reason Y invalid, date Z impossible. This is exactly what an LLM does when you ask it to “answer in JSON with this schema”: it works most of the time, it fails a non-negligible percentage, and you have no formal guarantee of anything.

The second way is to give them an electronic form where the fields are not free text. The name accepts any text, but the ID number has a mask that only lets you type eight digits followed by a letter; the date is a picker that only lets you choose valid dates; the reasons are a dropdown with five closed options. The person can type whatever they want, but the form does not accept invalid characters at any point. The result is parseable by construction: when you receive the completed form, the ID number has exactly the expected format, the date is a real date, the reason is one of the five. This is constrained decoding.

The analogy holds on four mappings:

The person filling it in = the LLM producing logits over the vocabulary.
The characters the form lets you type in each field = the bitmask applied to the logits before sampling.
How the form knows which characters to allow depending on which field you are in = the automaton (FSM or PDA) that holds the current state of the grammar.
The fact that the dropdown is precomputed when the page loads rather than recomputed each time = the precomputed table of valid tokens per FSM state, which makes the per-step cost amortised O(1).

The problem structured output solves

The operational problem is the contract between the LLM and the code that consumes its output. There are three naive approaches, all with documented failures:

Pure prompt engineering. “Answer only with valid JSON, no comments or prose”. It works most of the time for good models; it fails between 5 % and 25 % of the time depending on the model, the temperature, the complexity of the schema and the length of the output. The model adds a trailing comma, escapes a quote badly, wraps the JSON in a ```json markdown block, hallucinates a field that was not in the schema, ignores a required field. SqueezeBits measures a ≤72 % correct rate without constraining for some models on JSON Schemas of moderate complexity.

Post-hoc validation plus retry. The server receives the output, tries to parse it, and if that fails returns the error to the model and asks it to try again. Cost: 2-3× latency in the worst case (typically 2-3 retries before giving up), no guarantee of termination, noise in the logs, hard to test deterministically.

JSON repair libraries (json_repair, the fast-json-repair Rust port). Heuristic patches for common errors: trailing commas, missing quotes, prose interleaved with JSON. Useful as a fallback for approach 2; they are not a contract.

The accumulated operational cost: latency inflated by retries, noise in production, painful debugging of intermittent parse errors, broken contracts with downstream clients that assumed clean parsing.

Constrained decoding: the principle

At each decode step, the model produces a vector logits ∈ R^V where V is the vocabulary size (Llama 3: 128,256, GPT-4o: ~200,000, Qwen 3: 152,064). The conventional sampler applies softmax plus a sampling strategy (greedy, top-k, top-p, temperature) over the V tokens.

Constrained decoding interposes one operation before the softmax: it applies a bitmask that sets to -∞ the logits of the tokens that would violate the target grammar at this step. The result: the softmax only assigns probability mass to admissible tokens; sampling, whatever its strategy, can only pick a valid one.

The two operational questions are always the same:

How do you know which tokens are valid at each step? → hold the current state of an automaton (FSM/PDA) built from the grammar; look up the table (state → set of valid tokens).
How much does it cost to compute and apply the mask? → this is where Outlines, XGrammar, LLGuidance and LM Format Enforcer compete.

The four families of backends

Backend	Origin	Algorithm	Grammar formats	Default in	Notes
Outlines	dottxt (2023)	FSM + precomputed token trie	regex, JSON Schema, Lark CFG	HF TGI	The Rust version (outlines-core) closes part of the gap with XGrammar
XGrammar	CMU + MLC (2024-25)	Byte-level PDA + adaptive ctx-indep/dep cache	regex, JSON Schema, EBNF/CFG	vLLM v1, SGLang, TensorRT-LLM, NIM, MLC-LLM	Speedup vs naive: up to 100× CFG, 3× JSON; <40 µs/token JSON
LLGuidance	Microsoft Research	Earley parser + regex derivatives	regex, JSON Schema, Lark	OpenAI Structured Outputs (internally), Chromium	~50 µs CPU/token; underneath Guidance, llama.cpp, mistral.rs
LM Format Enforcer	noamgat	char-level parser + tokenizer prefix tree	JSON Schema, regex	(deprecated default in many engines, fallback in NIM)	Slower than XGrammar (3.5× worse JSON, 10× worse CFG)

Three operational observations:

XGrammar is the de facto default in May 2026 across the open-source ecosystem (vLLM v1, SGLang, TensorRT-LLM, NIM, MLC-LLM). Its byte-level PDA with an adaptive cache gives it almost zero overhead when well integrated.
LLGuidance is the least known but most used piece on the market, because it sits underneath OpenAI Structured Outputs (confirmed in the README of the repo itself). 50 µs of CPU per token for 128k tokenizers.
Outlines was the first, it still holds the conceptual mindshare (papers, canonical blog posts), but it lost operational ground to XGrammar. The Rust version (outlines-core) closes part of the gap.

XGrammar in detail (what sits under vLLM and SGLang)

The paper by Dong, Yin, Ruan and Chen (arXiv:2411.15100, November 2024) introduces a partitioning technique that is key to understanding why XGrammar is 3-100× faster than the alternatives.

Partitioning the vocabulary at each PDA state:

Context-independent tokens (~99 % of the vocab in a typical JSON Schema): their validity is decided only by the current PDA position, without needing to inspect the full stack. These are precomputable and stored as bitmasks in a cache.
Context-dependent tokens (~1 %): they require inspecting the PDA stack at runtime. They are handled case by case at a higher cost.

That partition is the fundamental reason why XGrammar works in production at low TPOT. 99 % of the lookups go to a precomputed table in O(1); only the residual 1 % pays the real cost.

Other combined techniques: a pushdown automaton for full CFGs (not just regex), a persistent stack for fast branching/rollback, JIT compilation plus an Earley parser in XGrammar-2.

Reported speedups (paper):

Up to 100× over previous solutions on CFG.
3× on JSON Schema versus Outlines.
Latency <40 µs per token on JSON Schema; <200 µs on XML/Python DSL.
When well integrated in vLLM/SGLang/TRT-LLM: near-zero end-to-end overhead.

XGrammar-2 (May 2026, arXiv:2601.04426) adds two pieces that matter for agents:

Structural Tag: a composable JSON protocol that unifies OpenAI harmony, tool calling and reasoning channels.
Cross-Grammar Cache for reuse at sub-structure level → it allows dynamic switching between sub-grammars in agentic loops without recompiling.
6× faster compile time than XGrammar-1.

The maths of the bitmask

Three numbers drive the operational decision.

Bitmask size per step. If the vocabulary has V tokens, the bit-packed bitmask takes V/8 bytes:

$$\text{bitmask size} = \frac{V}{8} \text{ bytes}$$

Llama 3 (V=128,256): 16 KB per bitmask per request per step.
GPT-4o tokenizer o200k (V≈200,000): 25 KB.
Llama 2 (V=32,000): 4 KB.

For a batch of 32 requests with structured output active on an H100, we are talking about ~512 KB of bitmasks per step, trivial against the GBs the forward pass moves.

Cost of applying the bitmask to the logits. A simple CUDA kernel, O(V) complexity, ~5-10 µs of latency on an H100. Negligible.

Cost of computing the bitmask (CPU-side). This is where backends differ:

Backend	CPU latency per token (Llama 3, V=128k)
LLGuidance	~50 µs
XGrammar	~40 µs (JSON Schema), ~200 µs (XML/DSL)
outlines-core	comparable to XGrammar after 2024
Outlines Python (legacy)	200-1000 µs
LM Format Enforcer	intermediate, degrades with a large vocab

With a model forward pass on the order of 10-50 ms per token in decode, a 40-50 µs mask is <0.5 % overhead, invisible. The operational key is that the CPU-side mask computation can be overlapped with the GPU-side forward pass: while the GPU computes the logits for token t, the CPU precomputes the mask for token t+1 based on the FSM state after token t-1. SGLang does this explicitly; vLLM v1 improved substantially over v0.

Compile-time cost. Precomputing the FSM/PDA and the cache:

Simple schemas (1-5 fields): <100 ms.
Deep JSON Schemas with many $defs: seconds.
OpenAI structured outputs: “10s typical, up to 1 minute for complex schemas”, cached after the first call.
XGrammar-2: 6× faster than XGrammar-1.
LLGuidance: ~2 ms startup.

Operational best practice: pre-cache the known schemas when the server starts, to avoid latency spikes on the first request for each new schema.

Does it degrade the model’s reasoning?

It is the most interesting question in the field and the answer is not obvious.

The theoretical argument for degradation: forcing the structure changes the model’s probability distribution; if the structured path pushes towards low-probability paths, the model gets “stuck” in a suboptimal branch with no room to explore.

The paper Let me Speak Freely? (Tam et al., arXiv:2408.02442, EMNLP 2024 Industry Track) reported significant degradation on reasoning tasks under format constraints (JSON/XML/YAML). The stricter the format, the greater the degradation. The paper itself, however, acknowledged an improvement on classification tasks with forced structure.

The dottxt rebuttal, Say What You Mean (official blog post): a methodological critique. The prompts in the original paper were different between structured and unstructured (not apples to apples). The JSON prompts in the original experiment gave less information than the unstructured ones. Re-running with equivalent prompts (Llama-3-8B-Instruct), dottxt does not reproduce the degradation. The conclusion: the paper confused the format constraint with the prompt engineering of that constraint.

Emerging consensus in May 2026 (gathered from technical blogs and community empiricism):

For extraction, classification, function calling and routing: constrained decoding improves exactness and reduces hallucinations. It is the right tool.
For multi-step reasoning (maths, logic, code review): use two-pass:
1. First call: free reasoning with Chain-of-Thought in natural text (generous max_tokens, no constraint).
2. Second call: pass it the reasoning plus the schema, and ask it to produce structured output with constrained decoding.

The “reason then structure” pattern gives the best of both worlds: reasoning without a straitjacket plus an output guaranteed to be parseable.

Real implementations in May 2026

Engine	Default backend	Others available	API
vLLM v1	XGrammar (auto)	Outlines, Guidance (llguidance), LM Format Enforcer	`guided_json`, `guided_regex`, `guided_choice`, `guided_grammar` in the request
SGLang	XGrammar	Outlines, LLGuidance	`response_format.json_schema`, `extra_body.regex`, `extra_body.ebnf`
TensorRT-LLM	XGrammar	LLGTRT (Rust llguidance)	Official integration since Jan 2025
NVIDIA NIM	XGrammar (switched from Outlines in 2025)	LM Format Enforcer (requires `NIM_ENABLE_KV_CACHE_REUSE=0`)	Pluggable multi-backend
llama.cpp	Native GBNF	LLGuidance	Each candidate token is tested against the parse state
HF TGI	Outlines	XGrammar (experimental)	“Guidance” feature with `/generate` and `/chat/completion` with `tools`
MLC-LLM	XGrammar (native, same team)	—	Its own API

vLLM example:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")

response = client.chat.completions.create(
 model="meta-llama/Llama-3.1-70B-Instruct",
 messages=[{"role": "user", "content": "Extract name and age from: María is 34 years old"}],
 extra_body={
 "guided_json": {
 "type": "object",
 "properties": {
 "name": {"type": "string"},
 "age": {"type": "integer"}
 },
 "required": ["name", "age"]
 }
 }
)

SGLang equivalent:

response = client.chat.completions.create(
 model="meta-llama/Llama-3.1-70B-Instruct",
 messages=[...],
 response_format={
 "type": "json_schema",
 "json_schema": {
 "name": "person",
 "strict": True,
 "schema": {...}
 }
 }
)

Usage patterns in production

Function calling / tool use. The LLM produces {tool_name: enum, arguments: object} according to a schema. The dominant case today (OpenAI, Anthropic, open models). The schema guarantees that tool_name is in the set of available tools and that arguments has the right types.

Entity extraction. A schema with {name, address, phone, ...} from unstructured text. It moves schema adherence from 76% to 98% in vendor benchmarks.

Routing / classification. The LLM picks between N options (enum in the schema). The bitmask shrinks to a handful of valid tokens → almost zero overhead, maximum reliability.

Constrained SQL. An SQL grammar as GBNF or Lark → it avoids injection and syntax errors. Useful in text-to-SQL agents.

Code generation with a valid AST. A grammar for the target language (Python, Rust, a custom DSL). It guarantees that the output is compilable or parseable code.

Agentic loops. XGrammar-2 Structural Tag for dynamic switching between formats (tool call → reasoning channel → tool result) without recompiling the grammar.

Operational pitfalls

Compile time for large schemas. JSON Schemas with many $defs can take seconds to compile. This causes slow startup if they are loaded at boot, or latency spikes on the first request if they are loaded on demand. Mitigation: warm the cache at boot with the fleet’s known schemas.

Tokenizer-specific FSM/PDA. A schema precompiled for Llama 3 (128k tokenizer) does not work for Qwen (152k). The schema cache must be keyed by (tokenizer_hash, schema_hash). Changing model = invalidating the cache.

Schema changes. Version schemas explicitly. Breaking changes → rebuild plus warm cache. Do not silence compile errors in production.

Streaming SSE. Structured output with streaming requires parsers tolerant of partial output (partial-json-parser, json-stream). Strict Pydantic v1 fails; v2 with partial validation works. Older clients may not handle incremental validation, so test with the real client.

Token healing. Tokens that cross boundaries (:// as a single token versus : + //) can break pure constraints. Outlines and XGrammar mitigate this internally; llama.cpp and others require explicit token healing. If the model sneaks a “strange” character in after the structure, this is probably why.

Real known bugs in vLLM 2025-26:

vLLM 0.8.4: XGrammar rejects minItems in JSON Schema (issue #16880).
vLLM with Qwen 2.5 VL: Outlines/XGrammar does not respect the schema in some cases (issue #13038).
The vLLM apply_token_bitmask_inplace bitmask backend is always auto, not exposed to the user.

OpenAI subset limitations: if you are going to port a schema validated on OpenAI structured outputs to another backend, check that you are not using constructs that OpenAI rejects but others allow (deep $ref, pattern, default, depth >5, anyOf as root).

Implications on on-premise hardware

On an RTX 4090 (24 GB). Any model you serve with vLLM or llama.cpp can carry structured output at practically negligible cost. The CPU-side mask latency (~40 µs) is trivial compared with typical TPOT (30-100 ms on consumer hardware). The interesting case: function calling over Llama 3 8B or Qwen 3 14B INT4 → reliable tool use without retries, with no need for a hosted API.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). Here XGrammar as the vLLM v1 / SGLang default is the way:

Llama 3 70B FP8 + XGrammar JSON Schema: TPOT P95 stable under 60 ms even with structured output active across the whole batch. It supports hundreds of distinct cached schemas.
DeepSeek-V3 + XGrammar Structural Tag: agentic tool calling with native MTP and constrained decoding combined; the mask cost overlaps with the MoE forward pass.
Multi-tenant function calling: each client can have its own set of tools (that is, its own schemas); compile time is amortised through caching, runtime is invariant.

The rule of thumb for May 2026: XGrammar by default, pre-cache the fleet’s schemas at boot, two-pass for reasoning tasks.

What we have not covered

Dynamic tool routing with XGrammar-2 Structural Tag: the detail of how TagDispatch picks sub-grammars at runtime.
Constrained beam search and its interaction with grammar: theoretical quality degradation versus greedy.
Grammars for code generation with a full AST: production-grade Python/Rust grammars, performance.
JSON Schema → Pydantic → grammar pipelines: tooling to reduce human error.
Inhibition decoding (the Inhibition Decoding paper, 2025): a variant that penalises but does not forbid certain tokens, useful for soft safety constraints.

References

Willard, B., Louf, R. Efficient Guided Generation for Large Language Models (Outlines). 2023. https://arxiv.org/abs/2307.09702
Dong, Y., Yin, X., Ruan, F., Chen, T. XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. 2024. https://arxiv.org/abs/2411.15100
XGrammar-2: Dynamic Structured Generation for Agentic LLMs. 2026. https://arxiv.org/abs/2601.04426
Tam, Z. et al. Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models. EMNLP 2024 Industry. https://arxiv.org/abs/2408.02442
dottxt blog, Say What You Mean: https://blog.dottxt.ai/say-what-you-mean.html
OpenAI, Introducing Structured Outputs in the API: https://openai.com/index/introducing-structured-outputs-in-the-api/
Outlines repo: https://github.com/dottxt-ai/outlines
outlines-core (Rust): https://github.com/dottxt-ai/outlines-core
XGrammar repo: https://github.com/mlc-ai/xgrammar
LLGuidance (Microsoft Research) repo: https://github.com/guidance-ai/llguidance
LM Format Enforcer repo: https://github.com/noamgat/lm-format-enforcer
llama.cpp grammars: https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md
vLLM Structured Outputs docs: https://docs.vllm.ai/en/stable/features/structured_outputs/
SGLang Structured Outputs docs: https://docs.sglang.io/advanced_features/structured_outputs.html
TensorRT-LLM guided decoding (Triton): https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/tensorrtllm_backend/docs/guided_decoding.html
NIM Structured Generation: https://docs.nvidia.com/nim/large-language-models/1.12.0/structured-generation.html
MLC blog, XGrammar: https://blog.mlc.ai/2024/11/22/achieving-efficient-flexible-portable-structured-generation-with-xgrammar
MLC blog, XGrammar-2: https://blog.mlc.ai/2026/05/04/xgrammar-2-fast-customizable-structured-generation
SqueezeBits, Guided decoding performance: vLLM vs SGLang: https://blog.squeezebits.com/guided-decoding-performance-vllm-sglang
Red Hat, Structured outputs in vLLM: https://developers.redhat.com/articles/2025/06/03/structured-outputs-vllm-guiding-ai-responses
Aidan Cooper, Constrained Decoding guide: https://www.aidancooper.co.uk/constrained-decoding/

Anatomy of an on-premise LLM inference stack: the seven layers that have to hold each other up

Sat, 30 May 2026 16:00:00 +0200

TL;DR

The on-premise LLM inference stack is not a model server: it is a building of seven layers that hold each other up. The inference layer (vLLM / SGLang) serves tokens, but without a dedicated embeddings layer RAG does not work; without the gateway layer the client couples its SDK to the specific engine; without an LLM-aware observability layer (Langfuse + OpenTelemetry GenAI) any quality degradation goes unnoticed; without a GitOps control plane layer (Flux + Forgejo) any manual change leaves invisible debt; and without a dependency tracking layer (Hubble + intent-based policies) decommissioning a Service silently breaks applications nobody remembers were using it. This post is born from a concrete incident: a pipeline that reported status: completed and matched_jobs: 0 for days because it kept calling an Ollama that had already been scaled to zero, while a badly written except labelled the generic ConnectError as “ChromaDB indexing error” and the vector store, innocent, took the blame. Each of the three symptoms was the cry of a layer that was missing or badly designed. The body of the post walks the seven layers with their canonical OSS piece, the design decisions that break them, the sizing maths on a generic 4×H100 SXM cluster (320 GB of VRAM, NVLink) and a final diagram of the connected stack. The thesis: a stack that passes the incident test is not measured by its peak throughput, it is measured by how long it takes to shout when something drifts from the design.

You are here: the seven layers seen from above

Before going into detail, the map. The seven layers are not seven servers: they are seven responsibilities the stack has to cover. A layer can collapse into one pod (gateway) or spread across several components (observability = traces + metrics + flow logs). What it cannot do is be missing.

Layers 1–4 sit on the request path: if they fall, the client notices in seconds. Layers 5–7 sit on the path of design and operations: if they fall, there is no immediately visible error, and that is why they are the ones that produce silent incidents, which is how most serious incidents end up. This post argues that the quality of the stack is measured in layers 5, 6 and 7, because 1–4 are commodities where everyone picks roughly the same pieces.

The analogy: the office building with shared services

Picture a twelve-storey office building with several tenants. It has a concierge desk (gateway: filters who comes in), it has lifts (LLM inference: they move the heavy load), it has stairs and a goods lift (embeddings: light but constant traffic, hardly anyone notices them until they break), it has plumbing and water tanks (vector store: what actually holds the state), it has a distribution board and sensors (observability: what warns you when something is drawing more than planned), it has a building manager (GitOps control plane: the only legitimate authority to move anything), and it has a tenant register (dependency tracking: who is connected to which shared service).

When a new client moves into the third floor and asks to install a server that needs more amperage than planned, the problem is not electrical: it is administrative. If the client can go straight to the board and plug in whatever they like, the building survives for a while and then a breaker trips at three in the morning. If the client has to go through the manager, the manager consults the tenant register (is anyone else hanging off that same circuit?), reviews the electrical planning (are we at the limit?) and either authorises or redirects. The building stays standing not because of its electrical installation but because of the discipline of going through the manager.

The LLM inference stack works identically. The physical layers (1–4) are the visible ones, the ones marketing people put on the slide. The governance layers (5–7) are the ones that distinguish a platform from a pile of pods with good luck.

Now to the incident that motivates the whole post.

The hook: the log that lied for six days

The application is called jobhunter internally. It is a cron pipeline that every six hours sweeps public job-ad sources, filters by EU geography, embeds the new ads, indexes them in a vector store and matches them against search profiles. The last step fires notifications.

For six days the pipeline reported the same thing on every run:

status: completed
total_found: 756
new: 23
matched_jobs: 0 ← zero, run after run

And in the logs:

[INFO] httpx: POST chromadb:8000/api/v2/.../collections/<id>/delete → 200 OK
[INFO] matcher: Purged 12 expired jobs from ChromaDB
[ERROR] pipeline: ChromaDB indexing error: All connection attempts failed
[INFO] httpx: POST chromadb:8000/api/v2/.../collections/<id>/get → 200 OK
[ERROR] pipeline: Matching error: All connection attempts failed

This is a log that invites you to blame ChromaDB. And in fact the first post-mortem written internally pointed at a version incompatibility between the Python client and the v2 server. A reasonable hypothesis, technically plausible, completely false.

The real cause: weeks earlier the general LLM on the platform had been migrated from Ollama to vLLM. The migration was clean for the two applications that depended directly on the large model, their manifests were pointed at the new endpoint. What nobody did was look at who else was calling the Service ollama.ollama.svc:11434. jobhunter called it to generate the embeddings for the ads. When the Ollama deployment was scaled to zero, the Service was left empty, and any outbound connection received a generic ConnectError("All connection attempts failed") from httpcore. The try/except wrapping the whole matching stage caught the exception and labelled it “ChromaDB indexing error”, because that was the lexical wrapper of the block, not because ChromaDB had anything to do with it. ChromaDB was answering 200 to delete and get in those same logs.

Three factors kept the incident alive for six days:

The except named the wrapper rather than the stage that failed. The log said “ChromaDB indexing error” when the error was an embeddings call against a nonexistent Service.
The pipeline returned status: completed even when there were errors. No alerting based on status fired. The metric that would have fired (matched_jobs stuck at zero) was not instrumented.
The pipeline image ran as :latest with no versioning, no SBOM, no reproducibility. When it started failing, it was impossible to know which Ollama client it had been built with.

Each of the three symptoms is the cry of a layer that was missing. The first asks for LLM-aware observability that distinguishes stages (layer 5). The second asks for the pipeline logic and the Prometheus metrics to behave like contracts (layer 5, SLI/SLO dimension). The third asks for GitOps with pinned images and an auditable SBOM (layer 6). And the whole incident, decommissioning a Service without knowing who consumes it, shouts dependency tracking (layer 7).

The rest of the post walks the seven layers through that lens: what each one solves, which OSS piece implements it in 2026, and which typical design decision breaks it.

Layer 1 — Gateway: the client SDK must not couple to the engine

What it has to solve. That the client sends POST /v1/chat/completions with the standard OpenAI SDK and never learns which engine (vLLM, SGLang, TensorRT-LLM), which specific model, which LoRA adapter or even which GPU pool is serving the request. Authentication, rate limit, routing by body.model and by tenant, header injection for tracing.

Canonical OSS piece in 2026. Envoy AI Gateway (Envoy with GenAI extensions: routing by model, token-based rate limit, fallback chains) or Cilium Gateway API with your own filters. For cases where the client wants multi-provider without distinguishing on-prem from SaaS, LiteLLM Proxy is the lightweight equivalent.

What breaks the layer. Exposing the inference engine endpoint directly. If clients call http://vllm-prod.svc:8000, any change of engine, model or pool forces you to touch the code of every app. The rule: the engine changes, the contract does not. The OpenAI SDK is standard; the routing behind the gateway is where design freedom lives.

Where jobhunter failed. There was no gateway. The app called ollama.ollama.svc:11434 directly. When Ollama died, there was no intermediate layer that could answer with a fallback, a retry against another pool, or at least a 503 error with a descriptive body.

Layer 2 — LLM inference: vLLM as the default choice, SGLang when prefix caching rules

What it has to solve. Serving tokens with throughput and latency under control: continuous batching, PagedAttention so the KV cache does not fragment the VRAM, FP8 so a 32B model fits with room to spare, multi-LoRA for per-tenant customisation without replicating the base, structured output for function calling with schema guarantees.

Canonical OSS piece in 2026. vLLM is the default choice: it covers the state of the art (PagedAttention, continuous batching, FlashAttention, FP8 quantisation, multi-LoRA, structured output, speculative decoding). SGLang comes in when the workload has high prefix caching (long chat with large system prompts, agents with repeated instructions), because its RadixAttention composes better than standard prefix caching. For very specialised inference with proprietary NVIDIA kernels, TensorRT-LLM, accepting the hardware lock-in.

What breaks the layer. Assuming a single model “does everything”. A 32B chat LLM does not serve /v1/embeddings; if you try, vLLM answers BadRequestError: "The model does not support Embeddings API". Assuming it did was one of the concrete wounds of the incident: the app expected an endpoint the model did not implement.

The design decision that costs more later. Serving the LLM with aggressive quantisation (INT4) without a quality eval calibrated for your corpus. INT4 with AWQ or GPTQ saves VRAM, but it degrades answers in technical or legal Spanish in a measurable way. The rule: any quantisation change goes through the same golden eval as a model change.

Layer 3 — Embeddings: separate from the LLM, fixed dimension, a life of its own

What it has to solve. Generating dense vectors for the RAG corpus and for retrieval queries. The embeddings model is a different thing from the chat LLM: different architecture (encoder, not decoder), different size (hundreds of millions of parameters, not tens of thousands of millions), different API (an /embeddings endpoint that takes text and returns a fixed-dimension vector).

Canonical OSS piece in 2026. Infinity or Hugging Face Text Embeddings Inference (TEI) to serve models of the bge-*, multilingual-e5-*, nomic-embed-* families with high throughput and multi-model support. OpenVINO Model Server when Intel hardware is available. sentence-transformers as a fallback embedded in the application itself when the corpus is small and the deployment is constrained.

What breaks the layer. Treating it as “the LLM does that too”. It does not if it is chat-only; and even when it does (some models have a dual endpoint), mixing chat and embeddings serving in the same process punishes the throughput of both. The physical separation is the design.

The technical fact people forget. The vector store and the embeddings model form an indivisible unit. Changing the model from multilingual-e5-large (1024 dim) to multilingual-e5-small (384 dim) is not a substitution: it is creating a new collection (my_corpus_v2) and re-embedding the whole corpus. If you upsert into the old collection, you hit a dim mismatch at runtime that takes the pod down. This seems obvious and is violated constantly, because the embeddings model is chosen once and then forgotten.

Layer 4 — Vector store + relational data + storage: what actually holds the state

What it has to solve. Persisting the vectors with efficient filters (tenant_id, created_at, source), persisting the relational metadata (users, configs, versioned prompts, traces), and persisting the model weights, the LoRA adapters, the datasets and the original corpora.

Canonical OSS pieces in 2026.

Qdrant for large collections (>200 k vectors), payload-aware filters, multi-tenancy via collection or via field.
pgvector for small collections with mandatory relational joins (wanting to do WHERE doc.author = ... AND vector <=> $1 in the same SQL).
PostgreSQL operated by CloudNativePG (CNPG) for the relational side: Barman Cloud backups, replication, connection via pooler.
MinIO for S3-compatible objects: bucket per tenant, cross-site replication, weights and adapters versioned by sha256.
Redis for queues, rate-limit counters and a cache of frequent retrievals.

What breaks the layer. Assuming the vector store is stateless ephemeral. It is exactly the opposite: it is the component where losing state costs the most. Without verified backups of the vector store, an index corruption forces you to re-embed the whole corpus, and on a corpus of millions of documents that is hours or days of GPU.

The design decision that pays later. Forgetting to version the collection by dimension and embeddings model. Suggested convention: my_corpus__embed-multie5l__1024d__v3. The name carries metadata; any change in any of the three attributes forces a new collection. It is ugly but it protects against an accidental upsert with the wrong dim.

Layer 5 — Observability: LLM-aware traces + infrastructure metrics + flow logs

What it has to solve. That any inference can be recovered from its trace_id, with all the gen_ai.* attributes (semantic conventions) plus your own attributes (tenant_id, adapter_id, priority_tier), a breakdown of latency (queue → prefill → decode → network), tokens consumed, tools invoked, exact model and adapter. And in parallel: Prometheus metrics from vLLM (vllm:num_requests_running, vllm:gpu_cache_usage_perc, vllm:prefix_cache_hit_rate), from the GPU (DCGM Exporter), from the network (Hubble flow logs with drops and NetworkPolicy enforcement).

Canonical OSS pieces in 2026.

OpenTelemetry Collector as the single transport for traces, metrics and logs, with OTLP receivers and separate exporters per destination.
Langfuse self-hosted for the LLM-aware side: tracing, prompt versioning, evals with LLM-as-judge (post and post).
VictoriaMetrics + Grafana for high-throughput TSDB metrics with long retention.
Hubble (Cilium) for L3/L4/L7 flow logs and NetworkPolicy visualisation.
DCGM Exporter for GPU metrics.

What breaks the layer. Wrapping the whole pipeline stage in a single try/except that labels the exception with the wrapper’s name. The operational rule: one try/except per stage, with the stage label in the message and a Prometheus metric with labels={"stage": "<name>"}. That way “ChromaDB indexing error” would never have been the log for an embeddings failure; it would have been “Embeddings call failed: ConnectError(ollama:11434)”.

Complementary rule. The pipeline returns status: completed if and only if there were no errors. With errors it returns completed_with_errors or failed, and the metric pipeline_errors_total{stage} is incremented. An alert based on increase(pipeline_errors_total[1h]) > 0 fires before the second failed run. Without this discipline, observability exists but does not warn.

Layer 6 — GitOps control plane: the only legitimate authority

What it has to solve. That the cluster state is the state declared in git. That any divergence between git and the cluster is visible and, in critical components, auto-reconciled or auto-alerted. That every deployed image has an immutable tag (sha digest or semver pin), an SBOM (Trivy) and traceability back to the commit that introduced it.

Canonical OSS pieces in 2026.

Flux (or ArgoCD) as the reconciler.
Forgejo (or Gitea, GitLab CE) as the self-hosted OSS forge.
cert-manager + Trust Manager for internal PKI.
External Secrets Operator + SOPS for encrypted versioned secrets.
Kyverno (or OPA Gatekeeper) for binding policies: deny :latest images, deny pods without a NetworkPolicy, deny Services without an owner label.
Trivy for SBOM and vulnerability scanning of images in the CI pipeline.

What breaks the layer. Images with a mutable tag (:latest, :main). Any kubectl edit in production that is not reflected in git. main branches with write permissions for humans without review. The rule: if a human can mutate the cluster without going through a signed commit, you do not have GitOps, you have somebody’s whiteboard.

How it applies to the incident. If the pipeline image had been pinned to a sha digest (registry.interno.local/jobhunter@sha256:9af2...), the team could have audited immediately which Ollama client it carried. With :latest, not even that.

Layer 7 — Dependency tracking: the layer the incident exposed

What it has to solve. Knowing who calls which Service, both declaratively (what the gitops repo says) and observed (what has been seen crossing the network over the last N days). And, on mature platforms, propagating that information as policy: if nobody declares and nobody observes traffic to the Service ollama.ollama.svc, decommissioning it is safe; if somebody declares it or uses it, decommissioning it opens a ticket.

Canonical OSS pieces in 2026.

Hubble (Cilium) for observed flow logs: hubble observe --to-namespace ollama --since 14d gives the list of source namespaces that have talked to Ollama in the last two weeks.
Otterize for intent-based policy: each Deployment declares “I need to talk to ollama-svc”, and the operator generates the corresponding NetworkPolicy and maintains a browsable catalogue of who is trying to talk to what.
manual kubectl-grep as a fallback: kubectl get deployments,cronjobs,statefulsets -A -o yaml | grep -E 'ollama[.-]' produces the declarative list.
NetworkPolicy as code reviewed in CI: every PR that touches a Service requires the associated policy to be kept or explicitly updated.

A pre-decom checklist that the incident suggests codifying as a CI hook:

SVC="ollama.ollama.svc.cluster.local"

# (a) declarative grep in the gitops repo
git -C $GITOPS_REPO grep -l "$SVC" || echo "OK declarative"

# (b) observed grep in Hubble (last 14 days)
hubble observe --to-fqdn "$SVC" --since 336h --output json \
 | jq -r '.source.namespace' | sort -u || echo "OK observed"

# (c) live grep in the cluster
kubectl get all -A -o yaml | grep -E "$SVC" || echo "OK live"

If all three return empty, the decom is safe. If any of them has content, there is downstream debt still open. The jobhunter incident is exactly what happens when this check does not exist: the team that decommissioned Ollama looked at the list of applications it knew depended on it directly; nobody looked at the list of those that depended on it silently.

The maths that matter: sizing the stack on 4×H100 SXM (320 GB)

The generic reference cluster for everything that follows: 4×H100 SXM 80 GB, NVLink between the four, 640 GB of system RAM, 2×NVMe NVMe-oF for local storage of weights and caches, redundant 25/100 GbE to the switch. Aggregate VRAM: 320 GB.

The VRAM budget is not free. A first reasonable split for a stack that serves one large general LLM, a medium code-specialised model, embeddings and a reranker, with room for multi-LoRA and for the KV cache:

Component	Reference model	Quant	Model weights	Reserved KV cache	Total VRAM
General LLM (TP=4)	70B-instruct	FP8 W8A8	70 GB	60 GB	130 GB
Code LLM (TP=2)	32B-coder	FP8 W8A8	32 GB	28 GB	60 GB
Embeddings	multilingual-e5-large	FP16	1.3 GB	n/a	8 GB (×2 replicas)
Reranker	bge-reranker-v2-m3	FP16	0.6 GB	n/a	4 GB
Multi-LoRA pool (over the general LLM)	up to 16 adapters	bf16	16 × 0.4 GB ≈ 6 GB	reuses the LLM’s KV	6 GB
Reserved for fragmentation + overhead					~30 GB
Total committed					~238 GB / 320 GB
Free headroom					~82 GB

The 26 % of free headroom is not waste: it is what lets the vLLM scheduler avoid preempting requests under moderate pressure, lets continuous batching group large batches without aborting, and lets a failover from the other site promote a standby without OOM.

Expected throughput, with the 70B general LLM in FP8 and tensor parallel 4, on an H100 SXM with continuous batching active and prefix caching at 35–55 % (typical in multi-turn chat with shared system prompts):

$$ \text{aggregate tokens/second} \approx 1500 \text{ to } 2500 $$

for concurrency between 32 and 64 requests, with sub-second TTFT P95 on short prompts (<2k tokens) and TPOT P95 around 40–60 ms/token as perceived by the client. These numbers are reasonable orders of magnitude, not guarantees: real throughput depends on the prompt mix, on whether speculative decoding (EAGLE-3) is active and burnt-in, and on the network cost between gateway and inference pods.

Embeddings throughput on two replicas of multilingual-e5-large with dynamic batching:

$$ \text{embeddings/second} \approx 3000 \text{ to } 6000 \quad (\text{optimal batch} \sim 64) $$

Enough to reindex a corpus of 1 million documents in roughly an hour, assuming chunks of 512 tokens and two chunks per document on average. For corpora of tens of millions of documents, re-embedding is done by delta via CDC over the source (covered in RAG corpus curation), not by full sweep.

Retrieval latency on Qdrant with HNSW (M=16, ef_construct=200) over a collection of 5 million 1024-dim vectors filtered by tenant_id:

$$ \text{P95 latency retrieve top-50} \approx 8 \text{ to } 25 \text{ ms} $$

Below the cost of cross-encoder reranking (bge-reranker-v2-m3 over top-50 = roughly 30–60 ms more), and below any call to the LLM. The bottleneck in a well sized RAG pipeline is never the vector store: it is the LLM’s decoding.

Final diagram: the complete connected stack

The solid lines are the request path: client → gateway → inference engine → vector store/embeddings → response. The dashed blue lines are telemetry: each component emits OTel to the collector, which routes traces to Langfuse, metrics to VictoriaMetrics and logs to Loki. The dashed red lines are reconciliation: the GitOps control plane keeps any layer in its declared state and warns of divergence.

The diagram is not decorative: every arrow is a stable contract between two layers. If one layer changes (vLLM → SGLang, multilingual-e5 → bge-m3, Qdrant → pgvector), the arrows hold. That stability of contracts is the architectural property that lets a team migrate components without breaking downstream apps.

Typical design decisions that break the stack

A short list of mistakes seen repeatedly in stacks that looked well designed on paper:

1. Coupling the client SDK to the inference engine. Dropping the gateway because “vLLM already speaks OpenAI-compatible” works on day one and hurts the day you have to add a fallback, a canary or a second model.

2. Sharing the LLM and embeddings endpoint. A qwen2.5-32b-Instruct is chat-only; BadRequestError: "The model does not support Embeddings API" is the cry of a design that confused the two layers.

3. Reusing the vector store collection when changing the embeddings model. Different dimensions do not allow upsert. Versioning the collection by (model, dim, version) is ugly but it saves the day of the change.

4. A try/except wrapping an entire pipeline with the wrapper’s label. The log lies because the label is lexical, not causal. Every stage in its own try/except with its own label and its own metric.

5. status: completed with errors. The pipeline has to distinguish completed, completed_with_errors and failed, and the alerting has to fire on the last two. Without this, observability exists in theory and does not warn in practice.

6. Images with a mutable tag. :latest and :main are not tags, they are aliases. Without a sha digest there is no reproducibility and no auditable SBOM.

7. Decommissioning a Service without a pre-decom check. The three-grep check (declarative + observed + live) takes two minutes and costs six days of incident when it is skipped.

8. Default limits.memory on pods that load models. A sidecar loading sentence-transformers + torch + tokenizer needs 2–4 GB; with limits.memory: 1Gi you hit OOM on the first pod restart, and sometimes with no alert because the liveness probe answers through another route.

All of them are variants of the same principle: the stack does not fail in its most expensive layer (inference, where nobody underestimates the cost), it fails in the cheap, boring layers (gateway, observability, GitOps, dependency tracking) where it is tempting to save.

Applied to typical on-premise hardware: a 4×H100 SXM cluster

On the generic reference cluster (4×H100 SXM 80 GB, NVLink, 640 GB RAM), the suggested split into pods:

gpu-node-01 (4×H100 SXM, intra-node NVLink)
├── vllm-llm-general (TP=4) ~130 GB VRAM (4 GPUs)
└── (shares GPUs with the multi-LoRA pool on the same deployment)

gpu-node-02 (4×H100 SXM, second node)
├── vllm-llm-code (TP=2) ~60 GB VRAM (2 GPUs)
├── infinity-embeddings (×2) ~16 GB VRAM (shared on 1 GPU with optional MIG)
├── tei-reranker ~4 GB VRAM (cohabitant)
└── failover reserve ~120 GB VRAM free for canary / standby

Not everything fits comfortably on a single node; two nodes with two H100 SXM each would be enough for the conservative setup, and the rest of the cluster (CPU-bound: gateway, vector store, observability, control plane) runs on GPU-free nodes.

The operational rule: inference concentrates, the rest of the stack distributes. Concentrating inference maximises the use of NVLink (cross-GPU tensor parallel without going through PCIe); distributing the rest prevents an event on the GPU node from taking the control plane down with it.

An even more conservative configuration, for SMEs with a single 4×H100 SXM node as a starting point, serves the general LLM (TP=4) and embeddings/reranker cohabiting with MIG (Multi-Instance GPU to partition an H100 into hardware-isolated slices). The code LLM is deferred to a second phase. It is viable and cost-aware; what is not viable is doing without layers 5, 6 and 7.

What we have not covered (upcoming posts)

This post focuses on the static design of the stack. Some pieces still deserve an article of their own:

The multi-site active/standby plane: Cilium Cluster Mesh, cross-cluster Qdrant replication, realistic RTO/RPO, when active-active pays off and when it does not.
The continuous fine-tuning plane: how the LoRA pipeline closes the loop from production feedback to a promoted adapter, in the spirit of the retrain post.
The safety/guardrails plane: where Llama Guard fits, Presidio for PII and XGrammar for guaranteed structured output, connected to layer 1, the gateway.
The cost plane: gen_ai.usage.* instrumentation at tenant and model level, tokens/euro dashboards, GPU elasticity decisions via KEDA.
The compliance plane: how the stack maps to ENS High, NIS2 and ISO/IEC 42001 without turning the deployment into a compliance exercise that paralyses delivery.
Below the engine: the mini-series that opens the stack’s basement — the interconnect (NVLink/NCCL), the host (NUMA, hugepages, CPU isolation) and the RKE2 resource managers that pin each pod to the right NUMA node.

Each one falls into a different series on the blog and will be covered with the same discipline: a concrete piece, a justified decision, the typical mistake seen in practice.

References

The six-stage LLMOps pipeline — the general framework this stack operates in.
The OSS catalogue for LLMOps in six stages — a card per tool in the stack.
Anatomy of an LLM request in production — the same architecture seen from the perspective of an individual request.
OSS vs hyperscalers in LLMOps — a comparison with the AWS/Azure/GCP stacks.
LLM tracing with OpenTelemetry GenAI — the backbone of layer 5.
Multi-LoRA serving — the piece that gives per-tenant customisation without replicating the base.
RAG corpus curation and Reranker and hybrid retrieval — everything that goes into layers 3-4 so that RAG does not degrade.
Quantisation for LLM inference — the reason for the FP8 W8A8 in the sizing.

Relevant official documentation

vLLM Production Stack — docs.vllm.ai
SGLang RadixAttention — github.com/sgl-project/sglang
Envoy AI Gateway — aigateway.envoyproxy.io
Langfuse self-hosted — langfuse.com/docs/self-hosting
OpenTelemetry Semantic Conventions for GenAI — opentelemetry.io/docs/specs/semconv/gen-ai
Hubble flow observability — docs.cilium.io/en/stable/observability/hubble
Otterize intent-based access — docs.otterize.com
Flux GitOps toolkit — fluxcd.io

Continuous batching: the salon with 8 chairs that does not wait for the slow customer — Orca, vLLM, chunked prefill and goodput

Sat, 30 May 2026 16:00:00 +0200

This post complements those on KV cache (the artefact continuous batching manages), PagedAttention (the memory piece that makes it viable), Disaggregated serving (the next layer of optimisation), Speculative decoding, Multi-LoRA and MoE (the three extensions that coexist with the scheduler in production).

You are here: DEPLOY

TL;DR

The static batching of the original HuggingFace Transformers (the pre-2022 era) systematically under-used the GPU for two structural reasons. First: the scheduling unit was the complete request; the longest one in the batch blocked all the others until it finished (severe head-of-line blocking, P95 TTFT five to ten times worse than reasonable). Second: each batch slot reserved memory for max_seq_len even though the real output was much shorter; documented padding waste sat between 60 % and 80 % and sustained GPU SM utilisation on real workloads fell to 20-40 %. Orca (Yu et al., OSDI 2022, FriendliAI + Seoul National University) introduced the idea that unjammed everything: the scheduling unit stops being the request and becomes one decoder iteration, a single token. After each iteration the scheduler can add new requests to the batch and remove the finished ones. vLLM (Kwon et al., SOSP 2023, UC Berkeley) materialised it open-source and production-grade thanks to PagedAttention, which solves the KV cache fragmentation that theoretical continuous batching would cause with dynamic allocation. SARATHI / Sarathi-Serve (Microsoft Research India, OSDI 2024) closed the last gap: the prefill stalls that paused active decodes when a new request arrived, via chunked prefill (splitting a long prefill into small chunks and mixing them with decodes in the same step) and stall-free batching. DistServe (Zhong et al., OSDI 2024) reformulated the key metric: what matters is goodput (requests/s meeting the TTFT and TPOT SLOs), not raw throughput. In May 2026, vLLM v1 brings a unified scheduler with chunked prefill always-on; SGLang adds RadixAttention, which gives cross-request prefix-cache hits; TensorRT-LLM calls it in-flight batching; llama.cpp supports it natively. The three operational tensions are speculative decoding (nested raggedness), multi-LoRA (each request with its adapter) and MoE (each expert sees very few tokens per step at a typical batch size). This post takes apart the mechanism, the maths (GPU utilisation, goodput vs throughput), the three variants (Orca → vLLM → Sarathi-Serve), the pitfalls (preempt-on-OOM, starvation, inverse HoL) and the real numbers with production configurations.

The analogy: the salon with 8 chairs

A hair salon with 8 chairs and a single brilliant stylist who moves between them. Customers arrive with very different needs: some want a 15-minute cut, others a 2-hour colour with a base, others a 90-minute straightening. The question is how to organise the flow.

The traditional strategy (what HuggingFace Transformers did in its original generate()) is to seat 8 customers at once, all at the same time, and accept nobody new until the last one finishes. If one of those 8 is a 2-hour job, the 7 who wanted the 15-minute cut sit there doing nothing for 1 hour and 45 minutes. The stylist finishes with the quick ones and is left staring at empty chairs until the colour is done. When everyone is ready, another 8 come in. That is static batching, and the only thing that stops it from being worse is that the GPU does not complain like a human customer.

The continuous strategy (Orca, vLLM) changes the scheduling unit. The stylist does not think “I am going to do one whole customer and then the next”; he thinks “on each tick I take one step of work at every occupied chair and, every time a chair frees up, I call the next customer in the queue without waiting for the others to finish”. The quick-cut customer leaves after 15 minutes, his chair is filled immediately with the next one, and the slow jobs proceed at their own pace without delaying anyone. The stylist is never idle.

The continuous strategy with chunked prefill (SARATHI / Sarathi-Serve) adds a subtler distinction. Some customers need a long initial phase (a 10-minute hair analysis before the cut; in LLM terms, the prefill of the prompt). Without chunked prefill, the stylist had to stop all the other chairs to do the new customer’s analysis in one go, and that was a stall visible in the TPOT of the active ones. With chunked prefill, the analysis is split into 2-minute pieces interleaved between the other customers’ active cuts. The customers already under way no longer notice pauses; the new customer takes slightly longer to start his cut proper, but the whole salon does not freeze.

And the metric that matters: the owner does not want to maximise “customers served per hour” at the cost of some leaving furious. He wants to maximise “customers served per hour within the time SLA”, and that is goodput, DistServe’s contribution.

The problem continuous batching solves

There are two structural pathologies of static batching that deserve to be explained with concrete numbers.

Padding waste. Each batch slot reserved memory for max_seq_len (prompt + max output), even though the real output ended in far fewer tokens. For a batch of 32 with heterogeneously distributed output lengths (half of them ≤50 tokens, a long tail up to 4,000), the typical memory waste was 60-80 %. That translated directly into wasted concurrency: with the same VRAM, instead of serving 32 requests with smart allocation, you served 8.

HoL blocking (Head-of-Line). The scheduling unit was the complete request. A batch containing one request of 500 tokens and 31 of ≤50 tokens ran 450 extra “empty” iterations (the GPU executed forward passes for all 32 slots, even though 31 had already finished). Wasted compute cost: ~84 % of the tail time in the example.

Measurable result. Sustained GPU SM utilisation on real workloads under static batching: 20-40 %. That is, ~70 % of the datacenter’s compute went unused. When the first Orca and vLLM benchmarks came out showing 10-24× throughput improvement, it was not marketing exaggeration; it was recovering all that wasted compute.

Orca (OSDI ‘22): the idea that changed everything

The paper by Yu, Jeong, Kim, Kim and Chun at OSDI 2022 (“Orca: A Distributed Serving System for Transformer-Based Generative Models”, from Seoul National University + FriendliAI) introduced two contributions that have remained the basis of everything that followed.

Iteration-level scheduling. Instead of scheduling at the level of the complete request, it schedules at the level of one decoder iteration: the step that generates ONE token. After each iteration the scheduler can (a) add new requests to the batch, (b) remove requests that have generated EOS or reached max_tokens, (c) reorder priorities. The compute engine executes exactly one iteration over the current batch.

Selective batching. Here is the non-obvious subtlety. The technical problem with batching requests of different lengths and different KV cache states is that some operations (the GEMMs of the Q, K, V projections and the FFN) are insensitive to position and can be batched by concatenating tokens, whereas attention is sensitive to per-request state (each request has its own KV cache of a different length). Orca’s solution: batch the GEMMs (concatenate all the step’s tokens into a [total_tokens, hidden] tensor) and run attention sequentially per request.

Paper result: up to 36.9× throughput vs FasterTransformer on GPT-3 175B at the same latency level. Orca is not open-source, it is only documented in the paper. FriendliAI commercialises it as the Friendli Engine. But the idea was published and everyone adopted it.

vLLM (SOSP ‘23): the open-source materialisation

What Orca described as a concept, vLLM materialised in production. The paper by Kwon, Li, Zhuang, Sheng et al. (UC Berkeley Sky Computing Lab, SOSP 2023) introduces PagedAttention, covered in detail in the PagedAttention deep dive, but it also consolidates continuous batching as universal practice.

The reason PagedAttention is a prerequisite for practical continuous batching is fragmentation. If you are going to insert and remove requests from the batch dynamically, and each request has a KV cache that grows on every iteration, traditional contiguous allocation fragments the HBM until it is unusable. PagedAttention splits the KV cache into fixed-size blocks (16 tokens by default) allocated on demand from a global pool. Memory waste falls from ~60-80 % to less than 4 % (only the last partially filled block per sequence).

vLLM paper metrics (2023):

vs HuggingFace Transformers (static, no continuous batching): up to 24× throughput.
vs HuggingFace TGI (which already had primitive continuous batching): ~3.5×.
vs FasterTransformer: 2-4× at the same latency.

What operationally distinguishes vLLM from Orca: vLLM runs attention in a single fused CUDA kernel (paged_attention_kernel) over non-contiguous blocks; Orca described attention as sequential, request by request. And vLLM exposes OpenAI-compatible APIs that let you drop the engine into existing stacks without touching the client.

Chunked prefill (SARATHI / Sarathi-Serve, OSDI ‘24)

There is a detail the original Orca/vLLM continuous batching did not solve: when a new request enters the batch, its prefill (processing the whole prompt at once) can take hundreds of milliseconds. During that time, the active decodes of the other requests are essentially paused, the GPU being dedicated to the new prefill. This showed up as spikes in TPOT (“inter-token latency”) every time a long request came in, which broke strict SLAs.

SARATHI (Agrawal et al., arXiv 2308.16369, August 2023) and later Sarathi-Serve (same group at Microsoft Research India, OSDI 2024, arXiv 2403.02310) introduce two combined ideas:

Chunked prefill. A long prefill (8,192 tokens, say) is split into chunks (2,048 tokens, say) processed one per iteration. Instead of one 200 ms step processing 8K tokens, four 50 ms steps processing 2K each.

Decode-maximal batching (“stall-free”). On each iteration, the scheduler first fills the batch with the active decodes (each costing 1 token), and only the leftover space goes to new prefill chunks. The result: active decodes keep advancing 1 token per iteration without pausing, while the new request completes its prefill in small bites.

The observation that justifies it: prefill is compute-bound (it processes N tokens at once and saturates FLOPs) while decode is memory-bound (1 token per step, under-using compute, with the GPU waiting on HBM). Mixing prefill chunks with decodes in the same step exploits the arithmetic-intensity slack: the decodes piggyback on the free compute of the prefill chunk.

Numbers:

Original SARATHI (LLaMA-13B on an A6000): decode throughput +10×, end-to-end +1.33×.
Sarathi-Serve (Mistral-7B on an A100): 2.6× serving capacity vs plain vLLM. Yi-34B on 2×A100: 3.7×. Falcon-180B with pipeline parallel: 5.6×.

Adoption in May 2026: always-on in vLLM v1 (default since v0.8.0, January 2025), SGLang, TensorRT-LLM. The key setting in vLLM is --max-num-batched-tokens (the token budget per step; default 2048). Raising it prioritises throughput, lowering it prioritises low TPOT.

Goodput: the metric that matters (DistServe, OSDI ‘24)

The classic “throughput” metric (requests/s or tokens/s) has a problem when there are SLOs. A server can report 1,000 req/s while P99 TTFT is 30 seconds and the SLO is 1 second, so only about 200 req/s actually meet the contract.

DistServe (Zhong et al., OSDI 2024) formalises goodput as the right metric: goodput = max sustained request rate meeting the SLOs (TTFT bound AND TPOT bound). The practical definition is usually: maximum rate with ≥90 % of requests within both SLOs.

Why it matters for the scheduler:

Optimising raw throughput leads to maximising batch size, which inflates P99 TPOT.
Optimising goodput limits the batch size when TPOT starts violating the SLO, prefers small requests if the batch already has a tail, and leaves resources available for new requests.

DistServe result: up to 7.4× more requests served or a 12.6× tighter SLO vs vLLM at the same SLO attainment. The gain comes from disaggregating prefill and decode onto different GPUs (removing the interference between phases), but the idea of optimising for goodput is independent and applicable to any scheduler.

Operationally this translates into monitoring:

goodput_proxy = histogram_quantile(0.95, vllm:time_to_first_token_seconds_bucket) < SLO_TTFT
AND histogram_quantile(0.95, vllm:time_per_output_token_seconds_bucket) < SLO_TPOT

The iterative scheduler in action

The maths that matter

Three formulas explain much of the operational behaviour.

GPU utilisation under static batching. With a batch of size B whose seq_len_i are the real lengths and max(seq_len_i) is the length that defines the padding:

$$U_{\text{static}} = \frac{\sum_i \text{seq len}_i}{B \cdot \max_i \text{seq len}_i}$$

For B=32, 30 sequences of 50 tokens and 2 of 500: U = (30·50 + 2·500) / (32·500) = 2500/16000 = 15.6 %. Four out of every five GPU cycles wasted.

GPU utilisation under continuous batching (idealised).

$$U_{\text{continuous}} \approx 1 - \frac{T_{\text{scheduler}}}{T_{\text{iteration}}}$$

With scheduler overhead of ~50-200 µs and an iteration time of ~10-30 ms: U > 95 %. The unit of loss is no longer padding, it is scheduling overhead, and that is negligible compared with the forward pass.

Goodput vs throughput.

$$\text{Goodput}(R) = R \cdot P(\text{latency} < \text{SLO})$$

where R is the offered request rate. Typical curve: goodput grows linearly with R up to the saturation knee, then falls because P(SLO) collapses when the system congests. The optimal point is just before the knee, not at peak throughput.

Example: at R=100 req/s with P(SLO)=0.99, goodput = 99. At R=200 req/s with P(SLO)=0.4, goodput = 80. More offered load, less useful goodput.

Chunked prefill token budget. On each vLLM step with chunked prefill active:

$$\text{prefill tokens this step} = \text{max num batched tokens} - \text{num decodes active}$$

Each active decode costs 1 token of the budget; the rest is filled with new prefill chunks. If max_num_batched_tokens = 2048 and num_decodes_active = 200, there are 1,848 tokens for prefill (one chunk of 1,848 or several small chunks).

Real implementations in May 2026

Engine	Current scheduler V	Chunked prefill default	Relevant notes
vLLM v1 (default ≥0.8.0)	unified V1	always-on	EngineCore isolated in a separate process; prefix caching with O(1) eviction; preempt-mode `recompute` by default; xgrammar/outlines backends.
SGLang	own (PyTorch ecosystem)	yes	RadixAttention gives cross-request prefix-cache hits; non-blocking CPU scheduler; leader in stable latency at high concurrency.
TensorRT-LLM	proprietary “in-flight batching”	yes	Policies `GUARANTEED_NO_EVICT` (conservative, default) and `MAX_UTILIZATION` (aggressive, risk of a pause when the KV is full). Compile-time vs runtime.
Triton + tensorrtllm_backend	`gpt_model_type: inflight_fused_batching`	yes	`max_queue_delay_microseconds` to group newly arrived requests. Decoupled mode for SSE streaming.
llama.cpp (llama-server)	own	`--cont-batching` ON since 2024	`-np N` parallel slots; no PagedAttention (contiguous KV per slot), so less flexible but simpler. Endpoint `:8080/metrics`.

A typical production-ready vLLM v1 configuration:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
 --tensor-parallel-size 4 \
 --max-num-batched-tokens 4096 \
 --max-num-seqs 256 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --preemption-mode recompute \
 --scheduling-policy fcfs \
 --gpu-memory-utilization 0.92

SGLang equivalent:

python -m sglang.launch_server \
 --model meta-llama/Llama-3.1-70B-Instruct \
 --tp 4 \
 --chunked-prefill-size 4096 \
 --max-running-requests 256 \
 --enable-radix-cache

The three operational tensions

Continuous batching + speculative decoding. Speculative decoding produces 1 to γ+1 tokens per step depending on the acceptance rate. The batch stops being uniform in tokens produced per iteration: nested raggedness. PagedAttention absorbs it (the KV cache can grow at different rates per request in the same step), but the scheduler loses symmetry. At low QPS (a conversational assistant) the combination is excellent: vLLM reports up to 2.8× speedup. At high QPS, the draft consumes slots from the decode pool and can reduce aggregate goodput. Rule of thumb: disable speculative decoding when gpu_cache_usage > 0.85. Full detail in Speculative decoding.

Continuous batching + multi-LoRA. Each request in the batch can use a different adapter (via SGMV, see Multi-LoRA serving). Worst case: every request in the batch with a different adapter and a different rank, and throughput falls by up to 50 % vs the base with no LoRA. Best case: all requests on the same adapter, equivalent to the base with no LoRA. Practical mitigation: group adapters by rank in the routing ahead of the engine; set --max-lora-rank to the maximum actually served, not generously above it.

Continuous batching + MoE. Each expert sees batch · k / N tokens per step. With DeepSeek-V3 (256 experts, k=8) and batch=32 in decode, each expert processes only 1 token on average: total compute starvation. To match the per-GPU throughput of a dense model, MoE needs batches »10× larger, which puts pressure on the KV cache. Wide-EP (see MoE inference) distributes the experts across many GPUs and allows larger effective batches per expert, at the cost of all-to-all comms that add milliseconds per step.

Metrics to monitor

The Prometheus metrics exposed by vLLM (prefix vllm:, vllm_ when scraped):

vllm:time_to_first_token_seconds (Histogram) — TTFT including queue time.
vllm:time_per_output_token_seconds (Histogram) — TPOT.
vllm:e2e_request_latency_seconds (Histogram) — end-to-end.
vllm:num_requests_running (Gauge) — active batch.
vllm:num_requests_waiting (Gauge) — queue depth.
vllm:num_requests_swapped (Gauge) — preempted to CPU.
vllm:gpu_cache_usage_perc (Gauge) — fraction of KV cache occupied.
vllm:gpu_prefix_cache_hit_rate (Gauge) — prefix cache hits.
vllm:num_preemptions_total (Counter) — preemptions. Any sustained value is a red flag.

Practical operational rules:

Stable zone under sustained load: gpu_cache_usage_perc ∈ [0.7, 0.9].
Warning at >0.95 (preemption imminent).
Critical if num_requests_waiting grows faster than num_requests_running: the server is not absorbing the load; scale out.

Operational pitfalls

Preempt-on-OOM. When gpu_cache_usage reaches ~1.0 with pending requests that need to grow their KV, vLLM preempts. V1 does RECOMPUTE by default (discards the KV, regenerates it on return); V0 did SWAP (moves it to CPU). RECOMPUTE is better for short sequences (cheap to regenerate); SWAP is better for long ones. The vllm:num_preemptions_total metric should be zero or near zero in steady state.

Inverse HoL blocking (memory monopoly). A very long request occupies many KV blocks, so small requests do not fit in the batch even though compute is free. Chunked prefill mitigates the compute blocking during a new prefill, but it does not solve the memory monopoly. Partial solution: per-request limit policies (an aggressive max_tokens) or priorities.

Starvation. FCFS can leave requests pending for a long time if the active ones do not finish. vLLM supports --scheduling-policy priority with an x-priority header. Recent work (NeurIPS 2024 Efficient LLM Scheduling by Learning to Rank, arXiv:2501.14312 Locality-aware Fair Scheduling) proposes schedulers with quantum-based starvation prevention; not yet integrated into vLLM mainline.

Badly calibrated chunk size. A small chunk (512) gives low TPOT, high TTFT and memory overhead from more KV accesses. A large chunk (8192+) gives low TTFT and TPOT spikes during the chunk. Rule: start at 2,048, measure P95 TPOT, adjust.

Batch size cap. A high --max-num-seqs gives more concurrency but P99 TPOT explodes. A low one wastes throughput. Rule of thumb: max_num_seqs ≈ HBM_for_KV / (avg_seq_len × bytes_per_token_KV).

Implications for on-premise hardware

On an RTX 4090 (24 GB). llama.cpp with --cont-batching -np 4-8 is the natural pattern. Typical models: Llama 3 8B Q4_K_M with ~8 parallel slots, aggregate throughput on the order of hundreds of tok/s. vLLM also works if the weights fit (Llama 3 8B BF16 does; the 70B does not fit whole), although PagedAttention on consumer hardware gives less return than in the datacenter.

On a generic 4×H100 SXM cluster (320 GB, NVLink). Here vLLM v1 / SGLang are the de facto standard. Typical configurations:

Llama 3 70B FP8 + TP=4: dozens of concurrent sessions with P95 TPOT under 50 ms, tens of thousands of aggregate tok/s at a moderate batch size.
Llama 3 70B AWQ-INT4 + TP=2 plus the rest of the cluster for additional concurrency or multi-LoRA with SGMV.
DeepSeek-V3 needs larger setups (8-16 H100) to fit whole in FP8; with Wide-EP, continuous batching starts operating on much larger batches and the economics change (see MoE).

The rule of thumb for May 2026: vLLM v1 with chunked prefill always-on and prefix caching enabled is the sensible default configuration for any dense model that fits comfortably; SGLang offers better stable latency at high concurrency thanks to the overlap between the CPU scheduler and the GPU step; TensorRT-LLM gives peak throughput at high concurrency with the rigidity of compile-time.

What we have not covered

Locality-aware fair scheduling (arXiv:2501.14312) and learning-to-rank schedulers (NeurIPS 2024): the next generation of algorithms closing the fairness vs prefix-cache locality trade-off.
Smooth goodput (arXiv:2410.14257): a refinement of the DistServe metric using max slowdown instead of a binary SLO.
Triton tensorrtllm_backend in production: decoupled mode for streaming, ensembles with pre/post-processing, autoscaling with KServe.
vLLM speculators v0.3.0 and the vLLM-compatible drafter training framework.

References

Yu, G.-I., Jeong, J., Kim, G.-W., Kim, S., Chun, B.-G. Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022. https://www.usenix.org/system/files/osdi22-yu.pdf
Kwon, W. et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. https://arxiv.org/abs/2309.06180
Agrawal, A. et al. SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. 2023. https://arxiv.org/abs/2308.16369
Agrawal, A. et al. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. OSDI 2024. https://arxiv.org/abs/2403.02310
Zhong, Y. et al. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024. https://arxiv.org/abs/2401.09670
Sheng, Y. et al. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. MLSys 2024. https://arxiv.org/abs/2311.03285
Efficient LLM Scheduling by Learning to Rank. NeurIPS 2024.
Locality-aware Fair Scheduling. 2025. https://arxiv.org/abs/2501.14312
vLLM V1 alpha release (Jan 2025): https://blog.vllm.ai/2025/01/27/v1-alpha-release.html
vLLM Anatomy (Sep 2025): https://blog.vllm.ai/2025/09/05/anatomy-of-vllm.html
vLLM Large-Scale Serving (Dec 2025): https://blog.vllm.ai/2025/12/17/large-scale-serving.html
vLLM speculators v0.3.0 (Dec 2025): https://blog.vllm.ai/2025/12/13/speculators-v030.html
Anyscale, Continuous Batching for LLM Inference (2023): https://www.anyscale.com/blog/continuous-batching-llm-inference
vLLM metrics docs: https://docs.vllm.ai/en/latest/design/metrics/
vLLM optimization: https://docs.vllm.ai/en/stable/configuration/optimization/
SGLang: https://github.com/sgl-project/sglang
TensorRT-LLM performance tuning: https://nvidia.github.io/TensorRT-LLM/performance/performance-tuning-guide/tuning-max-batch-size-and-max-num-tokens.html
Red Hat, 5 steps to triage vLLM performance (Mar 2026): https://developers.redhat.com/articles/2026/03/09/5-steps-triage-vllm-performance
NVIDIA blog, Chunked Prefill with TensorRT-LLM: https://developer.nvidia.com/blog/streamlining-ai-inference-performance-and-deployment-with-nvidia-tensorrt-llm-chunked-prefill/

Multi-LoRA serving: the single translator with a thousand glossaries — shared base, thousands of concurrent adapters and the SGMV kernel

Sat, 30 May 2026 14:00:00 +0200

This post complements Continuous fine-tuning in production. Continuous fine-tuning is the producer of the adapters; multi-LoRA serving is the consumer that puts them to work. Without this layer, the whole feedback cycle breaks in the last mile. It also crosses over with Modern alignment (DPO/KTO/ORPO/SimPO) (each alignment policy can live as a separate adapter) and Quantisation (a quantised base frees memory for many more adapters).

You are here: DEPLOY

TL;DR

The dominant pattern in 2026 is not one model per customer but a single general-purpose base model plus N fine LoRA adapters per task, customer, language or domain. The reason is obvious: a rank-16 LoRA over Llama-3-70B takes about 400 MB; a full fine-tune takes about 140 GB. Dozens or hundreds of adapters per base is manageable; dozens or hundreds of bases is prohibitive. What is not obvious is how to serve them concurrently without reloading weights every time the adapter changes (killing batching) or replicating the base (killing memory). The answer crystallised in 2024 with two complementary papers: S-LoRA (Sheng et al., Stanford + UC Berkeley, MLSys 2024) introduced unified paging, where adapter weights live in the same memory pool as the KV cache, both pageable, and heterogeneous batching, where a batch can hold requests with different adapters and different ranks without padding; Punica (Chen et al., UW + Duke, MLSys 2024) introduced the CUDA kernel that has become the de facto standard: SGMV (Segmented Gather Matrix-Vector multiplication), which computes Y += Σ_i X_i · A_i · B_i in a single pass, grouping requests by adapter. SGMV sits today underneath vLLM, LoRAX (Predibase), SGLang and TGI. The measurable operational result: up to 2,000 concurrent adapters on a single GPU (S-LoRA paper), up to 4× throughput vs naive vLLM and up to 30× vs HuggingFace PEFT. The price: typical overhead of 10-30 % latency per layer with an active adapter in a heterogeneous batch, practically zero when every request in the batch uses the same adapter, and 20-40 % in the worst case. This post takes apart the mechanism, the maths (memory per adapter, overhead per rank), the comparison table of implementations, the pitfalls (cold start, mismatched rank, fragmentation) and the real economics on an H100 with a Llama-3-70B FP8 base + 200 adapters.

The analogy: the single translator with a thousand glossaries

Imagine a specialised translation agency with a single senior translator, brilliant, fluent in fifteen languages and across all the general technical domains. That translator is expensive to hire and expensive to train: it took years of formation and experience that is not easily replicated. But the agency receives texts from very different clients: a law firm using legal terminology specific to its jurisdiction, a manufacturer with an internal parts nomenclature, a hospital with its own clinical abbreviations. Every client has its own jargon.

The agency does not hire one translator per client; that would be ridiculous, 90 % of the work is shared. What it does is keep one glossary per client: a small notebook, easy to update, containing the specific terms and how they are translated for that client. When the translator receives a text, they open the glossary for the client in question and work with it alongside. As they translate each word, they first check whether it is in the glossary; if it is, they use the specific version; if not, they use their general knowledge.

The glossaries live on a shared shelf, ordered by recent use: the most consulted ones within reach, the old ones in the archive. When a new client arrives, their glossary is brought from the archive to the shelf. When the desk fills up, the least used glossary goes back to the archive.

And the most important part: the translator can have several glossaries open at once because they are working in parallel on five texts from five clients. It is not one glossary per document; it is one glossary per client, and client A’s documents use A’s glossary, client B’s use B’s, all on the same desk.

The analogy holds on five mappings:

The single translator = the base model (Llama-3-70B, Qwen2.5-72B). Expensive to train, a single copy in VRAM.
Each glossary = a LoRA adapter. Small (~150-400 MB), specific, easy to update.
The shelf with the glossaries within reach = the pool of adapters cached in VRAM (typically 50-200 at a time with an FP8 base on an H100 80 GB).
The archive = adapter storage on MinIO/S3/HF Hub. Hundreds or thousands, fetched on demand.
Working in parallel with several glossaries open = a heterogeneous batch with SGMV. The kernel does the grouped lookup into the right glossary for each word in the batch.

The bare mechanism: what a LoRA does and why it can be served multi-tenant

A LoRA adapter modifies a matrix W of the base model by adding a low-rank product to it:

$$W' = W + B A$$

where W ∈ R^{d_out × d_in} is the original matrix (the base weights), A ∈ R^{r × d_in} and B ∈ R^{d_out × r} are the adapter’s trainable matrices, and r is the rank (typically 8, 16, 32 or 64, always much smaller than d_in and d_out).

In a forward pass, instead of computing y = W' x, it computes:

$$y = W x + B(Ax)$$

That is: the base computation (Wx) happens exactly as before; the adapter adds two cheap matmuls (Ax and then B(·)) that supply the correction. The matrix BA is never explicitly materialised.

What this enables in serving: if you have the base loaded and N different adapters, Wx is computed only once for all the tokens in the batch (the base is the same). What changes between tokens is only the delta B_i(A_i x). If the tokens in the batch use different adapters, different deltas have to be applied per token, and that is what the SGMV kernel does in a single pass.

Without a specialised kernel this degenerates: you need to launch N separate matmuls (one per adapter), pay kernel launch overhead N times and lose the batching. With a specialised kernel (SGMV), every delta is computed in a single pass grouped by adapter.

SGMV: the kernel that holds it all up

SGMV (Segmented Gather Matrix-Vector multiplication) is the CUDA kernel Punica introduced and that vLLM, LoRAX, SGLang and TGI have adopted as their multi-LoRA engine.

Its job is to compute, given a batch of tokens with mixed adapters:

$$y_t = W x_t + B_{a(t)} A_{a(t)} x_t \quad \forall t \in \text{batch}$$

where a(t) is the adapter assigned to token t. SGMV operates in two phases:

SGMV-shrink: projection d_in → r with the corresponding A_{a(t)} matrix.
SGMV-expand: projection r → d_out with the corresponding B_{a(t)} matrix.

Internally, SGMV groups the batch’s tokens by adapter (it segments them), and for each segment it uses the optimal kernel for the size: for large segments (several requests with the same adapter) it goes through tensor cores; for small segments (one request per adapter) it uses the batched gather path that minimises launch overhead.

The result, in a single kernel pass, is the correct delta for every token in the batch, whatever its rank or its adapter. Punica reported up to 12× throughput vs vLLM/FasterTransformer/HF Transformers/DeepSpeed in heterogeneous multi-tenant scenarios; when every request uses the same adapter, SGMV is practically equivalent to the base without LoRA because it reduces to the “single large segment” case that is optimal for tensor cores.

S-LoRA refined SGMV with two kernels specific to different serving phases: MBGMM (Multi-size Batched Gather Matrix-Matrix) for prefill, MBGMV for decode. Both support different ranks among requests in the same batch, which was a limitation in the original SGMV.

The maths that matters

Three numbers drive every operational decision with multi-LoRA.

Memory per adapter. For a d_in × d_out matrix with rank r and b bytes per parameter (BF16/FP16 = 2):

$$\text{bytes per matrix} = (d_{\text{in}} \cdot r + r \cdot d_{\text{out}}) \cdot b$$

Summing over all the target matrices in each layer and multiplying by the number of layers gives the adapter size. A concrete calculation for Llama-3-70B, rank 16, BF16, all the matrices (Q, K, V, O, gate, up, down):

Matrix	Dimension	Bytes
Q (8192→8192)	8192·16 + 16·8192	524,288
O (8192→8192)	8192·16 + 16·8192	524,288
K (8192→1024)	8192·16 + 16·1024	294,912
V (8192→1024)	8192·16 + 16·1024	294,912
gate (8192→28,672)	8192·16 + 16·28,672	1,179,648
up (8192→28,672)	8192·16 + 16·28,672	1,179,648
down (28,672→8192)	28,672·16 + 16·8192	1,179,648
Sum per layer		5,177,344 ≈ 4.94 MB
Adapter total (80 layers)		~395 MB

If it is limited to attention-only (Q, O, V, K): about 125 MB per adapter. The choice of which matrices get LoRA belongs to whoever trains it; in serving it is inherited and it determines the cost.

Memory per rank. Linear: rank 8 → ~200 MB; rank 16 → ~400 MB; rank 32 → ~800 MB; rank 64 → ~1.6 GB. The simple sizing rule is: max_lora_rank should be the maximum rank you are going to serve, no more. Setting it higher wastes reserved memory in every slot.

How many adapters fit. For an H100 SXM 80 GB with a Llama-3-70B FP8 base (~70 GB), about 10 GB are left after a minimal KV cache → around 25 fully targeted r=16 adapters or around 80 attention-only. With an AWQ INT4 base (~35 GB), about 45 GB are left → hundreds of adapters. The rule: quantising the base does not just free memory, it multiplies the economics of the platform.

Latency overhead per adapter. Under real reported conditions:

Case	Typical overhead
All requests in the batch on the same adapter	~0 % (equivalent to a static merge)
Heterogeneous batch, similar ranks (e.g. all r=16)	10-30 % per layer
Heterogeneous batch, mismatched ranks (r=8 with r=128)	up to +84 % P95 TTFT for the lower rank
Naive PEFT LoRA (no SGMV)	250-950 % extra

It scales linearly with rank: rank 8 ≈ baseline; rank 64 ≈ 3-4 × overhead.

The real implementations in May 2026

Implementation	Base kernel	Hot-swap	Quantised base + LoRA	Operational notes
vLLM	SGMV + ext	Yes (LoRAResolver, S3/HF/FS plugins)	AWQ/GPTQ yes, bnb 4-bit offline only	The de facto default. `--enable-lora --max-loras N --max-lora-rank R --max-cpu-loras M`
LoRAX (Predibase)	Optimised SGMV	Yes (dynamic loading)	Yes	Designed specifically for multi-LoRA. Supports Medusa adapters per adapter (spec-dec per adapter).
SGLang	SGMV / csgmv	Yes	Yes	`--enable-lora-overlap-loading` cuts TTFT by up to 78 % on LoRA-heavy workloads
TensorRT-LLM	LoRA Executor (C++)	Pre-compile at build time	INT4 + LoRA common	Peak throughput on H100/B200, less flexible than vLLM
HF TGI	Punica fork	Yes (`LORA_ADAPTERS=...`)	Yes	In maintenance mode as of May 2026; HF recommends vLLM or SGLang
NVIDIA NIM	TRT-LLM under the hood	Static or dynamic (`NIM_PEFT_REFRESH_INTERVAL`)	Yes	Adapter store per model; polling for hot-add/remove

Three operational observations:

vLLM dominates open-source serving in 2026 thanks to the combination of a mature SGMV + LoRAResolver plugins + support for a quantised base. The critical parameter is --max-lora-rank: many installations set it to the maximum “just in case” and waste memory silently.
LoRAX wins in production operations with thousands of rarely used adapters thanks to its dynamic loading that does not block concurrent requests. A public case: Convirza with 60+ concurrent adapters and sub-2s P95.
SGLang wins on latency when cold starts are frequent thanks to --enable-lora-overlap-loading (async H2D during the compute of the previous request).

Combined pattern with quantisation and disaggregated serving

With quantisation (Quantisation). The canonical stack in May 2026 is base in FP8 (Hopper/Blackwell) or INT4 AWQ + adapters in BF16/FP16. The adapters are not quantised: they are small, the memory saved is irrelevant, and quantisation noise would accumulate badly with the delta. Quantising the base frees massive amounts of memory for more adapters with no significant loss (<1 % in typical MMLU with AWQ INT4, somewhat more in math/code/reasoning). An adapter trained with a BF16 base works with an FP8/INT4 base at inference with marginal loss, and that is what makes QLoRA operationally trivial: train with a 4-bit base and deploy with a consistent 4-bit base.

With disaggregated serving (Disaggregated serving). Multi-LoRA + prefill/decode disaggregation adds a management layer: each pod needs access to the request’s active adapter. The 2026 strategy: replicate hot adapters across every pod (prefill and decode), evict the cold ones. Cold adapters not present in the target pod are transferred on demand, accepting the extra cost in TTFT. Recent work (InfiniLoRA, FASTLIBRA, LoRAServe) automates this balancing, but the simple rule of thumb works in most deployments.

Operational pitfalls

Cold start. The first request to a sleeping adapter involves a fetch (from S3/MinIO/HF Hub) → CPU load → H2D copy. For an adapter of around 400 MB: 0.5-5 s typically, depending on bandwidth. Under high concurrency it can be up to 35 % of E2E latency (Predictive-LoRA paper, arXiv:2512.20210). Established mitigations: SGLang --enable-lora-overlap-loading (cuts TTFT by 35-78 %); vLLM pre-warming with a dummy request when an adapter is registered; predictive prefetching based on patterns (Predictive-LoRA cuts cold start by 68 %).

Heterogeneous worst case. If every request in the batch has a different adapter and a different rank, SGMV loses its advantage because each segment has size 1. Throughput can fall by up to 50 % vs the base without LoRA. Practical mitigation: group adapters by rank in the upstream routing, trying to get requests of the same rank into the same batch step.

Mismatched rank. Co-batching rank 8 with rank 128 penalises the smaller one: +84 % P95 TTFT for the rank-8 requests (Serving Heterogeneous LoRA Adapters paper, arXiv:2511.22880). Practice: normalise the rank across the fleet whenever possible (train every adapter at the same rank, or at least within a narrow range).

Eviction. LRU is the default. If you have more adapters than fit in CPU RAM, the evicted ones get fetched again from cold storage. Monitoring cold_starts_per_minute and cache_hit_ratio per endpoint is basic hygiene.

Base versioning. Every adapter is tied to an exact base version (Llama-3-70B-Instruct ≠ Llama-3.1-70B-Instruct). The routing must validate the base+adapter pair before serving, or the output will be silent rubbish.

Memory fragmentation. Without paged management (the pre-S-LoRA / pre-vLLM case), evicting and inserting adapters fragments HBM until it becomes unusable. Unified Paging solves it: the LoRA weights and the KV cache live in the same pool of blocks, interchangeable.

Implications on on-premise hardware

On an RTX 4090 (24 GB). The classic case: base Llama-3-8B FP8 (~8 GB) or Llama-3-8B AWQ-INT4 (~5 GB) + dozens of r=16 adapters (~80 MB each, around 3 MB per adapter for Llama-3-8B with r=16 attention-only). 50-100 active adapters fit easily on the 4090. It is the natural setup for multi-tenant demos, fine-tuning over an 8B base and platform prototypes.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). This is where the serious setup comes in:

Llama-3-70B FP8 (~70 GB) fits on 2 H100s; the other 2 GPUs are available for batch + adapters.
Llama-3-70B AWQ-INT4 (~35 GB) fits on 1 H100; the rest of the cluster serves more concurrency or more adapters.
~200 fully targeted r=16 adapters fit with a comfortable budget on the cluster, enough for a SaaS platform with dozens of tenants and simultaneous A/B testing.
QLoRA training + consistent serving: the train→deploy adapter cycle takes hours, not days, because the adapter is around 400 MB instead of around 140 GB.

The rule of thumb on an H100 cluster in May 2026: an FP8 or INT4 base + 100-500 adapters per cluster is operationally trivial with vLLM or LoRAX; going past a thousand concurrent adapters starts to require serious tuning of eviction and prefetch.

Typical production stack

[API Gateway]
↓ (JWT with tenant_id / API key)
[Router]
↓ injects adapter_id into the request
[vLLM / LoRAX with --enable-lora]
--max-loras 16
--max-lora-rank 32
--max-cpu-loras 200
+ LoRAResolver → s3://adapters/{tenant_id}/{version}/
+ Base: Llama-3-70B-FP8 loaded once on 4×H100 TP=4
↓
[GPU]
FP8 base (70 GB) + ~150 BF16 adapters hot in HBM + ~1,000 warm in RAM
[MinIO / S3]
Cold storage for thousands of adapters
[CI pipeline]
→ trains new QLoRA adapter → push MinIO → notify server → warm-up
[Observability]
Prom: active_adapters, cache_hit_ratio, cold_starts_per_minute,
per_adapter_throughput, P99_with_lora vs P99_base

Routing pattern: Authorization: Bearer <key> → middleware extracts tenant_id → maps to adapter_id (Postgres or Redis) → POST /v1/completions with model: "<adapter_id>".

What we have not covered

DoRA (Weight-Decomposed LoRA): decomposes the update into magnitude + direction, closing part of the quality gap with full fine-tuning. Supported by TensorRT-LLM and others, but the serving pattern is identical to LoRA.
MoE + LoRA: how adapter fine-tuning is done over an MoE, what happens to the routing. Not trivial, an active research area in 2026.
Activated LoRA (arXiv:2512.17910): a variant that reuses the KV cache between compatible adapters, cutting the cost of cold start with shared prefixes.
LoRA for speculative decoding: each adapter brings its own Medusa head, supported by LoRAX as “Turbo LoRA”.
Compress-then-Serve (arXiv:2407.00066): quantising the adapters themselves to serve even more concurrently. Still a marginal practice in production as of May 2026.

References

Hu, E. et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. https://arxiv.org/abs/2106.09685
Sheng, Y. et al. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. MLSys 2024. https://arxiv.org/abs/2311.03285
Chen, L. et al. Punica: Multi-Tenant LoRA Serving. MLSys 2024. https://arxiv.org/abs/2310.18547
Dettmers, T. et al. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. https://arxiv.org/abs/2305.14314
Brüel-Gabrielsson, R. et al. Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead. 2024. https://arxiv.org/abs/2407.00066
Serving Heterogeneous LoRA Adapters in Distributed LLM Inference. 2025. https://arxiv.org/abs/2511.22880
InfiniLoRA: Disaggregated Multi-LoRA Serving. 2026. https://arxiv.org/abs/2604.07173
Predictive-LoRA: Proactive Fragmentation-Aware Serverless LoRA Serving. https://arxiv.org/abs/2512.20210
S-LoRA repo: https://github.com/S-LoRA/S-LoRA
Punica repo: https://github.com/punica-ai/punica
LoRAX repo (Predibase): https://github.com/predibase/lorax
vLLM LoRA docs: https://docs.vllm.ai/en/stable/features/lora/
SGLang LoRA docs: https://docs.sglang.io/advanced_features/lora.html
TensorRT-LLM LoRA docs: https://nvidia.github.io/TensorRT-LLM/advanced/lora.html
NVIDIA NIM PEFT: https://docs.nvidia.com/nim/large-language-models/latest/peft.html
HF TGI Multi-LoRA blog: https://huggingface.co/blog/multi-lora-serving
LMSYS S-LoRA recipe blog: https://www.lmsys.org/blog/2023-11-15-slora/
Predibase LoRAX blog: https://predibase.com/blog/lorax-the-open-source-framework-for-serving-100s-of-fine-tuned-llms-in

MoE inference: the call centre with 256 specialists and 8 taking each call — fundamentals, expert parallel and the economics of DeepSeek-V3

Sat, 30 May 2026 08:00:00 +0200

This post goes into an architectural decision in the Deploy stage of the six-stage LLMOps pipeline. It complements those on KV cache (which does not change with MoE), FlashAttention (still the attention kernel, also dense in every current MoE), Quantization (FP8/NVFP4 over expert weights is what lets DeepSeek-V3 fit into a modest cluster) and Speculative decoding (MTP in DeepSeek-V3 is native speculative decoding).

You are here: DEPLOY

TL;DR

A Mixture of Experts (MoE) replaces the transformer’s dense FFN layer with N parallel experts plus a router that picks the k experts relevant to each token. The operational consequence is that the model’s total parameters (how many were trained and how many must be loaded into VRAM) decouple from the active parameters per token (how many take part in each forward). DeepSeek-V3 has 671 B total but only 37 B active per token; Qwen3-235B-A22B is 235 B total and 22 B active; Mixtral 8x22B is 141 B and 39 B; Llama 4 Maverick is 400 B and 17 B. Compute per token falls to 2·N_active FLOPs, half that of a dense Llama-3-70B in DeepSeek-V3’s case, but memory stays proportional to N_total: MoE sparsity is compute sparsity, not memory sparsity. Since each expert sees only batch·k/N tokens per step, the memory-bound regime persists at a much higher batch than in dense. Serving DeepSeek-V3 671 B FP8 at low latency requires expert parallel (EP) at scale, EP=32 in prefill and EP=144 in decode in DeepSeek’s production deployment, where the dominant bottleneck stops being HBM and becomes the all-to-all of dispatch and combine between GPUs. As of May 2026, the pieces that make this viable are DeepEP (FP8-native all-to-all CUDA kernels, intra-node NVLink plus inter-node RDMA), EPLB (balancing by replicating hot experts) and Wide-EP on NVL72 (72 GPUs in one coherent NVLink domain). This post takes apart the mechanism, the maths (FLOPs/token, all-to-all bytes), the updated table of MoE models as of May 2026, the parallelisms (TP / EP / DP+EP / Wide-EP), the operational pitfalls (memory wall, cold experts, imbalance) and the real reproducible numbers on H100, B200 and GB200 NVL72.

The analogy: the call centre with 256 specialists

A serious call centre handling complex technical queries runs on specialists, not generalists. You have 256 operators on the payroll, each an expert in their sub-domain: employment law, mortgages, income tax, probate, and so on. But each incoming call only needs the attention of eight specific specialists (top-8 routing): a client’s query about tax relief on a mortgage for an inherited second home involves the income tax person, the mortgage person, the capital gains person, and so on. The receptionist (the router) listens to the first second of the call, decides which eight specialists are relevant, conferences them in and combines their answers with weights.

Three immediate consequences, identical to those of MoE in production:

You pay 256 salaries (every expert is loaded in VRAM) but you only spend the time of 8 (only k experts take part in the forward of each token). Your payroll cost scales with N_total; your cost per minute of call scales with N_active.
If all the operators fit in one office (a single GPU), fine: paper just moves between desks. But if they do not fit, you have to open branches: the conference call between operators in different branches goes over a telephone line between buildings (intra-node NVLink, inter-node InfiniBand) and that line becomes the dominant bottleneck once dozens of branches are coordinating.
If one week every client asks about income tax, the three income tax operators saturate while the probate ones sit idle. You need a system that replicates the hot operators across several branches to absorb peaks, and that pairs hot with cold so that no branch goes idle.

The receptionist is the router. The branches are the GPUs. The conference call between branches is the all-to-all (dispatch plus combine). The system for replicating hot operators is EPLB. The good telephone line is DeepEP. And when someone designs a building where all 72 branches fit together with very fast internal cabling, that is NVL72.

The bare mechanism

A classic MoE layer replaces the dense FFN (y = FFN(x) = down(act(up(x) · gate(x))) in SwiGLU) with:

Router: a linear projection g(x) = W_router · x produces N scores. A softmax over those scores gives affinity probabilities.
Top-k selection: the k experts with the highest score are chosen. Their softmax_topk weights are normalised to sum to 1.
Parallel compute: each of the k selected experts runs its own FFN over x. The N - k not selected sit idle for this token.
Weighted combination: the output is y = Σ_{i ∈ topk} w_i · Expert_i(x).

Variants the market consolidated in 2025-2026:

Shared experts: on top of the k routed ones, there are s experts that always run (1 or 2). They capture general knowledge every token needs. DeepSeek-V3 uses 1 shared + 8 routed; Mixtral uses 0 shared.
Fine-grained experts: more small experts instead of a few large ones (DeepSeek: 256 vs Mixtral: 8). This allows better specialisation because the top-k combination covers more sub-domains.
Auxiliary-loss-free routing (DeepSeek-V3): instead of adding a balancing loss during training, it dynamically adjusts a bias in the routing. It keeps balance without polluting the main loss.

Attention is still dense in every current MoE model (Mixtral, DeepSeek, Qwen3-MoE, Llama 4, Kimi K2). Only the FFN is replaced. That is why FlashAttention v3/v4 is still used as-is; DeepSeek’s MLA is an optimisation orthogonal to MoE.

The relevant MoE models in May 2026

Model	Date	Total	Active	Experts	Top-k	Shared	Notes
Mixtral 8x7B	Dec 2023	47 B	13 B	8	2	0	The one that normalised MoE in open weights
Mixtral 8x22B	Apr 2024	141 B	39 B	8	2	0	—
Grok-1 (open)	Mar 2024	314 B	~86 B	8	2	0	Weights opened by xAI
DeepSeekMoE 16B	Jan 2024	16 B	2.8 B	64+2	6	2	Introduces fine-grained + shared
DeepSeek-V2	May 2024	236 B	21 B	160+2	6	2	+ MLA
Snowflake Arctic	Apr 2024	480 B	17 B	128	2	10 B dense residual	Dense-MoE hybrid
Hunyuan-Large	Nov 2024	389 B	52 B	16+1	1+shared	1	Cross-layer KV cache
DeepSeek-V3	Dec 2024	671 B	37 B	256+1	8	1	MLA + MTP, aux-loss-free, FP8 training
Llama 4 Scout	Apr 2025	109 B	17 B	16	1+shared	1	Designed for 1 H100 node
Llama 4 Maverick	Apr 2025	400 B	17 B	128	1+shared	1	—
Qwen3-235B-A22B	Apr 2025	235 B	22 B	128	8	0	—
Kimi K2 (Moonshot)	2025	1 T	32 B	384	8	(MLA)	Trained with MuonClip
DeepSeek-V3.2-Exp	Sep 2025	671 B	37 B	256+1	8	1	+ DeepSeek Sparse Attention

Three operational observations:

The active/total ratio falls with every generation: Mixtral 8x7B was 28 %, DeepSeek-V3 is 5.5 %, Llama 4 Maverick is 4.3 %. What the industry discovered is that sparsity can be very aggressive without losing quality, provided N is large enough and the routing is well learned. Fine specialisation is worth more than per-expert capacity.
The flagships of the large providers (OpenAI, Anthropic, Google) are almost certainly MoE, but the architectures are not published. The only semi-credible leak (Hotz, Jul 2023) claimed GPT-4 was an 8×~220 B MoE; no official confirmation.
DeepSeek-V3 marks the inflection point: open weights, frontier-class, MoE with aggressive sparsity and compatible with FP8 quantisation. It is the model that forced the ecosystem (vLLM, SGLang, TensorRT-LLM) to optimise wide-EP in 2025.

The maths that matters

Three numbers drive the whole architectural decision with MoE.

Compute per token. A dense transformer spends roughly 2N FLOPs per token in the forward pass (with N = total parameters). A MoE spends 2·N_active FLOPs per token (the N_total - N_active experts do not take part). For DeepSeek-V3: 2 × 37 B = 74 GFLOPs/token. For dense Llama-3-70B: 2 × 70 B = 140 GFLOPs/token. DeepSeek does half the compute per token of Llama-3-70B while offering capacity equivalent to a model almost 10 × larger.

Total memory. But all the experts must be loaded in VRAM at some point in the forward pass. The sparsity is in compute, not memory. Weight memory:

$$\text{Memory}_{\text{weights}} \approx N_{\text{total}} \cdot \text{bytes per param}$$

For DeepSeek-V3 FP8 (1 byte): 685 GB. It does not fit in 8×H100 SXM (640 GB). You need either extra quantisation or 16+ GPUs. For Mixtral 8x22B FP8: 141 GB, comfortably inside 2×H100. For Qwen3-235B-A22B FP8: 235 GB, fits in 4×H100.

All-to-all bytes per MoE layer. When experts are spread across GPUs (EP), each token has to travel to the GPUs where its k experts live. The volume per MoE layer is roughly:

$$\text{bytes a2a/layer} \approx 2 \cdot \text{batch} \cdot \text{seq len} \cdot d_{\text{model}} \cdot k \cdot \text{bytes}$$

(the factor 2 is dispatch plus combine). For DeepSeek-V3 with d_model = 7168, k = 8, batch = 4096, FP8 (1 byte): each MoE layer moves ~470 MB; multiplied by 61 MoE layers in DeepSeek-V3, the complete step adds ~29 GB of inter-GPU traffic in MoE comms alone. That is of the order of a GPU’s HBM bandwidth in a single step. This is why bandwidth and NVLink/InfiniBand topology determine how large EP can be before comms saturate.

Persistent memory-bound regime. Here is the least intuitive consequence. In dense, raising the batch moves the operation from memory-bound to compute-bound: with enough tokens per batch, the weights are amortised over more operations. In MoE, each expert sees only batch · k / N tokens. For Qwen3-235B (k=8, N=128) with batch=128, each expert processes 8 tokens per step. To reach the compute-bound regime of the tensor cores (arithmetic intensity ~100-200 FLOPs/byte) you need batches in the thousands, around 1,600 according to recent estimates for Qwen3 (Memory-Bound MoE Serving, arXiv:2512.09277). The operational consequence: MoE scales throughput with batch more linearly than dense, but it also needs far larger batches to get close to its compute ceiling.

Expert parallel and the all-to-all bottleneck

There are three main ways to parallelise a MoE across GPUs:

Tensor Parallel (TP, Megatron-style). Splits each weight matrix across GPUs by width or height. Every GPU processes all tokens and all layers. Comms per layer: 2 all-reduces. It works well for dense; for MoE it leaves the routing sparsity unexploited (each GPU holds fragments of every expert).

Expert Parallel (EP). Splits whole experts across GPUs: 256 experts / 32 GPUs = 8 experts per GPU. Each token, after routing, travels to the GPUs where its top-k k experts live. Comms per MoE layer: an all-to-all dispatch (tokens towards experts) plus an all-to-all combine (outputs back). The cost grows with EP size and becomes the bottleneck when EP > 8.

DP + EP / Wide-EP. Replicates the model across DP groups; inside each group, EP shards the experts. Wide-EP takes EP to the full dimension of a rack (NVL72, 72 GPUs in a coherent NVLink domain). Each expert receives more tokens per step, so arithmetic intensity in each expert’s GEMMs improves and per-GPU throughput rises. vLLM reports 1.8× throughput per GPU with wide-EP against smaller setups on DeepSeek-V3 (vLLM blog, Dec 2025).

Quick decision table:

Case	Recommendation
Small MoE that fits in 1 node (Mixtral 8x7B, Llama 4 Scout)	TP only or EP=2 intra-node
Medium MoE (Mixtral 8x22B, Qwen3-235B) on 1-2 H100 nodes	EP=8 intra-node, optionally TP=2
Large MoE (DeepSeek-V3) on a multi-node cluster	TP × EP cross-node (e.g. TP=4 × EP=8 on 32 GPUs)
Large MoE with NVL72 available	Wide-EP=72 (decode), EP=32 (prefill)

DeepSeek’s production deployment combines Prefill EP=32 (4 nodes × 8 GPUs) with Decode EP=144 (18 nodes × 8 GPUs). The prefill/decode split fits conceptually with Disaggregated serving: prefill is compute-bound and benefits from moderate EP; decode is memory-bound and benefits from Wide-EP, which raises the tokens per expert per step.

DeepEP, EPLB and Wide-EP: the pieces that unblocked 2025

DeepEP (DeepSeek, opened in February 2025) is a library of all-to-all CUDA kernels optimised specifically for MoE EP. Four important properties:

Native FP8 dispatch: tokens travel in FP8, halving the bandwidth consumed.
Mixing intra-node NVLink with inter-node RDMA: the kernel picks the optimal route according to topology.
CPU bypass: the GPU→GPU dispatch goes over RDMA without touching the host.
Aligned with DeepSeek-V3’s group-limited gating: the routing prefers experts in the same node group where possible, minimising inter-node traffic.

In 2025 Tencent contributed optimisations that added +30 % on the normal kernels. There is a ROCm port for AMD.

EPLB (Expert Parallelism Load Balancer) solves the runtime imbalance problem: even when the model is well balanced in global distribution, a specific batch can heavily activate certain experts (prompts in a single language, a single domain). EPLB replicates the hot experts across several GPUs (redundant experts) and packs them heuristically with cold experts to minimise load variance. It has two modes: hierarchical (for prefill with medium EP) and global (for decode with Wide-EP). It is integrated into SGLang and vLLM.

Wide-EP on NVL72 is the peak setup of 2026: 72 Blackwell GPUs in a coherent NVLink domain (1.8 TB/s bidirectional per GPU, intra-rack). Each expert receives many more tokens per step, so arithmetic intensity improves and so does throughput per GPU. Combined with NVFP4 over expert weights and FP8 over attention, SGLang reported on GB200 NVL72 26,156 tok/s/GPU in prefill and 13,386 tok/s/GPU in decode for DeepSeek-V3 (LMSYS, Sep 2025), a 3.8× prefill and 4.8× decode versus H100.

Operational pitfalls

HBM fragmentation from expert weights. With EP=8 in 1 node, each GPU holds N/8 experts, each a tuple of matrices (gate, up, down). If N=256, that is 32 FFNs per GPU. The allocator can fragment; in some cases you need a more conservative --gpu-memory-utilization than usual.

Cold experts. In real distributions some experts are activated by <1 % of tokens. EPLB compensates by replicating hot ones, but the cold ones sit occupying HBM without taking part, “dead” memory. When real demand diverges a lot from the training distribution, this shows.

Continuous batching. It works with MoE, but the effective batch size per expert is batch_total · k / N. With large N and a moderate batch, each expert sees very few tokens per step. You need a batch_total far larger than in dense to match per-GPU throughput.

Speculative decoding + MoE. They interact well. MoE sparsity keeps the memory-bound regime at high batch, which is exactly the regime where speculative decoding wins most. DeepSeek-V3 integrates MTP (Multi-Token Prediction), speculative decoding native to the model, with no external draft, with acceptance around 85-90 % on the second token (see Speculative decoding for the mechanism).

MLA and FlashMLA are not MoE. DeepSeek-V3 combines MoE with Multi-head Latent Attention (MLA), an attention variant that reduces the KV cache ~10× against standard MHA. MLA and MoE are orthogonal: MLA optimises attention, MoE optimises the FFN. For serving DeepSeek you use the specific FlashMLA kernel (not FA3 directly).

DSA (DeepSeek Sparse Attention) in DeepSeek-V3.2-Exp (Sep 2025) introduces sparsity in attention too. The first time at the frontier that sparsity is applied in both attention and FFN.

Implications for on-premise hardware

On an RTX 4090 (24 GB). Large MoEs are out of reach. What does fit: Mixtral 8x7B AWQ-INT4 (~24 GB of weights, no headroom for KV cache, so it really requires TP=2 across two 4090s); DeepSeekMoE 16B BF16 (~33 GB, does not fit whole; INT4 ~8 GB, fits with room to spare). The interesting case is llama.cpp with --n-cpu-moe: expert weights offloaded to RAM (CPU), attention and shared experts on the GPU. It lets you run DeepSeek-V3 IQ4 (~400 GB) on a workstation with 1-2 GPUs plus 512 GB of RAM at modest throughput, but it opens the door to a low-budget setup.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). These fit comfortably:

Mixtral 8x22B FP8 (~141 GB) with TP=2 or EP=4. Low latency, excellent throughput.
Qwen3-235B-A22B FP8 (~235 GB) with TP=4 or EP=4. Plenty of room.
Llama 4 Scout 109B FP8 (~110 GB) with TP=2 or TP=4. Designed by Meta to serve well on one H100 node.

What does not fit in 4×H100: DeepSeek-V3 FP8 (~685 GB) without aggressive quantisation. Options: (1) scale to 8-16 H100; (2) use AWQ INT4 over routed experts plus FP16 over activated ones (~400 GB), which fits in 8×H100; (3) wait for Blackwell or use AMD MI300X (192 GB/GPU, so DeepSeek-V3 FP8 on 4×MI300X).

The rule of thumb as of May 2026: one 4×H100 node comfortably serves MoEs up to ~200 B total in FP8; for DeepSeek-V3 you have to scale to 2 nodes or wait for B200 NVL72 (where the whole model fits with room for enormous batches).

What we have not covered

MLA and FlashMLA in detail: DeepSeek’s KV cache optimisation, orthogonal to MoE.
DeepSeek Sparse Attention (DSA) introduced in V3.2-Exp: the first production implementation of attention sparsity for frontier models.
MoE during training: load balancing losses, drop policies, auxiliary terms vs aux-loss-free, Kimi K2’s MuonClip.
MoE + LoRA: how adapter fine-tuning is done on a MoE, and what happens to the routing.
LLM-d and other open-source platforms that materialise Wide-EP on Kubernetes.

References

Shazeer, N. et al. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR 2017. https://arxiv.org/abs/1701.06538
Lepikhin, D. et al. GShard: Scaling Giant Models with Conditional Computation. ICLR 2021. https://arxiv.org/abs/2006.16668
Fedus, W., Zoph, B., Shazeer, N. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. JMLR 2022. https://arxiv.org/abs/2101.03961
Gale, T., Narayanan, D., Young, C., Zaharia, M. MegaBlocks: Efficient Sparse Training with Mixture-of-Experts. MLSys 2023. https://arxiv.org/abs/2211.15841
Mistral AI. Mixtral of Experts. 2024. https://arxiv.org/abs/2401.04088
DeepSeek-AI. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. 2024. https://arxiv.org/abs/2401.06066
DeepSeek-AI. DeepSeek-V2. 2024. https://arxiv.org/abs/2405.04434
DeepSeek-AI. DeepSeek-V3 Technical Report. 2024. https://arxiv.org/abs/2412.19437
Tencent. Hunyuan-Large. 2024. https://arxiv.org/abs/2411.02265
DeepEP repo: https://github.com/deepseek-ai/DeepEP
EPLB repo: https://github.com/deepseek-ai/EPLB
DeepSeek open-infra-index, V3/R1 Inference System Overview: https://github.com/deepseek-ai/open-infra-index
vLLM blog, Large-Scale Serving DeepSeek (Dec 2025): https://blog.vllm.ai/2025/12/17/large-scale-serving.html
vLLM blog, WideEP en GB200 (Feb 2026): https://blog.vllm.ai/2026/02/03/dsr1-gb200-part1.html
LMSYS blog, DeepSeek 96 H100 PD+EP (May 2025): https://www.lmsys.org/blog/2025-05-05-large-scale-ep/
LMSYS blog, DeepSeek GB200 NVL72 part II (Sep 2025): https://www.lmsys.org/blog/2025-09-25-gb200-part-2/
NVIDIA Dynamo + GB200 NVL72 for MoE: https://developer.nvidia.com/blog/how-nvidia-gb200-nvl72-and-nvidia-dynamo-boost-inference-performance-for-moe-models/
Tensor Economics, MoE Inference Economics from First Principles: https://www.tensoreconomics.com/p/moe-inference-economics-from-first
Cohere, Why MoE models get more from speculative decoding: https://cohere.com/blog/mixture-of-experts-models-get-more-from-speculative-decoding

FlashAttention v1/v2/v3/v4: the librarian who never clears the desk — IO-awareness, async and Blackwell's asymmetry

Fri, 29 May 2026 17:00:00 +0200

This post complements KV cache: the working memory and PagedAttention deep dive. The KV cache post explains what is stored; PagedAttention, how it is managed in memory; FlashAttention, how the computation is executed. They are three different layers of the same problem and they stack multiplicatively.

You are here: DEPLOY

TL;DR

The standard attention of a transformer has a structural problem on modern GPUs: when you look at it with a profiler, it is not compute-bound, it is memory-bound. The S = QK^T matrix of size N × N, with N the sequence length, does not fit in the fast on-chip SRAM and forces round trips to HBM that dominate the total time. FlashAttention is the family of kernels that avoids materialising that matrix by tiling over Q, K and V, computing the softmax block by block with the online version and keeping everything inside SRAM. Each version raises the utilisation ceiling: FA1 (Dao et al. 2022) abolished the N×N matrix and brought memory complexity down to O(N); FA2 (Dao 2023) parallelised along the sequence dimension and cut non-matmul FLOPs to get close to 70 % utilisation on A100; FA3 (Shah, Bikshandi, Zhang, Thakkar, Ramani, Dao 2024) exploited three Hopper-specific mechanisms, async WGMMA, TMA and FP8, to reach 740 TFLOPS BF16 (75 % of peak) and 1.2 PFLOPS FP8 on H100; FA4 (March 2026) rewrote the kernel from scratch for Blackwell, where the tensor core scales 2.25× but the SFU (where the softmax exp runs) and the SMEM bandwidth do not scale at all. The answer is a software-emulated exponential that runs on tensor cores. The result on B200: 1,605 TFLOPS BF16, 1.3× faster than cuDNN 9.13 and 2× faster than FA3 on the same GPU. This post takes apart the why (memory roofline, IO complexity), the master analogy of the librarian, the minimum mathematics and the real numbers on H100 and B200.

The analogy: the librarian who never clears the desk

A large library with two zones: a very fast but small work desk (the librarian’s desk, 200 books fit on top of it), and a giant shelving system spanning three floors where everything lives (50 million books). The librarian has to cross-reference all the books in one thematic room and produce a summary.

A naive librarian does it the direct way: fetches every relevant book from the shelves, piles them on the work desk, and since they do not fit, leaves half of them on the floor, on the chair, on top of boxes. The day is spent running between the floor and the desk, opening and closing books, never able to concentrate. The work desk is extremely fast but underused, because most of the time the librarian is moving books between the floor and the desk. That is what standard attention does: it materialises the S = QK^T matrix in HBM and keeps coming back for it piece by piece.

A FlashAttention v1 librarian changes strategy: asks for one shelf at a time, brings it to the desk, reads what is needed, jots notes in a compact notebook (“from this shelf I care about this, this and this, with these relative weights”), returns the shelf to its place and brings the next one. The notebook is the only thing that carries over between shelves. Nothing is ever piled higher than the desk can hold. The trick that makes this possible is the online softmax: instead of needing all the content at once in order to normalise, it keeps a running max and a running sum that are updated shelf by shelf.

A FlashAttention v2 librarian realises that several topics can be worked in parallel, because the desk is big and the shelves are independent along some axes. Three assistants are put to work, each with a notebook, each covering a different block of the room, and the results are combined at the end.

A FlashAttention v3 librarian manages to mechanise the flow: a conveyor belt with two stations is installed. While station A reads the current shelf and takes notes, station B already has the next shelf in transit from the shelving system. When A finishes, B hands over the new shelf with no waiting. This is the ping-pong producer/consumer pattern: the TMA offloads the data loading and the consumer warps do the work in parallel with the loads. On top of that, the notes are written in lower-precision shorthand (FP8), because the pages that matter have already had an orthogonal pre-treatment that stops them losing precision where it hurts.

A FlashAttention v4 librarian discovers something new about the remodelled library (Blackwell): two much faster conveyor belts have been installed (tensor cores 2.25×), but the shorthand typewriter has not been upgraded (the SFU is unchanged). Now the bottleneck is writing the notes, not fetching the shelves. The fix is elegant: instead of using the shorthand machine, the notes are written with polynomial formulas that the tensor core itself can evaluate (software-emulated exponential). The belt no longer sits idle waiting for the machine.

The analogy holds up with four mappings:

Work desk = SRAM per SM (228 KB on H100, 256 KB on B200).
Giant shelving system = HBM (3.35 TB/s on H100, 8 TB/s on B200).
Notebook with running max and sum = online softmax stats (m, ℓ).
Conveyor belt with two stations = TMA + WGMMA producer/consumer pipeline.

Why standard attention was the bottleneck

The naive intuition, “attention is matmuls, and GPUs are good at matmuls”, is correct but incomplete. There are two matmuls (QK^T and then softmax(S) V), and in between a non-matmul operation (softmax) that requires materialising the intermediate matrix S of size N × N.

Modern GPUs have a very concrete roofline. For H100 SXM5:

Compute peak: 989 TFLOPS BF16 (tensor core, dense, without sparsity).
Memory bandwidth: 3.35 TB/s HBM3.
Break-even point (arithmetic intensity break-even): 989 × 10¹² / (3.35 × 10¹²) ≈ 295 FLOP/byte.

Any operation that fails to perform 295 operations for every byte it moves from HBM is memory-bound: the tensor core sits waiting for data. On B200 the ratio is similar (~281 FLOP/byte) because both compute and bandwidth went up.

Standard attention materialising S reads the matrix twice (once for the softmax, once to multiply by V) and writes it once. For Llama 3 70B with head dim d = 128 and context N = 128K:

Matrix S per head per layer: N × N × 2 bytes = 128K × 128K × 2 = 34.36 GB.
80 layers × 64 Q-heads → aggregate HBM traffic (if it were materialised serially) on the order of TBs, prohibitive in transient.
Even if it is not all materialised at once, the round trips dominate the time: effective arithmetic intensity well below 295 → memory-bound operation → tensor core underused at around 25 %.

FlashAttention does not change the mathematics of attention, it changes the order of the operations so that S is never materialised in HBM. Its IO complexity is Θ(N²·d²/M) with M = SRAM size per SM, against Θ(N·d + N²) for standard attention. With d = 128 and M = 228 KB: an approximate reduction factor of M/d² ≈ 14× less HBM traffic. That is what moves the operation from memory-bound to nearly compute-bound.

The trick that made it all possible: online softmax

Without online softmax there is no FlashAttention. The idea comes from Milakov and Gimelshein, 2018 (the paper “Online normalizer calculation for softmax”, arXiv:1805.02867), and it allows softmax([x_1, ..., x_N]) to be computed in one incremental pass without needing to know the global maximum before starting.

The standard softmax is:

$$\text{softmax}(x_i) = \frac{e^{x_i - m}}{\sum_{j} e^{x_j - m}}, \quad m = \max_j x_j$$

The online trick: keep a running maximum m^{(t)} and a running sum ℓ^{(t)}. When a new block of values arrives with local maximum m_{\text{new}}:

$$m^{(t+1)} = \max(m^{(t)}, m_{\text{new}})$$ $$\ell^{(t+1)} = e^{m^{(t)} - m^{(t+1)}} \cdot \ell^{(t)} + \sum_{j \in \text{new}} e^{x_j - m^{(t+1)}}$$

And the accumulated partial outputs are rescaled by the same factor e^{m^{(t)} - m^{(t+1)}}. At the end, dividing by the final ℓ gives exactly the same result as the standard softmax. It is mathematically exact, not an approximation.

This is what allows K to be traversed block by block without materialising the whole S matrix. Each block updates the stats and the accumulated outputs, and is then discarded. The desk never fills up.

The four versions (May 2026)

	FA1 (2022)	FA2 (2023)	FA3 (2024)	FA4 (2026)
Target GPU	A100 / Ampere	A100 / H100	H100 / Hopper	B200 / Blackwell
Core idea	Tiling + online softmax	Sequence parallelism + work partitioning	Async WGMMA + TMA + FP8	Polynomial exp + 2-CTA tensor cores
Memory	O(N) (vs O(N²))	same	same	same
Typical peak util	~25 % A100	~70 % A100, ~35 % H100	75 % H100 BF16, 60 % H100 FP8	71 % B200 BF16
Effective TFLOPS	—	225 TFLOPS A100 BF16	740 H100 BF16, 1,200 H100 FP8	1,605 B200 BF16
Speedup vs previous	2-4× standard	2× FA1	1.5-2× FA2 (BF16), 2.6× (FP8)	2× FA3 on B200
Paper	arXiv:2205.14135	arXiv:2307.08691	arXiv:2407.08608	arXiv:2603.05451

FA1 — changing the order of the operations

Three ideas combined: tiling of Q/K/V into blocks that fit in SRAM, online softmax over those blocks, and recomputation in the backward pass. The N×N matrix S is not stored, only the stats (m, ℓ), and in the backward pass S is recomputed block by block from Q, K and the stats. Result: 7.6× speedup on GPT-2 versus PyTorch standard attention, memory O(N).

FA2 — parallelising along the sequence

FA1 parallelised only over batch × heads. With a small batch (1-4) and models with few heads or aggressive GQA, the GPU was left with idle SMs. FA2 also parallelises along the sequence dimension: different SMs process different stretches of Q at the same time. It also rewrites the algorithm to minimise the non-matmul operations (softmax rescaling), because those do not go through tensor cores. And it improves work partitioning between warps (split-Q instead of split-K reduces shared memory traffic). Result: around 2× over FA1 on H100 and A100, 225 TFLOPS on A100 (72 % MFU). On H100 it stays around 30-35 % of BF16 peak because it does not exploit async WGMMA.

FA3 — the Hopper moment

This is where FlashAttention stops being an algorithm and becomes a Hopper-specific artefact. Three pillars:

Async WGMMA: Hopper’s new tensor core instructions let a warpgroup fire a GEMM and have the rest of the warpgroup do something else (the softmax, for instance) while the tensor core keeps working. This is the trick that unlocks matmul/softmax overlap.
TMA (Tensor Memory Accelerator): dedicated hardware for copying tiles between HBM and SRAM. It frees the SM from computing indices and predicating out-of-bounds accesses, which used to consume SM cycles. It is the equivalent of hiring warehouse staff: the librarian no longer has to carry the books personally.
FP8 with block quantisation + incoherent processing: quantising Q and K to FP8 doubles tensor core throughput. The precision loss is mitigated with two tricks: a per-tile scale (64×d) instead of a whole-tensor scale, and a pre-multiplication by a random orthogonal Hadamard-based matrix that “spreads” the outliers before quantising. Documented result: numerical error 2.6× lower than the FP8 baseline.

These three pillars combine with producer/consumer warp specialisation (producer warps do TMA loads; consumer warps do WGMMA + softmax) and ping-pong scheduling with two warpgroups taking turns so that there are never pipeline bubbles. When WG1 does softmax, WG2 does GEMM; then they swap.

Numbers: 740 TFLOPS BF16 on H100 (75 % of the 989 peak), 1.2 PFLOPS FP8 (60 % of the 1978 FP8 dense peak). For sequences ≥ 1K it beats cuDNN. Speedup over FA2: 1.5-2× BF16, 2.6× FP8.

FA4 — Blackwell’s asymmetry

Blackwell scaled everything unevenly:

Tensor core BF16 throughput: 1 PFLOP H100 → 2.25 PFLOPS B200 (2.25×).
SFU count (where the softmax exp runs): unchanged.
Shared memory bandwidth: unchanged.

In other words, if FA3 runs as-is on a B200 with no changes, the matmul goes twice as fast but the softmax stays exactly the same, and that stalls the pipeline. It was only a matter of time before someone fixed the imbalance.

FA4 (March 2026, same team of Dao + Princeton + Together AI + Meta + NVIDIA + Colfax) is a ground-up rewrite with three ideas:

Software-emulated exponential: a polynomial approximation of exp that runs on the tensor core instead of on the SFU. It loses a little precision (carefully bounded and compensated by the rest of the kernel) but keeps the conveyor belt moving.
Conditional softmax rescaling: avoids rescaling accumulators when the running max does not change significantly. A “lazy” style optimisation: it only pays the cost when the cost is needed.
2-CTA tensor core: two CTAs (Cooperative Thread Arrays) cooperate to feed the tensor cores with larger tiles. It gets more out of Blackwell’s new capabilities.

Written in CuTeDSL (NVIDIA’s CUTLASS Python DSL, not CUDA C++ directly). Result on B200 BF16: 1,605 TFLOPS (71 % of the 2250 peak). 1.3× over cuDNN 9.13. 2.7× over Triton. 2× over FA3 run as-is on B200 (which was the previous baseline). It is the first attention kernel to pass 1 PFLOPS.

Note: there is recurring confusion about “FP4 attention”. Blackwell’s NVFP4/MXFP4 extensions apply to weights, not to attention. FA4 can be combined with NVFP4 weights, but the attention computation itself is still BF16 or FP8 depending on the configuration. FP4 quantisation of QK^T does exist in some proprietary kernels (Fireworks AI’s FireAttention V4 combines it) but it is not standard practice.

Implementations and libraries in 2026

Dao-AILab/flash-attention (the canonical repo): supports SM 8.0 (Ampere) with FA2, SM 9.0 (Hopper) with FA3, SM 10.0 (Blackwell datacenter B100/B200/B300) with FA4. The consumer Blackwell version (5090, SM 12.0) has partial support as this post goes out.
FlashInfer (flashinfer-ai/flashinfer, arXiv:2501.01005): an attention engine aimed at serving (not training). Its conceptual contribution is the Block-Sparse Row (BSR), a unified abstraction covering paged KV cache, the radix tree of prefix caching and the tree masks of speculative decoding. Internally it can call FA2/FA3, cuDNN, CUTLASS or trtllm-gen FMHA kernels depending on the case. It JIT-compiles specific variants at runtime. Integrated into vLLM, SGLang and TensorRT-LLM.
vLLM (May 2026): automatic backend selection by GPU. Default FA4 on SM 10.0+, FA3 on SM 9.0, FA2 on the rest. Fallbacks on Blackwell: TRT-LLM Ragged → FlashInfer → TokenSpeed MLA. For FP8 KV cache on B200, FlashInfer is competitive.
SGLang: uses FlashInfer as its attention backend; RadixAttention is the prefix caching layer on top (a radix tree over the KV cache).
TensorRT-LLM: its own fused kernels (trtllm-gen FMHA). XQA is NVIDIA’s own optimisation for GQA in decode.
PyTorch SDPA and FlexAttention: torch.nn.functional.scaled_dot_product_attention selects the backend automatically. FlexAttention (new) lets you define custom masks declaratively and compiles to kernels that can use FA4 as a backend.
xFormers: still alive but residual. The built-in PyTorch SDPA covers most cases.

Cases where FlashAttention does not help

Very short contexts (N < 512): the tiling and kernel launch overhead does not pay off; cuDNN can win.
Non-standard custom masks: FA ships causal, sliding window and ALiBi only. Arbitrary masks need FlexAttention or the JIT variants of FlashInfer.
Non-standard head dim: FA optimises for d = 64, 128, 256. Odd dimensions (d = 96, d = 192) fall into slow paths.
GQA/MQA with extreme ratios: natively supported, but the speedup versus pure MHA depends on the Q-heads : KV-heads ratio.
Cross-attention: supported but less optimised; self-attention is where the gain is largest.
FP8 without block quantisation or incoherent processing: it loses several points in benchmarks. If your serving framework does not implement the two tricks from the FA3 paper, FP8 attention can be a bad idea.

Implications on on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace, SM 8.9)

The 4090 is Ada Lovelace, not Hopper. It runs neither FA3 nor FA4; it runs FA2. That means around 70 % utilisation in BF16 attention (~250 effective TFLOPS against the 4090’s 330 TFLOPS BF16 peak). It is not a disaster, FA2 is already very good compared with standard attention, but the ceiling is clearly below that of an H100. For consumer deploys on a 4090 with Llama 3 8B BF16 or any 14B-32B INT4 AWQ, FA2 is what you will be using, and it is perfectly reasonable.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

This is where FA3 shines and it is what vLLM/SGLang/TRT-LLM will select by default. Two common configurations:

Llama 3 70B FP8 with FA3 FP8 attention: 1.2 PFLOPS peak on the GPU, aggregate cluster throughput on the order of 8,000-12,000 tokens/s at medium batch depending on TP and context. For FP8 attention to deliver its full performance it is crucial to use the block quantisation + incoherent processing techniques from the FA3 paper (they are enabled by default in vLLM).
DeepSeek-V3 671B FP8 + MLA with FlashInfer: DeepSeek uses Multi-head Latent Attention (MLA), a different variant of standard attention. FlashInfer has specific kernels (FlashMLA). The typical stack is vLLM/SGLang + FlashInfer + FlashMLA + an FA3 fallback for the non-MLA layers.

If the infrastructure is Blackwell (B200/B300, which some clusters start receiving in 2026), FA4 is the right option and it should be enabled by default in vLLM 0.16+ and SGLang 0.5.11+.

What we have not covered

MLA (Multi-head Latent Attention) from DeepSeek and the specific FlashMLA kernels: they optimise KV cache compression but require different kernels.
Flexible masking and the FlexAttention use cases (PyTorch 2.5+): how to declare arbitrary masks without paying the cost of a custom kernel.
Hardware assistance for sparse attention (NVIDIA 2:4 sparse tensor cores) and why sparse attention has not established itself as a higher ceiling than dense FA.
FA in the backward pass of fine-tuning: this post focuses on inference, but FA3/FA4 also go through the backward pass and they are what makes training models with long contexts viable on H100/B200.

References

Dao, T., Fu, D., Ermon, S., Rudra, A., Ré, C. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 2022. https://arxiv.org/abs/2205.14135
Dao, T. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. 2023. https://arxiv.org/abs/2307.08691
Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., Dao, T. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision. NeurIPS 2024. https://arxiv.org/abs/2407.08608
Dao, T. et al. FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling. 2026. https://arxiv.org/abs/2603.05451
Milakov, M., Gimelshein, N. Online normalizer calculation for softmax. 2018. https://arxiv.org/abs/1805.02867
Ye, Z. et al. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving. MLSys 2025. https://arxiv.org/abs/2501.01005
Tri Dao FA3 blog: https://tridao.me/blog/2024/flash3/
Tri Dao FA4 blog: https://tridao.me/blog/2026/flash4/
PyTorch FlashAttention-3 announcement: https://pytorch.org/blog/flashattention-3/
PyTorch FlexAttention + FA4: https://pytorch.org/blog/flexattention-flashattention-4-fast-and-flexible/
Together AI FA4 blog: https://www.together.ai/blog/flashattention-4
Colfax Research FA3: https://research.colfax-intl.com/flashattention-3-fast-and-accurate-attention-with-asynchrony-and-low-precision/
Colfax Research FA4: https://research.colfax-intl.com/flashattention-4-algorithm-and-kernel-pipelining-co-design-for-asymmetric-hardware-scaling/
Repo Dao-AILab/flash-attention: https://github.com/Dao-AILab/flash-attention
Repo flashinfer-ai/flashinfer: https://github.com/flashinfer-ai/flashinfer
vLLM attention backends: https://docs.vllm.ai/en/latest/design/attention_backends/
NVIDIA Hopper Architecture in Depth: https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/

Speculative decoding: the secretary who types ahead of the boss — fundamentals, maths and state of play in May 2026

Fri, 29 May 2026 11:00:00 +0200

This post complements KV cache: the working memory and Quantisation for LLM inference. Both are acceleration levers inside a single forward pass. Speculative decoding is the complementary lever: instead of making each forward pass cheaper, it tries to produce more tokens per forward pass. It is orthogonal to quantisation and to the KV cache, and it stacks multiplicatively with both.

You are here: DEPLOY

TL;DR

Autoregressive LLM inference has a structural problem: every new token needs a complete forward pass of the model, and decode forward passes are memory-bandwidth-bound, not compute-bound (the detail is in KV cache). The GPU spends most of its time waiting for the next weight to arrive from HBM. Speculative decoding uses that dead time by doing two things in parallel: a small, cheap model (draft) generates γ tokens autoregressively, fast but imprecise, and the large model (target) verifies those γ tokens in a single parallel forward pass, at almost the same cost as a single-token forward pass. An acceptance rule based on rejection sampling decides how many draft tokens are accepted, and the maths proves, it does not approximate, it proves, that the output distribution matches sampling directly from the target exactly. The technique has a ceiling: you never generate more than 1/(1-α) tokens per step, with α the acceptance rate. The five dominant families in May 2026 are vanilla SD (Leviathan 2023), Medusa (extra parallel heads), EAGLE-1/2/3 (a draft that operates at the level of hidden states), MTP (multi-token prediction native to DeepSeek-V3) and P-EAGLE (a draft that produces the γ tokens in a single forward pass, integrated in vLLM 0.16+). On low-concurrency workloads with short prompts and long outputs, the typical case of the on-premise conversational assistant, the real speedup sits between 2× and 4× on modern hardware.

The analogy: the secretary who types ahead and the boss who validates

Picture a crowded press conference with a real-time transcription setup. Two people are working:

The secretary sits at the keyboard. They know the subject, they have read the briefings and they know the boss’s verbal tics. As soon as the boss opens their mouth, the secretary starts typing what they think is coming. They are fast, three or four words ahead of the boss, but they sometimes get it wrong, especially when the boss takes an unexpected turn.

The boss periodically reads the secretary’s screen. They read a whole block at once, three or four words, and mentally compare what is written with what they would have said. If it matches their intent, they leave the text and carry on. If the secretary drifted at some point, they correct right there, and whatever the secretary typed after the divergence is discarded (it could be wrong because of the earlier error). If the whole block was fine, the boss takes the chance to add one more word while correcting.

The result: the final text is identical to what the boss would have dictated alone, but it comes out faster because the secretary works ahead.

The analogy holds on four details:

The secretary is the draft model. Small, cheap, fast, approximate.
The boss is the target model. Slow, expensive, but the only authority on output quality.
The block the boss reads at once is the target’s parallel forward pass over prompt + γ draft tokens. Verifying γ tokens costs almost the same as generating one, because the bottleneck is loading the weights from HBM, not the operations.
The “correct right where it diverged” rule is the rejection sampling that preserves the target distribution.

From here we go into the mechanism and into why quality is preserved exactly, not approximately.

The bare mechanism

Call p the target distribution (what it would produce if it were the only one speaking) and q the draft’s. The speculative decoding iteration has three steps:

Step 1 — Draft. The draft generates γ tokens autoregressively: x_1, x_2, ..., x_γ. Each one through its own forward pass. Since the draft is small, the γ steps cost little. Typically γ ∈ [4, 8].

Step 2 — Verify. The target runs a single forward pass over the complete sequence prompt + x_1 ... x_γ. Because attention is causal, that forward pass simultaneously produces the distributions p(·|prompt, x_<i) for each position i. In other words, in a single step it obtains the verification of the γ tokens and, on top of that, an extra distribution p(·|prompt, x_1...x_γ) for the next token.

Step 3 — Accept/reject token by token. For each x_i from left to right it applies:

If p(x_i) ≥ q(x_i): always accept.
If p(x_i) < q(x_i): accept with probability p(x_i)/q(x_i).
If it rejects: stop, sample a replacement token from the normalised residual distribution norm(max(0, p − q)), and discard the rest.
If it reaches the end having accepted the γ tokens: append a bonus token sampled directly from p(·|prompt, x_1...x_γ), which is already in the logits of the target’s forward pass.

Result per iteration: between 1 and γ+1 new tokens. In the best case (all accepted + bonus) γ+1 tokens are generated at the cost of a single target forward pass plus γ draft forward passes, against γ+1 target forward passes in the version without speculative.

Why quality does not degrade (the proof)

This is the part of the technique that confuses people most the first time. It looks like magic: how can sampling from an arbitrary draft and then “validating” produce exactly the same distribution as sampling from the original target?

The proof fits in two lines. For any token x, the final probability of emitting it is the sum of two disjoint events: the draft proposes it and it is accepted, or the draft proposes something else, it is rejected, and sampling from the residual yields x. Formally:

$$P(\text{emit } x) = q(x) \cdot \min\!\left(1, \frac{p(x)}{q(x)}\right) + P(\text{reject}) \cdot \frac{\max(0, p(x)-q(x))}{\sum_y \max(0, p(y)-q(y))}$$

The first term is min(p(x), q(x)). The second is max(0, p(x) − q(x)) (the total rejected mass Σ max(0, p−q) cancels the denominator exactly). Adding them: min(p(x), q(x)) + max(0, p(x) − q(x)) = p(x). The output is statistically indistinguishable from sampling directly from the target, down to floating-point numerical differences.

This matters operationally: it means speculative decoding is not perceptual compression, it is not a quality-speed trade-off, it does not require extra validation with evals. If the target would have passed a given eval, the system with speculative passes it too.

The maths that matters: ceiling and speedup

Call α the acceptance rate: the expected probability that an individual draft token is accepted. Assuming that acceptances of consecutive tokens are independent (a reasonable approximation in practice), the expected number of tokens generated per iteration is:

$$E[\text{tokens per step}] = \sum_{k=0}^{\gamma} \alpha^k + \alpha^{\gamma+1} = \frac{1 - \alpha^{\gamma+1}}{1 - \alpha}$$

And the theoretical speedup relative to generating one token per target iteration, with c = T_draft / T_target the relative cost of the draft, is:

$$\text{Speedup} = \frac{1 - \alpha^{\gamma+1}}{(1 - \alpha)(\gamma c + 1)}$$

There is a ceiling that many implementations try to beat and cannot: as γ → ∞, the speedup converges to:

$$\lim_{\gamma \to \infty} E[\text{tokens per step}] = \frac{1}{1 - \alpha}$$

It is an algorithmic ceiling, not a hardware one. With α = 0.7 you never generate more than 3.33 tokens per iteration no matter how much the hardware improves. With α = 0.8 → 5. With α = 0.9 → 10. That is why EAGLE-3, which targets α > 0.8 on many benchmarks, is not an incremental improvement over vanilla SD (α ≈ 0.5-0.6): it is a change of regime, because it raises the ceiling instead of merely getting closer to it.

A concrete numerical example, Llama 3 70B as target with a draft giving α = 0.75, γ = 5, c = 0.1:

Expected tokens per step: (1 − 0.75⁶) / (1 − 0.75) = (1 − 0.178) / 0.25 = 3.29
Speedup: 3.29 / (5 × 0.1 + 1) = 3.29 / 1.5 = 2.19×

If we raise α to 0.85 with the same configuration: tokens per step = (1 − 0.85⁶) / 0.15 = (1 − 0.377) / 0.15 = 4.16, speedup = 4.16 / 1.5 = 2.77×. A change of α = 0.75 → 0.85 (10 absolute points) raises the speedup by 27 %. That is why the research of the last two years focuses on pushing α: that is where the prize is.

The five modern families (May 2026)

Family	Year	Core idea	Draft size	Typical α / τ	Speedup
Vanilla SD	2023	Same-family target + draft pair, rejection sampling	1/10 – 1/100 of target	0.5 – 0.8	2-3×
Medusa	2024	N extra heads in parallel predict t+1, t+2, …; tree attention verifies several candidates	no draft, +1-2 % params	top-5 > 0.8	2.18-2.83×
EAGLE-1/2/3	2024-25	Autoregressive draft at the level of features (hidden states), not tokens. Reuses the target’s embedding	1 transformer block (~0.5-2 % params)	EAGLE-3: α > 0.8, τ up to 7.5	up to 6.5× peak
MTP (DeepSeek-V3)	2024	Multi-token heads trained from scratch as part of the model; at inference they act as a “free” draft	14B params in V3 671B	α > 0.8 (MTP1)	1.5-1.8×
P-EAGLE	2026	EAGLE but producing the γ drafts in a single forward pass (parallel, not autoregressive)	same as EAGLE	+30 % over EAGLE-3	4-5× vs AR

Three operational observations:

EAGLE dominates in production because its overhead is minimal (one transformer block, ~1 % of the target in parameters) and its α is high. The draft does not need its own “complete” KV cache because it shares features with the target.
Medusa mattered historically, it showed that speculative was possible without a separate draft, but EAGLE beat it on every published benchmark through 2024-2025.
MTP is special: it is not something you add to an existing model. It is something the model trained natively. If you buy DeepSeek-V3, MTP comes free and gives around 1.8× without touching anything. If you buy Llama 3, there is no MTP to be had; use EAGLE-3.

There are also two related techniques that deserve a brief mention, both without a draft model: Lookahead decoding (Fu et al. 2024), which formulates decoding as Jacobi iteration and extracts n-grams from the trajectory; and REST (He et al. 2024), which keeps an n-gram datastore and proposes drafts by longest-prefix match against the last generated tokens. Both give 1.5-2× with no extra VRAM. Useful when you have no trained draft model and do not want to maintain one.

Real implementations in May 2026

vLLM v0.16+: unified support for EAGLE-1/2/3, P-EAGLE, Medusa, native MTP (DeepSeek-V3 and variants), n-gram/suffix decoding without a draft, an arbitrary draft model and MLP speculators. The canonical flag is --speculative-config '{"method":"eagle3", "model":"...", "num_speculative_tokens": 5}'.
SGLang: native EAGLE-3 support with --speculative-algorithm EAGLE3. For DeepSeek-V3 it uses MTP through an EAGLE adapter. It has its own draft training framework (SpecForge).
TensorRT-LLM: Medusa, EAGLE (a simplified variant without a tree), ReDrafter, Lookahead. They report around 2.2× with EAGLE.
llama.cpp: basic speculative only with a draft model (--model-draft). No native EAGLE/Medusa/MTP as far as I have verified. Typical speedup 1.5-2.5× single-user.

When speculative does NOT help

The technique has three important blind spots:

Large batch. The GPU moves from memory-bound (decode at low concurrency) to compute-bound (decode at high concurrency). In the compute-bound regime the “almost free” target forward passes stop being free: γ tokens start costing γ times more instead of nearly 1. The typical crossover is at batch 16-32 for dense models; for MoE with few active parameters per token the crossover happens later. At load peaks, speculative can worsen aggregate throughput per GPU.

Prefill / TTFT. Speculative decoding does not touch prefill. The phase that processes the whole prompt stays identical. TTFT does not improve and can get marginally worse because of the draft setup. If the SLA is TTFT-bound (assistants with short outputs, search, RAG with brief answers), this is not the right tool.

Short outputs. If the model generates 10-20 tokens and the answer is over, the fixed setup overhead is not amortised. Speculative shines with long outputs (300+ tokens): generative assistants, extended code completion, drafting.

Operational pitfalls

Extra VRAM for the draft. In vanilla SD, loading a complete draft plus its KV cache is expensive. Llama 3 8B in BF16 as the draft for a 70B is about 16 GB of weights plus KV cache. On an H100 80 GB with the target already nearly filling it, that can force you to shrink the target’s KV cache and lower maximum concurrency. EAGLE solves this: the draft is one transformer block (~1 GB for a 70B) and reuses the target’s features.

Draft quantisation. Quantising the draft to INT4 with GPTQ degrades α substantially (errors in the logits accumulate in the comparison against p). AWQ holds up better but also lowers α. Common practice in 2026: target in FP8 or INT4, draft in FP16/BF16. The draft is small enough that the VRAM saved by quantising it does not compensate for the drop in α and therefore in speedup. The detail of each format is in Quantisation.

Interaction with continuous batching. It does not break it, but it creates nested raggedness: each request in the batch can accept a different number of tokens on each iteration. vLLM’s PagedAttention absorbs it, but the scheduler loses efficiency. At low QPS (conversational assistant, low simultaneous concurrency) the combination is excellent. At high QPS there is real tension, and work such as Goodput-optimized speculative decoding (Liu et al., 2024) optimises γ dynamically according to the state of the batch.

Sampling temperature. α falls with high temperatures. At T = 1.0 with creative outputs (free-form writing), α can drop 10-15 points relative to T = 0 with the same model-draft pair. The speedup scales accordingly.

Implications on on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace)

The classic use is vanilla SD with two quantised models: for instance, Llama 3 70B AWQ-INT4 as target (~35 GB → needs TP=2 across two 4090s) and Llama 3 8B AWQ-INT4 as draft. In practice, the most realistic single-card case is Llama 3 8B target + a 1B model as draft or Qwen 3 14B target + Qwen 3 0.5B as draft: they fit comfortably and give 1.8-2.5× on conversational tasks. For EAGLE on consumer cards, the official drafts for popular families (Llama 3, Qwen 3) are published on Hugging Face and take up 0.5-2 GB extra.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here EAGLE-3 shines:

Llama 3 70B FP8 + EAGLE-3 draft in FP16: the draft takes about 0.7 GB; the target about 70 GB with TP=2 (35 GB per GPU). The speedup observed in reproducible benchmarks is between 2.5× and 4× at batch 1-4, falling to almost break-even at batch 32.
DeepSeek-V3 671B FP8 + native MTP: the model ships with MTP trained in; there is nothing to add. The speedup is 1.5-1.8× with zero extra VRAM. It is the most efficient option operationally: zero additional pieces.
Combining with disaggregated serving: as Disaggregated serving explains, prefill and decode can live in separate pods. Speculative applies only in the decode pods, which fits the separation perfectly (prefill is compute-bound and would not benefit).

The rule of thumb on an H100 cluster in May 2026: if the model is DeepSeek-V3 / V4 → native MTP, nothing else; if it is Llama 3 / Qwen 3 → EAGLE-3 with the official draft; if it is exotic → vanilla SD with a draft from the same family.

What we have not covered

Training custom EAGLE drafts with SpecForge: how to collect target trajectories and train the draft on-policy.
Speculative Prefill (arXiv:2502.02789): a variant for accelerating TTFT, a different mechanism from the decode one described here.
Tree attention in detail: how Medusa and EAGLE-2 verify several candidates at once with specific attention masks.
MoE + speculative: the combination has non-trivial interactions with the expert router. Low activated params keep the memory-bound regime in place even at high batch, which changes the rules.

References

Leviathan, Y., Kalman, M., Matias, Y. Fast Inference from Transformers via Speculative Decoding. ICML 2023. https://arxiv.org/abs/2211.17192
Chen, C., Borgeaud, S., Irving, G., Lespiau, J.B., Sifre, L., Jumper, J. Accelerating Large Language Model Decoding with Speculative Sampling. DeepMind 2023. https://arxiv.org/abs/2302.01318
Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J.D., Chen, D., Dao, T. Medusa: Simple LLM Inference Acceleration with Multiple Decoding Heads. ICML 2024. https://arxiv.org/abs/2401.10774
Li, Y., Wei, F., Zhang, C., Zhang, H. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. ICML 2024. https://arxiv.org/abs/2401.15077
Li, Y., Wei, F., Zhang, C., Zhang, H. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. EMNLP 2024. https://arxiv.org/abs/2406.16858
Li, Y., Wei, F., Zhang, C., Zhang, H. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. NeurIPS 2025. https://arxiv.org/abs/2503.01840
DeepSeek-AI. DeepSeek-V3 Technical Report — Multi-Token Prediction. https://arxiv.org/abs/2412.19437
Fu, Y., Bailis, P., Stoica, I., Zhang, H. Break the Sequential Dependency of LLM Inference Using Lookahead Decoding. ICML 2024. https://arxiv.org/abs/2402.02057
He, Z., Zhong, Z., Cai, T., Lee, J.D., He, D. REST: Retrieval-Based Speculative Decoding. NAACL 2024. https://arxiv.org/abs/2311.08252
vLLM speculative decoding docs: https://docs.vllm.ai/en/latest/features/speculative_decoding/
SGLang speculative decoding docs: https://docs.sglang.ai/advanced_features/speculative_decoding.html
TensorRT-LLM speculative sampling: https://nvidia.github.io/TensorRT-LLM/advanced/speculative-decoding.html
vLLM blog Speculative Decoding in vLLM (oct 2024): https://blog.vllm.ai/2024/10/17/spec-decode.html
vLLM blog P-EAGLE: Parallel Speculative Decoding in vLLM (mar 2026): https://vllm.ai/blog/2026-03-13-p-eagle
Official EAGLE repo: https://github.com/SafeAILab/EAGLE
Official Medusa repo: https://github.com/FasterDecoding/Medusa

Quantization for LLM inference: FP8, INT4 (GPTQ, AWQ) and GGUF — the model's accounting zoom

Wed, 27 May 2026 11:30:00 +0200

This post complements KV cache: the working memory of LLM inference, where cache quantisation is mentioned as a savings lever; here we go into the whole method, model weights and cache, and into why each format does what it does.

TL;DR

Quantising is a change of representation: instead of storing every model weight as a float16 or bfloat16 (2 bytes), it is stored as a short integer (1 byte INT8, half a byte INT4) with a scale factor that reconstructs a value close to the original. The price is loss of numerical precision; the reward is 2-4× less VRAM, 2-3× more throughput and, on Hopper and Blackwell, a radically lower compute cost because FP8/FP4 units execute in fewer cycles than BF16 ones. The four dominant formats in May 2026 are FP8 (E4M3/E5M2, datacenter), INT4 GPTQ (Hessian-aware reconstruction), INT4 AWQ (activation-aware) and GGUF (the llama.cpp family). Each has a sweet spot: FP8 when the datacenter is Hopper/Blackwell and quality matters; GPTQ and AWQ when serving runs on Ampere/Ada and 4 bits are mandatory; GGUF when the target is edge or a consumer GPU. This post explains the minimum maths, the algorithms behind each format, what each one loses measured in perplexity and MMLU, and how it all applies on a 4090 versus an H100 cluster.

You are here: DEPLOY

The analogy: JPEG with an edge detector

A JPEG compresses an image by reducing the precision with which pixels are stored, but it does not reduce it uniformly. Where there is flat sky, thousands of very similar pixels, it throws away detail without anyone noticing. Where there is a sharp edge, the outline of a face, it keeps the fidelity. The trick is to detect which parts are sensitive before compressing.

Quantization of an LLM works the same way. You do not take every model weight and say “all of them in 4 bits”. Some weights are very important, projection weights that move the output a lot when they change, and others are less so. Modern techniques (GPTQ, AWQ) are basically edge detectors: they identify which weights can be quantised aggressively and which need more bits or special treatment, and they apply the quantisation with that information.

The analogy holds up in three details:

Calibration with a small dataset = the phase in which the JPEG encoder analyses the image before choosing blocks.
A block of 128 weights with a shared scale = the JPEG 8×8 block with its DCT.
Outliers preserved with more precision = the high frequencies of an edge are preserved more than the flat areas.

From there on, what follows is the maths and the operational detail.

The minimum maths: scale and zero-point

Quantising a weight vector w ∈ ℝ^n (in BF16) to INT4 means finding two things:

A scale s ∈ ℝ (in BF16 or FP16).
For each weight, an integer code q ∈ {0, 1, ..., 15} that fits in 4 bits.

And an approximate reconstruction formula:

$$\hat{w}_i \approx s \cdot (q_i - z)$$

where z is the zero-point (the integer that defines which code represents the original zero). The zero-point exists in asymmetric INT4/INT8 so that half the range is not wasted on negative values when the weight distribution is not symmetric.

The choice of s and z for a block of weights w_block:

$$s = \frac{\max(w_\text{block}) - \min(w_\text{block})}{2^{b} - 1}, \quad z = -\frac{\min(w_\text{block})}{s},$$

with b = number of bits (4 in INT4). Encoding:

$$q_i = \text{clip}\!\left(\text{round}\!\left(\frac{w_i}{s} + z\right),\, 0,\, 2^b - 1\right).$$

And decoding at inference time:

$$\hat{w}_i = s \cdot (q_i - z).$$

Numerical example

Take 8 real weights from a linear layer: w = [0.31, -0.12, 0.78, -0.05, 1.42, -0.91, 0.23, 0.66]. We want to quantise to INT4 (16 levels).

max = 1.42, min = -0.91. Range = 2.33.

$$s = \frac{2.33}{15} \approx 0.1553, \quad z = -\frac{-0.91}{0.1553} \approx 5.86 \to 6.$$

Encoding of each weight:

`w_i`	`w_i/s + z`	`round`	`q_i`	`ŵ_i = s·(q-z)`	error
0.31	8.00	8	8	0.311	+0.001
-0.12	5.23	5	5	-0.155	-0.035
0.78	11.02	11	11	0.776	-0.004
-0.05	5.68	6	6	0.000	+0.050
1.42	15.14	15	15	1.398	-0.022
-0.91	-0.06	0	0	-0.932	-0.022
0.23	7.48	7	7	0.155	-0.075
0.66	10.25	10	10	0.621	-0.039

Mean squared error: MSE ≈ 0.0015. For a single layer with millions of weights, the aggregate effect is what calibration tries to minimise.

Storage: instead of 8 values × 2 bytes = 16 bytes (BF16), we have 8 × 4 bits = 4 bytes + 2 bytes (BF16 scale) + 0.5 byte (INT4 zero-point) ≈ 6.5 bytes. 2.5× less, yet the useful data is still recoverable with a small error.

PTQ vs QAT: when quantisation happens

Two different operational regimes.

Post-Training Quantization (PTQ) is applied after training, on a model already trained in BF16/FP16. It reads a small dataset (typically 128-512 examples) to calibrate the scales, runs the quantisation algorithm (GPTQ, AWQ, and so on) and produces the quantised weights. Cost: minutes to a few hours. Typical loss: 0.05-0.3 PPL in perplexity (~0.5-2 % in MMLU) for INT4 with modern methods.

Quantization-Aware Training (QAT) introduces the quantisation operations inside the training loop. During training the model “sees” that its weights get quantised and learns to be robust to it. Cost: retraining the model (expensive), but little is needed, a short fine-tune on top of the already quantised PTQ model. Typical loss: ~0 (quantisation becomes indistinguishable from the original model).

When to use which:

PTQ = default. 90 % of production cases. The model arrives in BF16, you quantise it in 1-2 hours on one GPU, you deploy it.
QAT = when PTQ loses too much and the difference matters (typical case: INT2/INT3, or sensitive models such as specific reasoning ones).

The dominant formats in 2026

FP8: the Hopper/Blackwell datacenter format

FP8 is not “INT8 + sign”: these are two 8-bit floating point formats.

E4M3 (4 exponent bits, 3 mantissa bits): range ±448, reasonable precision around ±1.0. Typically used for weights and for activations in most layers.
E5M2 (5 exponent, 2 mantissa): range ±57,344, lower precision. Used for gradients during training or for activations with large outliers at inference time.

Why FP8 left INT8 behind in the datacenter: the tensor cores on H100/H200/B200 execute FP8 operations natively with 2× the throughput of BF16 and 4× that of FP16. And because FP8 preserves logarithmic dynamics (just like FP16), matrices with values spread out in magnitude, typical of transformers, quantise with less error than with INT8.

The loss measured in production is minimal: for a Llama 3.1 70B FP8 versus BF16, perplexity rises by ~0.03 and MMLU drops by ~0.5 points. It is the default option in any modern deployment on H100/B200.

Microscaling: NVFP4 and MXFP4

Blackwell (B100/B200, 2025) introduces NVFP4 and MXFP4, 4-bit formats with small-block scaling (typically 16 or 32 elements per scale, against 128 in INT4 GPTQ). The scale is FP8 instead of FP16/BF16, which reduces storage further.

The result: 4 bits with quality close to FP8. In 2026 NVFP4 is becoming the default option for very large models (200B+) on Blackwell clusters. For 4×H100 SXM, Hopper rather than Blackwell, FP8 remains the sweet spot.

INT4: GPTQ vs AWQ

The two algorithms that dominate 4-bit quantisation solve the same problem with different strategies.

GPTQ (Frantar et al. 2022)

The idea: layer-by-layer quantisation, explicitly minimising the error at the output of each linear layer using information from the Hessian matrix (the second derivative of the loss). For each layer:

Estimate the Hessian H = X^T X where X are the calibration activations.
Quantise one weight at a time in order (typically the most sensitive one first).
Update the remaining weights to compensate for the error of the weight just quantised.

Step 3 is what makes GPTQ better than naive round-to-nearest: weights compensate for their neighbours’ errors. The official implementation quantises a Llama 3 70B in ~3-4 hours on one H100 with 128 calibration samples.

AWQ (Lin et al. 2023)

The AWQ observation: within a layer, not all channels (weight columns) are equally important. Roughly 1 % of the channels account for most of the impact on the activations. AWQ identifies them by measuring the mean magnitude of the activations that multiply them, and scales them before quantising so they are better preserved.

Concretely: if a channel c has large mean activations, AWQ multiplies that channel’s weights by a factor s_c before quantising, and when quantising the input to the next layer it divides by s_c. The maths cancels out, but the important weights end up with more resolution inside the INT4 range. No retraining, no Hessian, faster than GPTQ (~1-2 h for 70B on an H100).

Which one to choose

Criterion	GPTQ	AWQ
Quantisation speed	Slower	Faster
Quality preserved in INT4	Good	Slightly better (~0.05-0.1 PPL)
Supported hardware	Broad (Ampere+)	Broad (Ampere+)
Ecosystem	Mature, widely integrated	More recent, gaining ground
Default in 2026	When GPTQ artefacts already exist	Default for new quantisations

The practical rule in May 2026: AWQ by default for new INT4. GPTQ when a GPTQ-Int4 artefact published by the community already meets your requirements.

GGUF: the llama.cpp ecosystem

GGUF is not a quantisation algorithm, it is a file format, and a whole tooling ecosystem, around the llama.cpp runtime.

Its value: universal compatibility. The same GGUF file runs on:

Pure CPU (Intel/AMD x86, ARM).
Apple Silicon (M1/M2/M3/M4) with Metal acceleration.
Consumer GPUs (RTX, AMD Radeon) with layer offload to VRAM.
Edge devices (Jetson, ARM phones).

That is what llama.cpp allows and vLLM/TensorRT-LLM do not. The trade-off: lower peak throughput on datacenter GPUs than vLLM.

The most used GGUF sub-formats in 2026:

Sub-format	Effective bits	Relative quality	Typical use
`Q8_0`	8.5	Almost lossless	Baseline validation
`Q6_K`	6.6	Very small loss	High quality + savings
`Q5_K_M`	5.7	Small loss	Quality/size sweet spot
`Q4_K_M`	4.8	Moderate loss	Consumer default
`Q4_K_S`	4.5	Moderate-to-high loss	When Q4_K_M does not fit
`Q3_K_M`	3.9	Noticeable loss	Very constrained hardware
`Q2_K`	3.3	Large loss	Last resort

The _K_M suffix indicates the degree of mixing: inside the file, certain layers (typically attention.wv, feed_forward.w2) are stored with more bits than others. It is the equivalent of the JPEG “edge detector” applied layer by layer with a pre-established heuristic.

KV cache quantization

Quantising the model weights is half the problem. The KV cache, covered in detail in KV cache: the working memory, typically consumes 20-50 % of VRAM in production with concurrency. Quantising the cache is a lever too:

--kv-cache-dtype=auto (BF16/FP16, the default). 2 bytes per dimension × num_heads × head_dim × 2 (K and V).
--kv-cache-dtype=fp8 (E4M3 or E5M2 depending on hardware). 1 byte. Halves the cache with a typical loss of < 0.5 % on quality benchmarks.
--kv-cache-dtype=int4 (with blocks of 128). 0.5 bytes plus scale overhead. Divides the cache by ~3.5. Measurable loss (1-2 %) but acceptable with long contexts.

KV cache quantisation is orthogonal to weight quantisation: you can have BF16 weights and an FP8 cache, or INT4 weights and an FP8 cache, and so on. The dominant combination in 2026 on H100: FP8 weights + FP8 cache, which is almost indistinguishable from BF16 in quality and doubles concurrency capacity.

Measured quality loss (Llama 3.1 70B Instruct, reference)

A representative table for Llama 3.1 70B Instruct with a WikiText-2 calibration dataset (128 samples). Figures aggregated from public sources; they may vary by ±0.05 PPL and ±0.5 MMLU depending on implementation and seed.

Format	Model VRAM	Perplexity (WikiText-2)	MMLU (5-shot)	Relative speed (H100)
BF16 (baseline)	140 GB	4.85	82.1	1.00×
FP8 (E4M3)	70 GB	4.87 (+0.02)	81.6 (-0.5)	1.85×
INT8 SmoothQuant	70 GB	4.92 (+0.07)	81.0 (-1.1)	1.65×
INT4 AWQ	35 GB	4.99 (+0.14)	80.4 (-1.7)	2.50×
INT4 GPTQ	35 GB	5.05 (+0.20)	80.0 (-2.1)	2.40×
GGUF Q5_K_M	49 GB	4.94 (+0.09)	81.1 (-1.0)	n/a (llama.cpp)
GGUF Q4_K_M	42 GB	5.08 (+0.23)	79.8 (-2.3)	n/a (llama.cpp)
GGUF Q3_K_M	33 GB	5.45 (+0.60)	77.5 (-4.6)	n/a (llama.cpp)

Three lessons worth keeping:

FP8 is almost free in quality terms. If your hardware supports it, there is no debate.
INT4 AWQ is noticeably better than INT4 GPTQ in preserved quality, at comparable speed.
Q3 is already in measurable-loss territory; Q2 should no longer be used except for experiments or extreme demos.

Implications for on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace, no native FP8)

Llama 3.1 8B Instruct fits comfortably in BF16 (16 GB), but that leaves little headroom for a KV cache under concurrency. The usual sweet spot:

Llama 3.1 8B AWQ-INT4: ~5 GB of weights, 19 GB free for the KV cache → 4-8 concurrent sessions with a moderate context.
Llama 3 70B GGUF Q4_K_M: ~42 GB. It does not fit on a whole 4090; it requires CPU offload with llama.cpp (slow decode but workable for a single user).
Llama 3 70B AWQ-INT4 with TP=2 (two 4090s): ~17 GB per GPU → it fits and leaves headroom.

The 4090 does not support native FP8 (Ada Lovelace has the instruction but not Hopper’s accelerated throughput). In practice, FP8 on a 4090 works but without the speed gain: the sensible choice is INT4 AWQ.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here FP8 shines:

Llama 3.1 70B FP8 with TP=2: ~35 GB/GPU. Comfortable, leaving a huge amount of room for the KV cache → dozens of concurrent sessions.
Llama 3.1 405B FP8 with TP=4: ~200 GB/GPU. It just fits, with prefill+decode in the same pool.
Llama 3.1 405B INT4 AWQ with TP=2: ~100 GB/GPU. It allows serving the large model without saturating the cluster; there is headroom left for the cache and for serving another model at the same time.

The rule of thumb on an H100 cluster in 2026: FP8 if quality matters and the model fits; INT4 AWQ if the model does not fit in FP8 or if you want more concurrency at the cost of 1-2 MMLU points.

What we have not covered (upcoming articles)

Speculative decoding: the other big acceleration lever in inference. Orthogonal to quantization, it multiplies the speedup.
MoE quantization: Mixture-of-Experts models (Mixtral, DeepSeek V3, Qwen3-235B-A22B) have different quantisation patterns, experts are not quantised uniformly and there is dynamic routing.
Calibration dataset matters: how to choose the 128-512 calibration samples. The common mistake of grabbing a random dataset off the internet, and how to avoid it.
Multimodal quantization: vision-language models have heterogeneous layers (a CNN vision encoder, a transformer language model) that need separate treatment.

References

Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (ICLR 2023).
Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Han, S. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (MLSys 2024).
Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs (NeurIPS 2023). Introduces NF4 and double quantization.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., Han, S. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (ICML 2023).
NVIDIA. FP8 Formats for Deep Learning — E4M3/E5M2 white paper: https://arxiv.org/abs/2209.05433.
Rouhani, B. et al. Microscaling Data Formats for Deep Learning — MXFP4/MXFP8: https://arxiv.org/abs/2310.10537.
llama.cpp GGUF spec: https://github.com/ggerganov/llama.cpp/blob/master/docs/gguf.md.
vLLM quantization docs: https://docs.vllm.ai/en/latest/quantization/.

The OSS catalogue for LLMOps in six stages: tool by tool, what each one does and when to choose it

Sat, 23 May 2026 07:30:00 +0200

TL;DR

For each of the six LLMOps stages (Data, Tune, Eval, Deploy, Observe, Retrain) and the two cross-cutting components (prompt + data versioning), the open source ecosystem has canonical pieces that this blog has been citing again and again. This post gathers them in one place with cards of roughly 150 words per core tool: what it does, how it differs from the alternatives inside the same bucket, its licence and governance model, and a typical gotcha that you only learn in production. Plus more alternatives as bullets, a decision matrix per stage according to the case (small / large corpus, single tenant / multi-tenant…), a diagram of the connected OSS stack and a master table of licences / EE offerings. The intention: that the reader closes the post knowing what is available, which company maintains it, what gap each piece fills, and when to choose it. This is not opinion: it is a curated catalogue.

You are here: every stage, but by OSS column

This post shares a map with the two previous ones in the series, the six stages and the two cross-cutting pieces are all active, but it zooms in on the open source column.

The analogy: the electrician’s toolbox

A professional electrician arrives at an installation with a box organised into compartments. Nothing is improvised: for every type of cable there is a specific wire stripper, for every screw a screwdriver of the exact gauge, for every measurement a multimeter and a clamp meter, for every connection the right terminal block or connector. The difference between a professional electrician and a handyman is not that one knows more theory, the handyman has often read the manuals, it is that the right tool is within arm’s reach and he knows when to use each one. The day the specific stripper is missing, improvising with a box cutter breaks the insulation, leaves a badly terminated cable, and the panel comes back under warranty two months later.

The OSS LLMOps stack works the same way. For every canonical problem, versioning a dataset, indexing a corpus for retrieval, serving tokens with dynamic batching, propagating trace_id end-to-end, managing prompts with a production label, orchestrating retraining pipelines, there is a canonical piece of the open source ecosystem that solves it, maintained by a serious community or foundation, with a clear licence and a well documented gotcha. The consultant who knows which tool to use for each job builds a robust system in weeks; the one who improvises with “whatever the team already knows” pays later in operations, usually once the system is carrying real load and any replacement is expensive.

This post opens the toolbox and shows each card. It is not a user manual, that is what the deep-dive posts linked at the end are for; it is the curated catalogue.

Diagram of the connected reference OSS stack

The catalogue makes sense once you see how the pieces connect into a single coherent architecture, the one this blog has been describing throughout the series. The boxes do not float; they talk to each other over stable contracts (HTTP, gRPC, OTel, Kafka, S3/MinIO API).

Solid arrows mark data / control flow; the dashed blue ones are OTel traces. The K8s plane holds everything up. The control plane at the bottom is where the retraining pipelines, the evals in CI, the versioned prompts and the lineage live. The data plane on the left feeds both the serving side (RAG, configs) and the control plane (datasets, lineage). The observability plane receives from serving and from everything else.

Now we go stage by stage. Each one opens with a context paragraph, then cards for the core tools (about 150 words each), bullets with relevant alternatives, and a specific decision matrix at the end.

Stage 1 — Data + cross-cutting Data versioning

The Data stage solves three distinct problems that beginners confuse: versioning datasets (so that (dataset_id, version, hash) exists and propagates), storing and serving the operational corpus (object store + vector index + structured text), and moving it between systems with CDC and stable schemas. Covered in detail in the posts on data versioning with DVC and lakeFS, PostgreSQL + Qdrant in ingestion and RAG over Kafka.

DVC (Data Version Control)

DVC puts datasets under version control with the same discipline that git applies to code. The .dvc pointers live in git (plain text, about 200 bytes per dataset), the bulk content lives in a remote object store (S3, MinIO, Azure Blob, GCS). Each dvc add computes a SHA-256 hash of the dataset, uploads it to the remote and stores the pointer. The key line: the dataset_hash becomes the luggage tag that travels to the trainer, to experiment tracking and to lineage. The same dataset retrained twice produces the same hash, and therefore reproducible experiments. DVC integrates with MLflow and W&B as an input artefact. Gotcha: it works well for datasets that change by replacement (I swap train.jsonl for a new version) and worse for datasets with thousands of small files that change individually. For that case, combine it with lakeFS. Licence Apache 2.0, maintained by Iterative.ai since 2017. There is DVC Studio (managed) and dvc data (pure CLI) on different planes.

lakeFS

lakeFS brings git semantics (branch, commit, merge, rollback) to an entire S3/MinIO/ADLS bucket. Where DVC versions individual files as pointers in git, lakeFS versions the whole bucket: you can branch the corpus, ingest new data into the branch, validate that it passes checks (recall@10 over golden queries for embeddings, completeness for a tabular corpus), and only then merge into main. It is the piece that makes continuous RAG safe: the production corpus is always on main, updates are tested on branches. It offers hooks (pre-merge, pre-commit) that fire automatic validations, and time-travel to reproduce the state of the bucket at a past date. Gotcha: the manifest overhead on huge buckets (hundreds of millions of objects) deserves sizing; lakeFS keeps metadata in its own Postgres, not in the bucket. Licence Apache 2.0, maintained by Treeverse since 2020. Managed offering: lakeFS Cloud.

MinIO

MinIO is the S3-compatible object store that fills the “S3 on-premise” gap without surprises. API identical to S3 (the AWS SDKs work by pointing them at a different endpoint), its own CLI client (mc), erasure-coded mode for fault tolerance, bucket-to-bucket replication, encryption at rest. It is the base on which the other data plane components are built: DVC remote, lakeFS underlying storage, Postgres snapshots, MLflow artefacts, eval datasets, saved models, distributed KV cache fabric. In small deployments it runs single-node multi-disk; in serious ones, distributed clusters. Gotcha: the licence changed to AGPLv3 in 2021 (it was Apache 2.0 before), which means distributing software connected to MinIO obliges you to open the code that connects to it. For internal on-premise use this is not a problem; for a vendor packaging MinIO inside a commercial product, it is. Maintained by MinIO Inc. with a SUBNET enterprise offering and a community fork called AIStor launched in 2025.

Qdrant

Qdrant is the OSS vector database most aligned with this blog’s “RAG corpus per tenant with strict ACLs” pattern. Written in Rust, it exposes a REST + gRPC API, indexes with HNSW + scalar/binary quantisation to cut memory, supports efficient payload filtering (it is not post-filtering: it integrates the filter into the HNSW search), and allows collections isolated per tenant. For the multi-tenant chatbot scenario, Qdrant is where the tenant_<id>_kb_v3 collections with strict ACL live. It scales well horizontally (sharding by payload) and vertically (millions of chunks on a node with 64GB RAM). Gotcha: binary quantisation is aggressive, it cuts VRAM by 32× but degrades recall by 10-20 %; enabling it without re-tuning the threshold breaks retrieval silently. Licence Apache 2.0, maintained by Qdrant Solutions GmbH (Germany). There is Qdrant Cloud (managed) and EU-only support for ENS cases.

PostgreSQL + pgvector

Postgres 18 with the pgvector extension is the stack’s “hidden vector database”: when the corpus is small (under a million embeddings) and Postgres is already in production for operational data, running a separate Qdrant is expensive operations. pgvector adds a vector(dim) type, HNSW and IVF indexes, and the operators <->, <#>, <=> for cosine, L2 and dot product. Combined with tsvector (Postgres full-text search) it allows hybrid search, dense + sparse, in a single SQL query. Version 0.8 (2025) introduced halfvec and bit support to reduce size by 4×-8×. Gotcha: HNSW in pgvector consumes a fair amount of RAM to build the index (roughly 2× the size of the embeddings) and blocks inserts during the build; in production you build on a secondary, promote it, and discard the primary. Licence PostgreSQL License (permissive BSD-style) for both the core and pgvector. Maintained by the PostgreSQL Development Group plus pgvector by Andrew Kane + Crunchy Data + Neon.

Apache Kafka + Debezium

Kafka is the event bus where “everything that happens in the company is a stream” becomes real. For LLMOps in production it serves two functions: CDC from source systems (Debezium captures changes in Postgres / MySQL / MongoDB and publishes them as topics) and LLM event buffer (every request, every piece of feedback, every eval result ends up in a topic with the trace_id propagated). As the post on RAG over Kafka explains, the RAG corpus is kept fresh by capturing changes from the CMS / source system as CDC, running the embedding in Flink streaming, and ingesting into Qdrant continuously. Gotcha: a badly sized Kafka with long retention plus multi-client topics turns into a fast disk sink; measuring throughput per topic and key cardinality before production is mandatory. Kafka licence Apache 2.0 (ASF project); Debezium Apache 2.0 (project incubated by Red Hat). Kafka-compatible drop-in alternative: Redpanda (BSL, restricted commercial use).

Apache Flink (brief mention)

Flink processes streams with sub-second latency and exactly-once semantics. On the LLM plane it is used to: run embeddings in streaming (over CDC topics), aggregate online metrics, materialise features for retraining. Licence Apache 2.0, ASF. Common alternative: Spark Structured Streaming (also ASF, micro-batch latency).

More options for Data, mentioned on the blog:

Ceph — object store for large clusters with geo-distributed replication. LGPL/Apache licence, Red Hat / IBM.
Milvus — C++ vector database, an alternative to Qdrant; better for corpora of billions. Apache 2.0, Zilliz.
Karapace — Confluent-compatible OSS Schema Registry. Apache 2.0, Aiven.
DataHub / Apache Atlas / OpenMetadata — catalogue + lineage. Apache 2.0, Acryl Data / ASF / Collate respectively.
OpenLineage — cross-system lineage event standard. Apache 2.0, Linux Foundation AI&Data.

Decision matrix — Data:

If your case is	Choose
Corpus < 1M embeddings, you already have Postgres	pgvector (one component fewer)
Corpus 1M-100M, multi-tenant with ACL	Qdrant (integrated filtering, ACLs per collection)
Corpus > 100M, aggressive sharding	Milvus (scales linearly better into the billions)
Training datasets + experiment tracking	DVC over MinIO + MLflow integration
RAG corpus with controlled releases	lakeFS over MinIO + pre-merge hooks
You want both	DVC + lakeFS as complements (the blog’s recommendation)

Stage 2 — Tune

The Tune stage produces a new model_id, model_version, typically a LoRA adapter over a stable base, with lineage back to the dataset and experiment tracking to reproduce it. Detail in the post on continuous fine-tuning.

HuggingFace Transformers + PEFT

transformers is the canonical library for loading and training models of the decoder-only family (Llama, Mistral, Qwen, Gemma…) and encoder-decoder. peft (Parameter-Efficient Fine-Tuning) is the complement that adds declarative support for LoRA, QLoRA, IA3 and assorted adapters. Together they form the mandatory core of the OSS Tune stack: any higher-level framework (Axolotl, LLaMA-Factory) uses them underneath. PEFT allows you to train an adapter of about 280 MB (order of magnitude) instead of a full model of about 140 GB, with a functionally equivalent result on most style / domain adjustment tasks. Gotcha: PEFT with a badly configured target_modules trains an adapter that covers only Q and V of the attention, leaving out key, output proj and MLP. The result looks trained but performs poorly; adding target_modules=["all-linear"] fixes it (at the cost of a larger adapter). Licence Apache 2.0, maintained by Hugging Face SAS (a French company); open governance model with active external maintainers.

bitsandbytes

bitsandbytes implements weight quantisation to 8-bit and 4-bit with NF4 for models loaded with transformers. It reduces the 140 GB of Llama 3 70B FP16 to about 40 GB in NF4, allowing QLoRA training on a single H100 80GB. The trick is that the weights stay quantised in memory while the sensitive computations (attention, gradient updates in the adapter) are done in FP16/BF16 with on-the-fly dequantisation. Ideal for fine-tuning on limited hardware and for serving with vLLM when you want to cut VRAM. Gotcha: NF4 quantisation is lossy; on small models (< 7B) the quality degradation is noticeable. For production serving of models under 7B, INT8 is preferred (more memory, less loss) or FP8 if the hardware supports it (the H100 does). Licence MIT, maintained by Tim Dettmers (originally at U. Washington, now with support from Anthropic and HuggingFace).

MLflow Tracking

MLflow is the reference OSS experiment tracking tool: each trainer run records parameters (lr, batch size, epochs, target_modules), metrics (loss curves, eval scores), artefacts (model, tokeniser, configs) and, crucially, input artefacts (dataset_id, dataset_hash, parent_run). The model registry associates each model_version with a reproducible run_id. The line of continuity between Tune and Deploy passes through here: the deployment reads the model to serve from the registry, with its lineage made explicit. MLflow 2.x integrates MLflow Prompts (prompt registry) and MLflow Tracing (OTel-compatible spans), cutting the number of components needed. Gotcha: the default backend store is SQLite, which works for personal experiments and breaks on a shared cluster. In production: Postgres as the backend store + MinIO/S3 as the artefact store. Licence Apache 2.0, maintained by LF AI & Data (donated by Databricks in 2020).

Axolotl

Axolotl wraps transformers + PEFT + bitsandbytes + DeepSpeed + FSDP in a declarative YAML configuration: instead of writing a script of about 300 lines to configure a fine-tuning run, you define a config.yml with base model, dataset path, LoRA config and training hyperparameters, and run it in one line. It supports Llama, Mistral, Qwen, Gemma, Phi… loads. It keeps compatibility with the HuggingFace Hub to download models and datasets, and with MLflow / W&B for tracking. It is the convenience framework this blog cites when it talks about “productive fine-tuning without reinventing the wheel”. Gotcha: the pace of change in the community is fast; a config.yml that worked six months ago can break with a current version because of internal refactors. Pinning the exact Axolotl version in the environment mitigates this. Licence Apache 2.0, maintained by the OpenAccess AI Collective (community-driven). A very similar alternative, more widely used in China: LLaMA-Factory (Apache 2.0, Beihang U.).

Ray Train

Ray Train scales fine-tuning to multiple nodes by distributing workers across a Ray cluster. While DeepSpeed and FSDP are intra-job parallelism (several GPUs collaborating on one job), Ray Train is the orchestration plane that builds the cluster, launches workers, manages checkpoints, recovers from node failures, and integrates with Slurm or Kubernetes. For training runs beyond 8 GPUs on changing clusters, Ray Train avoids the operational burden of “manually launching N torchrun processes with NCCL”. It combines with MLflow for tracking. Gotcha: the Ray learning curve is real; for a single node with 4-8 GPUs, torchrun or Hugging Face Accelerate are simpler. Ray Train shines when there are N changing nodes. Licence Apache 2.0, maintained by Anyscale Inc. (commercial backer) + community. A more K8s-native alternative: Kubeflow Training Operator (Apache 2.0, LF AI & Data).

More options for Tune:

DeepSpeed — ZeRO parallelism in 3 stages, mixed precision, CPU/NVMe offload. MIT, Microsoft.
FSDP (Fully Sharded Data Parallel) — native PyTorch parallelism, an alternative to DeepSpeed. BSD, Meta.
LLaMA-Factory — equivalent to Axolotl with a focus on the Llama family. Apache 2.0, Beihang University.

Decision matrix — Tune:

If your case is	Choose
Fine-tune on 1 GPU with 24GB (RTX 4090)	QLoRA with bitsandbytes NF4 + Axolotl
Fine-tune on 1 H100 80GB, models < 13B	LoRA bf16 + Axolotl
Fine-tune on 4-8 GPUs, single node	transformers + PEFT + Accelerate + MLflow
Multi-node fine-tune on a K8s cluster	Kubeflow Training Operator or Ray Train
Reproducible tracking is mandatory	MLflow + DVC input artefact
You want the minimum viable	Axolotl + MLflow

Stage 3 — Eval + Guardrails

Eval validates candidates pre and post promotion against a golden set with operational metrics; Guardrails runs safety online. Detailed in the posts on evals and guardrails.

DeepEval

DeepEval is the “pytest-style” OSS eval suite: you define tests with assertions over faithfulness, answer relevancy, contextual precision, hallucination rate, summarisation quality… and run them in CI. Each metric is an evaluator: some rule-based, others LLM-as-judge with auditable prompts. The philosophy is “evals as unit tests”: parameterisable by dataset, failable in CI, integrable with GitHub Actions. Gotcha: LLM-as-judge metrics vary between judge model versions, so if the judge moves up a version, the thresholds lose their earlier statistical meaning. Explicit pinning of the judge model in config plus periodic threshold recalibration is mandatory discipline. Licence Apache 2.0, maintained by Confident AI (a company); parallel commercial SaaS offering. Comparable: TruLens (MIT, TruEra) and G-Eval (academic).

RAGAS (RAG Assessment)

RAGAS specialises in evaluating RAG pipelines. It defines four canonical metrics: faithfulness (the answer is supported by the retrieved chunks), answer relevancy (the answer addresses the query), context precision (the retrieved chunks are relevant), context recall (all the relevant chunks were retrieved). Each metric is computed with LLM-as-judge over a dataset of (query, context, expected answer). For a RAG system, RAGAS is the evaluator that measures whether retrieval is aligned with generation. It integrates with Langfuse and MLflow to store results. Gotcha: RAGAS works well with golden sets of under 1000 examples; over huge golden sets the cost of the LLM judge per evaluation explodes, and the practice is to sample. Licence Apache 2.0, maintained by Exploding Gradients (the authors’ company).

Promptfoo

Promptfoo is the declarative, CI-oriented evaluator: in promptfooconfig.yaml you define a set of prompts and a set of assertions (contains text X, does not contain Y, faithfulness > 0.8, judge approves…), point at a provider (OpenAI compatible, vLLM, Ollama…), and promptfoo eval runs the matrix of prompts × providers × assertions, returns a diff against the baseline and fails CI if anything regresses. It is the most “DevOps-friendly” piece of the evals ecosystem: it integrates trivially with GitHub Actions, GitLab CI or Jenkins. Gotcha: assertion thresholds have to be calibrated with real data; starting at a default > 0.5 produces false positives that erode the team’s trust. Calibrate after the first week. Licence MIT, maintained by Promptfoo, Inc. (a company); a commercial Promptfoo Cloud SaaS offering exists but the OSS version is complete.

NeMo Guardrails

NeMo Guardrails is NVIDIA’s framework for defining and enforcing policies in LLM systems through a DSL called Colang. It lets you express rules such as “if the user asks about topic X, answer with template Y” or “if the model tries to do Z, block it” in a conversational-script syntax rather than in Python. It runs as middleware between the app and the model: input rails (validate what comes in), output rails (validate what goes out), dialog rails (control the flow). Designed for complex multi-turn systems where the policies are nontrivial. Gotcha: Colang adds latency per turn (roughly 50-200 ms depending on the policy graph); for high-throughput conversational chat, dialog rails are disabled and only input + output remain. Licence Apache 2.0, maintained by NVIDIA.

Microsoft Presidio

Presidio is the most mature OSS PII (Personally Identifiable Information) detector in the ecosystem. It detects DNI, NIE, IBAN, phone numbers, emails, physical addresses, credit card numbers, personal names, dates of birth… with recognisers based on regex + NER (spaCy) + custom validators. It supports redaction (replace with placeholders), masking (asterisks) or deterministic anonymisation (repeatable hash). For ENS/NIS2 scenarios, it is the piece placed in front (on input) and behind (on output) the LLM to guarantee that no PII is processed or emitted. Gotcha: the built-in recognisers cover English well and the rest badly; for Spanish, Catalan and Basque you have to add custom recognisers, which is disciplined work but doable. Licence MIT, maintained by Microsoft.

More options for Eval:

Phoenix Arize OSS — combines tracing + evals, an alternative to Langfuse Evals. ELv2, Arize AI.
lm-eval-harness — academic suite with standard benchmarks (MMLU, HellaSwag…). MIT, EleutherAI.
HELM — holistic academic evals. Apache 2.0, Stanford CRFM.
Guardrails AI — a pythonic alternative to NeMo Guardrails. Apache 2.0, Guardrails AI Inc..
LlamaGuard / PromptGuard / ShieldGemma — safety models, not frameworks. Open weights, Meta / Google.

Decision matrix — Eval + Guardrails:

If your case is	Choose
Eval in CI, “pytest for LLMs” style	Promptfoo + GitHub Actions
Eval specific to a RAG pipeline	RAGAS + Langfuse datasets
General eval with custom metrics	DeepEval + MLflow dataset
Dialog policy with declarative rules	NeMo Guardrails (Colang)
PII redaction in/out only	Presidio (you do not need NeMo)
Open safety model in Spanish	LlamaGuard 3 or ShieldGemma

Stage 4 — Deploy

Deploy serves tokens to the user with predictable throughput and latency, adapter hot-swap and multi-tenancy where it applies. Covered in the posts on vLLM on K8s, LLM operators, multi-tenant cluster, KV cache, PagedAttention and disaggregated serving.

vLLM

vLLM is the reference OSS inference engine. It implements PagedAttention (KV cache paging in the style of virtual memory, avoiding fragmentation), continuous batching (requests join the batch as they arrive, rather than waiting for the next batch), prefix caching (common prefixes, system prompts, do not recompute the KV cache), LoRA hot-swap (--enable-lora allows loading and unloading adapters without restarting the engine), an OpenAI-compatible API, and disaggregated prefill/decode support since 2025. It covers almost everything from Llama 3 / Mistral / Qwen / DeepSeek. Gotcha: maximum throughput is only reached with --max-num-seqs and --gpu-memory-utilization tuned for the specific model and hardware; the defaults are conservative. The initial tuning session pays off: 2-3× throughput. Licence Apache 2.0, originated at UC Berkeley, today maintained by the vLLM Project / LF AI & Data plus a broad community (Red Hat, NVIDIA, AWS, IBM contribute). Serious alternatives in the same bucket: TGI (Apache 2.0, Hugging Face), SGLang (Apache 2.0, LMSys), TensorRT-LLM (Apache 2.0, NVIDIA, requires conversion).

KServe

KServe is the Kubernetes operator for serving ML models, LLMs included, in a declarative pattern: you define an InferenceService YAML with the model and predictor (which can be vLLM, TGI, Triton, or a custom container) and KServe handles scheduling onto GPU nodes, autoscaling (scale-to-zero included), traffic splitting for canary, and model registry integration. It is the layer that standardises “how a model is deployed on K8s” across multiple engines, instead of inventing engine-specific YAML. It supports multi-model with Inference Graphs (chaining preprocessor → model → postprocessor) and integrates with KEDA/Karpenter for GPU pool autoscaling. Gotcha: scale-to-zero on GPU works badly in practice because the warm-up (loading weights into VRAM) takes tens of seconds; minReplicas: 1 is better. Licence Apache 2.0, maintained by Kubeflow / LF AI & Data. Alternatives: KubeRay (Apache 2.0, Anyscale), llm-d (Apache 2.0, CNCF), KAITO (MIT, Microsoft Azure).

Triton Inference Server

Triton serves heterogeneous models on a single backend: LLMs (via the vLLM or TensorRT-LLM backend), traditional models (ONNX, TorchScript, TensorFlow), custom models. For systems that mix LLM inference with traditional classifiers, embedding encoders, reranking models, OCR and so on, Triton avoids having N different engines in N pods. It supports ensemble models (chaining models in a single request), dynamic batching, model versioning, model warmup. Gotcha: Triton is flexible but heavy to operate; for systems serving only LLMs, vLLM directly is simpler and more optimised. Triton shines when there is real heterogeneity. Licence BSD-3-Clause, maintained by NVIDIA.

Envoy AI Gateway

Envoy AI Gateway is the CNCF ecosystem’s “LLM-aware API gateway”. Built on Envoy Proxy, it adds knowledge of the OpenAI-compatible APIs (chat completions, embeddings, etc.), routing across multiple backends (local vLLM + OpenAI + Anthropic + Bedrock), token-based rate limiting (limits by tokens per minute, not by requests), intelligent retries, fallback between providers, and built-in OTel observability. It is the piece that makes “AI Gateway” real as an architectural category. Gotcha: integration with authentication (OIDC, JWT) is flexible but requires detailed Envoy configuration; an “out of the box” AI Gateway with no configuration produces an Envoy that passes everything through. Licence Apache 2.0, maintained by CNCF since the initial donation from Tetrate. Alternatives: LiteLLM Proxy (MIT, BerriAI), Portkey (MIT, Portkey AI), Kong AI Gateway (Apache 2.0 base + EE, Kong Inc.).

llama.cpp

llama.cpp serves LLMs on CPUs (and Apple Silicon, GPUs via Vulkan/Metal/CUDA) with very aggressive quantisation (GGUF format, down to 2-bit). It is the canonical option for inference on hardware without a dedicated GPU: edge devices, workstations, development machines. It covers everything from small models (Phi-3, Gemma 2B) to Llama 70B on hardware with enough RAM. Gotcha: latency on CPU is orders of magnitude worse than on a dedicated GPU, so it is useful for offline evals, drift checks and local development, not for production serving under real load. Licence MIT, maintained by Georgi Gerganov + community.

More options for Deploy:

TensorRT-LLM — maximum optimisation on NVIDIA Hopper/Ada. Apache 2.0, NVIDIA.
SGLang — good for workloads with structured generation and JSON. Apache 2.0, LMSys.
TGI — a mature alternative, focused on the HuggingFace ecosystem. Apache 2.0, HuggingFace.
NVIDIA Dynamo — multi-node disaggregated serving. Apache 2.0, NVIDIA.
llm-d — K8s operator specific to LLMs. Apache 2.0, CNCF.

Decision matrix — Deploy:

If your case is	Choose
Production serving on NVIDIA H100/A100	vLLM (the safe default)
Absolute throughput squeezing on Hopper	TensorRT-LLM + vLLM plugin or standalone
Edge / local dev without GPU	llama.cpp
Multi-model (LLM + classifiers + encoders)	Triton with the vLLM backend
Declarative K8s with autoscaling	KServe + vLLM as predictor
AI Gateway with token rate limiting	Envoy AI Gateway
Multi-node disaggregated GPU cluster	NVIDIA Dynamo over vLLM

Stage 5 — Observe

Observe propagates trace_id end-to-end, emits runtime metrics, runs an LLM judge over sampling and detects drift. Detailed in tracing with AgentSight, MCP observability with OTel and eBPF + drift.

OpenTelemetry Collector

The OTel Collector is the agent that receives traces, metrics and logs in OTel format (or in any other via receivers), processes them (filters, sampling, attribute enrichment, PII redaction), and routes them to one or several backends (Tempo, Jaeger, Prometheus, Loki, Langfuse…). It is the piece that decouples the apps from the observability backend: moving from Tempo to Jaeger means changing the Collector’s exporter, not the app. For LLMOps it matters especially because the OTel GenAI semantic conventions spec defines the attributes gen_ai.request.model, gen_ai.prompt.version, gen_ai.response.tokens and so on, which stitch the trace_id to the system’s lineage. Gotcha: the Collector configuration tends to grow; without discipline and periodic review it ends up as an 800-line YAML that nobody understands. Modularising with extensions helps. Licence Apache 2.0, maintained by the CNCF / OpenTelemetry Project.

Tempo (traces) + Jaeger

Grafana Tempo is the distributed trace backend optimised for cost: it uses an object store (S3/MinIO) instead of Elasticsearch, deduplicates by trace_id, and integrates natively with Grafana for visualisation. For LLMOps, where a real request generates 10-30 spans (gateway, prompt pull, RAG retrieval, prefill, decode N times, scoring), Tempo copes with high volumes at a reasonable cost. Jaeger is the more established CNCF alternative, better for cases under 100k traces per day, worse for native object store. Gotcha: Tempo has no traditional indexing; searches like “traces that took more than 5s and touched tenant X” require TraceQL + Grafana and are not as fast as Jaeger with Elasticsearch. For immediate ad-hoc diagnosis it is worth keeping a parallel Jaeger with aggressive sampling. Licences AGPL 3.0 (Tempo) and Apache 2.0 (Jaeger), maintained by Grafana Labs and CNCF respectively.

Prometheus + Grafana

Prometheus is the time-series metrics foundation of the ecosystem. Pull model (it scrapes /metrics endpoints), PromQL for queries, exporters for everything (Postgres, Kafka, NVIDIA GPU via dcgm-exporter, vLLM natively). Grafana visualises Prometheus + Tempo + Loki on a single plane. For LLMOps, the critical metrics are gpu_utilization, kv_cache_usage_pct, tokens_per_second, prefill_latency_p95, decode_latency_p95, queue_depth, aggregated by tenant. Gotcha: Prometheus is very good up to about 1M active series; above that it is worth moving to Thanos or Mimir for long retention and horizontal scalability. For the blog’s typical LLM cluster (4-8 H100), Prometheus alone is enough. Licences Apache 2.0 (Prometheus, CNCF) and AGPL 3.0 (Grafana 10+, Grafana Labs).

Langfuse

Langfuse is the LLM-specific OSS observability + prompt management tool. It captures spans with LLM semantic conventions (input, output, model, tokens, latency, score, user_id, session_id), visualises them as conversational traces (not just span trees), manages prompts versioned with a production label and allows curated datasets + evals from the same UI. For serious LLMOps, Langfuse fills the gap that neither Tempo nor Jaeger covers: a tracing UI designed to be LLM-first. Gotcha: Langfuse maintains its own store (Postgres + ClickHouse for high volume); on large clusters the operational side of ClickHouse deserves attention. To get started, Postgres alone copes. Licence MIT for the OSS core, EE Enterprise Edition with additional features (SSO, audit logs, advanced RBAC). Maintained by Langfuse GmbH (Berlin, German). There is Langfuse Cloud (SaaS).

Phoenix Arize OSS

Phoenix is Arize AI’s OSS for LLM observability + evals, an alternative to Langfuse with a different emphasis: more oriented to evaluation and visual debugging (embedding drift, cluster analysis), less to prompt management. A good pairing with Langfuse when you want a dual approach: Langfuse for “production conversational traces”, Phoenix for “exploratory investigation of model behaviour”. Gotcha: Phoenix duplicates functionality with Langfuse and with MLflow; having all three in production multiplies the operational burden. Pick one as the main tool and the others as complements. Licence Elastic License 2.0 (not strictly OSI), maintained by Arize AI.

Cilium Tetragon + Hubble

Tetragon (eBPF runtime security observer) and Hubble (eBPF network observer) are the low-level pieces that give real runtime visibility into the cluster: which processes run in which pods, what syscalls they make, what network connections they open, in real time. For ENS/NIS2 environments that demand “prove what ran in production”, Tetragon is the irrefutable audit layer: every process execution with its parent, its capabilities, its K8s context. Hubble visualises network flows by pod, namespace, service. Gotcha: the volume of events generated is high; without kernel-side filtering (which Tetragon supports with TracingPolicy), it saturates the observability plane fast. Discipline in policies. Licence Apache 2.0 for both, maintained by Cilium / CNCF / Isovalent.

Evidently AI

Evidently is the OSS library for drift detection: it compares input and output distributions between two time windows (training vs production, current week vs previous week), applies statistical tests (KS, PSI, Wasserstein, chi-square) and generates HTML reports. For LLMOps it detects when the distribution of prompts changes (new topics, new lengths, new languages) or when the model starts answering shorter / longer / differently. Gotcha: Evidently is oriented to tabular data and embeddings; for raw text it is best combined with an encoder embedder that produces vectors before applying the tests. Licence Apache 2.0, maintained by Evidently AI (a company). Alternatives: NannyML (Apache 2.0, NannyML BV), Alibi Detect (Apache 2.0, Seldon).

More options for Observe:

Loki — Prometheus-style logs backend for Grafana. AGPL 3.0, Grafana Labs.
Pixie — auto-instrumented eBPF observability. Apache 2.0, CNCF.

Decision matrix — Observe:

If your case is	Choose
Minimum viable stack	OTel Collector + Tempo + Prometheus + Grafana + Langfuse
Traces with strong ad-hoc search	Add Jaeger with aggressive sampling
ENS / NIS2 compliance runtime audit	Tetragon + Hubble + mandated retention
Exploratory investigation of the model	Phoenix Arize OSS alongside Langfuse
Statistical drift detection	Evidently over embeddings + inputs
Cluster beyond 1M Prometheus series	Mimir (Grafana Labs) or Thanos

Stage 6 — Retrain + cross-cutting pieces

Retrain closes the loop feedback → triage → enriched dataset → new adapter. Prompt versioning and data versioning stitch lineage across stages. Detailed in retrain, prompt versioning and data versioning.

Apache Airflow

Airflow is the most established OSS DAG scheduler. You define workflows as Python code (DAGs), each DAG with tasks (operators) that run according to declared dependencies plus a cron schedule. For retraining: a weekly DAG that extracts feedback from Postgres, triages it with an LLM-as-classifier, enriches the enriched dataset in DVC, launches the fine-tuning job on Kubernetes, runs evals against the golden set, and promotes if it passes the gates. There is a huge ecosystem of operators for everything (S3, Postgres, Kafka, Slack, K8s, Spark…). Gotcha: Airflow 2.x improved a great deal over the chaos of 1.x, but the scheduler is still a component that deserves operational attention (Postgres backend, executor pool, sidecar workers); for simple pipelines it is over-engineering. Licence Apache 2.0, maintained by the ASF.

Argo Workflows

Argo Workflows is the K8s-native equivalent of Airflow: each step is a container, the DAGs are defined as K8s YAML, the executor is Kubernetes itself. For environments where everything is K8s, Argo fits without an extra component to maintain. Long tasks (a six-hour fine-tuning run) execute as Pods that survive control plane failures. It integrates trivially with Kubeflow Pipelines (which is built on top). Gotcha: Argo’s YAML syntax is verbose; for complex DAGs, Argo feels less productive than Airflow in Python. Solutions: Hera (Python DSL for Argo, a DataBricks contribution) or Argo + custom CRDs. Licence Apache 2.0, maintained by CNCF.

Kubeflow Pipelines

Kubeflow Pipelines is the layer above Argo Workflows oriented specifically to ML: artefact tracking, experiment tracking, reusable pipeline templates, versioned components. Built on Argo, it adds the ML conceptual model (input artefact, output artefact, metrics) that raw Argo does not have. For cyclical retraining on a K8s cluster, it is the most “ML-ready” option in the OSS ecosystem. Gotcha: Kubeflow as a full suite is heavy (10+ components); many organisations install only Pipelines + Training Operator + Katib and skip Notebook Server / legacy KFServing. Licence Apache 2.0, maintained by CNCF / LF AI & Data.

Feast

Feast is the most widely used OSS feature store. It defines feature views over batch sources (BigQuery, Postgres, Parquet) and online sources (Redis, DynamoDB, Postgres with an extension), exposes a consistent API for read-during-training and read-during-inference (point-in-time correctness), and guarantees that the features of the model in production are the same ones it was trained with. For LLMOps where the model needs consistent user / session / context features (latest plan, tenure as a customer, recent tickets), Feast provides the discipline. Gotcha: for many pure LLM systems (a RAG chatbot with no complex features), Feast is over-engineering, Postgres is enough. When there are real features (recommendation, scoring, ranking), Feast shines. Licence Apache 2.0, maintained by LF AI & Data.

Argilla

Argilla is the OSS annotation + HiL (human-in-the-loop) platform most aligned with modern LLMOps. It creates annotation projects with templates (classification, ranking, span annotation, RLHF preference, free-form text), connects to HuggingFace datasets, and integrates with Langfuse to import traces from production as cases to annotate. It supports multiple annotators with reconciliation, kappa scoring and quality control. To enrich retrain datasets with cases from the “blunt tone” cluster of the Retrain post, Argilla is the front end. Gotcha: Argilla requires Elasticsearch for production performance; for small experiments SQLite is enough. Licence Apache 2.0, maintained by Argilla, Inc. (acquired by Hugging Face in 2024). Alternative: Label Studio (Apache 2.0, HumanSignal), more generalist, less LLM-first.

Langfuse Prompts + MLflow Prompt Registry

Langfuse Prompts manages prompts as versioned entities with labels (production, staging, experiment). The client reads the active prompt from Langfuse on the request path (with a local cache of a few seconds) and propagates prompt_id, prompt_version to the OTel span, exactly as the forensic post does. MLflow Prompt Registry does the same with a slightly different conceptual model (no labels-as-pointers; it uses stages, like the Models registry). Both are valid; the choice depends on which tracking tool is already in place. Gotcha (Langfuse): labels are mutable, so changing production points at another version with no explicit audit; prompts are better deployed via PR against the configs repo, not manually in the UI. Licences and governance covered above.

More options for Retrain + cross-cutting pieces:

Prefect — “modern” Python DAGs, an alternative to Airflow. Apache 2.0, Prefect Tech.
Dagster — DAGs with a strong focus on data assets. Apache 2.0, Dagster Labs.
Label Studio — generalist annotation. Apache 2.0, HumanSignal.
OpenLineage — cross-system lineage event standard. Apache 2.0, LF AI & Data.
DataHub / Apache Atlas / OpenMetadata — catalogue + lineage with a UI. Apache 2.0.

Decision matrix — Retrain + cross-cutting pieces:

If your case is	Choose
Simple pipelines with a catalogue of operators	Airflow
Everything is K8s, minimal components	Argo Workflows
ML pipelines with artefact tracking	Kubeflow Pipelines
HiL annotation for LLM retraining	Argilla + Langfuse integration
Features shared between training and inference	Feast
No complex features, only prompts + LLM	Skip Feast
Lightweight prompt registry	Langfuse Prompts
MLflow is already centralised	MLflow Prompt Registry

Master table: licence, governance and enterprise offering

Tool	Licence	Governance / maintainer	Commercial EE / SaaS
DVC	Apache 2.0	Iterative.ai	DVC Studio
lakeFS	Apache 2.0	Treeverse	lakeFS Cloud
MinIO	AGPL v3	MinIO Inc.	SUBNET / AIStor
Qdrant	Apache 2.0	Qdrant GmbH	Qdrant Cloud
pgvector	PostgreSQL License	Andrew Kane + community	— (built-in Postgres clouds)
PostgreSQL	PostgreSQL License	PostgreSQL Global Dev Group	several managed (Crunchy, Neon, Aiven, EDB)
Apache Kafka	Apache 2.0	ASF	Confluent Cloud
Debezium	Apache 2.0	Red Hat / ASF	Debezium Server / Confluent Connectors
Apache Flink	Apache 2.0	ASF	Ververica Platform, Aiven
HF Transformers	Apache 2.0	Hugging Face SAS	HF Inference Endpoints / Enterprise Hub
PEFT	Apache 2.0	Hugging Face SAS	— (part of the HF offering)
bitsandbytes	MIT	Tim Dettmers + community	—
MLflow	Apache 2.0	LF AI & Data	Databricks MLflow
Axolotl	Apache 2.0	OpenAccess AI Collective	—
Ray (Train)	Apache 2.0	Anyscale + community	Anyscale Platform
DeepSpeed	MIT	Microsoft	—
DeepEval	Apache 2.0	Confident AI	Confident AI SaaS
RAGAS	Apache 2.0	Exploding Gradients	—
Promptfoo	MIT	Promptfoo, Inc.	Promptfoo Cloud
NeMo Guardrails	Apache 2.0	NVIDIA	NeMo Microservices
Presidio	MIT	Microsoft	—
Phoenix (Arize)	Elastic v2	Arize AI	Arize Platform
vLLM	Apache 2.0	vLLM Project / LF AI & Data	several (Red Hat, AWS, IBM, NVIDIA)
TGI	Apache 2.0	Hugging Face SAS	HF Inference Endpoints
SGLang	Apache 2.0	LMSys + community	—
TensorRT-LLM	Apache 2.0	NVIDIA	NVIDIA AI Enterprise
llama.cpp	MIT	Georgi Gerganov + community	—
Triton Inference Server	BSD-3	NVIDIA	NVIDIA AI Enterprise
KServe	Apache 2.0	LF AI & Data (Kubeflow)	—
Envoy AI Gateway	Apache 2.0	CNCF / Tetrate	Tetrate Service Bridge
LiteLLM	MIT	BerriAI	LiteLLM Cloud
OpenTelemetry	Apache 2.0	CNCF	several vendors (Honeycomb, Datadog, Grafana)
Tempo	AGPL 3.0	Grafana Labs	Grafana Cloud Tempo
Jaeger	Apache 2.0	CNCF	—
Prometheus	Apache 2.0	CNCF	Grafana Cloud, AMP, GCP Managed Prom, Azure
Grafana	AGPL 3.0	Grafana Labs	Grafana Cloud, Grafana Enterprise
Loki	AGPL 3.0	Grafana Labs	Grafana Cloud Loki
Langfuse	MIT (core) / EE	Langfuse GmbH	Langfuse Cloud
Tetragon	Apache 2.0	Cilium / CNCF / Isovalent	Isovalent Enterprise
Hubble	Apache 2.0	Cilium / CNCF	Isovalent Enterprise
Evidently AI	Apache 2.0	Evidently AI	Evidently Cloud
Apache Airflow	Apache 2.0	ASF	Astronomer, MWAA, Cloud Composer
Argo Workflows	Apache 2.0	CNCF	—
Kubeflow Pipelines	Apache 2.0	CNCF / LF AI & Data	—
Feast	Apache 2.0	LF AI & Data	Tecton (commercial)
Argilla	Apache 2.0	Hugging Face	HF Hub features
OpenLineage	Apache 2.0	LF AI & Data	—
DataHub	Apache 2.0	Acryl Data	Acryl Cloud

The pattern to watch for when reading the table: AGPL 3.0 and Elastic v2 are the ones that add the most friction in companies with strict licensing policies (legal asks for a specific review). The Apache 2.0 ones pass compliance without argument. Those with an “EE Enterprise” tier or equivalent hide a decision: the OSS version is functionally complete for production, but team features (SSO, audit, advanced RBAC) live in the commercial version. For ENS customers under a HIGH declaration, the EE features (SSO with corporate SAML/OIDC, immutable audit logs) are usually mandatory, so it is worth knowing the price beforehand.

When to move from the “minimum stack” to the “full stack”

The whole catalogue can be intimidating. But you do not build all of it on day one. There is a reasonable order that this blog has been validating in posts throughout the series. The minimum viable stack that serves an LLM API with acceptable discipline:

Serving: vLLM on Kubernetes plus an Envoy AI Gateway in front.
Data: Postgres + pgvector (no Qdrant), MinIO for object store, no Kafka.
Tune: Axolotl + MLflow, no Ray Train.
Eval: Promptfoo in CI, no RAGAS and no judge in production.
Observe: OTel Collector + Prometheus + Grafana + Langfuse, no Phoenix and no Tetragon.
Retrain: feedback in Postgres + crontab scripts, no Airflow.
Versioning: prompts in Langfuse + datasets in DVC over MinIO, no lakeFS.

That is about 8-10 components and it serves a reasonable LLM system for a single tenant with moderate traffic. As the system grows, there are identifiable moments where adding each piece pays off:

Trigger	Component to add
Multi-tenant with isolated corpora	Qdrant (collections per tenant, ACL)
The corpus is refreshed often and periodically breaks	lakeFS (branches with pre-merge hooks)
The embedding pipeline needs streaming	Kafka + Debezium + Flink
Retraining moves from monthly to weekly	Airflow or Argo Workflows
Shared features appear (customer profile, scoring)	Feast
Annotation outgrows informal capacity	Argilla
RAG eval needs specific metrics	RAGAS + Langfuse datasets
ENS compliance demands runtime audit	Tetragon + Hubble
Drift is invisible and shows up late	Evidently
A single stack stops covering multi-model	Triton or KServe with several predictors
Multiple simultaneous multi-tenant adapters	vLLM Production Stack + a dedicated Operator

Each jump adds 1-2 components and is worth the cost only when the trigger is clear. Adding Kafka “just in case” when the corpus is updated once a month is net negative work.

What we have not covered (yet)

Some pieces still deserve a post of their own:

Schema Registry for LLM data and prompts (Confluent OSS, Karapace, JSON Schema Registry).
Catalogue + lineage in depth: DataHub vs Atlas vs OpenMetadata + OpenLineage taken seriously.
Federated learning over OSS (Flower, FedML) for scenarios where the data is not centralised.
OSS MCP Servers and their place in the stack as a tools / actions layer.
“Agentic” evals specific to multi-step systems with tool use.
Upgrade best practices for each component (vLLM every 6 weeks, Kafka major every 18 months, etc.).

References

vLLM · vLLM Production Stack · TGI · SGLang · TensorRT-LLM · llama.cpp — OSS inference engines.
Triton Inference Server · KServe · Envoy AI Gateway · LiteLLM — orchestration and AI gateway.
Qdrant · pgvector · Milvus — vector databases.
DVC · lakeFS · MinIO — versioning and object store.
Apache Kafka · Debezium · Apache Flink — streams and CDC.
Hugging Face Transformers · PEFT · bitsandbytes · Axolotl — fine-tuning.
MLflow · Ray Train · Kubeflow Training Operator — training orchestration.
DeepEval · RAGAS · Promptfoo · NeMo Guardrails · Presidio — evals and guardrails.
OpenTelemetry · OTel GenAI semconv · Tempo · Prometheus · Grafana · Loki — observability foundation.
Langfuse · Phoenix Arize · Evidently AI — LLM observability and drift.
Cilium · Tetragon · Hubble — eBPF runtime.
Apache Airflow · Argo Workflows · Kubeflow Pipelines · Feast · Argilla — orchestration + retrain + annotation.

Vllm on lo0 — Tech Blog

Sizing for agents: 275 input tokens for every output token, and why the bottleneck is still decode

TL;DR

You are here: the arithmetic that comes before buying

The analogy: the repair shop and the front desk

Part 1. The real load profile

The three sources

The figures you can actually use

What to measure before calculating

Part 2. Prefill, decode and the cache trap

Where it falls over

Co-batching, which is the other effect nobody measures

Part 3. The KV budget, with the numbers of the agentic case

Part 4. From people to concurrent requests

The amplification

Part 5. The gateway, which also has to be sized

A worker is a universe

Postgres

The token counter, which is the agentic trap

Streaming and admission

Resources and autoscaling

Part 6. What to watch: the metrics that work and the ones that broke

At the gateway

At the engine

The SLO worth chasing

Part 7. The engine levers, ordered by return

The full calculation, from start to finish

A reference config.yaml

Checklist

Traps and things that are not what they seem

Closing

See also

Sources

Prefix routing: what LiteLLM does not do, who does, and how much it really improves

TL;DR

You are here: between the gateway and the engine

The analogy: the library with four counters

The correction: what I wrote in June

What LiteLLM does have: four affinities

prompt_caching: why it is no use for this

session_affinity: the one to use

The routers that do read the prompt, and why they are no good

What did land: admission control

What the engine exposes upwards

The ZMQ stream

The hash that has to be replicated

The hits are invisible by default

The metrics, and what broke

The last token

The ones that do do it

vLLM production-stack

llm-d

NVIDIA Dynamo

The rest of the field

Comparison table

How much it really improves

The five failure modes

The architecture that comes out of all of the above

When not to build this

Checklist

Traps and things that are not what they seem

Closing

See also

Sources

Humans and agents on the same gateway: why `priority` does not prioritise, the 429 that brings down the whole pool, and the two ways to separate traffic

TL;DR

You are here: the gateway layer with two classes of traffic

The analogy: the registry desk and the courier with two hundred files

The two kinds of traffic, measured

What does not work: priority in LiteLLM

What does work: two pools

By credential: model_group_alias at key or team level

By User-Agent: tag_regex

A different strategy per pool with routing_groups

Admission control: the new piece in the 1.101 branch

Concurrency per key, and the three settings that do nothing

Session affinity: the KV cache that gets thrown away

The engine: what vLLM can and cannot do

Retries: the product nobody calculates

A month’s budget in twenty minutes

A reference `config.yaml`

`prompt_caching`: why it is no use for this

`session_affinity`: the one to use

What does not work: `priority` in LiteLLM

By credential: `model_group_alias` at key or team level

By `User-Agent`: `tag_regex`

A different strategy per pool with `routing_groups`

A reference `config.yaml`

A `Deployment` serves tokens, but offers no contract

`InferenceService`: predictor, transformer, explainer

`ServingRuntime` and `ClusterServingRuntime`: the catalogue of trains

The storage initializer and `storageUri`

The LLM branch: `LLMInferenceService`

The `generate` extension and why it did not win