Inference on lo0 — Tech Blog

The blog map: from the utility feed to the token served

Sat, 29 Aug 2026 10:00:00 +0200

This blog started in March 2026 with a handful of articles about Kubernetes and networking, and by now it has piled up more than 150 pieces that, read one at a time, look like unconnected topics: PagedAttention, diesel generators, OpenCost, SPIFFE, immersion cooling, RAGAS. They are not. All of them describe the same object from different altitudes: a sovereign inference factory, that is, the complete machine that turns megawatts from the grid into tokens served to a user, with ownership and control of the data kept in house.

This article is the index of that machine. It orders what has been published into ten layers, bottom up, and links what is written in each one. It is good for three things: finding a specific topic, seeing which layers are covered in depth and which are half done, and choosing a reading route according to the problem in front of you.

The map

The map reads bottom up, and the criterion for ordering it is physical dependency: there is no inference engine without a powered and cooled GPU, no RAG without an engine serving embeddings, no demonstrable regulatory compliance without observability recording what happened. Layer 10 is the exception, and that is why it appears cutting across everything: cost, performance and energy are not a floor of the building, they are the measure of the whole building.

Layer 0 · Physical infrastructure

The most recent vertical, and the one that closes the circle: the path of the watt from the utility feed to the chip, and the path of the heat from the chip to the atmosphere.

The power chain: redundancy, Tier, kW against kVA, PUE, AI density and the limit of the grid.
Generators: NFPA 110, transients, sizing, diesel against gas, fuel and paralleling.
Switching: ATS against STS, the ITIC window, open and closed transition, and the A and B paths.
The UPS and ride-through: IEC 62040-3 topologies, eco-mode, flywheel, lithium against lead and the pulsing loads of AI.
The thermal challenge: why AI density broke air, and the spectrum of solutions.
Air and its limits: CRAC and CRAH, the geometry of the flow, the cube law, containment, ASHRAE and rear doors.
Direct liquid to the chip: cold plate, CDU, TCS and FWS loops, coolant chemistry and leaks.
Immersion and the megawatt horizon: single-phase and two-phase, the PFAS wall and microfluidics in the silicon.

Layer 1 · The node

Inside the machine, before any process exists: the buses, the memory and the topology that decide how much performance actually reaches the model.

PCIe topology, GPUDirect and ACS: the corridors the tensors travel through.
NUMA, hugepages and CPU isolation: why the neighbour on your socket costs you latency.
NVLink, NVSwitch and NCCL: the cable a large model is split across.
Network NUMA with Cilium and DraNet: the kitchen door nobody watches.
CPU, Memory and Topology Manager in RKE2: how to ask for a GPU with affinity and no surprises.
SM, CUDA streams and CUDA graphs: where time is lost inside the accelerator.
Mixed NVIDIA and Intel environments: from the homogeneous cluster to the real estate.
Measuring the power of a GPU: sensor metrology, which belongs to this layer and to layer 10 at the same time.

Layer 2 · Storage

State of the art, performance, security and availability: the set of four on the storage that holds weights, corpus and checkpoints.
From disk to HBM: why starting a model takes as long as it takes.
Speeding up the cold start: from minutes to seconds with serialisation and cache.

Layer 3 · The cluster

vLLM on Kubernetes: the inference piece that does scale.
LLM inference operators: OME, vLLM Production Stack and company.
The GPU cluster as a multi-tenant platform: from shared machine to service.
Sharing a GPU: time-slicing, MPS and MIG: the three ways to split an accelerator.
Volcano and Kueue: gang scheduling, queues and quotas.
Autoscaling with HPA and KEDA: scaling on metrics that mean something.
Serving several models on one GPU: co-residency, swap and sleep.
On-premise capacity planning: from the SLO to the number of nodes.
GitOps for the stack with Flux: operating the platform as code.
Cilium BGP on RKE2 and eBPF from zero to Cilium: the network underneath everything above.

Layer 4 · The inference engine

The densest block of the blog: vLLM cut open, piece by piece.

KV cache and PagedAttention: the working memory and its block manager.
Continuous batching and the scheduler step: how each round of work is assembled.
Optimising prefill and optimising decode: the two phases and their controls.
The attention backend and FlashAttention: the core of the computation.
Prefix cache: hit rate engineering and batch sizing with grid search: the two settings that give away the most performance.
Disaggregated serving and one big replica or many small ones: the topology decisions.
Long context and KV offloading: the KV memory hierarchy when the window does not fit in HBM.
Structured output and function calling: holding the model’s output in place.
Serving reasoning models: the invisible draft you pay for all the same.
Multimodal on-premise with vLLM and serving embeddings and rerankers with TEI: the other models in the stack.
The inverted roofline: why a small model is not optimised the same way as a large one.

Layer 5 · The model

Everything done to the model before serving it, or while it is being served.

Quantization for inference, FP8 end-to-end, aggressive sub-4-bit quantisation and quantising on the fly.
Pruning and distillation: reducing without amputating, and teaching a small model.
MoE in inference and device-native architectures.
Multi-LoRA serving, QLoRA and multi-LoRA at the limit and the complete QLoRA runbook.
Speculative decoding and self-speculative with early exit.
Modern alignment: DPO, KTO, ORPO and SimPO.
Continuous fine-tuning and retrain: closing the loop.

Layer 6 · Data and RAG

Document ingestion from the PDF to the indexed chunk and corpus curation.
Embeddings in 2026 and reranker with hybrid retrieval.
PostgreSQL and Qdrant in ingestion, Debezium and CDC and RAG over Kafka.
Semantic cache: the receptionist with a memory.
RAG on CPU and aggressive RAG on small models.
Evaluating a RAG without fooling yourself: RAGAS and the golden dataset.
Ontologies and knowledge graphs and data versioning with DVC and lakeFS.

Layer 7 · Control plane and delivery

Choosing the OSS gateway and the L7 inference router: the switchboard in front of the engine.
KServe and the Open Inference Protocol: the standard control plane.
Model registry and distribution with OCI and ORAS: where the bytes live.
Canary, blue-green and shadow: deploying a model without breaking the service.
Prompt versioning: the contract that avoids the scare.
MCP from the inside and adding authentication with Keycloak.
Operating the control plane (September 2026 run): the LiteLLM and Langfuse pair, day 2 of the proxy and virtual keys, budgets and limits and humans and agents on the same gateway, the prefix routing LiteLLM does not do and the MCP gateway as a second front door, sizing for agents and the seams with identity and traces: the gateway no longer as a piece you choose, but as a piece you operate.
Identity: Keycloak in an AI platform, with its position in the architecture and the MCP standards it does not implement, and how to complete it for MCP by building the protected resource side.
Operating Langfuse (a run of eight, September 2026): what goes into a trace, the first article about day 2 of version 4.
Building an end-to-end sovereign assistant, with all of the above pieces working together, is the exercise that closes this layer and is pending publication.

Delivery to the teams that use it is its own family of pieces:

Backstage as a self-service portal: the catalogue, the templates and the golden path, with the real cost of operating it.
Kubeflow in depth: which pieces of the ML toolbox are worth it and which ones you already have.
Knative and scale-to-zero: when switching off the GPU saves money and when the cold start violates your SLO.

Rebuilding the cloud experience on your own iron is a run of three:

From public cloud to private: what you lose when you leave the cloud and which projects fill each gap, with the business case and the sovereignty drivers.
Crossplane as a control plane: the declarative provisioning API that on-premise does not give you, and the cost of operating your own control plane.
KubeVela and Score: application-centric self-service by CLI, the friendly face Crossplane deliberately does not offer.

Layer 8 · Observability and evaluation

Tracing with OpenTelemetry GenAI and instrumenting vLLM with OTel.
Langfuse from the inside: the sorting centre for traces.
GPU observability with DCGM and the anatomy of DCGM and vLLM metrics.
Evals: the layer after tracing and LLM-as-judge.
Incident response runbooks: from the alert to the action.
Hubble, AgentSight and drift detection with eBPF: observing without instrumenting.

Layer 9 · Security, trust and compliance

Guardrails and safety and LLM Guard: the lines of defence around the model.
Hardening and secrets of the sovereign stack.
Isolating AI agents from the workspace and the runbook with bubblewrap and Tetragon, with Tetragon as the base.
Signing, provenance and AIBOM and identity and isolation with SPIFFE and Confidential Containers.
ISO/IEC 42001, the EU AI Act article by article and the cross-mapping of ENS, 42001 and the AI Act.
Compliance in healthcare and in defence: the same infrastructure facing two sector frameworks.

Layer 10 · Cost, performance and energy

The data track, written in a different format from the rest of the blog: tables, figures and a verifiable source, no analogies. These are the articles that hold up an investment decision.

FinOps: cost model and state of the art, OpenCost in depth, Kubecost against OpenCost, from the GPU-hour to the cost per token, chargeback and showback, the cost of the idle GPU, cloud GPU and neoclouds, full on-premise TCO and multi-tenancy FinOps with LiteLLM.

Benchmarking: frameworks and metrics, the tool catalogue, GuideLLM and the SLO under load, GenAI-Perf in depth, how to read MLPerf Inference, measurement bias and reproducibility, quality benchmarks and the Pareto frontier between engines.

Energy: measuring GPU power, energy per token in Spain, energy benchmarking frameworks, measuring in production with Kepler and DCGM, MLPerf Power, Wh per token leaderboards, from the watt to carbon, efficiency levers and energy in the TCO and in regulation.

Synthesis: the three axes and the identity that binds them, sovereign on-premise against hyperscalers, with data, from the SLO to the number of GPUs and the reproducible harness.

To that list you can add two cost pieces that are not about GPUs: the cost of agents with durable execution and the gateway’s cost per request.

The master maps

Five earlier articles work as the general plan and are still the best way in for anyone arriving new:

The six-stage LLMOps pipeline: the master map of the life cycle.
The seven layers of the on-premise stack: the architecture version of this same map.
Anatomy of an LLM request in production: a real request crossing every layer.
Five maturity levels of the platform and seven greenfield deployment phases: where you are and what comes next.
The OSS catalogue entry by entry and the parallel catalogue against the hyperscalers: which tool for each stage.
MLOps for LLMs in 2026: the starting landscape.

Four routes

I am about to build the first platform. Landscape and the six-stage pipeline, the seven layers of the stack, the seven deployment phases, capacity planning, vLLM on Kubernetes, gateway, observability with OTel and Langfuse, and guardrails before opening the door.

I have the cluster and the performance is not there. Anatomy of a request, KV cache and PagedAttention, prefill and decode, prefix cache, batch sizing, GuideLLM to find the saturation point, measurement bias so you do not fool yourself with the numbers, and the Pareto frontier between engines.

I have to justify the investment to management. The three axes, on-premise TCO, cost per token, idle GPU, on-premise against hyperscalers, sizing and justifying, and the reproducible harness so the figures survive an audit.

My job is the room, not the software. The power chain, the generators, the thermal challenge, air and its limits, direct liquid to the chip, immersion, and from there to storage and node topology.

What is missing

The map also serves to show the gaps, and three are identified, plus a line that is half open:

A comparison of vector databases: Milvus, Qdrant, pgvector and OpenSearch with data, beyond the scattered mentions.
Lakehouse for RAG: Iceberg and Trino as the substrate of the corpus.
Out-of-band management: OpenBMC, Redfish and DCIM, which is exactly the bridge between layer 0 and layer 3.
Sovereign substrate: the CLI self-service part is already covered by the layer 7 run (Crossplane, KubeVela and Score), and what remains is the IaaS-type substrate underneath, the sovereign alternatives to the hyperscaler (Spinifex, Harvester, OpenStack) on which everything else runs.

MLPerf Inference: how to read it, what LoadGen enforces and what comparability it offers

Sat, 29 Aug 2026 09:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

MLPerf Inference is maintained by MLCommons on a six-monthly cadence. It imposes three things no home-made benchmark has: a mandatory load generator (LoadGen, compiled from a tagged revision and unmodified), model-equivalence rules that bound what can be optimised, and an audit process that reviews up to two submissions per round. The four scenarios (SingleStream, MultiStream, Server, Offline) have a minimum duration of 600 s and different metrics: Server reports the sustained Poisson rate and is the only one with a latency constraint. For the LLM benchmarks that constraint is not a request latency but two simultaneous SLOs at the 99th percentile: TTFT and TPOT, with per-model figures ranging from 2000 ms / 200 ms on Llama 2 70B to 450 ms / 40 ms on its Interactive variant. Quality is required relative to the reference model (99 % or 99.9 %) and the generation length is audited as well (Llama 2 70B: more than 90 % of 294.45 tokens per sample). Three reading rules sum up the real comparability: results are only compared within the same version, division, category and scenario; per-accelerator is not an official MLCommons metric, it is derived; and neither cost nor energy enters the primary metric, to the point that round v5.1 had 2 power submissions against 27 participating organisations.

What it is and how it is organised

MLPerf Inference is maintained by the MLCommons Association through the MLPerf Inference Working Group, whose participation is limited to MLCommons members and affiliates (MLPerf Inference Working Group). The declared cadence for publishing results is roughly six months.

Round	Results published
v4.0	27 March 2024
v4.1	28 August 2024
v5.0	2 April 2025
v5.1	9 September 2025
v6.0	1 April 2026
v6.1	in preparation as of this data sheet

The suite splits into two system categories: Datacenter and Edge, the latter defined in the rules as everything that is not datacenter (inference_rules.adoc). A Datacenter system has two requirements that do not apply in Edge:

Mandatory ECC on DRAM and HBM, active during all performance and accuracy runs. There is no requirement on SRAM.
Mandatory networking since round v3.0, with a minimum bandwidth computed from the achieved throughput. For Llama 3.1 405B the minimum ingress is throughput × 20000 × dtype_size bytes/s; for Llama 2 70B the minimum egress is throughput × 1024 × dtype_size.

Edge additionally allows inferred results: a MultiStream derived from SingleStream counts as 8 times the p99 latency; an Offline derived from MultiStream, 8000 divided by the mean latency in milliseconds.

The four scenarios

This is the normative table, transcribed from the rules:

Scenario	Query generation	Duration	Samples/query	Latency constraint	Tail latency	Reported metric
SingleStream	The next query is issued when the SUT completes the previous one	600 s	1	None	90 %	p90 latency with early stopping
Server / Interactive	LoadGen issues queries following a Poisson distribution	600 s	1	Benchmark-specific	99 %	Maximum Poisson parameter supported
Offline	All samples are delivered at the start in a single query	1 query, 600 s	At least 24,576	None	N/A	Measured throughput
MultiStream	The next query is issued when the SUT completes the previous one	600 s	8	None	99 %	p99 query latency with early stopping

Four operational clarifications that change how each number is interpreted:

In Server, LoadGen does not measure a latency: it runs a binary search over the target rate. For a given value it generates queries at that QPS with Poisson arrivals; if the run fails the latency gate, it lowers the value and repeats. The final published metric is, per the official FAQ, scheduled samples per second, not completed samples nor the target entered by the user.
In MultiStream, the latency of a query is the maximum of the latencies of its samples, and the 8 samples of each query are contiguous in the order in which they were loaded. The value multi_stream_samples_per_query = 8 is fixed in the official configuration.
In Offline there is no latency constraint at all. The rules themselves forbid techniques that only work in fixed-length experiments except in the Offline scenario, which is a normative acknowledgement that an Offline result says nothing about behaviour under an SLO.
The percentiles in the table are the lower theoretical bound for runs with very many queries. With early stopping, the percentile actually computed is somewhat above the nominal one, as a penalty for processing few queries.

The mandatory scenarios depend on the benchmark. In Datacenter, the summarisation, question answering, text generation and VLM tasks require (Server | Interactive), Offline, that is, the submitter chooses between Server and Interactive. Reasoning and recommendation require Server, Offline. Medical segmentation, node classification, speech-to-text and RAG require only Offline.

LoadGen: what it measures and what it does not

LoadGen is the module that generates the load and computes the metrics. Its use is mandatory for all submissions, and it must be compiled from a tagged and approved revision of the repository, unaltered; the README is explicit that local modifications of the C++ library are not accepted.

Its four declared responsibilities are generating the queries according to the scenario, tracking the latency of each query, validating the accuracy of the results and computing the final metrics. Latency is defined as the time from when LoadGen was scheduled to pass a query to the system under test until it receives the response.

Equally important is what it declares out of scope: LoadGen does not know the model, does not know the data formats, cannot score accuracy and does not know the scenario constraints of the MLPerf rules. The consequence is written in its own README: being model-agnostic, it cannot enforce the MLPerf requirements, for example the target percentiles and latencies. Those limits are injected from mlperf.conf and user.conf. The justification for making it mandatory appears in the founding ISCA 2020 paper: it establishes a clear boundary between the components that belong to the submitter and those that belong to MLPerf, and it measures the performance of the complete system instead of that of a single piece.

Duration, query count and early stopping

The minimum duration is 600,000 ms in all four scenarios. The minimum number of queries comes from a confidence interval table that the rules publish:

Tail percentile	Confidence	Margin of error	Inferences	Rounded
90 %	99 %	0.50 %	23,886	24,576
95 %	99 %	0.25 %	50,425	57,344
97 %	99 %	0.15 %	85,811	90,112
99 %	99 %	0.05 %	262,742	270,336

The submission checker applies Server: 270336, SingleStream: 1024 and MultiStream: 662. For Offline, the minimum is set per benchmark: 24,576 on Llama 2 70B, 15,000 on Mixtral 8x7B, 13,368 on Llama 3.1 8B, 8,313 on Llama 3.1 405B, 4,388 on DeepSeek-R1, 1,633 on Whisper and 43 on 3D UNet.

The early stopping mechanism allows runs to be shortened while keeping the statistical guarantee. With tolerance d = 0 and confidence c = 0.99, the algorithm resolves by binary search the smallest number of queries below the latency threshold that satisfies the criterion, given the number of queries observed above it.

LoadGen’s modes are SubmissionRun (accuracy followed by performance), AccuracyOnly, PerformanceOnly and FindPeakPerformance. The last, applicable only to Server, takes target_qps as a lower bound if it passes, estimates the upper bound at double and doubles it until failure, and then does a binary search. In performance mode LoadGen selects queries uniformly at random with replacement from a set of size QSL; in accuracy mode it uses a copy of the validation dataset, each sample exactly once. One accuracy run must be executed for every performance result submitted, and the same code in both modes.

The seeds are announced four weeks before the deadline and the mandatory generator is Mersenne Twister 19937. They change every round: in v5.1, qsl_rng_seed = 1780908523862526354; in v5.0 it was 6023615788873153749.

The compliance tests

They are activated by placing an audit.config file in the working directory, whose parameters override those of mlperf.conf and user.conf. Their logs are mandatory in the submission package. The declared purpose is to detect anomalies, not to diagnose their cause.

For the LLM benchmarks the required test is TEST06, specifically against the end-of-sequence token exploit. It runs 100 samples and demands three conditions: that the first token reported separately matches the first token of the model’s output, that the output ends with zero or a single EOS token, and that the number of tokens reported matches the real length of the generated sequence.

Divisions, categories and what can be touched

The three divisions

Closed: requires preprocessing, postprocessing and model equivalent to the reference implementation. It allows calibration for quantisation and does not allow retraining. It is the only one that can use the MLPerf name unqualified.
Open: allows arbitrary preprocessing, postprocessing and model, including retraining. The accuracy, latency and scenario constraints do not apply: instead, the accuracy obtained and the latency constraints under which the performance was obtained must be reported. The model may have any origin, be quantised in any way and be sparsified in any way. It must use the same validation dataset as the corresponding Closed and use all of it.
Network: inherits all the Closed requirements, applies only to Datacenter, and separates the LoadGen node from the system under test through a fabric. The submitter’s dispatch library cannot preprocess, postprocess, batch, pad or cache. Bus interconnects are forbidden by name (PCIe, CXL, CCIX, HyperTransport, NVLink, QPI, UPI, ICI) and only Ethernet, IEEE 802.11, InfiniBand and 3GPP are admitted, with a requirement to work chassis to chassis at more than ten metres.

The availability categories

Category	Hardware	Software
Available in cloud	Available for cloud rental	Available
Available on premise	Available for purchase	Available
Preview	Must be available for the next submission, or the one after 140 days, whichever is longer	Available except what is needed for substantially new hardware
RDI	Does not meet the above	Does not meet the above

Available demands four cumulative conditions: price available, having been rented or shipped to at least one third party, public evidence of availability and reasonable availability to additional third parties on the submission date. A Preview result obliges a resubmission as Available with equal or better performance, tolerating up to 2 % degradation from noise; if there is no resubmission, the Preview result is marked invalid. RDI components cannot be submitted as Available until the cycle after the next one, or 221 days, whichever is longer.

What Closed allows and forbids

MLPerf delivers the weights in fp16 or fp32. The central rule is that the submitter may perform purely mathematical and reproducible quantisation, using only the calibration data and the tensors of the delivered model, to any numerical format that reaches the required quality, and that the method must be described publicly at a level that allows it to be reproduced. The test against back doors is elegant: the description of the quantisation method must be much smaller than the non-zero weights it produces.

Allowed, among other things: any framework or runtime, arbitrary data layout, varying the matrix multiplication algorithm, mathematically equivalent transformations, polynomial approximations of transcendentals, processing queries out of order within what the scenario admits, replacing dense operations with mathematically equivalent sparse operations, hand-picking different precisions per operation, fusing and unfusing, dynamic batching, and mixtures of experts combining weights with different quantisation.

Forbidden: replacing or supplementing weights wholesale, discarding non-zero weights, including pruning, caching queries or responses, coalescing identical queries, modifying weights during the timed portion, quantisation algorithms of size comparable to the weights they produce, hardcoding the total number of queries, using knowledge of the LoadGen implementation to predict peaks or troughs in the Server scenario, changing the number of search beams, and incorporating statistics from the performance or accuracy sets.

Two clarifications that matter to anyone serving LLMs in production: the KV cache is allowed just as in the reference model provided it is not applied across queries; PagedAttention is admitted if the block is reused only within the batch; continuous or dynamic batching is allowed; and KV cache entries are treated as activations for quantisation purposes, and cannot be pruned. Speculative decoding is only admitted in the benchmark and scenario combinations explicitly listed, with the reference head at the same precision, and implementations that artificially manipulate the acceptance rate are forbidden.

The SLOs almost nobody cites: TTFT and TPOT

This is the point where an MLPerf table is most often misread. In the LLM benchmarks of the Server scenario, the constraint is not a request latency: the rules set target_latency = 0 and activate use_token_latencies = 1, so that the gate is formed by two simultaneous metrics, both evaluated at the 99th percentile:

TTFT (time to first token): latency of the first token.
TPOT (time per output token): mean interval between all generated tokens.

The figures are in mlperf.conf and change per round and per benchmark:

Benchmark (Server scenario)	TTFT	TPOT	Round
`llama2-70b`	2000 ms	200 ms	v5.0, v5.1, v6.x
`llama2-70b-interactive`	450 ms	40 ms	v5.1, v6.x
`mixtral-8x7b`	2000 ms	200 ms	v5.0, v5.1, v6.x
`llama3_1-405b`	6000 ms	175 ms	v5.0, v5.1, v6.x
`llama3_1-405b-interactive`	4500 ms	80 ms	v6.x
`llama3_1-8b`	2000 ms	100 ms	v5.1, v6.x
`llama3_1-8b-interactive`	500 ms	30 ms	v6.x
`deepseek-r1`	2000 ms	80 ms	v5.1, v6.x
`deepseek-r1-interactive`	1500 ms	15 ms	v6.x
`gpt-oss-120b`	3000 ms	80 ms	v6.x

A TPOT of 40 ms is equivalent to 25 tokens per second per user, which is the order of magnitude of a conversational interface; one of 200 ms is equivalent to 5 tokens per second, which is a batch use case dressed up as a service. Two results for the same model under the Server and Interactive labels are not the same measurement. Two documented discrepancies between primary sources remain: the llama2-70b-interactive variant does not appear in the mlperf.conf tagged as v5.0 despite having been announced in that round, and in the v6.x cycle the TPOT of gpt-oss-120b-interactive appears as 15 ms in mlperf.conf and as 20 ms in the text of the rules.

For contrast, the constraints of the non-LLM benchmarks in the Server scenario are conventional request latencies: ResNet-50 15 ms, RetinaNet 100 ms, BERT 130 ms, DLRMv2 60 ms, DLRMv3 80 ms, RNN-T 1000 ms, GPT-J and Stable Diffusion XL 20,000 ms.

Quality: the 99 % and the token count

The quality targets are expressed relative to the reference model, never in absolute terms, and each benchmark demands one variant, the other or both. In the current Datacenter suite, 3D UNet, Llama 3.1 8B and Whisper ask for 99 % and 99.9 % of FP32, which generates two distinct results; Llama 2 70B asks only for 99.9 %; Llama 3.1 405B, Mixtral, DeepSeek-R1, GPT-OSS-120B, Qwen3-VL, RGAT and WAN-2.2 ask only for 99 %; and E2E-RAG is the exception at 97 %. In the checker this materialises as separate benchmarks (bert-99 and bert-99.9), each with its computed numerical threshold.

On top of that metric there is a second LLM-specific constraint that is cited very rarely: the generation length is audited as well.

Benchmark	Length constraint
Llama 2 70B	tokens per sample > 90 % of 294.45
Llama 3.1 405B	tokens per sample between 90 % and 110 % of 684.68
Mixtral 8x7B	tokens per sample between 90 % and 110 % of 144.84
Llama 3.1 8B	total generated length > 90 % of 8,167,644

The FAQ settles the obvious shortcut: reducing the maximum output length below the reference is not allowed, and truncating tokens to improve performance or reach the accuracy target is not permitted. The inference parameters are fixed in Closed (max_new_tokens = 1024 on Llama 2 70B, 20000 on Llama 3.1 405B and DeepSeek-R1), as is the decoding algorithm: greedy search in the Llama family, Mixtral and DeepSeek-R1; sampling with temperature = 1.0 and top_p = 1.0 in GPT-OSS-120B and Qwen3-VL. Accuracy is reported to five significant figures with round-half-to-even.

The current Datacenter suite

Area	Task	Model	Dataset	QSL	Required quality
Vision	Medical segmentation	3D UNet	KiTS 2019	42	99 % and 99.9 % of FP32 (DICE 0.86330)
Language	Summarisation	Llama 3.1 8B	CNN DailyMail v3.0.0	13,368	99 % and 99.9 % of FP32
Language	Question answering	Llama 2 70B	OpenOrca	24,576	99.9 % of FP32
Language	Text generation	Llama 3.1 405B	LongBench, Ruler, GovReport	8,313	99 % of FP16
Language	QA, maths and code	Mixtral 8x7B	OpenOrca, GSM8K, MBXP	15,000	99 % of FP16
Language	Reasoning	DeepSeek-R1	`mlperf_deepseek_r1`	4,388	99 % of FP16 (exact match 81.9132 %)
Language	QA, maths and code	GPT-OSS-120B	AIME25, GPQA Diamond, LiveCodeBench v6	6,396	99 % of 83.13 %
Vision	Vision-language model	Qwen3-VL-235B-A22B	Shopify product catalogue	48,289	99 % of BF16 (hierarchical F1 0.7824)
Commerce	Recommendation	DLRMv3	Synthetic Streaming 100B	34,996	99.9 % of FP32
Generative	Text to video	WAN-2.2-T2V-A14B	VBench prompts	248	99 % of BF16 (VBench 69.7752)
Graphs	Node classification	RGAT	IGBH	788,379	99 % of FP32 (72.86 %)
Audio	Speech to text	Whisper	LibriSpeech	1,633	99 % and 99.9 % of FP32 (WER 2.0671 %)
Language	End-to-end RAG	E2E-RAG	FRAMES	824	97 % of FP32

The E2E-RAG benchmark deserves attention from anyone designing a RAG platform, because it fixes the whole pipeline: a FAISS HNSW index with mandatory parameters M = 32, efConstruction = 200 and efSearch = 100; e5-base-v2 embeddings; ColBERTv2 reranking; generation and query rewriting with GPT-OSS-120B; evaluation with Llama 3.1 8B as judge; and a maximum of 5 retrieval iterations, mandatory for all submitters. Model loading is not timed; building the vector database is.

The definition of sample changes per model and is a frequent source of confusion: in the LLMs it is a sequence, in DLRMv3 it is a request with a user history and 2048 candidates, in WAN-2.2 it is a pair of positive and negative prompts, and in PointPainting it is five images and a lidar point cloud.

How to read a results row

The columns of the official table in the Closed division are Submitter, Software, System, Benchmark Results, Processor/Count, Details, Accelerator/Count and Code. The Open division adds Model Used and Notes. Rows with power measurement add System Power in Server and Offline, or Energy Per Stream in the stream scenarios. Any citation must use the identifier in the format major-version.minor-version.entry.benchmark, of the style 5.1-0053, and carry a footnote with suite, version, division, benchmark, scenario, date and source.

Per-accelerator is not an official metric

There is no per-accelerator column. It is a derived metric, and the MLCommons messaging guidelines are blunt: any comparison based on a different or derived metric, such as power, cost, model size or accuracy, must make the basis of comparison clear in the text and in a footnote, and secondary and derived metrics cannot be presented as official or verified MLPerf metrics. NVIDIA itself states this with the standard formula in its round blogs: per-GPU performance is not a primary MLPerf Inference metric and is computed by dividing the reported throughput by the number of reported accelerators. The same applies to combining results from several benchmarks: MLCommons allows it but does not endorse it, and the composite cannot be presented as an official result.

The comparison rules

MLPerf results can only be compared with compatible MLPerf results: same benchmark, same scenario and compatible versions per the normative compatibility table.
MLPerf results cannot be compared with non-MLPerf results.
When comparing, any difference in version, division, category, verification status, scenario or number of chips must be clearly identified. When comparing Open with Closed, it must be identified in what sense the Open result would not qualify as Closed.
Submitters cannot publish results of a version before its official date; non-submitters must wait two weeks from that date.
The sanctions regime goes as far as barring an infringer from submitting results in the future and marking their results as non-compliant permanently in the database, with a deadline of three working days to withdraw the infringing content.

How much audit sits behind a row

In each round up to two submissions are audited: one chosen at random from all of them and zero or one chosen by the review committee. And there is a constraint that conditions the reading of the whole table: only Available submissions in the Closed division are auditable. Preview, RDI and Open fall outside the process. There is an exemption from the random audit if the system is equivalent to another already audited and neither the aggregate performance nor the per-accelerator performance differ by more than 10 % from those of the previous audit. The deadlines are 28 days to select an auditor, 30 for the report after signing the confidentiality agreements and around 90 days for the complete process, with two days of hardware access for the auditor. The rules close with three sentences that define the standard of proof: results that cannot be replicated are not valid results, benchmark detection is not allowed and input-based optimisation is not allowed either.

The recent rounds, in figures

Round	Organisations	Performance results	Power results	New benchmarks
v4.0	23	more than 8,500	900	2
v4.1	22	964	31	1
v5.0	23	17,457	n/a	4
v5.1	27	n/a	2	3
v6.0	24	n/a	n/a	5 of 11 Datacenter tests

These counts do not form a time series: the counting criteria differ between press releases and none documents the criterion, to the point that v4.0 and v4.1 differ by a factor close to nine.

The published performance jumps are comparable within each announcement. Between v4.0 and v5.0, twelve months apart, the number of Llama 2 70B submissions multiplied by 2.5, the median score doubled and the best score was 3.3 times faster. Between v5.0 and v5.1, in six months, the best systems improved by up to 50 % in some scenarios. Round v6.0 changed axis and its press release talks about scale rather than speed: 30 % more multi-node submissions, 10 % of the systems with more than ten nodes against 2 % in the previous round, and a system larger than 72 nodes and 288 accelerators, quadrupling the previous maximum node count.

Hardware that debuted in each round: MI300X, TPU v6e, Xeon Granite Rapids and B200 in v4.1; MI325X, Xeon 6980P, GB200 and Jetson AGX Thor in v5.0; MI355X, Intel Arc Pro B60, GB300 and RTX Pro 6000 Blackwell Server Edition in v5.1.

What MLPerf Inference does not measure

Cost. There is no cost or TCO metric, and cost is explicitly classified as a derived metric that cannot be presented as official.
Energy, except as an optional extension. MLPerf Power is regulated in a separate document, requires measuring at the wall with the power group’s workflow and PTDaemon, and forbids any other method. Its real adoption is marginal: 2 power submissions in v5.1 against 27 participating organisations. In addition, MLCommons bounds what that figure means: the measured power is only valid for the benchmark it accompanies, and any other reference to power, such as a TDP setting or a power supply rating, is neither measured nor validated by MLCommons. The messaging rule forbids submitters from publishing per-watt normalised comparisons using any proxy other than the measured power.
Configurations not optimised by the vendor. The bias is acknowledged in the audit rule itself, which contemplates submissions whose performance is not consistent with the known characteristics of the hardware, or where the committee lacks visibility into how it was achieved, or where hardware and software are not reasonably available to the public.
Barrier to entry. Only MLCommons members and test partners can submit results for review. Anyone measuring with the same code without submitting must label every figure as unverified with the corresponding legend.

Usage flow for an on-premise platform

MLPerf Inference is useful in a purchasing decision if it is used for what it is, a test bench with written rules, and not as a ranking.

Filter by division and category before looking at any number: keep Closed and Available. A Preview or RDI result describes hardware that cannot yet be bought, and neither of the two is auditable.
Choose the scenario that matches the load pattern: Server or Interactive if there are users waiting, Offline if the work is batch. An Offline result does not authorise promising a latency.
Read the SLO before the throughput: in any LLM benchmark, the number of queries per second only means something alongside its TTFT/TPOT pair. The same machine publishes very different figures under Server and under Interactive.
Check the quality variant: 99 % and 99.9 % are different results for the same model, and the second usually costs performance.
Translate to your own configuration carefully: dividing by the number of accelerators is a derived metric, not an official one, and it has to be declared as such.
Close the gap MLPerf leaves open with your own measurement: cost, energy and behaviour under real traffic. That is exactly the function of the reproducible harness and of the track B bench tools, and the reason no platform decision stands on a public table alone.

Cross-links from the benchmarking track

B1/B2 — LLM inference benchmarking: frameworks, metrics and state of the art: the TTFT, TPOT and goodput metrics outside the MLPerf corset, and which tool measures each one.
B2 — LLM benchmark tool catalogue: the tools with which you reproduce at home what MLPerf standardises.
B3 — GuideLLM and SLO validation under load: the practical equivalent of LoadGen’s binary search in the Server scenario.
B6 — Measurement bias and reproducibility: why LoadGen exists, told from the side of what happens when there is none.
B8 — Serving engines compared on the Pareto frontier: the comparison MLPerf does not make, with engines instead of systems.
C4 — MLPerf Power: the energy extension of this same machinery, with its certified measurement at the wall.

Sources

MLCommons, MLPerf Inference Working Group — https://mlcommons.org/working-groups/benchmarks/inference/
MLCommons, MLPerf Inference Rules (inference_rules.adoc) — https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc
MLCommons, MLPerf Inference Power Measurement (power_measurement.adoc) — https://github.com/mlcommons/inference_policies/blob/master/power_measurement.adoc
MLCommons, MLPerf Submission Rules (submission_rules.adoc) — https://github.com/mlcommons/policies/blob/master/submission_rules.adoc
MLCommons, MLPerf Results Messaging Guidelines — https://github.com/mlcommons/policies/blob/master/MLPerf_Results_Messaging_Guidelines.adoc
MLCommons, LoadGen README — https://github.com/mlcommons/inference/blob/master/loadgen/README.md
MLCommons, loadgen/test_settings.h — https://github.com/mlcommons/inference/blob/master/loadgen/test_settings.h
MLCommons, loadgen/mlperf.conf (master) — https://github.com/mlcommons/inference/blob/master/loadgen/mlperf.conf
MLCommons, loadgen/mlperf.conf (tag v5.1) — https://github.com/mlcommons/inference/blob/v5.1/loadgen/mlperf.conf
MLCommons, loadgen/mlperf.conf (tag v5.0) — https://github.com/mlcommons/inference/blob/v5.0/loadgen/mlperf.conf
MLCommons, tools/submission/submission_checker.py (tag v5.0) — https://github.com/mlcommons/inference/blob/v5.0/tools/submission/submission_checker.py
MLCommons, compliance/nvidia TEST06 README (tag v5.1) — https://github.com/mlcommons/inference/blob/v5.1/compliance/nvidia/TEST06/README.md
MLCommons, Benchmark results: Inference Datacenter — https://mlcommons.org/benchmarks/inference-datacenter/
Reddi et al., MLPerf Inference Benchmark, ISCA 2020 — https://arxiv.org/abs/1911.02549
MLCommons, MLPerf Inference v5.0 Benchmark Results (abril 2025) — https://www.businesswire.com/news/home/20250402313932/en/MLCommons-Releases-New-MLPerf-Inference-v5.0-Benchmark-Results
MLCommons, MLPerf Inference v5.1 Benchmark Results (septiembre 2025) — https://www.globenewswire.com/news-release/2025/09/09/3147136/0/en/MLCommons-Releases-New-MLPerf-Inference-v5-1-Benchmark-Results.html
MLCommons, MLPerf Inference v6.0 Benchmark Results (abril 2026) — https://www.globenewswire.com/news-release/2026/04/01/3266801/0/en/MLCommons-Releases-New-MLPerf-Inference-v6-0-Benchmark-Results.html
MLCommons, MLPerf Inference v4.1 Benchmark Results (agosto 2024) — https://www.businesswire.com/news/home/20240828886616/en/New-MLPerf-Inference-v4.1-Benchmark-Results-Highlight-Rapid-Hardware-and-Software-Innovations-in-Generative-AI-Systems
NVIDIA Developer Blog, NVIDIA Blackwell delivers massive performance leaps in MLPerf Inference v5.0 — https://developer.nvidia.com/blog/nvidia-blackwell-delivers-massive-performance-leaps-in-mlperf-inference-v5-0/
NVIDIA Developer Blog, NVIDIA Blackwell Ultra sets new inference records in MLPerf debut — https://developer.nvidia.com/blog/nvidia-blackwell-ultra-sets-new-inference-records-in-mlperf-debut/
MLCommons, MLPerf Automotive (octubre 2025) — https://arxiv.org/html/2510.27065v1
Tschand et al., MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts — https://arxiv.org/abs/2410.12032

Cost, performance and energy: the three axes of LLM inference and the identity that ties them together

Sat, 13 Jun 2026 03:30:00 +0200

Notation: amounts in euros (N €), decimals with a point. Cost and energy references are European (France, Germany, Spain), because the aim is to defend a sovereign European architecture; when a source quotes prices in dollars it is flagged as “USD”. The dollar symbol is not used (on this site it is a formula delimiter).

What this series is

This is the first instalment of a series of 28 data articles (not opinion pieces) about the three dimensions that decide whether a sovereign European LLM platform is sustainable: cost (FinOps), performance (benchmarking) and energy. The aim is to have a body of verifiable figures, with their methodology, to support architecture decisions with numbers rather than intuition. This first article sets the frame: what each axis measures, how they link to one another, an end-to-end worked example with European data and the 2026 references that will be used as a base.

The quantitative thesis, in one sentence: the three axes are not independent. Throughput (tokens per second) is the common denominator of cost per token and energy per token, and energy is, on top of that, between 30 % and 50 % of the cost itself. Optimising one moves the other two. And in Europe, where the cluster sits (France, Germany, Spain) changes both the electricity cost and the carbon per token at once.

The three axes and their metrics

Axis	Question	Key metrics	Unit	Tool family
Cost (FinOps)	How much does each unit of work cost?	cost/1M tokens (CPM), cost/request, €/GPU-hour, utilisation	€/1M tok, €/h	OpenCost, Kubecost, CloudZero
Performance	How much work per second and at what latency?	TTFT, TPOT/ITL, throughput, goodput, P50/P99	tok/s, ms	GuideLLM, AIPerf, LLMPerf, MLPerf Inference
Energy	How many watts per unit of work?	energy/token, average power, PUE, carbon	J/tok, Wh, kWh, gCO₂	Zeus, CodeCarbon, MLPerf Power, Kepler

Each axis has its own in-depth introductory article in this same batch; here only how they fit together matters. It is worth breaking each one down a little more before crossing them.

Cost axis in detail

Metric	Definition	Why it matters
CPM (cost/1M tokens)	cluster cost ÷ tokens produced	the comparable unit on-prem vs cloud
Cost per request	cost attributed to one complete request	billing by real usage
€/GPU-hour	hourly cost of a GPU (amortised or rented)	the basis of any calculation
Utilisation	fraction of the GPU actually used	the lever that moves everything else
Fixed vs variable cost	amortised capex vs energy/operations	defines the break-even point

Performance axis in detail

Metric	Definition	Dominant phase
TTFT	latency to the first token	prefill
TPOT / ITL	time between output tokens	decode
Throughput	tokens or requests per second of the system	both
Goodput	throughput that meets the SLO (not the raw figure)	the honest number
P95 / P99	latency percentiles (not the mean)	the real experience

Energy axis in detail

Metric	Definition	Unit
Average power	average instantaneous consumption	W
Energy per token	energy per generated token	J/token
Total energy	integral of power over time	Wh, kWh
PUE	datacenter overhead (cooling, losses)	ratio ≥ 1
Carbon	energy × grid intensity	gCO₂eq

The identity that ties the three axes together: throughput

The cost per million tokens (CPM) is the cluster’s cost divided by what it produces:

$$\text{CPM} = \frac{\text{cluster cost per hour}}{\text{throughput (tok/s)} \times 3600 / 10^6}$$

Energy per token has exactly the same form, with power in the numerator:

$$\text{energy per token (J)} = \frac{\text{average power (W)}}{\text{throughput (tok/s)}}$$

The two share the same denominator: throughput. Hence the key operational consequence: every performance improvement that raises tokens/s lowers both cost per token and energy per token at the same time. These are not three separate optimisations; throughput moves them together. That is why performance benchmarking (axis 2) is, at bottom, also a FinOps tool and an energy efficiency tool.

The CPM↔throughput relationship is direct: the higher the throughput, the lower the CPM. It is the identity on which the whole cost analysis of the series is built.

Worked example: unit economics of an 8×H100 node in Europe

To see the identity in action, an end-to-end calculation on an example node (8×H100 SXM, Llama 3.1 70B FP16 served with vLLM), with European prices. Order-of-magnitude figures; the methodology is what matters.

Starting data:

Parameter	Value	Source
European cloud rental (Scaleway, per H100)	2.73 €/h	Scaleway
→ 8×H100 node on-demand	~21.84 €/h	8 × 2.73
Amortised on-prem cost (estimate)	~10–13 €/h	capex 24–36 months + opex
Throughput	2,800 tok/s	reference benchmark
Power per H100 (TDP)	up to 700 W	NVIDIA
Reference PUE	1.4	typical of a datacenter

Cost per million tokens. At 2,800 tok/s, hourly throughput is 2,800 × 3,600 / 10⁶ = 10.08 million tokens/hour. Therefore:

$$\text{CPM (cloud)} = \frac{21{.}84}{10{.}08} \approx 2{.}17 \text{ €/1M tok}$$

With the amortised on-prem cost (~11 €/h) it drops to ~1.09 €/1M tok, the difference between renting from a provider and amortising your own iron.

Energy per token. The power of the 8 cards is 8 × 700 = 5,600 W (board only). Over 2,800 tok/s:

$$\text{energy per token} = \frac{5,600}{2,800} = 2 \text{ J/token (board)}$$

Over a million tokens that is 2 × 10⁶ J = 0.56 kWh (GPU only). Applying PUE 1.4: ~0.78 kWh per million tokens. At French wholesale prices (~0.058 €/kWh) that is ~0.045 € of electricity per million tokens; in Germany (~0.091 €/kWh), ~0.071 €, some 57 % more, because of the grid, not the model.

The link. If a serving optimisation raised throughput from 2,800 to 4,200 tok/s (+50 %), the CPM would fall to ~1.45 €/1M tok and energy per token to ~1.33 J, both at once, through the same denominator. That is the identity, with numbers.

Energy is inside the cost

The second link: energy is not an axis separate from cost, it is a large component of it. 2026 reference data:

Figure	Value	Source
Electricity as a share of TCO (5 years)	30–40 % (>50 % at high density 24/7)	Spheron
Day-ahead wholesale France (Mar-2026)	~58 €/MWh (0.058 €/kWh)	TradingEconomics
Day-ahead wholesale Spain	~54 €/MWh (0.054 €/kWh)	TradingEconomics
Day-ahead wholesale Germany	~91 €/MWh (0.091 €/kWh)	TradingEconomics
Electricity cost/token at 80 % util	¼ of that at 20 % util	Spheron

A methodological warning: the wholesale price (day-ahead) is not what a company pays; industrial retail adds grid charges and taxes (roughly ×1.5–2). But the relative ordering holds: France and Spain, cheap; Germany, expensive.

The third link, utilisation, closes the triangle: power is consumed even when the GPU is idle, so spreading the fixed cost (energy included) over more tokens (more throughput, more occupancy) lowers all three axes at once. The electricity cost per token at 80 % utilisation is a quarter of what it is at 20 %.

Sensitivity analysis: what moves the cost per token

Four levers dominate the CPM, and it is worth knowing their elasticity before defending any figure:

Lever	Change	Effect on CPM	Mechanism
Throughput	+50 % tok/s	−33 % CPM	direct denominator
Utilisation	20 % → 80 %	up to −75 % on the electricity portion	spreads fixed cost
Location (energy)	Germany → France	lowers the electricity portion (~37 % less €/kWh)	wholesale FR < DE
Amortise vs rent	21.84 → 11 €/h	−50 % CPM	removes the provider’s margin
Quantisation (FP8/INT4)	+throughput, −VRAM/KV	lowers CPM and J/token	same identity

The reading: the CPM is not a number, it is a function of utilisation, throughput, location (energy price) and ownership model. Any on-prem vs cloud comparison that does not fix those assumptions is not comparable, and that is exactly the job of the synthesis article.

Simplified TCO model (on-premise)

The real on-premise cost is not the GPU alone: it is amortised capex + opex, over a hardware cycle of 24–36 months. Reference structure of an inference node:

Component	Type	Typical weight	Notes
GPUs	capex	the bulk	amortise over 24–36 months
Server, network, BMC	capex	medium	NICs, switching, out-of-band management
Electricity	opex	30–50 % of TCO	depending on country (FR/ES cheap, DE expensive) and PUE
Cooling	opex	inside the PUE	PUE 1.2–1.6 typical
Maintenance and support	opex	medium	spares, contracts, RMA
Staff and operations	opex	variable	the hidden cost that cloud rental includes

The difference between cloud rental (21.84 €/h in the example) and the real amortised cost (~10–13 €/h) is this breakdown: the provider charges a margin for absorbing capex, cooling and operations. The break-even point appears when sustained volume exceeds ~2M tokens/day, with typical amortisation in 6–12 months (Spheron). Below that threshold, renting usually wins; above it, amortising. Article A8 develops the full model and synthesis S3 connects it with sizing.

Reference: the price of a GPU-hour in European sovereign cloud

The other side of the on-prem comparison is the price of renting the GPU. For a sovereign European platform, what matters is not AWS/GCP/Azure (subject to the US CLOUD Act), but the European providers:

Provider	H100 (€/hour)	Location	Sovereignty
Scaleway	from 2.73	Paris, Warsaw	EU jurisdiction, exempt from US CLOUD Act
OVHcloud	+40–45 % vs Scaleway	France, Germany	EU jurisdiction, exempt from US CLOUD Act
Hyperscalers (reference)	3–7 USD	global	subject to US CLOUD Act

The data: Scaleway offers the H100 from 2.73 €/h in European datacenters (Paris, Warsaw), with models and data under EU jurisdiction, immune to extraterritorial regulation; OVHcloud serves it from France and Germany, ~40–45 % more expensive than Scaleway as of March 2026. Both are exempt from the US CLOUD Act, the foundational sovereignty requirement for data subject to GDPR (Scaleway, DeployBase).

The implication for the proposal: the comparison is not only cost, it is cost + sovereignty. An on-prem cluster of your own or a European cloud keep the data under EU jurisdiction; a US hyperscaler does not, however cheap the GPU-hour comes out. That is the leg that the cost axis on its own does not capture, and that synthesis S2 quantifies.

Energy and carbon reference: France, Germany, Spain

Here the choice of European country changes both the electricity cost and the carbon per token. It is the most important table for a sovereign proposal:

Country	Wholesale (€/MWh, Mar-2026)	Grid carbon (gCO₂/kWh)	Profile
France	~58	~20–60	nuclear: cheap and very clean
Spain	~54	~150–170	renewables + gas: cheap, medium carbon
Germany	~91	~363 (2024 average)	coal/gas + renewables: expensive and dirty

Sources: day-ahead wholesale prices (TradingEconomics), German carbon ~363 gCO₂/kWh (2024 average, Statista via Wikipedia), nuclear ~17–35 gCO₂e/kWh (EEA), Spain ~167 gCO₂/kWh (2020). The PUE multiplies all of that energy: at 1.4, for every kWh of GPU the datacenter spends 1.4 kWh with cooling included.

The calculation, on the example node (0.78 kWh per million tokens, with PUE): in France (~40 gCO₂/kWh) it is ~31 gCO₂ per million tokens; in Germany (~363 gCO₂/kWh), ~283 gCO₂ per million, a factor of ~9×. That is, the same load served in France emits of the order of 9× less carbon per token than in Germany, and costs ~37 % less in electricity, without touching a single line of the stack, purely because of the country’s grid. Siting the cluster in France or Spain is at once a cost lever, a carbon lever and a compliance lever. Hourly grid intensity is obtained from ElectricityMaps, the reference source for these calculations.

Scale context: global datacenters consumed ~460 TWh in 2025 (≈1.8 % of the world’s electricity), and the IEA projects passing 800 TWh in 2028, which is why energy efficiency per token stops being a detail and becomes a design constraint, and why carbon enters the European regulatory equation.

Sovereignty and regulation: the axis that does not appear in the CPM

For a European platform there is a fourth factor that the three numerical axes do not capture, and that can invalidate the lowest cost figure: jurisdiction. Three fronts that turn “the cheapest GPU-hour” into “the cheapest GPU-hour that complies”:

Front	What it requires	Architectural implication
GDPR + US CLOUD Act	personal data under a jurisdiction accessible to US authorities is exposed	European cloud (Scaleway/OVHcloud) or on-prem keep the data in the EU
EU AI Act	obligations for high-risk systems (traceability, control, documentation)	on-prem control makes auditability and compliance easier
CSRD / emissions reporting	reporting the carbon footprint (scope 2)	a low-carbon grid (France) improves the reported figure

The key point for the proposal: a US hyperscaler can offer a competitive GPU-hour, but it puts the data under the US CLOUD Act, which for data subject to GDPR is a compliance risk, not a saving. That is why this series’ comparison is not made in USD against US providers: it is made in euros, against options that keep data sovereignty, on-prem of your own or European cloud. Cost, energy and carbon are measurable axes; sovereignty is the constraint that decides which options even enter the table. This links with the mapping of the EU AI Act onto the on-premise architecture and with the technical controls ENS × ISO 42001 × EU AI Act.

The operational conclusion: France and Spain not only come out cheaper and cleaner than Germany; being EU jurisdiction, they meet the sovereignty constraint that a US hyperscaler does not meet at any price. Three advantages that reinforce each other, and that the proposal can defend with the same table.

Methodology: how to read (and not fool yourself with) these figures

Five warnings that apply to all the data in this series:

Board vs node. The GPU’s power (700 W per H100) is the floor; the complete node (CPU, NICs, power supplies, fans) consumes more. Always compare the same boundary.
Wholesale vs retail. The day-ahead price is not what the company pays; industrial retail adds grid charges and taxes. Use whichever matches your contract.
Measured vs estimated. A wattmeter figure and an nvidia-smi estimate are not the same thing; the difference is quantified in the energy articles.
Amortisation assumptions. The on-prem cost/hour depends on the hardware cycle (24 vs 36 months) and on the assumed utilisation; changing those assumptions moves the CPM more than any serving optimisation.
Comparability. Two CPM or J/token figures are only comparable if model, precision, context length, load and country match. Most public comparisons (and in USD) do not fix those assumptions, which is why the series reproduces the ones that matter with its own harness and European data.

The combined scorecard

The destination of the series is a single board where each model/configuration/location is scored on all the axes at once, to decide on a Pareto frontier rather than on a single number. An illustrative example (order-of-magnitude figures) on the same 8×H100 node:

Candidate	CPM (€/1M)	Goodput (tok/s)	Energy (J/tok)	Carbon (gCO₂/1M)	Sovereignty
A · 70B FP16, on-prem France	~1.09	~2,800	~2.0	low (grid ~40 gCO₂/kWh)	EU
B · 70B FP8, on-prem France	~0.75	~4,000	~1.4	lower	EU
C · Scaleway on-demand	~2.17	~2,800	n/a	low (FR)	EU
D · US hyperscaler	~3.0	~2,800	n/a	depends on region	not EU

The reading: B dominates A on the three intrinsic axes thanks to FP8. C (European cloud) is more expensive per token but shifts capex and operations, while keeping sovereignty. D (hyperscaler) may be competitive on cost but loses sovereignty, an axis that for GDPR data is not negotiable. No isolated number decides: the choice comes from seeing the whole row.

The trap the board avoids: optimising a single axis. Each cell will be filled in by an article in the series with measured data.

Where each axis is measured: from design to production

Each axis is measured at three different moments, with different tools and increasing precision. Confusing them is a common source of figures that do not add up:

Axis	Design (estimate)	Benchmark (controlled)	Production (continuous)
Cost	datasheet, GPU-hour price	CPM measured over synthetic load	OpenCost/Kubecost per team and token
Performance	theoretical throughput (roofline)	GuideLLM/AIPerf with SLO	OTel/Langfuse over real traffic
Energy	TDP × hours	Zeus/MLPerf Power on the bench	Kepler/DCGM per pod in the cluster

The rule: a design figure serves for sizing, but not for defending a proposal; for that you need a reproducible benchmark figure, and to operate, a continuous production one. When someone disputes a number in the proposal, the answer is not “a blog says so”, but “here is the bench, here is the methodology, reproduce it”.

Glossary of units and conversions

Quantity	Unit	Conversion / reference
Energy	joule (J), watt-hour (Wh)	1 Wh = 3,600 J; 1 kWh = 3.6 MJ = 3,600,000 J
Power	watt (W)	1 W = 1 J/s; H100 ≈ 700 W TDP
Energy per token	J/token	= average power (W) ÷ throughput (tok/s)
Throughput	tok/s	tokens generated per second by the system
Cost per token	€/1M tok (CPM)	= cost/hour ÷ (tok/s × 3,600 / 10⁶)
Latency	ms	TTFT (first token), TPOT/ITL (between tokens)
PUE	ratio (≥1)	total DC energy ÷ compute energy
Carbon	gCO₂eq	= energy (kWh) × PUE × grid intensity (gCO₂/kWh)

A useful conversion to keep in your head: at a constant 700 W, an H100 consumes ~0.7 kWh per hour, ~16.8 kWh a day and ~6,130 kWh a year (board only, no PUE). Multiplied by the PUE and your country’s energy price (FR ~0.058, ES ~0.054, DE ~0.091 €/kWh wholesale), that number is the basis of the electricity portion of the TCO.

How the rest of the series fits together

The 28 articles group into four tracks; this is the map:

Track	What it contributes	Closes with
FinOps (8)	measuring and allocating cost, TCO model	cost/token comparable on-prem vs European cloud
LLM benchmarking (8)	measuring performance reproducibly	Pareto frontier of engines
Energy (8)	measuring watts and carbon per token	energy in the TCO and in European regulation
Synthesis (4)	crossing the three axes	scorecard, on-prem vs cloud, sizing, harness

The three articles that follow this one in the batch introduce each axis in depth: their frameworks and the state of the art. From there, each article adds a layer of data to the final scorecard.

The thread running through all 28 instalments is the one in this article: cost, performance and energy are a single problem with three faces, tied together by throughput and by the energy that lives inside the cost, and in Europe, by the country where the cluster sits. Anyone who optimises one face while ignoring the others will end up with a pretty number and a platform that does not hold up. The work of the series is to produce the figures, measured, dated, European and reproducible, that allow a sovereign architecture to be defended before any committee with the only answer that closes a technical debate: the data and the bench to reproduce it.

Sources

Scaleway · H100 GPU instance (€/h price, EU sovereignty) — https://www.scaleway.com/en/h100/
DeployBase · OVHcloud GPU Pricing (European sovereignty) — https://deploybase.ai/articles/ovhcloud-gpu-pricing
TradingEconomics · electricity price France / Germany / Spain — https://tradingeconomics.com/france/electricity-price
EEA · emission intensity of electricity in Europe — https://www.eea.europa.eu/en/analysis/indicators/greenhouse-gas-emission-intensity-of-1
Electricity sector in Germany (carbon ~363 gCO₂/kWh) — https://en.wikipedia.org/wiki/Electricity_sector_in_Germany
Spheron · AI Inference Power & Cost Economics (2026) — https://www.spheron.network/blog/ai-inference-cost-economics-2026/
NVIDIA · H100 (TDP/specs) — https://www.nvidia.com/en-us/data-center/h100/

Serving reasoning models: the invisible scratchpad that decides your latency and your bill

Fri, 12 Jun 2026 05:30:00 +0200

TL;DR

A reasoning model generates, before the answer you see, a thinking block, hundreds or thousands of tokens enclosed between <think> and </think>, that the user does not read but that consumes the same compute, the same VRAM and the same bill as any other token. Serving them in production is not “serving a bigger model”: it is serving a model whose cost per request is variable and, by default, uncontrolled. Three levers change the equation: choosing well between think / non-think mode (the hybrid models of 2026 allow it within the same weights), setting a reasoning budget (thinking_token_budget in vLLM, s1-style budget forcing) and measuring the impact on the KV cache (a 30k-token reasoning chain can eat ~9 GB of KV in FP16). Managed well, you gain accuracy where it matters; managed badly, you pay up to 113× more energy for an answer that did not need that much thinking.

The analogy

Picture a candidate sitting a competitive exam. Before writing the good answer on the official sheet, they fill three sheets of scratch paper: they try approaches, correct themselves, discard. Those scratch sheets are not handed in, the board only reads the final answer, but the candidate has spent ink, time and half an hour of the exam on them.

A reasoning model does exactly that. The <think>…</think> block is the scratch paper: it reasons out loud, contradicts itself, backtracks, and finally emits the “clean” answer. The production problem is that you pay for the scratch paper too: every reasoning token takes a slot in the batch, grows the KV cache and adds to the bill just like an output token. And, like any nervous candidate, the model tends to overthink: it writes five sheets when one would have done.

The operational question in this post is not “does it reason well?”, but “how much scratch paper do I let it write, and when do I take the pen away?”.

What changes when the model “thinks”

In a normal model, a request is: prompt → output tokens. In a reasoning model the sequence has two stretches:

Reasoning block (<think> … </think>): the chain of thought (CoT). Long, variable, normally hidden from the end user.
Final answer: what the user sees.

vLLM models this explicitly. When started with a reasoning parser, the output carries a reasoning_content field separate from content:

vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
 --reasoning-parser deepseek_r1

The parser detects the thinking delimiters and separates both stretches in the API response (vLLM · Reasoning Outputs). This matters for serving for one practical reason: if your gateway or your UI does not separate reasoning_content from content, you will end up showing the scratch paper, or, worse, recording it in logs and traces without meaning to.

The 2026 landscape: from the model that “always reasons” to the switchable hybrid

The first generation (o1, DeepSeek-R1, QwQ-32B) were models that always reasoned. In 2026 the dominant pattern is the hybrid: a single set of weights that switches between think and non-think.

DeepSeek-V3.1 introduced a hybrid reasoning architecture in a single model: DeepSeek-V3.1-Think for complex problems and a faster direct mode for the trivial, switchable by the user (InfoQ). Important for serving: reasoning is off by default and is turned on with thinking=True in chat_template_kwargs.
Qwen3 brings Hybrid Thinking Modes and an explicit Thinking Budget to bound the reasoning compute; in its series thinking is on by default (like QwQ-32B).
gpt-oss shows a different reasoning structure from the R1/Qwen family, which in practice means that the parser and the delimiters are not universal: each family has its own.

The architectural consequence is clear: in 2026 you no longer decide “a model that reasons” vs “one that does not”, but when you let the same model reason. That turns reasoning into a routing and budgeting decision, not a model-catalogue one. It connects directly with the L7 inference router: the “cheap” route turns thinking off; the “expensive” one turns it on with a cap.

The economics: why the scratch paper drives up the bill

Here is the heart of the problem. Reasoning tokens are invisible to the user but real in compute and in billing. Providers charge them as output tokens; API providers bill those reasoning tokens at the output price, not for free (Test-Time Compute Quietly Changed the Economics of Inference). On-premise there is no external invoice, but there is a cost: they occupy batch slots, KV VRAM and watts.

The billable cost (or the equivalent cost in GPU·hours) of a request is:

$$\text{compute tokens} = \text{prompt} + \underbrace{\text{reasoning tokens}}_{\text{hidden, variable}} + \text{visible answer}$$

The middle term is the one that runs out of control. On hard tasks the reasoning/visible-answer ratio can be enormous, and empirical analysis has measured up to 113× the energy cost against standard inference on certain coding tasks (Towards Data Science · Inference Scaling).

And it does not always buy accuracy. The overthinking phenomenon is documented: more thinking does not guarantee a better answer, and beyond a certain point it makes things worse. There is work showing that shorter chains are up to 34.5 % more accurate than the longest one sampled for the same question (Don’t Overthink it), plus studies on the “mirage” of test-time scaling (Mirage of Test-Time Scaling) and on when more thinking hurts (When More Thinking Hurts). The first large-scale study of test-time scaling, more than 30 billion tokens generated with 8 open-source LLMs from 7B to 235B across 4 reasoning datasets, systematises where it scales and where it does not (The Art of Scaling Test-Time Compute).

The operational reading: reasoning is a cost you have to budget for, not a “more quality” switch you leave on full all the time.

Controlling the thinking budget

Three mechanisms, from coarsest to finest.

1. Switching think / non-think

The cheapest thing is not to reason when it is not needed. On hybrid models, turn thinking off for classification, extraction, formatting, trivial answers:

# DeepSeek-V3.1: reasoning OFF by default; turned on explicitly
chat_template_kwargs = {"thinking": True} # only on the "hard" route

This is 80 % of the saving for 20 % of the effort: route by difficulty and reserve thinking for what actually pays for it.

2. Setting a cap: `thinking_token_budget`

vLLM ships a sampling parameter thinking_token_budget that counts reasoning tokens and, on reaching the limit, forces the block to close (reasoning_end_str, typically </think>), making the model answer straight away. If it is not specified, there is no limit other than max_tokens (PR #37112, implementation note). Underneath, a ReasoningBudgetLogitsProcessor injects something like “Let me stop thinking and answer now.</think>” when the cap is reached.

# Define the delimiters of the reasoning block
vllm serve Qwen/Qwen3-32B \
 --reasoning-parser qwen3 \
 --reasoning-config '{"reasoning_start_str":"<think>","reasoning_end_str":"</think>"}'

// per request: cut the scratch paper off at 1024 tokens
{"messages":[...], "thinking_token_budget": 1024}

3. Budget forcing (state of the art)

The technique that popularised fine-grained control is budget forcing, from the s1 paper: to spend less, thinking is terminated by forcing the end delimiter; to spend more, the end is suppressed and the word “Wait” is appended, which pushes the model to review and often correct its own reasoning. With this, s1-32B (a Qwen2.5-32B-Instruct fine-tuned on just 1,000 examples) beat o1-preview by as much as 27 % on AIME24/MATH (s1: Simple Test-Time Scaling). The serving lesson: the budget is not only a cost cap, it is a bidirectional quality lever.

$$\text{accuracy} \approx f(\text{reasoning tokens}) \quad\text{with a maximum, not monotonic}$$

The curve rises, saturates and on many tasks falls. Your job is to operate it near the knee, not at the extreme.

The impact almost nobody measures: the KV cache

This is where reasoning really bites serving. Every token of scratch paper is one more token in the KV cache. A long reasoning chain inflates the working memory just like a long input context.

The KV size grows linearly with the sequence (reasoning included):

$$\text{KV bytes} = 2 \times L \times h_{kv} \times d_{head} \times s \times b$$

where $L$ is layers, $h_{kv}$ KV heads (few, with GQA), $d_{head}$ the per-head dimension, $s$ the sequence length and $b$ the bytes per element. The term reasoning blows up is $s$: a “normal” 500-token answer and one with 8,000 tokens of thinking do not take up the same space, not by a long way.

The real numbers are frightening: for a distilled reasoning model with standard attention such as DeepSeek-R1-Distill-Llama-70B, a single 30,000-token reasoning chain consumes ~9 GB of KV in FP16, which quantising the KV to FP8 reduces to ~4.5 GB, one of the highest-impact optimisations for chain-of-thought workloads (Spheron · KV Cache Optimization). Multiply that by concurrency and you understand why a cluster that served 200 simultaneous requests of a “normal” model drops to 40 with the same model reasoning.

Two state-of-the-art nuances worth knowing:

Prefix caching is not free here. The empirical study of reasoning-model serving finds that prefix caching clearly improves speed on models of 14B or more, but hurts on 7B models (cache management costs more than it saves) (Reasoning Language Model Inference Serving Unveiled). And since the bulk of the reasoning cost sits in the decode (generating tokens one by one), prefix caching, which only speeds up the prefill, barely touches the expensive phase. It connects with prefix cache: engineering the hit rate.
Reasoning-specific KV compression. Techniques such as R-KV have appeared, compressing the KV by exploiting the redundancy typical of chains of thought (the model repeats itself and rambles), recovering memory without touching quality (R-KV).

The most profitable and available lever today is still KV in FP8 (see FP8 end to end) combined with a reasoning budget that avoids 30k-token tails in the first place.

Reasoning + structured output: order matters

If you serve structured output (JSON with a schema) and reasoning at the same time, there is a subtle trap: the grammar must not be applied during the thinking block. The model needs free text to reason; forcing the schema inside the <think> breaks the reasoning. vLLM solves it by having the structured-output engine (xgrammar) use the reasoner’s end_token_id to skip the constraint while the block lasts and apply it only to the final answer (vLLM · Reasoning Outputs). If you build this by hand on another engine, remember: first let it think, then impose the form. It links with structured output.

Reference architecture

On the blog’s example cluster, a node of 4×H100 SXM (80 GB, NVLink), a sensible topology for serving reasoning without wrecking the SLO:

A cheap difficulty classifier up front (a small model or rules) that picks the route. Most of the traffic does not need to think.
Fast route (non-think): the same hybrid model with reasoning off. Low TTFT, predictable cost.
Reasoning route (think): reasoning on with thinking_token_budget by default (say 1–2k tokens) and the option to raise it per request for the genuinely hard cases.
KV in FP8 in the engine on the reasoning route, plus per-request reasoning-length metrics exported to your observability (instrumenting vLLM with OTel).

To prototype this routing outside the cluster, an RTX 5090 (Blackwell, 32 GB) serves a 7–14B reasoning model and lets you validate parsers, budgets and the reasoning_content/content separation before touching the H100s. Do not expect to serve a 32B in think mode at high concurrency on a consumer card: the reasoning KV eats the 32 GB in no time.

Sizing starts, as always, from an SLO; the difference is that now the “answer size” in your capacity planning has to count the scratch paper, not just the visible answer.

Operational pitfalls (and honest scepticism)

Leaking reasoning_content. If the gateway does not separate it, you end up showing or logging the scratch paper, which may contain wrong attempts, sensitive data or language you do not want in your UI. Always separate it at L7.
Serving reasoning “just in case”. Overthinking is real and sometimes lowers accuracy. Do not turn think on by default: turn it on per route, with a budget.
Forgetting the KV. The most common mistake: sizing concurrency from the length of the visible answer. The KV is dictated by the total sequence, reasoning included. Measure p50/p95 reasoning tokens per endpoint.
Prefix caching as a silver bullet. On small models it can make things worse, and in any case it does not touch the decode, which is where the reasoning cost goes.
Assuming universal delimiters. R1, Qwen3 and gpt-oss do not share a reasoning structure. Use the right --reasoning-parser per family and test streaming (the parser has to separate correctly token by token).
Believing more compute = more intelligence. The state of the art (s1, short-m@k, “When More Thinking Hurts”) points the other way in many cases: there is a knee, and operating beyond it is burning watts. Test-time compute is a tool with diminishing, and sometimes negative, returns, not a linear lever.

A note of caution for June 2026: the field of reasoning control (budget forcing, reasoning-specific KV compression, thinking early-exit) moves fast and almost all of it is very recent. The specific figures, the 113×, the 34.5 %, the 9 GB, depend on model, task and configuration; take them as an order of magnitude for design, not as constants. Measure on your own workload before committing to SLOs.

Closing

Serving a reasoning model means serving a model whose cost per request you decide with the thinking budget, not the catalogue. The chain goes: route by difficulty → turn think off where it adds nothing → set thinking_token_budget by default on the route that thinks → quantise the KV to FP8 → measure reasoning length as a first-class metric. Do that, and reasoning becomes a capability you pay for only when it buys accuracy. Do not, and your cluster will serve a fifth of the users while the model fills sheets of scratch paper nobody is going to read.

Sources

vLLM · Reasoning Outputs — https://docs.vllm.ai/en/stable/features/reasoning_outputs/
vLLM PR #37112 · reasoning_budget — https://github.com/vllm-project/vllm/pull/37112
Adding Reasoning Budget to vLLM — https://shuyo.wordpress.com/2026/03/26/adding-reasoning-budget-to-vllm-reasoning-token-limit/
s1: Simple Test-Time Scaling — https://arxiv.org/abs/2501.19393
The Art of Scaling Test-Time Compute — https://arxiv.org/abs/2512.02008
Don’t Overthink it (short-m@k) — https://arxiv.org/pdf/2505.17813
Mirage of Test-Time Scaling — https://arxiv.org/pdf/2506.04210
When More Thinking Hurts — https://arxiv.org/html/2604.10739v1
Reasoning Language Model Inference Serving Unveiled — https://arxiv.org/pdf/2510.18672
R-KV: Redundancy-aware KV Cache Compression — https://arxiv.org/pdf/2505.24133
Spheron · KV Cache Optimization Guide (2026) — https://www.spheron.network/blog/kv-cache-optimization-guide/
Test-Time Compute Quietly Changed the Economics of Inference — https://medium.com/@Elongated_musk/test-time-compute-quietly-changed-the-economics-of-inference-9d0b8d77641c
Inference Scaling (Test-Time Compute) · Towards Data Science — https://towardsdatascience.com/inference-scaling-test-time-compute-why-reasoning-models-raise-your-compute-bill/
DeepSeek-V3.1 Hybrid Reasoning · InfoQ — https://www.infoq.com/news/2025/09/deepseek-v31-hybrid/

Long context and KV offloading: when the notebook does not fit on the desk

Fri, 12 Jun 2026 05:00:00 +0200

TL;DR

Serving long context is not serving a smarter model, it is managing a notebook that does not fit on the desk. The context window is set by the model (and extended with RoPE scaling / YaRN), but the cost of serving it is set by the KV cache, which grows linearly with sequence length and, in production, explodes: a 300k-token contract on Llama 3 70B eats ~93 GB of KV, more than an entire H100, and a million tokens asks for ~125 GB. When the KV does not fit in HBM, there are only two ways out: recompute (extremely expensive, attention is quadratic) or offload the KV to a cheaper memory hierarchy: DRAM, NVMe, network. The 2026 OSS state of the art, LMCache, Mooncake (Kimi’s platform) and NVIDIA Dynamo/KVBM, turns that offload into a first-class KV layer, with reuse across requests, prefill/decode disaggregation and cache-aware routing. Reported results: 3×–10× less latency with LMCache and up to +525 % throughput in long-context scenarios with Mooncake. The price: every memory hop adds transfer latency, so offloading only pays off when what you save by not recomputing beats what it costs to move the bytes.

The analogy

A researcher works at a small desk. On it fit the papers they are pulling from right now, which is the GPU’s HBM, blazingly fast but tiny. When the case is long (a thousand-page file, a million-token context), the papers do not fit. There are three options:

Throw papers away and ask the archive for them again every time they are needed. That is recomputing the KV: correct, but agonisingly slow, because “asking the archive” in an LLM means running the whole prompt through attention again, and attention is quadratic.
Put a shelf next to the desk (the CPU’s DRAM) and, further out, a store room in the basement (NVMe) and a depot in another building (network/object). Moving papers between the desk and the shelf costs seconds, not hours. That is hierarchical KV offloading.
Have two researchers split the work: one reads and marks up the whole file (prefill), then passes their notes to the second, who only drafts (decode). That is the KVCache-centric disaggregated architecture.

This post is about options 2 and 3, the ones that make long context viable in production. Option 1 is what you pay by default if you do nothing.

Part 1 · Why long context is a memory problem

The KV cache grows with the sequence

For every token that goes in or out, the model stores its key and value vectors in each layer so it does not have to recompute them. The size is:

$$\text{KV bytes} = 2 \times L \times h_{kv} \times d_{head} \times s \times b$$

with $L$ layers, $h_{kv}$ KV heads (few, with GQA), $d_{head}$ the per-head dimension, $s$ the sequence length and $b$ the bytes per element. Everything is a model constant except $s$: the KV is linear in context length. Doubling the context doubles the KV. That is the basis of KV cache: the working memory of inference.

The numbers are frightening

At production scale, that linear term turns brutal:

A 300k-token contract on Llama 3 70B consumes ~93 GB of KV alone, more than the 80 GB of an entire H100 (DigitalOcean · Long-Context Inference Cost).
A 1M-token context needs ~125 GB of KV, which exceeds both an RTX 4090 (24 GB) and an 80 GB A100 (Introl · Long-Context LLM Infrastructure).
Even a 7B at 128k climbs to ~14 GB of KV (against ~6 GB at 4k).

The KV stops being an implementation detail and becomes the resource that dictates your concurrency.

And on top of that, attention is quadratic

The KV is linear, but the compute of attention is quadratic in length. When the prompt reaches 1M tokens, generating each token can take on the order of 1.765 seconds, with more than 96 % of the latency spent in attention (Introl). The aggregate effect is a throughput collapse of 10×–100× against short contexts. This is why long context is not “fixed” with more VRAM alone: there is a memory problem (KV) and a compute problem (attention) at the same time.

Extending the window ≠ serving it cheaply

A common confusion: “my model supports 1M of context” does not mean “I can serve 1M cheaply”. The window is extended with RoPE scaling techniques such as YaRN, the practical option for fine-tuning open source models to long context: it needs 10× fewer training tokens and 2.5× fewer steps than naive RoPE interpolation, and it took LLaMA-2 from 4k to 32k and 128k (YaRN/LongRoPE). But that solves the model attending to long context, not the KV fitting on your GPU. According to the 2026 round-ups, May 2026 marked the first generation of open million-token models (with families supporting 256k extensible to 1M via YaRN) (letsdatascience). Treat it as a trend, not a closed spec, and verify capabilities per concrete model.

Part 2 · The KV memory hierarchy

The core idea of offloading is old in systems: hierarchical caching. “Hot” KV (what is in use) lives in HBM; “warm” KV drops to DRAM; “cold” KV to NVMe; “shared across nodes” to network or object storage. Every hop multiplies capacity and divides bandwidth.

Working out when offloading pays off

Offloading is not magic: moving KV from DRAM/NVMe back to HBM costs time. The rule is simple. Offloading pays off when the cost of recomputing exceeds the cost of transferring:

$$t_{recompute}(s) \;>\; t_{transfer} = \frac{\text{KV bytes}}{BW_{enlace}}$$

Since $t_{recompute}$ grows with attention (quadratic) and $t_{transfer}$ grows only with the size of the KV (linear) divided by the link bandwidth, the longer the context, the more the balance tips in favour of offloading. That is why offloading is the lever for long context: it is exactly where recomputing becomes unbearable. The LMCache documentation says it without hedging: offloading only helps when the recomputation it avoids is larger than the overhead it introduces (LMCache benchmark).

Part 3 · KV offloading in practice (OSS)

The limit of out-of-the-box prefix caching

vLLM already caches prefixes in HBM (see prefix cache: hit rate engineering and PagedAttention). The problem: it only lives in HBM, which is small. When the useful KV exceeds HBM, it is evicted and, on the next request, has to be recomputed. There is no speedup even with prefix caching enabled, because the cache is no longer there (vLLM vs LMCache benchmark). Native prefix caching is necessary but insufficient for long context.

LMCache: the persistent KV layer

LMCache adds persistent storage backends to vLLM’s prefix cache (and also supports SGLang and NVIDIA Dynamo as engines). It pulls the KV out of HBM and shares it across engines and across queries, covering a three-level hierarchy: GPU HBM, CPU DRAM and NVMe SSD, with additional backends for Redis/Valkey, Mooncake, InfiniStore, S3 and NIXL/GDS (GitHub LMCache, arXiv 2510.09665). Because DRAM/NVMe hold far more KV than HBM, the hit ratio rises, and LMCache reports 3×–10× less latency combined with vLLM. Against vLLM’s basic CPU offload, LMCache loads the KV at chunk level with high-performance CUDA kernels, which cuts transfer overhead.

A minimal start, offloading to CPU:

# vLLM with LMCache as the KV layer (offload to DRAM)
pip install lmcache
LMCACHE_CONFIG_FILE=lmcache.yaml \
vllm serve meta-llama/Llama-3.1-70B-Instruct \
 --tensor-parallel-size 4 \
 --kv-transfer-config '{"kv_connector":"LMCacheConnector","kv_role":"kv_both"}'

# lmcache.yaml — hierarchy HBM -> DRAM -> NVMe
chunk_size: 256
local_cpu: true # warm KV in DRAM
max_local_cpu_size: 200 # GB
local_disk: "file:///nvme/lmcache" # cold KV on NVMe
max_local_disk_size: 2000 # GB

Offloading to CPU adds moderate overhead but retains a good part of the benefit against recomputing (LMCache docs). Mind the physical path: that HBM↔DRAM↔NVMe traffic goes over PCIe and across NUMA nodes. It is worth reading PCIe topology, GPUDirect and ACS and NUMA, hugepages and CPU isolation, because a badly wired offload can eat its own gain.

Combine it with FP8

Offloading reduces where the KV lives; quantization reduces how much it weighs. They are orthogonal and they add up: KV in FP8 halves the size (see FP8 end-to-end), which means half the bytes to transfer on every offload. In long context, FP8 + offload is the default combination.

Part 4 · KVCache-centric / disaggregated architecture

The next level is not just offloading KV, but redesigning serving around the KV.

Mooncake (Kimi’s platform)

Mooncake, the serving platform behind Kimi (Moonshot AI), is the reference KVCache-centric disaggregated architecture: it separates the prefill and decode clusters, and exploits the underused CPU, DRAM and SSD resources of the GPU cluster to build a distributed KV cache. Its core is a KVCache-centric scheduler that maximises throughput while respecting SLOs, with a prediction-based early rejection policy for overloaded scenarios (arXiv 2407.00079, USENIX FAST'25). The long-context numbers are striking: up to +525 % throughput in certain simulated scenarios while respecting the SLO, and in production Kimi handles 115 % and 107 % more requests on A800 and H800 clusters respectively against previous systems. The title of their FAST talk sums it up: “Trading More Storage for Less Computation”, exactly the rule from Part 2.

NVIDIA Dynamo and KVBM

NVIDIA Dynamo is the open source distributed serving framework with prefill/decode disaggregation, dynamic GPU scheduling and KV-aware routing: the PrefillRouter computes overlap scores between the incoming request and the KV blocks already cached, and routes to the GPUs that already have the relevant KV, avoiding recomputation (Dynamo · Disaggregated Serving, Dynamo · KV-aware routing). KV transfer between prefill and decode is handled by NIXL directly GPU-to-GPU over the best available transport (NVLink, InfiniBand). And its offload manager, KVBM (KV Block Manager), has a three-layer architecture (LLM runtime, logical block management, NIXL transport) that offloads cold KV to CPU RAM, NVMe or network storage to free up HBM (Dynamo · KV Cache Offloading). This connects with disaggregated serving: prefill and decode in specialised pods.

The llm-d project walks the same road, from vLLM’s prefix caching to KV-aware distributed scheduling (llm-d · KV-Cache Wins You Can See).

Part 5 · Reference architecture

On the blog’s example cluster, a node with 4×H100 SXM (80 GB, NVLink), local NVMe and plenty of DRAM, a sensible stack to serve long context without recomputing pointlessly:

KV in FP8 in the engine (vLLM), to start with half the bytes.
LMCache as the KV layer with a DRAM→NVMe hierarchy, so that long prefixes (a manual, a code base, a case file) are cached once and reused across sessions without recomputation.
KV-aware routing (Dynamo or llm-d) if you have several nodes: send the request to the GPU that already holds the prefix, not to any GPU at all.
Prefill/decode disaggregation once the prefill of enormous contexts starts stealing decode from the other requests. Prefilling 300k tokens monopolises the GPU and kills everyone’s latency.
Metrics for KV hit ratio, bytes transferred per level and prefill/decode ratio, exported to your observability stack (instrumenting vLLM with OTel). Without those three metrics, offloading is faith, not engineering.

To prototype the stack, validating the LMCache config, the backend formats, the hit/miss behaviour, an RTX 5090 (Blackwell, 32 GB) with NVMe is more than enough to serve a 7–14B and watch offloading work at contexts of 32–128k. Do not expect to serve 1M on a consumer card: the long-context problem is precisely that neither the KV nor the attention fit on small hardware. Real sizing starts from the SLO and from the context-length distribution of your traffic, not from the model’s maximum window, and that is where it links to capacity planning.

Part 6 · Operational pitfalls (and honest scepticism)

Believing that native prefix caching is enough. It only lives in HBM; in long context it is evicted and you recompute. You need a persistent layer (LMCache/KVBM) for the cache to survive.
Offloading without measuring the break-even. If your prefixes are not reused, offloading only adds transfer latency without saving recomputation. Offloading shines with shared prefixes (RAG over the same corpus, agents with the same system prompt, long sessions), not with one-off, never-repeated requests.
Forgetting that attention is still quadratic. Offloading fixes the memory of the KV, not the compute of attention. For real 1M tokens you also need efficient attention, sparse attention or retrieval approaches (RetrievalAttention, ShadowKV); offloading on its own does not bring down the 1.765 s/token.
Wiring the offload badly. HBM↔DRAM↔NVMe crosses PCIe and NUMA. An offload that crosses the wrong NUMA node or saturates a PCIe lane can be slower than recomputing. Measure the link’s real bandwidth, not the datasheet figure.
Treating the figures as constants. The 93 GB, the +525 %, the 3–10× all depend on model, length, hardware and reuse pattern. They are orders of magnitude for design; measure on your own load.
Trusting million-token model specs without verifying. The landscape of open 1M models is very recent (2026) and moves fast; the specific names and limits change between versions. Verify the window and the cost of serving it per model, do not take the headline on trust.

A note of caution for June 2026: the distributed KV layer (LMCache, Mooncake, KVBM, NIXL, llm-d) is setting right now, with APIs and backends that change release by release. It is exactly the kind of piece worth assembling behind an abstraction (vLLM’s kv_connector) rather than coupling to a single vendor.

Closing

Long context is a problem of memory and compute at the same time, and the KV cache is the bottleneck of both. The window is a gift from the model; serving it cheaply is systems engineering: quantise the KV (FP8), offload it by tiers (HBM→DRAM→NVMe→network) with a persistent layer, and reorganise serving around the KV (aware routing, separated prefill/decode) when scale demands it. The rule that governs it all is Mooncake’s: trade storage for computation. You store more KV in cheap places so you do not recompute expensive attention. Do it well and a 4×H100 node serves contexts that, by recomputing, would not even get started. Do it badly, or not at all, and every long request reads the whole case file from scratch again, while the GPU drowns in a quadratic attention you had already paid for once.

Sources

LMCache · GitHub — https://github.com/LMCache/LMCache
LMCache: An Efficient KV Cache Layer (arXiv 2510.09665) — https://arxiv.org/pdf/2510.09665
LMCache · Offload KV cache to CPU (docs) — https://docs.lmcache.ai/getting_started/quickstart/offload_kv_cache.html
vLLM Prefix Caching vs. LMCache: Benchmarking KV Reuse Tradeoffs — https://levelup.gitconnected.com/vllm-prefix-caching-vs-lmcache-benchmarking-kv-reuse-tradeoffs-944fbaf98b56
Mooncake: A KVCache-centric Disaggregated Architecture (arXiv 2407.00079) — https://arxiv.org/abs/2407.00079
Mooncake · USENIX FAST'25 (Trading More Storage for Less Computation) — https://www.usenix.org/conference/fast25/presentation/qin
Mooncake · GitHub — https://github.com/kvcache-ai/Mooncake/
NVIDIA Dynamo · Disaggregated Serving (docs) — https://docs.dynamo.nvidia.com/dynamo/design-docs/disaggregated-serving
NVIDIA Dynamo · KV-aware routing (docs) — https://docs.nvidia.com/dynamo/latest/user-guides/kv-cache-aware-routing
NVIDIA Dynamo · KV Cache Offloading / KVBM (docs) — https://docs.nvidia.com/dynamo/backends/v-llm/kv-cache-offloading
llm-d · KV-Cache Wins You Can See — https://llm-d.ai/blog/kvcache-wins-you-can-see
Long-Context Inference at Scale: The Hidden Infrastructure Cost · DigitalOcean — https://www.digitalocean.com/community/tutorials/long-context-inference-production-cost
Long-Context LLM Infrastructure · Introl — https://introl.com/blog/long-context-llm-infrastructure-million-token-windows-guide
YaRN / LongRoPE (arXiv 2402.13753) — https://arxiv.org/html/2402.13753v1

Speeding up model cold start: from minutes to seconds

Thu, 11 Jun 2026 13:00:00 +0000

This is the third batch of an operational series on squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB with NVLink. The sister pieces in this batch are Multimodal: serving a VLM on-premise with vLLM, adding vision to the same engine, and FinOps and GPU multi-tenancy with LiteLLM, sharing out and charging for the GPU across teams. This post takes the problem that From disk to HBM laid out conceptually and turns it into a runbook: how you actually bring the cold start down, knob by knob, so that elasticity is usable.

TL;DR

The post From disk to HBM left an uncomfortable idea behind: moving the weights from disk to HBM is only one of the five line items of cold start. The other four, starting the process, creating the CUDA context, setting up the allocator and, the most expensive one that hardly anyone looks at, capturing the CUDA graphs and compiling the JIT kernels, are untouched by any fast loader. That is why a server that loads the weights in 6 s can still take 90 s to be ready. This runbook attacks all five. For the weights: safetensors (mmap, zero-copy, no bounce through a host FP32 buffer) versus the pickle of torch.load; Tensorizer (CoreWeave), which serialises the model into one file and streams it tensor by tensor straight to the GPU from object storage/HTTP/S3; and the Run:ai Model Streamer (from NVIDIA), which reads concurrently and overlaps the read with the H2D copy. All three are switched on with one flag, --load-format, in vLLM. For the non-weight part: torch.compile/Inductor compilation cache (persisted and reused across start-ups), bounding the CUDA graph capture to the batch sizes you really use (the default capture can eat ~54 s), and sleep mode as a shortcut that skips all five line items. The maths: $t = W/B$ for the weights, the saving from overlapping read and H2D instead of doing them in series, and a 70B cold start broken down item by item. And the economics: when scale-to-zero with pre-warm beats keeping a warm replica. On the generic 4×H100 SXM 80 GB cluster.

The analogy: opening the kitchen in the morning

A restaurant does not open by flicking a switch. Anyone who has worked in one knows it: opening time is decided by everything that happens beforehand, and almost none of it is “bringing in the food”.

The first cook arrives at a cold, dark place. They have to: turn on the lights and start the systems (the process init); fire up the ovens and let them come up to temperature (the CUDA context, which takes its time to become operational); get the boards out, sort the cold room, organise the work space (the memory allocator, which reserves and structures the HBM); go down to the store and bring up the produce, the crates of vegetables, the meat, the fish, and put it in its place (loading the weights from disk to HBM, the entire subject of From disk to HBM); and, the most forgotten of all, sharpen every knife, set up the mise en place, make the stocks, pre-heat the mother sauces, all the preparation that is not an ingredient but without which not a single dish goes out (CUDA graph capture and JIT compilation of the kernels).

Here is the trap that From disk to HBM already anticipated: if you only optimise the produce, you still open late. You can hire the best delivery service in the world, a blazingly fast service lift, crates that put themselves away. But if the cook is still sharpening knives and waiting for the ovens to come up to temperature, the kitchen does not serve the first dish. The produce arrived early and the restaurant is still shut.

Today’s runbook is, literally, the list of everything that has to be done for the kitchen to open on time, and which of those tasks can be sped up, which can be left done from the night before (cached), and when it pays better never to switch the kitchen off (warm replica) than to light it again every morning (scale-to-zero).

The five line items of cold start

From the moment an inference pod is born until it returns its first token, the clock runs in five different places. It is worth naming them, because each optimisation attacks one or two, never all of them:

Process init. Starting the Python interpreter, importing the engine (vLLM and its dependencies), parsing the configuration. Seconds, and it grows with the size of the environment and the CUDA/PyTorch imports.
CUDA context. The first cudaSetDevice / cudaFree(0) initialises the CUDA runtime against the driver: it loads the context and maps the GPU. It is not instant, on the order of one to several seconds per GPU depending on driver and number of devices.
Allocator. vLLM sets up its memory manager over the HBM (PyTorch’s caching allocator plus the KV-cache layout). Reserving and structuring tens of GB has its cost.
Weight loading disk→HBM. The journey that From disk to HBM dissected: disk → page cache → host buffer → PCIe → HBM. What almost everyone believes cold start is, and it is only one of the five.
CUDA graph capture + kernel JIT. vLLM captures CUDA graphs for the various batch shapes and compiles kernels with torch.compile/Inductor (and backends such as DeepGEMM or FlashInfer). This item is the great ignored one: the default graph capture in vLLM takes on the order of 54 s because it covers a wide range of batch sizes (vLLM, torch.compile integration; Red Hat, vLLM with torch.compile).

The lesson the diagram is shouting: on a modern server with torch.compile and CUDA graphs, item 5 can be as big as or bigger than item 4. Speeding up weight loading from 40 s to 6 s is a huge advance, but if you leave graph capture at 50 s, the cold start falls from 104 s to ~70 s, not to 12 s. You have to attack both halves.

Speeding up item 4: loading the weights

safetensors versus pickle: the format rules

The first knob is the on-disk format. The old torch.load uses pickle, which has two problems. The security one is well known: deserialising a pickle executes arbitrary Python code, since the __reduce__ protocol lets the file invoke any callable on load, so every model download was a remote execution vector (HuggingFace, Safetensors). The performance one is what concerns us: loading a pickle rebuilds Python objects and usually passes through an intermediate host buffer before reaching the GPU.

safetensors, now under the PyTorch Foundation (HuggingFace, Safetensors joins PyTorch Foundation), solves both. The file is just a JSON header + raw tensor bytes: loading it cannot do anything except populate tensor buffers. And the key property for us is physical: the data region is aligned to a page boundary (the header is padded so that the first tensor starts at a multiple of the OS page size). That is what makes zero-copy possible: a loader can mmap the file, cudaHostRegister over the mapped region and DMA straight from the page cache into VRAM, with no torch.load deserialisation and no temporary host FP32 buffer (HuggingFace, Safetensors). It is the base format; everything else is built on top of it.

The mmap trap (already flagged in From disk to HBM): mmap reads nothing immediately, it defers the cost to the first access to each page. If you do not force the read, the cold start looks short and the first token pays the page faults. And the “blazingly fast second load” is the page cache lying to you: in production pods are born cold, on nodes where those files are not cached.

Tensorizer: streaming tensor by tensor straight to the GPU

Tensorizer, from CoreWeave, serialises the model’s weights and its tensors into a single file and, instead of loading the whole model into RAM before moving it to the GPU, streams the data tensor by tensor from disk, an HTTP/HTTPS endpoint or an S3 bucket, deserialising on the fly straight onto the GPU (vLLM, Loading Models with CoreWeave’s Tensorizer). The operational advantage: near-instant loading and low host RAM usage during initialisation, which matters especially in serverless and autoscaling scenarios, exactly the scale-to-zero case. And it decouples the weights from the container image: the model lives in object storage, not inflating the image pull.

In vLLM it is enabled with --load-format tensorizer (or load_format="tensorizer" via the API). It requires serialising the model once into Tensorizer’s format; from then on, any pod streams it.

Run:ai Model Streamer: reading and copying at the same time

The Run:ai Model Streamer (from NVIDIA) attacks the bottleneck from another angle: it reads the tensors concurrently, N OS threads reading from storage into the CPU buffer, while streaming them to VRAM, so that the storage read and the H2D copy overlap instead of happening in series (vLLM, Loading models with Run:ai Model Streamer; NVIDIA, Reducing Cold Start Latency). It reads safetensors directly, with no reconversion.

The numbers published by NVIDIA give the order of magnitude: the streamer reaches 4.88 s reading from S3 at concurrency 32 and 7.53 s from an SSD IO2 at concurrency 8; integrated into vLLM, the total time to ready drops to 23.18 s from S3, 28.28 s from SSD IO2 and 35.08 s from GP3 (NVIDIA, Reducing Cold Start Latency). Note the gap between the weight loading time (~5–8 s) and the total to ready (~23–35 s): those ~18–27 s of difference are the other four line items, above all item 5. The streamer fixed item 4 and the rest is still there, exactly what this post warns about.

Concurrency is the tuning parameter: it controls how many OS threads read tensors into the CPU buffer (and, for S3, how many client connections the host opens) (vLLM, Run:ai Model Streamer). 16 is usually enough for local NVMe; 32 for high-throughput object storage. One thread does not saturate a Gen5 NVMe; overlapping does.

# default safetensors (the slowest of the fast ones)
vllm serve <model> --load-format safetensors

# Run:ai Model Streamer (concurrent reads, overlaps read + H2D)
vllm serve <model> --load-format runai_streamer \
 --model-loader-extra-config '{"concurrency": 32}'

# Tensorizer (tensor-by-tensor stream straight to GPU from object storage/S3)
vllm serve <model> --load-format tensorizer

The maths of the weights: $t = W/B$ and the saving from overlapping

The floor of item 4 is simple: moving $W$ gigabytes at a bandwidth $B$ takes

$$t = \frac{W}{B}$$

Take a large model, a 70B. In BF16 ($b=2$ bytes/param) that is $W = 70 \times 10^9 \cdot 2 = 140$ GB; in FP8 ($b=1$), $W = 70$ GB. Bandwidth depends on the tier (see the table below): local Gen5 NVMe reads on the order of ~14 GB/s per disk; PCIe Gen5 x16 copies host→GPU at ~50 GB/s; network storage, ~1–3 GB/s. The theoretical floor for reading 140 GB from an NVMe is $140/14 = 10$ s; from the network at 2 GB/s, 70 s. The bottleneck rules.

The overlapping trick. The default loader does the two steps, reading from disk and copying H2D, in series: first it fills a buffer, then it copies, then the next one. The time is the sum:

$$t_{\text{series}} = t_{\text{read}} + t_{\text{H2D}}$$

Tensorizer and the Run:ai Model Streamer overlap them: while one thread copies a chunk to VRAM, another is already reading the next one from disk. With enough concurrency, the total time tends towards the maximum of the two, not the sum:

$$t_{\text{overlapped}} \approx \max(t_{\text{read}},\, t_{\text{H2D}})$$

With the 70 GB of the 70B in FP8, from NVMe at 14 GB/s and PCIe at 50 GB/s: $t_{\text{read}} = 70/14 = 5.0$ s, $t_{\text{H2D}} = 70/50 = 1.4$ s. In series, $5.0 + 1.4 = 6.4$ s; overlapped, $\max(5.0,\,1.4) = 5.0$ s. The saving here is modest (~22%) because the disk dominates comfortably. Overlapping shines when the two steps are comparable: from network storage at 5 GB/s, $t_{\text{read}} = 70/5 = 14$ s and $t_{\text{H2D}} = 1.4$ s, series 15.4 s vs overlapped 14 s; but with many concurrent streams saturating a fast disk and a matched H2D, going from the sum to the maximum can almost double the effective throughput. The flip side: overlapping only gives you what the physical bottleneck allows. If the disk only delivers 14 GB/s, no streamer brings you below 5 s for 70 GB, and that is what the other lever is for, moving fewer bytes (FP8 versus BF16 halves $W$).

The storage tier

Tier	Typical bandwidth	140 GB (70B BF16)	70 GB (70B FP8)	Notes
Local Gen5 NVMe	~14 GB/s per disk	~10 s	~5 s	the short path; one disk
Local Gen5 NVMe (several, RAID0)	~28–50 GB/s	~3–5 s	~1.5–2.5 s	if the loader saturates several streams
Object storage (S3/RGW)	~1–3 GB/s per stream	~47–140 s	~23–70 s	streamer concurrency helps a great deal
NFS / shared network	~1–2 GB/s	~70–140 s	~35–70 s	puts the network and its contention in the path

The operational conclusion is the same as in From disk to HBM: the weights that serve the cold start live on the node’s local NVMe, not on the network. Network storage is the repository; the inference node has a hot local copy (pre-pull with an initContainer or a per-node cache DaemonSet). Tensorizer is the interesting exception: it streams from object storage so efficiently that it sometimes makes serving from S3/RGW viable with no local copy, at the price of depending on the bandwidth of the storage network.

Speeding up item 5: everything that is not weights

Here is the half of the cold start that fast loaders do not touch. Three fronts.

Caching the torch.compile/Inductor compilation

torch.compile has a built-in cache system: the compiled artefacts are saved after the first start-up and reused across start-ups, even across machines if it is configured properly. In vLLM, the result of the Dynamo compilation is stored in ~/.cache/vllm/torch_compile_cache/ (vLLM Blog, Introduction to torch.compile; vLLM, torch.compile integration). The consequence for the runbook: if that directory is a persistent volume shared between pods (a PVC, or a hostPath on the node’s NVMe populated by the first start-up), subsequent pods skip the compilation and start faster. The first pod pays for the compilation; the rest inherit the cache.

It is exactly “sharpening the knives the night before”: the expensive preparation is done once and reused. The nuance: the cache is sensitive to the vLLM/PyTorch version, the GPU, the quantisation and the configuration, and changing any of them invalidates the cache and you pay for the compilation once again.

Bounding the CUDA graph capture

vLLM captures CUDA graphs for a wide range of batch sizes by default, and that is expensive: the default capture is around 54 s, and on large configurations up to 294 s has been measured. By limiting the capture to the batch sizes your workload really uses (for example 1, 2, 4, 8, 16, 24, 32, 64) the cold start is cut by more than 70%, from 294 s to ~82 s in the measured case (Red Hat, vLLM with torch.compile). The knob is the engine’s compilation/CUDA graphs configuration; the principle is not to capture graphs for batch sizes you will never see.

A real trade-off: fewer captured batch sizes = faster start-up but less coverage. A batch outside the captured set falls back to the eager path (slower per request). You bound to the frequent sizes of your traffic, not of any traffic. And there is always the extreme of disabling CUDA graphs (enforce_eager): near-instant start-up on item 5 at the cost of throughput in the hot state, valid for debugging or for very low-traffic services where cold start weighs more than steady state.

Warm pools: not paying for item 5 on the critical path

The definitive shortcut for all five items is not to run them while the user waits. A warm pool, a replica already started, with weights loaded, graphs captured and kernels compiled, waiting for traffic, turns the cold start into zero for that request. It is GPU cost (partly or wholly idle) in exchange for start-up latency; we analyse it in the economics section below.

Sleep mode: the shortcut that skips all five items

And there is a shortcut halfway between “start from scratch” and “keep a whole replica warm”: vLLM’s sleep mode, the subject of Serving several models on a single GPU. Instead of killing the process, it puts it to sleep: it parks the weights in host RAM (level 1) or discards them (level 2), but keeps the process alive, and with it the CUDA context, the allocator, the CUDA graphs and the already compiled JIT kernels. The wake pays for neither item 1, 2, 3 nor 5; it only puts the weights back into VRAM (item 4, and from RAM, not from disk). That is why a wake is 18–200× faster than a full cold start, and even level 2, which reloads the weights from the same disk, is still 23–45× faster, because it skips the other four items (vLLM Blog, Sleep Mode). Sleep mode is living proof of this post’s thesis: if preserving items 1, 2, 3 and 5 gives you 18–200×, then moving bytes was never the whole cost.

An example cold start, item by item

Let us put numbers on a cold start of 70B in FP8 (70 GB) from local NVMe and see which optimisation attacks each item:

#	Line item	Default	Optimisation	Optimised
1	Process init	~4 s	(little room; minimal environment)	~3 s
2	CUDA context	~6 s	(driver; fixed)	~6 s
3	Allocator	~4 s	(fixed)	~4 s
4	Weight loading	~25 s	Run:ai streamer / Tensorizer (concurrency + overlap)	~5 s
5	CUDA graphs + JIT	~54 s	torch.compile cache + bounded capture	~10 s
	Total	~93 s		~28 s

Two readings. First: optimising only item 4 (a fast loader and nothing else) brings the total down from 93 to ~73 s, a real but disappointing improvement, because item 5 is still intact. Optimising only item 5 (cache + bounded capture) brings it down to ~49 s. Attacking both brings it down to ~28 s, and that is what turns theoretical scale-to-zero into usable scale-to-zero. Second: items 2 and 3 are practically irreducible, they depend on the driver and the hardware; they constitute the floor of the cold start, on the order of 10 s, which no software optimisation crosses. Below that floor, the only answer is not to start from scratch: sleep mode (wake from sub-second to a few seconds) or a warm replica (zero).

The economics of scale-to-zero

The capacity question all of this serves is: switch off or keep warm? Scale-to-zero saves GPU while there is no traffic, but pays the cold start when it comes back. The decision is a balance between the cost of an idle GPU and the latency SLA.

The basic criterion: scale-to-zero is only viable if the cold start fits inside the start-up SLA your users tolerate, or if you have a pre-warming mechanism that starts the replica before the request arrives (load prediction, an early signal from the queue). Without pre-warm, a 90 s cold start means the first user after an idle period waits 90 s, unacceptable for almost any interactive SLO. With the cold start brought down to ~28 s it is still a lot for an interactive request, but it is already tolerable for batch workloads or for a pre-warm triggered by autoscaling with KEDA on queue depth.

The decision rule, in one sentence: keep a warm replica when the cost of the idle GPU during the troughs is lower than the cost of missing the SLA on every start-up; do scale-to-zero (with pre-warm) when the troughs are long and deep and the optimised cold start fits in your tolerance. For a 24×7 interactive service with irregular but continuous traffic, a floor of warm replicas almost always wins. For a large model invoked a few times a day (a 70B for hard tasks), scale-to-zero with an optimised cold start, or sleep mode if it shares a GPU with another model, is the sensible option. It is the same capacity planning boundary: cold start is a parameter of the replica cushion, not a start-up detail.

Applied to the generic 4×H100 cluster

Let us bring the runbook down to the 4 H100 SXM 80 GB with NVLink. The concrete decisions:

Loader and format. Weights in safetensors on disk (never pickle). For the short path from local NVMe, the Run:ai Model Streamer (--load-format runai_streamer, concurrency 16–32) is the default option: concurrent reads, overlapping read and H2D, and you only change one flag. Keep Tensorizer for the case where you serve from object storage (RGW/S3) and want to decouple the weights from the image with no local copy, since its tensor-by-tensor streaming straight to the GPU is what makes that pattern viable. In both cases, FP8 versus BF16 halves item 4 almost for free (measure the quality, do not assume it).

Storage tier. Model repository on the network (Ceph RGW), hot copy on the node’s local NVMe populated by pre-pull. The cold start that counts is the cold one, on a node where the files are not in page cache; serving that start-up from the network puts its latency and contention into the critical path.

Caching item 5. The torch_compile_cache directory on a persistent volume or a hostPath NVMe shared between pods of the same model: the first pod compiles, the rest inherit. And bound the CUDA graph capture to the real batch sizes of each service’s traffic.

When scale-to-zero with pre-warm vs a warm replica. The agent LLM (main service, continuous traffic) never does scale-to-zero: a floor of warm replicas on one or two GPUs. The occasional 70B does: scale-to-zero with an optimised cold start, or, if it shares a GPU with a mid-sized model, sleep mode so the wake is sub-second instead of a 28 s start-up (exactly the play in Serving several models on a single GPU). The optimised cold start in this post is what makes swap and failover between replicas tolerable: a replica that falls over and is replaced in 28 s degrades less than one that takes 90 s. And it connects directly with multi-tenancy: a short cold start lets you share GPUs between teams with per-tenant scale-to-zero without the first user of each tenant paying for an endless start-up.

The cross-cutting principle: cold start is the ceiling on elasticity. The five line items, with both halves attacked, a fast loader for the weights and cache plus bounded capture for the graphs, bring it down from minutes to seconds. And below the irreducible floor of ~10 s (CUDA context + allocator), the only way out is not to start from scratch: sleep mode or a warm replica.

Traps and things that are not what they seem

“I switched to a fast loader and start-up barely improved.” Your bottleneck probably was not item 4. If CUDA graph capture eats 54 s, bringing weight loading down from 25 to 5 s takes 20 s off 93, which is barely noticeable. Measure where the time goes before optimising: vLLM’s start-up logs break down weight loading versus graph capture.

“The second time it started in a flash.” The page cache (for the weights) and the torch.compile cache (for the graphs) lying to you at the same time. The start-up that matters is the cold one, on a clean node. Benchmarking the second start-up measures a situation that almost never occurs in the peak that triggers the autoscaler.

“Tensorizer/streamer will give me the 18–200× of sleep mode.” No. Those loaders speed up only item 4. The 18–200× of sleep mode comes from preserving the other four by keeping the process alive. They are tools for different problems: a fast loader is for a real cold start (a new process); sleep mode is for alternating models without killing the process.

"enforce_eager fixes the cold start." It fixes item 5 (no graph capture, near-instant start-up) but degrades throughput in the hot state, since CUDA graphs exist because they speed up the steady state. It is a trade: it wins for very low traffic services where start-up weighs more than the hot path; it loses for a service with sustained load.

“I bound the CUDA graphs to a single batch size and that is it.” You bound to the sizes your traffic uses, not to one. A batch outside the captured set falls back to the eager path and runs slower. Too aggressive and you penalise the steady state to gain a few seconds of start-up.

“Moving the weights to FP8 also speeds up item 5.” Not directly. FP8 halves the bytes (item 4) and doubles inference throughput, but graph capture and kernel compilation do not depend on the size of the weights, they depend on the shape of the graph and the batch sizes. Item 5 is attacked with caching and bounded capture, not with quantisation.

Conclusion

From disk to HBM said it conceptually: moving the weights is one of the five line items of cold start. This runbook turns it into action. For the weights, item 4, there is a clear menu: safetensors as the base (mmap, zero-copy, no pickle), Run:ai Model Streamer for the short path from NVMe (concurrent reads overlapping read and H2D), Tensorizer for streaming from object storage with no local copy, and the cross-cutting lever of moving fewer bytes with FP8. But the forgotten half of the start-up is item 5, CUDA graph capture and JIT compilation, which on a modern server can be as big as the weight loading: you attack it by caching the compilation across pods and bounding the graph capture to the real batch sizes. Attacking only one half leaves the cold start half done; attacking both brings it from ~90 s to ~28 s. And below the irreducible floor of ~10 s imposed by the CUDA context and the allocator, the only way out is not to start from scratch: sleep mode (sub-second wake) or a warm replica (zero). The kitchen opens on time when somebody took care of the produce and of sharpening the knives the night before. Optimising only the delivery from the store leaves the ovens cold and the restaurant shut.

References

HuggingFace, Safetensors (format, zero-copy, security versus pickle): https://huggingface.co/docs/safetensors/index
HuggingFace, Safetensors is Joining the PyTorch Foundation: https://huggingface.co/blog/safetensors-joins-pytorch-foundation
vLLM, Loading Models with CoreWeave’s Tensorizer: https://docs.vllm.ai/en/stable/models/extensions/tensorizer/
vLLM, Loading models with Run:ai Model Streamer: https://docs.vllm.ai/en/stable/models/extensions/runai_model_streamer/
NVIDIA, Reducing Cold Start Latency for LLM Inference with NVIDIA Run:ai Model Streamer (S3/SSD benchmarks, concurrency): https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer/
vLLM Blog, Introduction to torch.compile and How It Works with vLLM (compilation cache): https://blog.vllm.ai/2025/08/20/torch-compile.html
vLLM, torch.compile integration (cache in ~/.cache/vllm/torch_compile_cache/): https://docs.vllm.ai/en/latest/design/torch_compile/
Red Hat Developer, vLLM with torch.compile: Efficient LLM inference on PyTorch (CUDA graph capture ~54 s, bounding batch sizes −70%): https://developers.redhat.com/articles/2025/09/03/vllm-torchcompile-efficient-llm-inference-pytorch
vLLM Blog, Zero-Reload Model Switching with vLLM Sleep Mode (wake 18–200×): https://blog.vllm.ai/2025/10/26/sleep-mode.html

Multimodal on-premise: serving a VLM with vLLM (vision + language)

Thu, 11 Jun 2026 11:40:00 +0000

Third batch of an operational series on squeezing a generic on-premise LLM cluster of 4×H100 SXM. Its sibling pieces in this batch are FinOps and GPU multi-tenancy with LiteLLM, which puts a price on every token, and a visual token costs the same as a text one, and Speeding up cold start with Tensorizer, which matters twice as much here because a VLM loads two models (the vision encoder and the LLM). If the VLM ends up feeding a document assistant, that end-to-end setup with LibreChat and RAG is another story (in preparation).

TL;DR

A vision-language model (VLM) is not magic: it is a normal LLM with a sensory organ bolted on the front. Three pieces in series: a vision encoder (a ViT) that looks at the image, a projector or connector (typically an MLP) that translates the encoder’s output into the LLM’s embedding space, and the usual LLM, which receives those embeddings as if they were just more tokens. The consequence that governs the entire operational design is brutal in its simplicity: an image becomes visual tokens, and those tokens cost exactly the same as text tokens. A high-resolution document page is not “a couple of image tokens”: it is hundreds or thousands of tokens entering the context, inflating the prefill (compute-bound) and occupying KV cache (memory-bound) just as if you had pasted a thousand words. In Qwen2.5-VL the arithmetic is literal: the number of visual tokens is $H\times W / (14\times14\times4)$, so an $896\times896$ image is 1,024 tokens and an A4 page scanned at a decent resolution easily reaches 1,500–2,500 tokens for the image alone. This post opens up the anatomy, works through the token-cost maths and translates it into TTFT and VRAM on an 80 GB H100, explains vLLM’s multimodal support (--limit-mm-per-prompt, pixel budget via mm_processor_kwargs, chat template) and answers the question that really matters: classic OCR or VLM. The short answer: use Tesseract/PaddleOCR for the bulk of documents with clean text, and reserve the VLM, expensive in tokens and in GPU, for the subset that genuinely needs it: complex layout, nested tables, handwriting, stamps, charts. On the generic 4×H100 SXM cluster, with FP8 so it fits.

The analogy: the reader who sees the page versus the one who only hears the transcript

Picture two experts you ask to review a scanned contract. To the first you read the contract aloud, a blind transcript, word by word. It is fast, efficient, and for 90 % of the clauses it is enough: the text is the text. But there are things you cannot read out: that the signature at the end is handwritten and does not match the typed name; that there is a “PAID” stamp struck diagonally across the amount; that the table on page 3 has a merged cell that changes who each row applies to; that there is a pen annotation in the margin. The blind transcript loses all of that or flattens it into gibberish.

To the second expert you hand the whole page, to look at. They see the signature, they see the stamp, they understand the structure of the table because they perceive the lines and the cells, they read the marginal note. They grasp the document as a visual object with layout, not as a river of characters. They are strictly more capable.

Why not always use the second one, then? Because seeing costs more mental bandwidth. The first expert receives the contract as a flow of words: cheap, linear. The second receives it as an image that their brain has to parcel up, attend to region by region, and reconstruct. They process a great deal more information per page, most of which, in a clean text document, is redundant with what the transcript would already have given them. If you only need to read the clauses, hiring the expert-who-sees for every page is overpaying for capability you do not use.

That is the entire thesis of this post. The VLM is the expert who sees; OCR is the blind transcript; and “mental bandwidth” is tokens. The engineering consists of sending the expert-who-sees only the pages where seeing changes the answer, and resolving the rest with the cheap transcript.

Anatomy of a VLM: encoder, projector, LLM

A modern VLM of the Qwen-VL family (Qwen2.5-VL model card; Qwen3-VL Technical Report) is made up of three blocks in series:

Vision encoder (ViT). A Vision Transformer that receives the image, cuts it into patches and produces an embedding per patch (or per group of patches). In Qwen2.5-VL it is a dynamic-resolution ViT: it accepts images of arbitrary size, resizing them to multiples of 28 and splitting them into 14×14 pixel patches, with window attention to speed up the processing of large images. There is no “canonical size” everything is squashed to: a large image produces more patches than a small one.
Projector / connector. The ViT’s embedding space is not the LLM’s. The projector acts as the translator. In Qwen2.5-VL it is elegant and cheap: it groups every 2×2 block of adjacent patches (four ViT tokens), concatenates them and projects them with a two-layer MLP into a single fused token in the LLM’s dimension. That fusion factor of 4 is the reason for the $4$ that appears in the denominator of the token formula below: four ViT patches collapse into one visual token that the LLM sees.
The LLM. The usual language transformer. It receives a sequence of embeddings that is a mixture of text tokens (from your prompt) and visual tokens (from the image), all in the same space, and does what it knows how to do: attention over the complete sequence and autoregressive generation. To the LLM, a visual token and a text token are indistinguishable in cost: both go through the same QKV projections, both occupy an entry in the KV cache, both take part in the $O(C^2)$ attention.

The idea to internalise: the encoder and the projector are the sensory organ, but the inference cost lives in the LLM, and that cost is paid in tokens. The ViT adds a fixed prefill overhead (processing the image once), but the recurring bill, the LLM prefill and the KV cache throughout the whole generation, is set by the number of visual tokens the projector spits out. That is why the control lever is not “which encoder”, but how many pixels you let through.

The maths: what it costs to see a page

Let us put numbers on “an image costs many tokens”, with the real Qwen2.5-VL formula (Qwen team, Qwen2.5-VL blog):

$$N_{\text{tok}} = \frac{H \times W}{14 \times 14 \times 4} = \frac{H \times W}{784}$$

where $H$ and $W$ are height and width in pixels (rounded to the multiple of 28 the ViT imposes). The $14\times14$ is the patch size; the $\times4$ is the projector’s 2×2 fusion. A sanity check against the model card figure: an $896\times896$ image gives

$$N_{\text{tok}} = \frac{896 \times 896}{784} = \frac{802{,}816}{784} = 1{,}024 \text{ tokens.}$$

It checks out. Now an A4 page scanned at a resolution reasonable for reading small text, say $1120 \times 1568$ px (approximate height $\times$ width, already rounded to multiples of 28):

$$N_{\text{tok}} = \frac{1120 \times 1568}{784} = \frac{1{,}756{,}160}{784} \approx 2{,}240 \text{ tokens.}$$

A single page is ~2,240 visual tokens before you write a word of prompt. For context: that is roughly what 1,700 words of Spanish text would take up (at ~1.3 tokens/word). If that A4 page contains 600 words of actual text, the VLM is paying almost 4× in tokens to see it rather than read its transcript. If the page is a dense table or a handwritten document that OCR cannot read, that 4× is justified. If it is a paragraph of clean text, it is pure waste.

Qwen2.5-VL lets you control this with min_pixels / max_pixels: the number of tokens per image is dynamic and ranges from about 4 up to 16,384 by default, adjustable with those parameters (HF Qwen2.5-VL docs). Lowering max_pixels cuts tokens at the expense of resolution, and of the ability to read the fine print. It is exactly the lever of “how much mental bandwidth do I give the expert”.

From token cost to TTFT

Visual tokens are not free in latency. The prefill, processing the whole context before the first token, is compute-bound and its cost grows with the context length $C$ (linear in the projections, quadratic in the attention); this is worked through in detail in vLLM’s attention backend. Every visual token enters that $C$.

Let us model the TTFT as the time to process the prefill tokens at a given prefill throughput. Take a VLM of around 7–8B on an H100 with an illustrative prefill throughput of ~12,000 tok/s (an example figure; measure it, do not assume it, and it depends on how much the ViT forward pass weighs on top). For a request with an 80-token text prompt plus one page at 2,240 visual tokens:

$$\text{TTFT} \approx \frac{80 + 2240}{12{,}000 \text{ tok/s}} \approx \frac{2320}{12{,}000} \approx 0{.}19 \text{ s}$$

plus the fixed cost of the vision encoder’s forward pass over the ~9,000 ViT patches (another compute-bound slice of prefill). Now five pages in the same request (a short PDF):

$$\text{TTFT} \approx \frac{80 + 5 \times 2240}{12{,}000} = \frac{11{,}280}{12{,}000} \approx 0{.}94 \text{ s}$$

The TTFT multiplies almost fivefold, because the visual tokens completely dominate the context. And attention scales as $O(C^2)$: five pages are not “5× more expensive” in the attention component, but ~25× relative to one. The operational lesson: the number of images and their resolution are your TTFT budget, not a detail. Lowering max_pixels from 2,240 to, say, 1,100 tokens/page (lower resolution, enough for large text) cuts the prefill almost in half.

VRAM: the VLM’s weights and what is left for KV

On an 80 GB H100 SXM, let us budget a VLM of the Qwen-VL family. Take an 8B (LLM) plus the vision encoder (a ViT of a few hundred million parameters; call the whole thing ~8.5B effective parameters for weight purposes). In BF16 (2 bytes/parameter):

$$M_{\text{BF16}} \approx 8{.}5 \times 10^9 \times 2 \text{ B} \approx 17 \text{ GB.}$$

In FP8 (1 byte/parameter), the weights halve (Qwen3-VL-8B-Instruct-FP8, with reported quality almost identical to BF16):

$$M_{\text{FP8}} \approx 8{.}5 \times 10^9 \times 1 \text{ B} \approx 8{.}5 \text{ GB.}$$

For an 8B model this fits comfortably in BF16 on an H100; FP8 matters more for leaving VRAM free for KV cache, and here the KV is the problem, because visual tokens inflate it. The detail of end-to-end FP8 (weights and KV) is in FP8 end to end. What is relevant: one page is ~2,240 KV entries; serving several sessions that send long documents consumes KV at a rate a text-only case would never see. With a 32B VLM in FP8 (~32 GB of weights) on an 80 GB H100 you are left with ~45 GB for KV and activations, which with long multimodal contexts run out sooner than “text-only” intuition suggests. The rule: with a VLM, plan the VRAM counting the visual tokens in the KV, not just the weights.

Serving a VLM with vLLM

vLLM supports image multimodal input natively for the Qwen-VL family and many other VLMs (Multimodal Inputs, vLLM docs). The gears that matter:

--limit-mm-per-prompt. Limits how many multimodal items each prompt accepts, per modality. By default the limit is generous (999 per modality), but in production it is worth setting it low, --limit-mm-per-prompt '{"image": 2}', because every accepted image reserves token and KV budget. If you set a limit of 8 images and someone sends 8 A4 pages, that is ~18,000 visual tokens in a single request: it blows up max_model_len or the KV. The limit is a budget firebreak, not a cosmetic restriction. (A curious case: setting it to 0 for a modality lets you pass precomputed embeddings without loading the encoder module, saving VRAM, useful if you do the vision encoding outside vLLM.)

Pixel budget via mm_processor_kwargs. Here the token formula connects with the configuration. When serving Qwen2.5-VL you pass, for example, --mm-processor-kwargs '{"min_pixels": 12544, "max_pixels": 254016}' (vLLM Qwen2.5-VL discussion). With $254{,}016$ maximum pixels per image, the token ceiling per image is $254{,}016 / 784 \approx 324$ tokens, an aggressive trim that keeps the cost in check at the price of not being able to read very small print. This is the dial that really governs your token bill: raise it for dense documents, lower it for images where fine detail does not matter.

The chat template and the image placeholder. The model’s processor (loaded via AutoProcessor) inserts special tokens marking where the image goes in the sequence: in Qwen2.5-VL the sequence <|vision_start|><|image_pad|><|vision_end|>, where <|image_pad|> expands to the exact number of visual tokens the image produced. vLLM applies this template automatically; it is worth knowing there is a documented quirk, the image block tends to be placed before the user’s text in the sequence order, regardless of where you put it in the content (issue #15125, vllm-project/vllm), which rarely causes trouble but is worth bearing in mind if the image/text order is semantically important for your prompt.

Current models. As of June 2026, the reference open family is Qwen3-VL (dense 2B/4B/8B/32B variants and MoE 30B-A3B / 235B-A22B, interleaved contexts of up to 256K tokens combining text, image and video), with published FP8 checkpoints of almost identical quality to BF16 (Qwen3-VL Technical Report, arXiv 2511.21631; Qwen3-VL Usage Guide, vLLM recipes). Qwen2.5-VL remains perfectly serviceable and its token formula is the one we have used because it is public and clean; the cost figures are of the same order in both generations.

Classic OCR vs VLM: when to use which

Here is the decision that pays the bill. Parsing documents to feed them into a document ingestion pipeline has two paths, and choosing the split between them well is the difference between a healthy cluster and a choking one.

Classic OCR (Tesseract, PaddleOCR). Text-recognition engines that run on CPU, are cheap, fast and mature. Tesseract is the portable workhorse (edge, offline kiosks, embedded scanners). PaddleOCR is more capable on layout and Asian scripts, with its PP-Structure module for preserving structure, extracting tables and detecting key-value fields (PaddleOCR vs Tesseract, CodeSOTA 2026). On clean, well-scanned text, classic OCR is hard to beat on cost per page: zero GPU, millisecond latency, very high throughput. Its weakness is the “hard” document: complex multi-column layout, tables with merged cells, handwriting, stamps, charts, poor scan quality. There, pipeline-based OCR either fails or returns gibberish that destroys the structure.

VLM. It takes the image of the document and generates structured markdown or HTML in a single end-to-end pass, understanding the layout because it sees it. It shines exactly where classic OCR struggles: nested tables, reading charts, handwriting, understanding the spatial relationship between elements. The price is the one we have been calculating all through this post: expensive in tokens and in GPU. Worth noting a 2026 nuance: the frontier has moved. VLM models specialised in document parsing have appeared, compact (~1–3B), open source, scoring very high on benchmarks such as OmniDocBench (PaddleOCR-VL in the 94–96 point band; dots.ocr 3B at ~88 with full markdown including tables) (LLM-OCR vs Traditional OCR, Parsli 2026; Open-Source OCR Models, E2E 2025). In other words: for pure parsing, today you may not want a 32B generalist VLM, but a small specialised VLM-OCR. But it is still GPU and it still pays tokens per image: the cost calculation does not disappear, the model just gets cheaper.

The split rule. It is not “OCR or VLM” as a global choice, but per-document triage. Run everything through a cheap first filter: classic OCR already returns a confidence score; if it is high and the document has no tables or visual elements, the OCR transcript will do and you have spent CPU. If confidence is low, there are tables, there is handwriting or there are graphical elements, escalate that page to the VLM. In a typical corpus, 80–90 % of pages are resolved with cheap OCR and only the rest consumes GPU. This is what makes an on-premise VLM sustainable: you do not serve everything with it, you use it as the expensive specialist you send only the hard cases.

Applied to the generic 4×H100 cluster

How this fits into a cluster of 4×H100 SXM 80GB:

Where to place the VLM. A VLM deserves a whole H100 or a large slice, for a VLM-specific reason that text-only intuition does not anticipate: the token cost per image means not many concurrent sessions fit. If each request brings one or several pages, each consumes thousands of KV tokens; the KV cache runs out with few simultaneous sessions. Compared with a text-only chat service where dozens of short conversations fit per GPU, a document service saturates the KV with a handful of heavy requests. That is why it makes no sense to carve the GPU finely for the VLM with aggressive MIG (GPU sharing is covered in sharing a GPU: time-slicing, MPS and MIG): the bottleneck is not idle compute but the KV, and a small slice runs out of KV before it makes use of the compute.

FP8 so it fits and to free up KV. Serve the VLM in FP8: at equal reported quality, the weights take up half and you leave the maximum VRAM for the KV cache, which is the scarce resource with multimodal contexts. For a 32B in FP8 (~32 GB) on an H100, those ~45 GB free for KV are what determine how many concurrent pages you can handle.

The VLM only for the subset that needs it. The most important architectural piece: do not put the VLM on the critical path of every document. The triage from the previous section lives upstream; the bulk (clean text) is resolved with OCR/PaddleOCR on CPU, and in a cluster with 4×H100 there is probably spare CPU to run it in parallel with the GPUs, and only the hard subset reaches the VLM. This frees the H100s for what really needs them and keeps a reasonable cost per document. It connects directly with FinOps and GPU multi-tenancy with LiteLLM: when you put a price on every token, a visual token costs the same as a text one, so a page through the VLM can cost 4× its transcript, and that should show up in the chargeback of the team that decides to send everything to the VLM for convenience.

Double cold start. An operational detail that bites: a VLM loads two models into memory, the vision encoder and the LLM, so its cold start is heavier than that of an equivalent LLM. If you are going to do model swapping or on-demand scaling, fast loading matters twice as much; that is where speeding up cold start with Tensorizer comes in. If instead you serve several models on one GPU by rotating them, the swap+sleep pattern (serving several models on one GPU) has to account for the VLM taking up more room and loading more slowly.

Consumer GPU, for reference. If someone wants to prototype outside the cluster, an RTX 5090 (Blackwell, 32 GB) comfortably serves a 7–8B VLM in FP8 (~8.5 GB of weights) with KV to spare for a few pages per request; a 32B in FP8 (~32 GB) no longer fits with useful KV on that card and calls for the H100. The 5090 is good for validating the triage pipeline and the prompt; the real service belongs on the H100s.

What we have not covered

Video. Qwen3-VL handles video with dynamic FPS sampling; each frame is an image and the token cost multiplies by the number of sampled frames. One minute of video can be tens of thousands of tokens. It is the most expensive multimodal beast and deserves its own analysis.
Prefix caching of images. If the same image (a logo, a template) appears in many requests, can its KV be cached? It depends on the backend and on the stability of the placeholder; it connects with the attention backend.
Evaluation. How to measure that the VLM extracts the table correctly (it is not enough for it to “look right”): faithfulness against a set with structured ground truth.
Fine-tuning the VLM for a specific document domain (invoices, particular forms), which can put part of the domain’s “knowing how to see” into the weights and reduce what the prompt has to explain.

References

vLLM, Multimodal Inputs (limit_mm_per_prompt, precomputed embeddings): https://docs.vllm.ai/en/stable/features/multimodal_inputs/
vLLM, Vision Language Multi Image (mm_processor_kwargs, min/max pixels): https://docs.vllm.ai/en/latest/examples/offline_inference/vision_language_multi_image/
vLLM, Qwen3-VL Usage Guide (recipes): https://docs.vllm.ai/projects/recipes/en/latest/Qwen/Qwen3-VL.html
Qwen team, Qwen2.5-VL (blog, token formula, dynamic resolution): https://qwenlm.github.io/blog/qwen2.5-vl/
Qwen2.5-VL (Transformers docs, min/max pixels, token range): https://huggingface.co/docs/transformers/model_doc/qwen2_5_vl
Qwen3-VL Technical Report, arXiv 2511.21631: https://arxiv.org/abs/2511.21631
Qwen3-VL-8B-Instruct-FP8 (FP8 with quality almost identical to BF16): https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct-FP8
vLLM issue #15125 (image placeholder order): https://github.com/vllm-project/vllm/issues/15125
PaddleOCR vs Tesseract, CodeSOTA 2026: https://www.codesota.com/ocr/paddleocr-vs-tesseract
LLM-Based OCR vs Traditional OCR, Parsli 2026: https://parsli.co/blog/llm-ocr-vs-traditional-ocr
Open-Source OCR Models 2025, E2E Networks: https://www.e2enetworks.com/blog/complete-guide-open-source-ocr-models-2025

Serving embeddings and rerankers with TEI in production

Thu, 11 Jun 2026 09:40:00 +0000

Sixth piece in an operational series about squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB. If RAG on the CPU argued where the data plane runs and the embedder zoo decided which model you serve, this post looks at the engine that serves them: the embeddings and rerankers server. The sisters in this batch, document ingestion from PDF to indexed chunk, GitOps for the inference stack with Flux and hardening and secrets for the sovereign stack, build the rest of the system around this piece. The closer of the series, the end-to-end sovereign assistant with LibreChat + LiteLLM + RAG, is still in draft.

TL;DR

Serving embeddings is not model.encode(text) scattered through the ingestion code. Just as you do not put a transformers.generate() inside your API and call it vLLM, you do not put the embedder inline either: you put it behind a dedicated server that does batching, controls concurrency, emits metrics and exposes a single HTTP contract that both ingestion and query-time reuse. In the 2026 open source ecosystem that server is TEI, Text Embeddings Inference from Hugging Face: a Rust binary that serves embedding, reranker (cross-encoder) and sequence classification models, with token-based dynamic batching, and that exposes OpenAI-compatible endpoints (/v1/embeddings) on top of the native /embed, /embed_sparse, /rerank and /predict (TEI repo, docs). It runs on CPU (ONNX Runtime, Intel MKL) and on CUDA (with FlashAttention and cuda graphs), with the same contract on both sides, so the rest of the system does not know, or care, what silicon sits behind it. The technical piece that gives almost all the performance is dynamic batching: TEI groups requests that arrive separately into a batch that shares the fixed cost of the forward, controlled by --max-batch-tokens (how many tokens fit per batch) and --max-concurrent-requests (backpressure). Grouping raises throughput, going from serving 1 sequence to 32 multiplies tokens/s at almost constant cost until the hardware saturates, in exchange for some latency while the tray fills. The numbers: a 568M encoder (bge-m3, bge-reranker-v2-m3) takes ~1.1 GB in fp16 and ~0.57 GB in int8, so several replicas fit even in a small H100 slice; embedding a short query online costs tens of ms, but reranking the top-50 costs linearly in candidates because a cross-encoder evaluates each (query, doc) pair separately. The split across the 4×H100 cluster: TEI-CPU on the fleet for batch ingestion (no SLA, throughput/€), TEI-GPU in a MIG slice of one H100 for high-QPS online embeddings and rerank, talking to ingestion and to the LiteLLM gateway through the OpenAI contract.

The analogy: the stamping press that only stamps vectors

Picture a workshop with a specialised stamping press. It does not make varied parts; it does one single thing: it takes flat material and, in one stroke, stamps a shape into it. In our case it takes text and stamps a vector. And like every industrial press, it has one property that defines all its behaviour: the cost of the stroke is almost fixed, no matter how much material you put on the tray.

Lowering the press, heating the die, aligning, pressing and lifting costs, say, one second. If you put a single sheet in, you spend that whole second stamping one part: a waste. If you fill the tray with thirty sheets and lower the press once, you spend nearly the same second and come out with thirty stamped parts. The cost per part collapses. The press performs far better when it fills the tray before pressing.

This is exactly the dynamic batching of an embeddings server. The “stroke” is the model’s forward pass on the GPU or the CPU: an operation that loads the weights, multiplies them by the activations and emits the vectors, with a large fixed cost that does not scale with how many sequences you process at once, until you saturate memory bandwidth or the compute units. Serving requests one at a time is stamping one sheet per stroke. Grouping them into a batch is filling the tray.

But the press has an operational dilemma, and here is the heart of the post. If you wait until the tray is completely full before pressing, the first sheets that arrived sit waiting for the rest: latency. If you press as soon as one sheet drops so nobody has to wait, you go back to the waste of one stroke per part: low throughput. The embeddings server solves this with a bounded wait window: it gathers whatever arrives within a window of microseconds to milliseconds, or until the batch token budget is full, and then it presses. That is the policy that turns a pile of independent requests into a few efficient strokes without anyone waiting too long.

The rest of the post is, at bottom, about configuring that press properly: how many sheets fit on the tray (max-batch-tokens), how much queue is tolerated before work is rejected (max-concurrent-requests), whether the press lives on the cheap CPU fleet (nightly ingestion, pure throughput) or in a slice of the expensive GPU (online, bounded latency), and why the reranker is a different press whose cost does grow with the number of parts.

What TEI is, exactly

Text Embeddings Inference (TEI) is Hugging Face’s inference server for the family of models that does not generate tokens: embedders, cross-encoder rerankers and sequence classifiers. It is to encoders what vLLM is to generative LLMs: a high-performance server, written in Rust, that handles tokenisation, batching, concurrency, metrics and the HTTP contract, leaving the application code to only make calls (TEI repo).

Three model types, three jobs:

Embedding (bi-encoder): turns a text into a reusable vector. bge-m3, Snowflake Arctic Embed, multilingual-e5. The vector is stored in the index and reused on every search.
Reranker (cross-encoder): takes a pair (query, document) and emits a relevance score, not a vector. bge-reranker-v2-m3. It produces nothing reusable: every pair is evaluated again.
Sequence classification: emits labels/probabilities over a text. Useful for lightweight guardrails, routing, language or toxicity detection.

The endpoints

TEI exposes both an OpenAI-compatible endpoint and its native endpoints, verified against the current documentation (Quick Tour, DeepWiki):

Endpoint	What for	Model type
`/v1/embeddings`	Embeddings with the OpenAI contract (drop-in for clients that already speak OpenAI)	embedder
`/embed`	Dense embeddings, TEI’s native API	embedder
`/embed_sparse`	Sparse embeddings (lexical head of models like `bge-m3`/SPLADE)	embedder
`/rerank`	Orders a list of texts by relevance against a query	reranker (cross-encoder)
`/predict`	Sequence classification (labels/scores)	classifier
`/embed_all`	Per-token embeddings (no pooling)	embedder
`/similarity`	Direct similarity between texts	embedder
`/metrics`, `/health`, `/info`, `/tokenize`, `/decode`	Operation, observability and utilities	all

The operational key is in the first row: TEI speaking the OpenAI contract on /v1/embeddings means the embedder is swappable without touching client code. The same code that called OpenAI’s text-embedding-3-large points at your on-prem TEI by changing the base_url, and the gateway (LiteLLM) routes it as one more provider. /rerank, by contrast, is not part of the OpenAI standard, no such endpoint exists in their API, so the gateway treats it as a reranking-specific endpoint (LiteLLM TEI rerank).

The backends

TEI runs on CPU and on GPU with the same HTTP contract, which is exactly the property the CPU/GPU split of RAG needs (TEI docs, hardware):

CPU: cpu-* image. ONNX Runtime (recommended) or Intel MKL backend. It exploits the integer compute paths (AVX-512+VNNI, AMX on 4th-gen Xeon) when the model is in int8. It is the batch ingestion engine.
CUDA: architecture-specific images (cuda-*, with variants for different compute capabilities). It uses FlashAttention and cuda graphs to squeeze the GPU. It requires compute capability ≥ 7.5 (Volta is out) and CUDA 12.2+ compatible drivers. It is the high-QPS online engine.
There is also Metal support (Apple Silicon) and experimental ROCm (AMD Instinct), less relevant to the on-prem case of this post.

The golden rule: the same bge-m3 served by TEI-CPU and by TEI-GPU exposes an identical /v1/embeddings. The client does not notice the silicon. That is what lets you put ingestion on CPU and online on GPU without rewriting anything.

Why a dedicated server and not calling the model inline

The temptation, especially in the prototype, is to load the embedder into the application process and call model.encode() directly. It works in the demo and breaks in production for five reasons, all solved by the dedicated server:

Batching. Your ingestion code processes documents; your query code embeds one query at a time. Inline, each encode() is an independent forward, one press stroke per sheet. A dedicated server joins requests from different origins (ingestion + several concurrent queries) into a single batch and amortises the fixed cost. This is what really multiplies throughput, and you cannot do it if the model lives inside each isolated process.
Concurrency and backpressure. Under load, what happens when 10,000 requests arrive at once? Inline, you run out of memory or queue without control. TEI has --max-concurrent-requests: above that limit it rejects instead of degrading everyone, which is the correct way to manage overload (TEI CLI args).
Metrics. TEI exposes /metrics in Prometheus format: latencies, batch size, tokens/s, queued requests. Inline you have none of this without hand-instrumenting every encode(). To observe the data plane you need that endpoint.
A reusable HTTP contract. The same server is consumed by batch ingestion, by online query-time and by any other service that needs vectors. One copy of the model in memory, one version point, one place to change the model. Inline, each service loads its own copy and the version drifts on its own.
Lifecycle separation. The embedder is updated, restarted or scaled without touching the application. If tomorrow you swap bge-m3 for Snowflake Arctic, you change the server container, not the code of N services. (Remember the embedder drift trap: changing embedder forces a reindex; having a single server makes that change atomic and auditable.)

It is the same argument for why nobody serves an LLM with transformers.generate() in production: you serve it with a dedicated engine. The embedder is no different.

Dynamic batching: the mechanics of the press

Here is the performance engine. TEI does token-based dynamic batching: instead of processing requests one at a time or grouping them by a fixed number of requests, it groups them by token budget (TEI README, Discussion #367).

Two knobs rule:

--max-batch-tokens (the default depends on the build, typically 16384): the total tokens that fit in a batch. With max-batch-tokens=1000 you get 10 requests of 100 tokens, or one of 1000. The docs say it clearly: this number should be as large as possible until the model becomes compute-bound. It is the size of the press tray, measured in tokens, not in parts, because the real cost scales with tokens, not with number of texts.
--max-concurrent-requests (default 512): how many requests it admits in flight before rejecting. It is the backpressure control, not the batching one: it protects against surges by returning an error instead of queuing indefinitely.

The policy: TEI keeps an internal queue. When there is work, it fills a batch up to the max-batch-tokens budget (or until there are no more requests waiting) and sends it to the model in one stroke. Requests that arrive while the previous batch is being processed wait in the queue and go into the next one. Watch out for a real operational detail: if max-batch-tokens is smaller than the model’s maximum input length, you can trigger a loop where a long request never fits into a batch (Issue #723); the token budget has to be at least as large as the longest sequence you accept.

The number: throughput with batching vs without batching

Let us model the press. Let $C$ be the fixed cost of a forward pass (loading weights, launching kernels) and $c$ the marginal cost per sequence inside the batch. For a batch of size $B$, the stroke time is roughly:

$$t(B) \approx C + c \cdot B$$

as long as the batch fits in the hardware (memory-bandwidth-bound, which is the normal regime of a small encoder on GPU). The throughput, sequences per second, is:

$$\text{throughput}(B) = \frac{B}{t(B)} = \frac{B}{C + c \cdot B}$$

Let us put in concrete numbers, indicative but of the right order for bge-m3 (568M) on a GPU serving ~256-token sequences. Say $C = 4$ ms of fixed cost per stroke and $c = 0.5$ ms per marginal sequence.

Without batching ($B=1$):

$$t(1) = 4 + 0.5 = 4.5 \text{ ms} \quad\Rightarrow\quad \text{throughput} = \frac{1}{4.5\text{ms}} \approx 222 \text{ sec/s}$$

With batching ($B=32$):

$$t(32) = 4 + 0.5 \times 32 = 20 \text{ ms} \quad\Rightarrow\quad \text{throughput} = \frac{32}{20\text{ms}} = 1600 \text{ sec/s}$$

The batch of 32 multiplies throughput by ~7.2× compared with serving one at a time, because the fixed cost $C$, which at $B=1$ was 89% of the time, is spread across 32. And the asymptotic limit, as $B \to \infty$, is $1/c = 2000$ sec/s: the press cannot go faster than its marginal cost, and we are already at 80% of that ceiling with $B=32$. Going up to $B=128$ would give ~1969 sec/s, a marginal improvement in exchange for a good deal more latency and memory. Batching has diminishing returns: most of the gain is in going from 1 to a few dozen.

The latency trade-off is the other face. The request that arrived first in the batch of 32 does not see its response until the whole stroke finishes: it waits for the tray to fill plus the 20 ms of the forward. If the 32 requests arrived within a window of, say, 5 ms, the first request suffers ~5 ms of batching wait + 20 ms of compute = 25 ms of latency, against the 4.5 ms it would have had being served alone. The p99 latency gets worse, that is the price of the full tray, but in exchange the system absorbs 7× more load with the same silicon. That is why the correct configuration depends on the use case: in batch ingestion (no user waiting) you maximise max-batch-tokens and latency does not matter; online (user waiting) you bound the wait window to keep p99 under control even at the cost of some throughput.

Deployment: compose for embedder and reranker

TEI is deployed as a container. The key arguments: --model-id (the model from the Hub), --pooling (how it aggregates tokens into the vector: cls, mean, splade, last-token), --dtype (precision), and the batching knobs already seen. Two separate services, since the embedder and the reranker are different models and different endpoints, each with its own container.

Embedder (`bge-m3`) on GPU

# docker-compose: TEI serving bge-m3 as an embedder, on GPU
services:
 tei-embed:
 image: ghcr.io/huggingface/text-embeddings-inference:cuda-1.9
 command:
 - --model-id=BAAI/bge-m3
 - --pooling=cls
 - --dtype=float16
 - --max-batch-tokens=16384 # big tray: the press performs
 - --max-concurrent-requests=512 # backpressure: reject above this
 ports: ["8081:80"]
 volumes: ["./hf-cache:/data"]
 deploy:
 resources:
 reservations:
 devices: [{driver: nvidia, count: 1, capabilities: [gpu]}]
 # exposes /v1/embeddings (OpenAI), /embed, /embed_sparse, /metrics

Reranker (`bge-reranker-v2-m3`) on GPU

 tei-rerank:
 image: ghcr.io/huggingface/text-embeddings-inference:cuda-1.9
 command:
 - --model-id=BAAI/bge-reranker-v2-m3  # cross-encoder, ~568M
 - --dtype=float16
 - --max-batch-tokens=16384
 - --max-concurrent-requests=256
 ports: ["8082:80"]
 volumes: ["./hf-cache:/data"]
 deploy:
 resources:
 reservations:
 devices: [{driver: nvidia, count: 1, capabilities: [gpu]}]
 # exposes /rerank — invoked ONLY over top-k (20/50), not over hundreds

bge-reranker-v2-m3 is a ~568M cross-encoder built on bge-m3, multilingual, that takes (query, document) and emits a relevance score directly, without producing a reusable vector (model card, BGE docs). It carries no --pooling because it emits no embedding: it emits a scalar.

CPU for batch ingestion

For the CPU fleet, it is enough to change the image and the dtype; the HTTP contract is identical:

 tei-embed-cpu:
 image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.6
 command:
 - --model-id=BAAI/bge-m3
 - --pooling=cls
 - --dtype=float16  # on CPU, ONNX int8 if the export supports it
 - --max-batch-tokens=8192 # smaller tray: the CPU saturates sooner
 ports: ["8083:80"]
 volumes: ["./hf-cache:/data"]
 # ONNX/MKL backend: exploits AVX-512+VNNI / AMX on 4th-gen Xeon

Any OpenAI client embeds against the endpoint without knowing what is behind it:

from openai import OpenAI
client = OpenAI(base_url="http://tei-embed:80/v1", api_key="-") # api_key ignored
r = client.embeddings.create(model="BAAI/bge-m3", input=["user query"])
vec = r.data[0].embedding # same contract as OpenAI; the silicon is invisible

And the rerank, through its native (non-OpenAI) endpoint:

import requests
docs = ["retrieved chunk 1", "retrieved chunk 2", "...top-20 from retrieval..."]
r = requests.post("http://tei-rerank:80/rerank",
 json={"query": "user query", "texts": docs})
ranked = r.json() # [{index, score}, ...] ordered by relevance

Integration: how ingestion and the gateway consume it

The TEI server has two clients with opposite profiles, and that is precisely the reason to centralise it:

Ingestion (batch, no SLA) calls /v1/embeddings or /embed with large lots of chunks. Here throughput is what matters: ingestion pushes thousands of texts and TEI groups them into large batches against the high max-batch-tokens. It is the case of the press with a full tray. Ingestion then writes the vectors (dense + sparse) into the vector DB; the specific PDF-to-indexed-chunk pipeline is the sister piece of this series in preparation, and meanwhile the PostgreSQL + Qdrant schema in microservices covers the structure of the pipeline.

The gateway (query-time, online) consumes TEI through two different routes:

Embedding the query: a single call to /v1/embeddings with a short text. The gateway, LiteLLM or another L7 router, treats it as one more OpenAI embeddings provider, routed by model. That embedded query goes to the vector DB for the hybrid search.
Reranking the top-k: after retrieving 20-50 candidates from the retriever, the gateway calls /rerank on the reranker server with the query and the candidate list, and keeps the best ones. Since /rerank is not an OpenAI standard, LiteLLM exposes it through its own reranking contract (LiteLLM rerank with TEI).

The full query-time pattern: embed query (TEI dense) → search (vector DB, hybrid retrieval) → rerank top-k (TEI rerank) → send query + top-5 to the LLM. Two different TEI endpoints, one for dense and another for rerank, both behind the gateway, both swappable.

Sizing: memory, replicas and latency budget

Footprint of a 568M encoder

The weight footprint of bge-m3 (568M params) depends only on precision:

$$\text{fp16: } 568 \times 10^6 \times 2 \text{ B} \approx 1.14 \text{ GB} \qquad \text{int8: } 568 \times 10^6 \times 1 \text{ B} \approx 0.57 \text{ GB}$$

On top of this comes the space for the batch’s temporary activations (proportional to max-batch-tokens × hidden dimension) and, in TEI, the FlashAttention/cuda graphs structures. In practice, a bge-m3 in fp16 is served comfortably in ~6 GB of VRAM even at high batch, and in int8 on CPU the whole model (0.57 GB) fits in the cache and RAM of any server without blinking, activating the AMX/VNNI paths that make it viable, which is exactly the argument of RAG on the CPU.

How many replicas fit? If we reserve a 10 GB slice of an H100 for serving embeddings online, with an effective footprint of ~2.5 GB per replica (fp16 weights + activations + overhead), around 3-4 replicas of bge-m3 fit in that slice, splitting the QPS between them. In int8 the footprint drops and more fit, at the cost of a full point of quality. For the reranker, the weight footprint is the same (568M), but its batch pattern is different (query-doc pairs), so it is worth sizing it separately.

Latency budget: short query vs top-50 rerank

Here is the fundamental asymmetry between embedder and reranker, and it is what decides the sizing.

Embedding a short query (online): it is one forward over a text of tens of tokens. On GPU, inside a dynamic batch, it is tens of ms including the batching wait, the order of the earlier numerical example, ~5-25 ms. It is cheap and constant: a query is a query, no matter how many documents are in the corpus. That is why embedding online fits easily into an interactive latency budget.

Reranking the top-50 (online): a cross-encoder does not produce a reusable vector; it evaluates every pair (query, document) in a forward. The cost is linear in the number of candidates:

$$\text{cost}_{\text{rerank}} \propto k \times \text{forward}(\text{query} + \text{doc})$$

Reranking a top-50 is 50 forwards of (query + doc) pairs, where each pair is longer than the query alone (query + ~256-token chunk). TEI groups them into batches, and batching helps again there, but the total work is ~50× that of embedding one query. If embedding a query costs of the order of 10-20 ms, reranking 50 candidates costs of the order of hundreds of ms depending on the batch and the hardware. Hence the operational rule repeated throughout the series: broad, cheap recall in the retriever, precision rerank over FEW candidates. Reranking a top-20/50 fits in the online budget; reranking hundreds at high QPS does not, and that is where the reranker demands a dedicated GPU or a larger slice. The detail of why the cross-encoder is expensive but precise is in reranker and hybrid retrieval.

The query-time latency budget, added up:

Stage	Typical cost (order)	Scales with
Embed query (TEI dense)	~10-20 ms	constant (1 short text)
Hybrid search (vector DB, HNSW+sparse)	single-digit ms	index size (sublinear)
Rerank top-k (TEI cross-encoder)	~100-300 ms	linear in k
Generation (LLM, outside TEI)	seconds	output tokens

The reranker is the most expensive stage of the data plane, and the only one that scales badly with the width of retrieval. Size it as the critical component.

Applied to the generic 4×H100 cluster

Let us come down to the cluster of the series: 4×H100 SXM 80 GB plus a generic CPU fleet (Xeon with AMX, NUCs). TEI lives in two places depending on the load profile, and the identical OpenAI contract is what makes each place swappable.

TEI-CPU on the fleet → batch ingestion. Re-embedding the corpus is throughput-bound work with no latency SLA: its place is the CPU fleet with bge-m3 in int8 (ONNX/AMX). Here you maximise max-batch-tokens because nobody is waiting, and the press performs with the tray brimming. No H100 should spend a cycle reindexing a corpus that changes once a day, which is exactly the bad split that RAG on the CPU takes apart. The CPU fleet scales horizontally and cheaply; the corpus is chopped up and spread across nodes.

TEI-GPU in a MIG slice → high-QPS online embeddings and rerank. Query-time has a latency SLA and, if the system serves many requests per second, it needs the GPU’s throughput and low latency. But a 568M encoder does not deserve a whole H100: it takes ~2.5 GB effective, and leaving it 80 GB wastes 97% of the card. The right answer is a MIG slice: partition an H100 into hardware-isolated instances and give TEI one of the small slices (a 1g.10gb, for example), leaving the large slices for generation or for other tenants. The online embedder and reranker live there, with memory and compute isolation guaranteed by the partitioning. The how of the partitioning, MIG, MPS, time-slicing and when to use each, is in sharing a GPU; the reading for TEI is direct: MIG gives the hard isolation that an online service with bounded latency wants, while batch ingestion on CPU does not even touch the GPU.

The split, in one sentence: ingestion squeezes throughput/€ on CPU with a full tray; online squeezes latency/QPS in a MIG slice of an H100; both speak the same /v1/embeddings and /rerank, so moving load from one side to the other is changing a URL in the gateway.

If at some point you need a 7B embedder (gte-Qwen2, NV-Embed) whose quality bge-m3 cannot reach, that is no longer small-encoder work: it drags an LLM’s cost profile with it and is served where 7Bs are served, with vLLM --task embed in a large GPU slice, not with TEI-CPU. But it is the occasional exception, not the base load.

Conclusion

TEI is the stamping press of the RAG data plane: a specialised server that only stamps vectors and scores, and that performs far better when it fills the tray before pressing. The technical piece that gives the performance is token-based dynamic batching, controlled by max-batch-tokens and max-concurrent-requests, with a clear trade-off between throughput and p99 latency that is resolved differently in ingestion (tray brimming, no SLA) and online (bounded window, p99 protected). The architectural virtue is the OpenAI-compatible contract on /v1/embeddings, which makes the embedder swappable without touching code, plus the native /embed_sparse and /rerank endpoints for the sparse head of bge-m3 and for the bge-reranker-v2-m3 cross-encoder. The numbers drive the sizing: a 568M encoder fits in gigabytes, not tens of gigabytes, so several replicas live in a MIG slice; embedding a query is cheap and constant, but reranking is linear in candidates and is the critical stage of the latency budget. On the 4×H100 cluster, the right split is TEI-CPU for batch ingestion and TEI-GPU in a MIG slice for high-QPS online: two presses, two silicons, one contract.

References

Hugging Face — Text Embeddings Inference (TEI), repo y docs (Rust, backends CPU ONNX/MKL y CUDA, endpoints OpenAI-compatibles). https://github.com/huggingface/text-embeddings-inference · https://huggingface.co/docs/text-embeddings-inference
Hugging Face — TEI Quick Tour (endpoints /embed, /embed_sparse, /rerank, /predict, /v1/embeddings). https://huggingface.co/docs/text-embeddings-inference/quick_tour
Hugging Face — TEI CLI arguments (--model-id, --pooling, --dtype, --max-batch-tokens, --max-concurrent-requests). https://github.com/huggingface/text-embeddings-inference/blob/main/docs/source/en/cli_arguments.md
Hugging Face — TEI README / Discussion #367 (token-based dynamic batching; semántica de max-batch-tokens). https://github.com/huggingface/text-embeddings-inference/blob/main/README.md · https://github.com/huggingface/text-embeddings-inference/discussions/367
BAAI — BGE-M3 model card (568M, XLM-RoBERTa, dense+sparse+colbert, MIT). https://huggingface.co/BAAI/bge-m3
BAAI — bge-reranker-v2-m3 model card (cross-encoder ~568M sobre bge-m3, multilingüe). https://huggingface.co/BAAI/bge-reranker-v2-m3 · https://bge-model.com/bge/bge_reranker_v2.html
Chen, J., et al. BGE M3-Embedding. arXiv:2402.03216. https://arxiv.org/abs/2402.03216
BerriAI — LiteLLM: proveedor TEI para /rerank. https://github.com/BerriAI/litellm/issues/8372

Serving several models on a single GPU: co-residency, model-swapping and sleep mode

Thu, 11 Jun 2026 02:40:00 +0000

This is the second piece of an operational series on squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB with NVLink. Its sibling, Sharing a GPU: time-slicing, MPS and MIG, splits the GPU so that several processes can use it at once; this one splits the complementary problem: you have more models than VRAM and you need them to coexist in time, not just in space. The third, The RAG data plane on CPU, moves off the GPU everything that does not need to be there; and the end-to-end sovereign assistant (fourth instalment, in preparation) assembles all of this behind LibreChat and LiteLLM. Here we assume that VRAM is the scarce resource and that keeping everything loaded at once is not an option.

TL;DR

You have an agent LLM serving almost all the traffic, one or two rerankers for RAG, an alternate model (another family, another language, a fine-tune) and, now and then, a large model for hard tasks. Added up, they do not fit in the 80 GB of an H100. There are three ways to make them coexist, and the key to not confusing them is to understand that they are three different states of where the weights live:

Co-residency: several models loaded at the same time in HBM, if the sum of their VRAM budgets fits in the physical memory. It is the ideal when they fit: zero switching latency. The catch: each engine must cap its own VRAM; if two engines both believe they own the whole 80 GB, the second one to ask for memory dies with OOM. On a GPU shared by time-slicing, sharing time does not protect memory (the processes still compete for the same HBM) and this problem gets worse, as we see in the sibling piece.
Model-swapping (with llama-swap): only one (or very few) resident at a time. The proxy looks at the request’s model field and, if another one is needed, unloads the current model and loads the requested one. Since there is only one inside, each model can use almost the whole GPU. The cost is a full cold start: seconds from local NVMe, minutes from network storage. It mixes engines: it exposes OpenAI-compatible endpoints including /rerank with llama-server on GGUF.
vLLM sleep mode (--enable-sleep-mode, endpoints /sleep and /wake_up): the model is not unloaded, it is put to sleep. The weights are parked in host RAM (level 1) or discarded while the process stays alive (level 2). The wake is 18–200× faster than a cold start because the process is still alive and keeps the CUDA allocator, the CUDA graphs and the compiled JIT kernels. The only thing rebuilt is the KV cache, which is discarded on sleep. It needs enough RAM for the sleeping weights.

The decision tree is short. Do they all fit? Co-residency. They do not fit, switching is infrequent and you mix engines? llama-swap. Everything is vLLM, there is RAM to spare and wake latency matters? Sleep mode. Sustained load on all of them at once? That is not swapping, it is more GPUs or replicas (see one big vs N small). This post puts numbers on each of them on an 80 GB H100.

The analogy: one stage, one spotlight

Picture a theatre with a single stage and a single spotlight, the GPU. You have more actors (models) in the company than fit under the light at once. There are three ways to stage the show:

Co-residency: several small actors on stage. If the cast for this scene is three slight actors, the main LLM, one reranker, another reranker, they all fit under the spotlight at the same time. Switching lines between them is instantaneous: they are already on stage. But the stage has a fixed size; add a fourth, burly actor (the large model) and they do not fit, so someone falls off the edge. That is the OOM.
Model-swapping: the actor goes home. When an actor finishes their part, they go home (the disk). If the next scene needs them again, they have to come back from home: get dressed, put the make-up on, travel to the theatre. That takes minutes. In exchange, while they are on stage they have the whole stage to themselves. That is the llama-swap model: maximum VRAM per actor, but re-entry costs a full round trip.
Sleep mode: the actor waits in the wings. Instead of going home, the actor steps out of the spotlight but stays backstage (host RAM), already dressed and made up. When their turn comes, they step in in two strides. No journey, no wardrobe: just crossing the curtain. That is the vLLM wake: the weights come back not from disk but from RAM, and everything that goes into “preparing the actor”, the allocator, the CUDA graphs, the kernels, is already done because they never left the building.

The operational moral is the same as in the theatre: the cost of an actor is not only their talent, it is how long they take to walk on when you need them. Co-residency pays for the permanent spot; swapping pays for the journey; sleep pays for the RAM of keeping them waiting nearby. The rest of this post is about choosing which, with a VRAM budget and a stopwatch in hand.

The three states: where the weights live

Co-residency: the VRAM budget

Co-residing is the default option when they fit: zero switching latency because every model is already in HBM. The whole question is do they fit?, and it is answered with a VRAM budget. The VRAM of an inference engine splits, roughly, into three items:

$$\text{VRAM}_{\text{model}} = \underbrace{P \cdot b}_{\text{weights}} + \underbrace{\text{KV}}_{\text{cache}} + \underbrace{A}_{\text{activations + overhead}}$$

where $P$ is the number of parameters, $b$ the bytes per parameter given the quantisation, KV the KV-cache budget (which scales with concurrency and context length) and $A$ the overhead of activations, CUDA buffers and fragmentation. The hard rule of co-residency is:

$$\sum_i \text{VRAM}_{\text{model } i} + \text{headroom} \;<\; \text{VRAM}_{\text{physical}}$$

If the sum overflows, no sharing scheme saves you: the first engine to ask for memory beyond the free gap dies with OOM. And here is the critical nuance that connects to the sibling piece: splitting the GPU by time does not split the memory. Under time-slicing, two processes take turns on compute but share the whole HBM with no isolation; if both assume they own the 80 GB, they collide. Only MIG gives memory partitions with a real boundary, as we see in Sharing a GPU. To co-reside properly, each engine must cap itself: in vLLM, --gpu-memory-utilization 0.45 tells it “do not use more than 45 % of the GPU”; in llama-server, you control the layers on GPU and the KV size. If you do not set those limits, vLLM claims 90 % of the GPU for itself by default, and there is no room left for anyone else.

A worked example: what fits in 80 GB

Take the realistic scenario. A 32B agent LLM in FP8 ($b = 1$ byte/param):

$$P \cdot b = 32 \times 10^9 \cdot 1 \text{ B} = 32 \text{ GB of weights}$$

On top of that comes the KV cache. To serve with decent concurrency and agent contexts (which are long: history, tools, documents), a KV budget of ~12 GB is reasonable, plus some ~2 GB of activations and overhead. Total for the main LLM: ~46 GB. That leaves ~34 GB of the 80. Let us see what fits there:

Service	Size (FP8/INT8)	KV + overhead	Total VRAM
32B agent LLM	32 GB	~14 GB	~46 GB
Reranker A (cross-encoder ~0.5B)	~0.5 GB	~0.5 GB	~1 GB
Reranker B (cross-encoder ~0.5B)	~0.5 GB	~0.5 GB	~1 GB
8B alternate model	~8 GB	~4 GB	~12 GB
Sum			~60 GB
Free headroom			~20 GB

With this they co-reside comfortably: the 32B LLM, the two rerankers (which are featherweights, a few hundred MB to 1 GB each) and even an 8B alternate, leaving 20 GB of cushion. Rerankers are the textbook case for co-residency: so small that they always fit alongside the main LLM, and rotating them would be absurd. Switching between them is instantaneous because they are all loaded.

When does it break? When the occasional large model arrives, a 70B in FP8:

$$70 \times 10^9 \cdot 1 \text{ B} = 70 \text{ GB of weights}$$

The 70B’s weights alone already eat 70 of the 80 GB. There is no way to co-reside it with the 32B, not remotely. That is where co-residency ends and rotation begins: the 70B can only come in if we evict the 32B from the GPU. That is the exact boundary: co-reside while the sum of budgets fits with headroom; rotate as soon as one model needs, on its own, more VRAM than is left over.

Model-swapping with llama-swap: the actor goes home

When they do not fit at once, the first answer is to rotate: keep one resident model and change it on demand. The canonical tool for this in the on-premise world is llama-swap, a Go proxy that sits in front of your inference servers (llama.cpp, vLLM, TabbyAPI and so on) and starts and stops them as needed.

The mechanism is elegant in its simplicity. Every OpenAI-compatible request carries a model field. llama-swap reads that field, looks at which upstream server it has configured for that model, and:

If the requested model is already loaded, it routes the request straight through.
If another one is loaded, it stops it (freeing its VRAM) and starts the right one.
When the new server reports ready, it forwards the request.

The cost of a swap is exactly a full cold start of the new model: unloading the current one (fast, just freeing memory) plus loading the new one (slow, moving the weights from disk to HBM). That second term is the painful one, and its magnitude is decided by where the weights are: from local NVMe it is seconds; from network storage (Ceph RGW, NFS) it can be minutes. The whole analysis of the load path, and why the default loader makes it slow, is in From disk to HBM; here the consequence is enough: a swap is only viable if it is infrequent, because every switch pays that toll in full.

The advantage in exchange: since there is only one resident, that model can use almost the whole GPU. The 70B that co-resides with nobody fits comfortably if it is the only one inside. And llama-swap mixes engines: you can have a vLLM configured for the big LLM, a llama-server with GGUF for the alternate, and two llama-server instances for the rerankers, which expose /rerank, /v1/rerank and /v1/reranking natively. That heterogeneity (GGUF + rerank + vLLM behind a single OpenAI endpoint) is exactly what vLLM alone does not give you, and the main reason to choose llama-swap.

Example ConfigMap: two rerankers rotating on the same port

A concrete and useful case: you have two rerankers, one multilingual and one specialised in code, that you do not need at the same time and would rather not keep both resident. With llama-swap they rotate on the same endpoint, triggered by the model field. The ConfigMap (mounted as the proxy’s config.yaml in a Kubernetes deployment) would be:

apiVersion: v1
kind: ConfigMap
metadata:
 name: llama-swap-rerankers
data:
 config.yaml: |
 # How long a model stays loaded after the last request
 # before llama-swap unloads it to free VRAM
 healthCheckTimeout: 60

 models:
 # Multilingual reranker (GGUF, via llama-server)
 "reranker-multilang":
 cmd: >
 /usr/bin/llama-server
 --model /models/bge-reranker-v2-m3.Q8_0.gguf
 --reranking
 --port ${PORT}
 --n-gpu-layers 99
 --ctx-size 8192
 # ttl: after 300 s idle, it is unloaded and the GPU is freed
 ttl: 300

 # Code reranker (GGUF, via llama-server)
 "reranker-code":
 cmd: >
 /usr/bin/llama-server
 --model /models/codereranker.Q8_0.gguf
 --reranking
 --port ${PORT}
 --n-gpu-layers 99
 --ctx-size 8192
 ttl: 300

A request to /v1/rerank with "model": "reranker-multilang" starts that server; the next one with "model": "reranker-code" stops the multilingual one and starts the code one on the same ${PORT} that llama-swap manages. Since both are small, the swap between them takes one or two seconds: quantised GGUFs weigh a few hundred MB and come from local NVMe. The ttl controls how long a model stays loaded after its last request: raising it avoids swaps if requests arrive in bursts; lowering it frees the GPU sooner for other uses.

An honest caveat: this pattern of two rerankers that rotate makes sense when they do not fit alongside the rest, or when you want to reserve the VRAM for something else. If they do fit (and a couple of 0.5 GB rerankers almost always fit, as we saw above), co-residing them is strictly better: zero swap latency. llama-swap shines when you rotate large models or mix engines that do not coexist well, not for juggling tiny models that would fit together.

vLLM sleep mode: the actor in the wings

Swapping has a problem: every switch pays the entire cold start, and a cold start is not just moving weights. As we saw in From disk to HBM, starting an inference engine includes initialising the Python process and the CUDA context, setting up the memory allocator, capturing the CUDA graphs and compiling the JIT kernels (DeepGEMM, FlashInfer, TorchInductor). Moving the weights is only one of five items, and often not the largest.

vLLM’s sleep mode (--enable-sleep-mode) attacks exactly that: instead of killing the process when a model stops being used, it puts it to sleep, leaving the process alive. There are two levels, and the difference is where the weights go:

Level 1: offloads the weights to host RAM (CPU) and discards the KV cache. The process stays alive. The wake copies the weights from RAM to VRAM, not from disk. Typical wake: ~0.1–0.8 s for small models, ~3–6 s for large ones. It needs enough RAM for the sleeping weights (on the order of the model’s size in GB).
Level 2: discards the weights entirely, keeping only small buffers (rope scaling tensors and the like). The wake does reload the weights from disk, but everything else, process, allocator, CUDA graphs, JIT kernels, is already done. Typical wake: ~0.8–2.6 s for small models. RAM use is almost nil (megabytes).

The key that explains the numbers: at both levels, keeping the process alive preserves the expensive infrastructure. That is why the benchmark in the vLLM blog (Oct 2025) reports that a wake is 18–200× faster than a full reload, and, most counter-intuitively, level 2 is still 23–45× faster than a cold start despite reloading the weights from the same disk, because it skips the other four items. In their measurements on an A100, a full cycle of 5 model switches goes from 357 s without sleep (≈48 s per switch) to 112 s with level 1 (wake of 0.26 s / 0.82 s) or 125 s with level 2 (0.85 s / 2.58 s).

The KV cache is always discarded on sleep. That is not a minor detail: it means the first response after the wake rebuilds the KV from scratch, paying a full prefill. That is why the wake is not “entirely free”: the model is available in sub-second time, but the first request is a little slower until the KV is repopulated. A one-request warm-up after the wake hides that cost.

# Start vLLM with sleep mode (admin endpoints, trusted networks only)
export VLLM_SERVER_DEV_MODE=1
vllm serve <model> --enable-sleep-mode --port 8001

# Sleep (level 1: weights to host RAM)
curl -X POST 'localhost:8001/sleep?level=1'

# Wake up
curl -X POST 'localhost:8001/wake_up'

Security warning (from vLLM’s own blog): the /sleep, /wake_up, /collective_rpc and /reset_prefix_cache endpoints require VLLM_SERVER_DEV_MODE=1 and should only be exposed on trusted networks, since they can take the service down. They are for internal orchestration (a controller that sleeps and wakes models according to the queue), not for the public plane.

The latency maths: why the wake wins

Let us put numbers on “the wake comes from RAM, not from disk”. The cost of having a model available is, in each strategy, the time to move its weights from wherever they are to HBM, plus, in the swap case, the other four cold-start costs. Take the 34 GB of weights of the 32B LLM in FP8 and compare the three paths.

Cold start from NVMe (swap). A reasonable Gen4/Gen5 NVMe gives on the order of ~5 GB/s effective per stream with the default loader (the disk’s theoretical floor is higher, but single-threaded deserialisation does not saturate it, see From disk to HBM). For 34 GB:

$$t_{\text{NVMe→HBM}} \approx \frac{34 \text{ GB}}{5 \text{ GB/s}} \approx 6.8 \text{ s just moving bytes}$$

And that is before adding CUDA graph capture and kernel compilation, which add several more seconds. The real cold start of a 32B is around 15–40 s depending on loader and storage. From the network (Ceph RGW), multiply.

Level 1 wake from RAM. The weights come not from disk but from host RAM, and travel over PCIe Gen5 x16, whose practical host→GPU bandwidth is ~50–64 GB/s. For the same 34 GB:

$$t_{\text{RAM→HBM}} \approx \frac{34 \text{ GB}}{55 \text{ GB/s}} \approx 0.62 \text{ s}$$

And there is nothing else to pay: the allocator, the graphs and the kernels are already there. The real wake of a model this size lands in the sub-second to a few seconds range that vLLM reports. The speed-up against an NVMe cold start is on the order of:

$$\frac{t_{\text{cold start}}}{t_{\text{wake}}} \approx \frac{15\text{–}40 \text{ s}}{0.6\text{–}3 \text{ s}} \approx 10\text{–}60\times$$

consistent with the 18–200× from the blog, whose measurements include smaller models, where the relative weight of the preserved CUDA graphs is even larger and the factor goes up.

Why the bandwidth difference explains almost everything. The key jump is not 5 vs 55 GB/s (a ~11× in transport). It is that the cold start also pays for rebuilding infrastructure, which the wake skips entirely. The table:

Path	Weight source	Bandwidth	34 GB bytes only	+ CUDA graphs / JIT	Realistic total
Cold start (swap) NVMe	disk	~5 GB/s	~6.8 s	yes (several s)	15–40 s
Cold start (swap) network	network	~1–2 GB/s	17–34 s	yes	30 s – min
Level 1 wake	host RAM	~50–64 GB/s	~0.6 s	no (preserved)	0.6–3 s
Level 2 wake	disk	~5 GB/s	~6.8 s	no (preserved)	7–10 s

Look at the level 2 row: it reloads the weights from the same disk as the swap (~6.8 s of bytes), but since it does not rebuild graphs or kernels, its total (~7–10 s) still beats a full cold start (15–40 s). It is the proof that moving bytes is only part of the cost, and the part sleep mode exploits.

Assumptions, honestly: the bandwidths are indicative. NVMe “5 GB/s effective” assumes the default loader; with a concurrent streamer it goes up. PCIe “55 GB/s” assumes Gen5 x16 with a pinned buffer and NUMA locality; if the buffer lands on the wrong socket, it drops. And the “15–40 s” cold-start range depends on the model, the quantisation and whether the files are in page cache (the “the second time it loaded fast” trap). The numbers are there to reason about orders of magnitude, not to size anything without measuring on your hardware.

The decision tree

The three strategies do not compete: each wins in a different regime. The tree, in order:

The node most often ignored is the one at the top right: “sustained load on all of them at once?”. If your four models get constant and simultaneous traffic, neither swapping nor sleep helps, since both assume the models take turns in time. Rotating under sustained load only adds switching latency without solving the underlying problem: there is not enough compute. The answer then is to scale horizontally (more replicas) or spread across more GPUs, a capacity decision analysed in One big vs N small. Swapping and sleep are tools for temporally unbalanced workloads: many models, but rarely active at the same time.

Applied to the generic 4×H100 cluster

Let us bring this down to the 4 H100 SXM 80 GB with NVLink. The winning strategy is not to pick one of the three, but to spread the models across GPUs according to their usage pattern and apply to each GPU the strategy it calls for. A reasonable split:

H100 #0, the workhorse (co-residency). The 32B agent LLM (main service, constant traffic) co-resides with the two rerankers and, if they fit, the 8B alternate. It is the GPU that never rotates: everything living here is used continuously and fits comfortably in 80 GB (the ~60 GB of the example above). Zero switching latency between the LLM and its rerankers, which is exactly what RAG needs, since a fast reranker is useless if you have to wait for a swap every time.

H100 #1, light services with MIG. If you have many small heterogeneous services, an embedding model, a classifier, a guardrail, an STT/TTS, splitting this GPU with MIG into isolated instances (each with its slice of HBM with a real boundary) gives co-residency with memory isolation, preventing a service that inflates its KV from taking the others down. The detail of when MIG beats time-slicing is in Sharing a GPU; the rule here: co-residing light services on one GPU makes sense when they fit and when isolating them is worthwhile, and MIG is the tool for the second part.

H100 #2, the occasional large model (sleep mode or swap). The 70B that is only invoked for hard tasks does not deserve a dedicated GPU kept awake, since it would sit idle most of the time, burning 700 W for nothing. Two options:

If this GPU also serves a medium model routinely and you only occasionally need the 70B, use sleep mode: put the medium one to sleep (level 1, weights to RAM), wake the 70B. Note that level 1 sleep frees the VRAM, the weights go to RAM, so yes, it fits even though the 70B would not fit alongside an awake medium model. Waking the medium one again is sub-second.
If the 70B comes as GGUF or you mix engines, llama-swap rotates between the medium model and the 70B by the model field. Each 70B invocation pays its cold start (seconds from local NVMe), acceptable if it is occasional.

H100 #3, replica / overflow. The fourth GPU absorbs peaks: a replica of the main LLM for when #0’s queue grows, or reserve capacity. There is no swapping here: it is pure capacity, the answer to the tree’s “sustained load” node.

The cross-cutting principle: co-reside what is used together and continuously (LLM + rerankers); isolate the light and heterogeneous with MIG; sleep or rotate the large and occasional; replicate what saturates. The four GPUs do not all do the same thing, each runs the strategy its load pattern calls for. And the NVLink between them matters for something else (tensor parallelism for the 70B if it would not fit even on one; see One big vs N small), but for this post’s problem, many models, one GPU, the lever is when each model needs to be awake.

Traps and things that are not what they seem

“Co-residing is always better if they fit.” Almost, but watch the KV cache: co-residing two models means splitting the KV budget between them. If the main LLM needs a large KV for high concurrency and long contexts, adding a flatmate cuts that KV and lowers its throughput. Sometimes it is better to give the whole GPU to the main model and rotate the secondary one. Co-residency is not free: the tenant takes room away from the cache of the one that matters.

“Sleep mode is like swapping but faster.” Not exactly. Swapping releases the process; you can have N models configured and only pay RAM/disk for the resident one. Sleep mode keeps one live process per sleeping model, so each sleeping vLLM still occupies its process slot, its RAM (level 1) and its management footprint. Sleep scales well to a few rotating models; for many (10+), level 2 (minimal RAM) or plain swapping fit better. Do not put 15 models into level 1 sleep and expect the RAM to hold.

“The wake is instantaneous, I lose nothing.” The model’s wake is sub-second, but the KV cache was discarded on sleep. The first request after the wake pays a full prefill to repopulate the KV, slower than normal. If your SLA is strict on the first response after an idle period, add an automatic warm-up after the wake. Prefix caching helps make that reprefill cheaper if there are stable prefixes.

“llama-swap with a low ttl saves me VRAM for free.” It saves you VRAM while nobody uses that model, but every time it comes back it pays the cold start. An aggressive ttl on a model with bursty traffic turns every burst into a load wait. The right ttl depends on the temporal pattern of the requests, not on how much VRAM you want to free. Measure it.

“Time-slicing lets me co-reside more models.” False and dangerous. Time-slicing shares out compute time, not memory, and all the processes still compete for the same HBM with no isolation. Co-residing via time-slicing does not give you more effective VRAM; it gives you more processes fighting over the same VRAM, and an OOM when the sum overflows. For real memory partitioning, MIG. The detail is in the sibling piece.

Conclusion

Having more models than VRAM is not a problem of insufficient hardware: it is a problem of managing a scarce resource over time. The intuition of “I need one GPU per model” is expensive and almost always wrong, because models are rarely used all at once. The three strategies are three answers to the same question, where do the weights of the models you are not currently using live: in HBM if they fit (co-residency, zero latency but they cost space), on disk if switching is rare (swapping, maximum VRAM per model but a return trip of seconds to minutes), or in RAM if switching latency matters (sleep mode, sub-second wake at the cost of tied-up RAM). Sleep mode is the most interesting addition of 2025 because it breaks the false dilemma of “everything loaded vs reload every time”: by keeping the process alive and preserving the allocator, the CUDA graphs and the kernels, it turns a 30–100 s cold start into a sub-second wake, and it does so even when it reloads the weights from the same disk (level 2), because moving bytes was never the whole cost. On the four-H100 cluster, the play is not to pick one strategy but to spread: co-reside what goes together, isolate the light stuff with MIG, sleep or rotate the large and occasional, replicate what saturates. The GPU is the stage with a single spotlight; the art is knowing which actor walks on, which goes home and which waits in the wings.

References

vLLM Blog (Embedded LLM), Zero-Reload Model Switching with vLLM Sleep Mode, 26 oct 2025: https://blog.vllm.ai/2025/10/26/sleep-mode.html
vLLM Docs, Sleep Mode: https://docs.vllm.ai/en/latest/features/sleep_mode/
mostlygeek, llama-swap (OpenAI/Anthropic-compatible model-swapping proxy): https://github.com/mostlygeek/llama-swap
llama-swap, Configuration: https://github.com/mostlygeek/llama-swap/blob/main/docs/configuration.md
NVIDIA, H100 Tensor Core GPU (HBM3 specs, 80 GB, ~3.35 TB/s): https://www.nvidia.com/en-us/data-center/h100/
NVIDIA, Reducing Cold Start Latency for LLM Inference with NVIDIA Run:ai Model Streamer: https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer/

Sharing one GPU between several workloads: time-slicing, MPS and MIG

Thu, 11 Jun 2026 02:00:00 +0000

This post opens an operational series on how to squeeze a generic on-premise 4×H100 SXM LLM cluster. The sibling pieces: Serving several models on one GPU: swap and sleep (what to do when the models do not fit at the same time and have to take turns in memory), RAG on CPU: separating the data plane from generation (moving retrieval off the GPU to free it up) and Sovereign end-to-end assistant with LibreChat, LiteLLM and RAG, the final assembly, a fourth instalment in preparation. Here we start with the most basic question: you have one GPU and you want to put several workloads on it. How do you split it?

TL;DR

You have one GPU, or a few, and several workloads that want to run on it: a chat model, an embeddings service, a reranker, a queue of dev jobs. The GPU is underused if it only runs one thing, but piling several on carelessly causes contention, OOM or cascading failures. There are three mechanisms and they share out different things. Time-slicing (replicas from the NVIDIA k8s device-plugin) multiplexes in time: it advertises the GPU as “N GPUs” and the processes take turns at the compute, but they share the full physical VRAM, with no memory isolation, no fault isolation and no QoS. Its trap is an OOM that does not show up in the Kubernetes scheduler but at run time, when the sum of VRAM allocations exceeds the real memory. MPS (Multi-Process Service) multiplexes in space: it shares out the SMs between processes that execute kernels concurrently, reduces context-switch overhead and allows SM and memory limits per process. It raises throughput when there are many small kernels, but fault isolation is still weak. MIG (Multi-Instance GPU) partitions in hardware: it cuts the Hopper GPU into up to seven instances with dedicated SMs, L2, memory and bandwidth, with real memory, fault and performance isolation; datacenter only (A100/H100/H200/B200), never on an RTX 5090. The rule: real isolation / multi-tenant / compliance → MIG (if it is Hopper); many small concurrent kernels and trust between workloads → MPS; dev, bursts, consumer GPU or no need to isolate → time-slicing. This post works it through with numbers: the VRAM budget of four vLLM instances on an H100 advertised as four replicas, and what fits into a 10 GB MIG instance.

The analogy: a shared hob, one kitchen with several cooks, several kitchens

Imagine you have a single professional hob and three orders to cook at the same time. There are three ways to organise it, and they are exactly the three mechanisms.

Time-slicing is one hob taken in turns, with no larder of your own. Each cook comes in, cooks their dish, leaves, and the next one comes in. The split is temporal: nobody cooks at the same time, they take turns. The problem is not the hob, which gets shared out fine, but the common larder: the ingredients sit in a single shared cupboard and nobody has their own. If the three cooks reserve more flour than there is in total, it is not that they wait their turn: it is that there is no flour. The service goes down for everyone. And if one cook leaves a pan burning and starts a fire, it burns the whole kitchen, not just their corner.

MPS is several cooks coordinated on the same worktop. Now they do cook at the same time, sharing out the space on the worktop (the SMs). A head chef (the MPS daemon) coordinates so they do not collide and so the worktop does not sit empty while one of them waits for water to boil. You can assign each cook a percentage of the worktop and a larder limit. They work faster as a group because the worktop does not sit idle between small tasks. But they still share the kitchen: if one starts a serious fire, the others notice.

MIG is several independent kitchens in the same building. A concrete wall separates each kitchen: its own hob, its own larder, its own door and its own electrical panel. What happens in kitchen 3, a fire, an empty larder, a slow cook, does not touch kitchen 1. It is the only split with genuine isolation. The price: you have to decide in advance how many kitchens and of what size, the walls are fixed, and only expensive buildings (datacenter) come ready to put them up.

The rest of the post is, essentially, when you want cheap turns, when you want coordinated cooks and when you need concrete walls.

An H100 SXM 80 GB does not fill up with just any workload. A bge-reranker-v2-m3 reranker takes a few hundred MB and saturates a handful of SMs; a bge-m3 embeddings service is just as small; a 1B guardrail model in INT4 fits in a couple of GB. Dedicating 80 GB of HBM3 and 132 SMs to serving embeddings is using a hydraulic press to hammer in a drawing pin, the same argument as in mixed environments, but now inside the GPU instead of moving the workload to different silicon.

The aim of sharing is to raise the useful utilisation of fixed capital. But sharing badly introduces three pathologies:

Compute contention: two workloads fight over the same SMs and both run slowly, with unpredictable latency jitter.
Memory contention: the sum of requested VRAM exceeds the physical amount and something dies with a CUDA out of memory.
Cascading failure: a workload that blows up (an illegal kernel, an OOM) can drag its neighbours down with it if they share a context.

The three mechanisms attack these pathologies to different depths. None of them solves all three except MIG, and MIG costs specific hardware. Let us look at them one by one.

Time-slicing: compute turns, shared larder

Time-slicing is temporal multiplexing in software. On Kubernetes, the NVIDIA GPU Operator configures the device-plugin to advertise N replicas of each physical GPU. An H100 declared with replicas: 4 appears to the scheduler as four nvidia.com/gpu resources, and Kubernetes can place four pods on it. Internally, the GPU scheduler hands out compute turns to each process: it runs a bit of process A, switches to B, to C, to D, then back to A. It is the same time-sharing an operating system does with the CPU.

The key idea, and the one that causes most confusion, is this: a replica is NOT a fraction of the GPU. It is a compute turn. NVIDIA’s documentation is explicit: unlike MIG, there is no memory or fault isolation between replicas. The four replicas of the H100 see the full 80 GB of VRAM, unpartitioned. There is no 20 GB per replica. There are 80 GB for all four, handed out in cudaMalloc arrival order.

This has three consequences you need to internalise:

It does not isolate memory. If the sum of what the four processes reserve exceeds 80 GB, the fourth cudaMalloc fails with OOM. The Kubernetes scheduler does not see it coming: it counted four available nvidia.com/gpu resources and happily placed four pods. The OOM appears at run time, not at scheduling time. This is trap number one of time-slicing.
It does not isolate faults. A process that triggers an unrecoverable CUDA error can leave the GPU context in a state that affects its neighbours. They share the same device with no barriers.
It gives no compute QoS. Under contention, the turn-taking does not guarantee a minimum fraction to anyone. Each workload’s latency suffers jitter proportional to how many active replicas are fighting over the GPU at that instant. A latency-sensitive workload (an interactive chat) can see its TTFT dance around depending on what its neighbours are doing.

What is it good for, then? For dev, bursts and low utilisation. If you have four developers who touch the GPU sporadically, advertising four replicas lets all four have access without fighting almost ever (they rarely overlap while active). For batch workloads that tolerate jitter. And, a decisive advantage, it works on consumer GPUs: an RTX 5090 32 GB does not support MIG, but it does support time-slicing. It is the only “Kubernetes-native” way to share a 5090 between several pods.

The VRAM budget in time-slicing (the calculation that avoids the OOM)

Here is the maths you have to do before deploying, because Kubernetes will not do it for you. Suppose an H100 80 GB advertised as 4 replicas and we want to run four vLLM instances on it, one per replica.

vLLM reserves memory with the --gpu-memory-utilization parameter, which is the fraction of the total physical VRAM that each instance keeps (for model weights plus KV-cache). The detail that kills: that fraction is computed over the 80 physical GB, not over a supposed “20 GB of my replica”, because the replica does not have 20 GB. Remember, there is no memory partitioning. Each vLLM sees the 80 GB and reserves its fraction of them.

The no-OOM constraint is therefore that the sum of fractions be less than 1:

$$\sum_{i=1}^{N} g_i < 1 \quad\Longleftrightarrow\quad \sum_{i=1}^{N} g_i \cdot V_{\text{HBM}} < V_{\text{HBM}}$$

where $g_i$ is the --gpu-memory-utilization of instance $i$ and $V_{\text{HBM}} = 80$ GB. It is wise to leave headroom (runtime overhead, fragmentation, CUDA context), so in practice you aim for the sum to stay comfortably below 1, say $\le 0.9$.

A case that works. Four vLLM instances at $g_i = 0.20$:

$$\sum_{i=1}^{4} 0.20 = 0.80 \quad\Rightarrow\quad 0.80 \times 80\ \text{GB} = 64\ \text{GB} < 80\ \text{GB} \quad\checkmark$$

Each instance reserves $0.20 \times 80 = 16$ GB. Four instances add up to 64 GB, leaving a 16 GB cushion. There is no OOM. Each vLLM has 16 GB for weights plus KV-cache: enough for a 7B–8B model in FP8/INT4 with a modest KV-cache.

A case that blows up. The same four instances, but somebody raises $g_i = 0.30$ thinking “I have four replicas, I can give each one more”:

$$\sum_{i=1}^{4} 0.30 = 1.20 \quad\Rightarrow\quad 1.20 \times 80\ \text{GB} = 96\ \text{GB} > 80\ \text{GB} \quad\times$$

The first instances start up and reserve $0.30 \times 80 = 24$ GB each. Three instances are already at $72$ GB. The fourth tries to reserve another 24 GB, there is nothing left, and it dies with CUDA out of memory. And worse: Kubernetes will reschedule it on the same GPU (it still sees four replicas), where it will die again, in a CrashLoopBackOff that makes no sense if you only look at the pod manifest.

The operational rule is brutally simple: in time-slicing, you manage the VRAM budget by hand, adding up the --gpu-memory-utilization values. The replica count controls how many pods fit in compute turns, but it does not reserve a single byte of memory. Confusing the two is the recurring mistake.

MPS: coordinated cooks on the same worktop

The Multi-Process Service (MPS) attacks a different problem. By default, when several processes use the same GPU without MPS, each one has its own CUDA context, and the GPU alternates between contexts (time-slicing at driver level): they do not execute kernels at the same time, they take turns, with context-switch overhead. If your kernels are small and do not fill the GPU on their own, this leaves SMs idle: process A uses 30 % of the SMs during its turn and the other 70 % is wasted.

MPS introduces a daemon that shares a single CUDA context between processes, so that their kernels can execute concurrently occupying different SMs at the same time. It is spatial sharing of compute: instead of taking turns at the whole worktop, each cook occupies a part and they work in parallel. This reduces context-switch overhead and raises throughput when there are many small concurrent kernels that individually do not saturate the GPU.

And, unlike pure time-slicing, MPS allows per-process limits, which gives you a form of QoS:

CUDA_MPS_ACTIVE_THREAD_PERCENTAGE limits the percentage of SMs an MPS client can use. By default each client gets $100 / \text{MaxSharedClients}$. Pinning it to, say, 40 % caps that process’s compute ceiling (MPS docs).
CUDA_MPS_PINNED_DEVICE_MEM_LIMIT imposes a memory cap per client (valid from CUDA 11.5). This is what time-slicing does not have: a per-process VRAM limit that the runtime enforces.

These two limits turn MPS into a resource provisioning mechanism that mitigates the noisy neighbour: you can guarantee that a process does not eat more than X % of the SMs or more than Y GB. The combination gives reasonable QoS, not perfect but real.

The limitation MPS does not solve: fault isolation is weak. Since clients share the daemon’s CUDA context, a fatal error in one client can affect the daemon and therefore the other clients (historically, a client dying dirtily could require restarting the daemon). It is better than time-slicing in this respect, but a long way from hardware isolation. That is why MPS fits when there is trust between the workloads, processes from your own team, not third-party tenants.

The canonical use case: many small concurrent inference requests that individually leave the GPU half empty. MPS overlaps them and raises aggregate throughput. Serving several small models, or several light replicas of the same model, on a datacenter GPU where you trust all the workloads, is MPS territory.

MIG: concrete walls

The Multi-Instance GPU (MIG) is the only one of the three that gives genuine isolation, because it cuts the GPU in hardware. Available on modern datacenter GPUs, A100 (Ampere), H100/H200 (Hopper), B200 (Blackwell), and never on consumer ones: an RTX 5090 (consumer Blackwell) does not support MIG, nor do GeForce cards in general.

MIG divides the GPU into up to seven instances (GPU Instances), and each instance gets a dedicated portion of:

SMs (compute slices): compute is split into 7 slices, each one roughly 1/7 of the SMs.
L2 cache and memory: each instance has its slab of HBM and its portion of L2 cache.
Memory bandwidth: dedicated, not shared.
Data paths and engines: with fault barriers between instances.

The result is that a MIG instance behaves like a smaller, independent GPU: whatever happens in one, an OOM, a kernel blowing up, a workload saturating its compute, does not affect its neighbours. Memory, fault and performance (QoS) isolation, the three things time-slicing does not give and MPS only half gives.

The H100 80GB profiles

MIG does not allow arbitrary sizes: it has fixed profiles. On the H100 80GB, the profile catalogue (notation <compute>g.<memory>gb) is:

Profile	Compute (slices)	Memory	Max. instances
`1g.10gb`	1/7	10 GB	7
`1g.20gb`	1/7	20 GB	4
`2g.20gb`	2/7	20 GB	3
`3g.40gb`	3/7	40 GB	2
`4g.40gb`	4/7	40 GB	1
`7g.80gb`	7/7	80 GB	1 (whole GPU)

(There is also 1g.10gb+me, a variant with media engines for video encoding.) The base memory unit on the H100 80GB is 10 GB per slice (80 GB / 8, with one slice reserved), and the compute unit is 1/7 of the SMs. The profiles combine these units. Note 1g.20gb: the same compute fraction as 1g.10gb (1/7 of the SMs) but twice the memory, useful when a workload needs more VRAM than compute.

One important detail: MIG partitions do not mix freely. The GPU is divided following a valid geometry (the profiles fit together like pieces in a grid), and the profiles are fixed when you configure the GPU; changing them requires draining and repartitioning. They are concrete walls: solid, but they do not move while hot.

The calculation: 7×1g.10gb against 1×7g.80gb

Let us compare the two extremes. On the left, seven 1g.10gb instances: seven isolated 10 GB GPUs. On the right, a single 7g.80gb: the whole H100, unpartitioned.

The operational question is what fits in 10 GB. An instance’s VRAM budget is split between model weights and KV-cache:

$$V_{\text{inst}} = V_{\text{weights}} + V_{\text{KV}} + V_{\text{overhead}}$$

Take a model of 7B parameters in FP8 (1 byte/parameter):

$$V_{\text{weights}} \approx 7 \times 10^9 \times 1\ \text{byte} = 7\ \text{GB}$$

On a 1g.10gb instance (10 GB), after the 7 GB of weights and subtracting roughly 0.5–1 GB of runtime overhead, about 2 GB are left for KV-cache. That is enough for a modest context window and low concurrency, fine for a guardrail service, a classifier or an extraction model that handles short prompts one at a time. A 7B in INT4 (~3.5 GB of weights) leaves ~5.5 GB of KV-cache, much roomier. But a 13B model in FP8 (~13 GB of weights) does not fit in a 10 GB instance: not even the weights get in. For that you need 1g.20gb, 2g.20gb or larger.

Against this, the 7g.80gb (whole GPU) gives you the 80 GB for one large model: a 70B in FP8 (~70 GB of weights) fits with a tight KV-cache, or a 70B with more room spread across several whole GPUs in tensor-parallel (see TP against replicas: one big one versus N small ones).

The reading is clear: fine partitioning (7×1g.10gb) maximises the number of small isolated workloads; not partitioning (1×7g.80gb) maximises the model size that fits. The KV-cache available per instance shrinks proportionally as you partition, so fine-grained MIG serves many light isolated services, not one large model chopped up. If your workload is a single large model, MIG is not for you: use the whole GPU or several in TP.

The decision tree

The three questions, in order:

Do you need REAL isolation?
(multi-tenant, compliance, one workload failing must not touch another)
│
┌────┴────┐
YES NO
│ │
Is it Hopper/ Many SMALL concurrent kernels
Ampere/ AND you trust every workload?
Blackwell? │
│ ┌────┴────┐
┌──┴──┐ YES NO
YES NO │ │
│ │ MPS Dev / bursts / consumer GPU
MIG │ (spatial, / no need to isolate?
│ │ per-process │
│ no real QoS) YES
│ isolation: │
│ rethink TIME-SLICING
│ (move to (temporal, cheap,
│ CPU, another works on a 5090)
│ GPU, or accept
│ the risk with
│ time-slicing)

And each branch in one sentence:

MIG when isolation is a requirement (compliance, multi-tenant, hard SLA) and you have datacenter hardware that supports it. Concrete walls cost money, but if you need them there is no substitute.
MPS when you have many small concurrent workloads that leave the GPU half empty and you trust all of them (same team, not third-party tenants). You raise throughput with reasonable QoS, accepting imperfect fault isolation.
Time-slicing when it is dev, bursts, low utilisation, a consumer GPU, or you simply do not need to isolate anything. Cheap and universal, but manage the VRAM budget by hand.

One nuance the recent documentation records: they can be combined. You can do time-slicing on top of a MIG instance (hardware isolation at the instance boundary, software turns inside), or use MPS inside a MIG instance. The layers are not mutually exclusive; the tree picks the primary strategy.

Applied to the generic 4×H100 cluster

Let us get down to numbers with a generic on-premise cluster of 4×H100 SXM 80 GB with NVLink. It is common to have a heterogeneous menu of workloads: a large chat model, light services (embeddings, reranker, guardrails) and a dev/experimentation queue. Each type calls for a different mechanism. A reasoned split:

GPU 0 and GPU 1, large model in tensor-parallel (not shared). A 70B model in FP8 takes ~70 GB of weights; served comfortably with a generous KV-cache it needs more than one H100. We spread it in tensor-parallel over two whole H100s joined by NVLink (intra-node bandwidth is what makes TP viable; the detail is in TP against replicas). Here we do not share: these two GPUs belong to the large model, full stop. Total isolation by dedication.

$$V_{\text{available}} = 2 \times 80 = 160\ \text{GB};\quad V_{\text{weights}} \approx 70\ \text{GB};\quad V_{\text{KV}} \approx 80\ \text{GB of KV-cache}$$

There is memory to spare for a long request queue and high concurrency.

GPU 2, split with MIG into small instances for light services. Embeddings (bge-m3), reranker (bge-reranker-v2-m3) and a couple of guardrail models (1B–3B) are different workloads, potentially from different teams, and you want a failure or a spike in one not to touch the others. Light multi-tenancy with isolation → MIG. A reasonable partitioning of the H100:

$$\underbrace{3 \times \texttt{1g.10gb}}_{\text{30 GB, 3/7 SMs}} \;+\; \underbrace{1 \times \texttt{4g.40gb}}_{\text{40 GB, 4/7 SMs}}$$

The three 1g.10gb instances (10 GB, 1/7 of the SMs each) host embeddings, reranker and a 1B INT4 guardrail, each isolated, with no noisy neighbour. The 4g.40gb (40 GB, 4/7 of the SMs) hosts an intermediate 7B–13B model with a decent KV-cache for a support service. Each service has its larder and its wall; if the reranker blows up, the chat never finds out.

GPU 3, time-slicing for dev and bursts. The developers touch the GPU sporadically: experiments, short fine-tunes, model trials. They do not need isolation (it is the same team) and they rarely overlap while active. We advertise it as 4 replicas via the device-plugin. Four dev pods fit in turns. VRAM budget with the formula above: if each dev brings up a vLLM at --gpu-memory-utilization 0.20, they add up to $4 \times 16 = 64$ GB < 80 GB, with no OOM. If somebody needs more, lower the replica count or coordinate with the team. The cost of flexibility is manual discipline.

Summary of the split:

Resource	Mechanism	Workload	Isolation
GPU 0 + GPU 1	Dedication (TP)	70B chat in tensor-parallel	total (dedicated)
GPU 2	MIG (3×1g.10gb + 1×4g.40gb)	embeddings, reranker, guardrails, 7B–13B	real hardware
GPU 3	Time-slicing (4 replicas)	dev, bursts, experiments	none (trust)

The logic is always the same: spend isolation (MIG) where you need it, spend cheap concurrency (time-slicing) where you do not, and reserve whole GPUs for what genuinely fills them. An H100 serving embeddings on a 7g.80gb would be as absurd as an RTX 5090 trying MIG: the tool does not match the workload.

What we have not covered

What happens when they do not even fit at the same time: if you have more models than VRAM and they have to take turns in memory (loading/unloading weights, not just taking turns at compute), you enter swap and sleep territory, the sibling piece Serving several models on one GPU.
NUMA-aware scheduling: on multi-socket nodes, which GPU goes with which CPU/memory matters for latency; see Kubelet resource managers on RKE2.
Autoscaling the replicas: how many instances to bring up according to real load, with KEDA and queue metrics; see Autoscaling LLMs on Kubernetes with KEDA.
A jitter benchmark under contention: how much TTFT really wobbles in time-slicing with 4 active replicas compared with MIG, material that deserves its own measurement, not an estimate.

References

NVIDIA — Time-Slicing GPUs in Kubernetes (GPU Operator). https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html
NVIDIA — Multi-Process Service (MPS) Overview. https://docs.nvidia.com/deploy/mps/index.html
NVIDIA — MPS: Tools and Interface Reference (CUDA_MPS_ACTIVE_THREAD_PERCENTAGE, CUDA_MPS_PINNED_DEVICE_MEM_LIMIT). https://docs.nvidia.com/deploy/mps/appendix-tools-and-interface-reference.html
NVIDIA — Multi-Instance GPU (MIG) User Guide. https://docs.nvidia.com/datacenter/tesla/mig-user-guide/
NVIDIA — Supported MIG Profiles (H100 80GB catalogue). https://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-mig-profiles.html
NVIDIA — k8s-device-plugin (time-slicing replicas). https://github.com/NVIDIA/k8s-device-plugin
vLLM — Engine Arguments (--gpu-memory-utilization). https://docs.vllm.ai/en/latest/serving/engine_args.html

Aggressive RAG in small models: trading parameters for retrieval

Tue, 09 Jun 2026 02:20:00 +0000

This post belongs to the series on inference performance in small models. Its sibling piece, The roofline inverts in small models, explains why the compute-bound prefill is the bottleneck that shapes the whole discussion here. It is worth reading first: here we assume that adding more context is not free.

TL;DR

An SLM (say 1B–8B parameters) knows fewer facts than a 70B–700B model, simply because it has fewer weights in which to memorise them. But its ability to reason over text in front of it, following instructions, extracting, synthesising, comparing, degrades far less with size than its encyclopaedic knowledge does. The operational consequence is direct: use the SLM as a reasoning engine over curated context, not as a database. Move knowledge from the weights into the context via retrieval. The problem is that “aggressive retrieval” is often read as “put in lots of chunks”, and that collides head-on with three facts about SLMs: shorter context windows, worse use of long context (the lost in the middle effect is more severe the smaller the model) and a compute-bound prefill whose cost grows with the context length $C$, linearly in the projections and quadratically in attention. You cannot simply add tokens. Each retrieved token is paid for twice, in degraded quality and in TTFT, and the SLM is the worst placed to absorb either cost. The way out is not to retrieve less, but to retrieve better: reranking for precision over recall, compressing the context before injecting it, prefix caching the stable documents, semantic caching of answers, and structured output with external tools that stand in for internal knowledge. This post works through the maths and gives a TTFT number before and after compressing a context from 4,000 to 1,000 tokens on an RTX 4090.

The analogy: the open-book exam

Two students sit the same exam. The first has a prodigious memory: they have memorised the entire syllabus, paragraph by paragraph. The second has an ordinary memory, forgetting dates, confusing names, but is allowed to bring in a crib sheet.

If the second student’s crib sheet is a chaos of piled-up photocopies, they lose: they take too long to find what they are after, get distracted by irrelevant pages and run out of time. But if their crib sheet is excellent, trimmed to the essentials, reordered by relevance, with the important things at the top and no filler, then they not only do not lose: they often win, because they reason just as well as the first student and, on top of that, work from verified material instead of from blurry recollections they may be inventing.

The moral has three layers, and each maps to an engineering decision:

Memorising everything is expensive. The first student invested months. A large model invests parameters, and VRAM, and inference FLOPs, in memorising facts.
The crib sheet matters more than its size. A well-made one-page crib sheet beats ten badly organised pages. More retrieved context is not better context: the material’s precision beats volume.
Knowing how to search and synthesise is a different skill from knowing. It is the one the SLM retains. The whole strategy consists of leaning on that skill and subcontracting the memory.

The rest of the post is, essentially, how to build an excellent crib sheet under the constraint that the student (the SLM) reads slowly and gets tired with long texts.

The capacity argument: how many facts fit in the weights

Let us start by justifying the thesis with an order of magnitude, not with faith. How much factual knowledge really fits in a model’s weights?

There is a recurring empirical estimate in the interpretability and memorisation literature: a dense model can store on the order of 2 bits of memorised information per parameter before saturating (the exact figure varies by study and training regime; take it as an order of magnitude, not as a law). An 8B-parameter model then has an information storage ceiling on the order of:

$$8 \times 10^9 \text{ params} \times 2 \text{ bits/param} = 1.6 \times 10^{10} \text{ bits} \approx 2 \text{ GB of information}$$

And that budget is not only for facts: the vast majority goes on grammar, syntax, reasoning ability, code, formatting, and only a fraction is left for encyclopaedic knowledge. Compare it with the other side: a retrievable corpus of several million documents, a corporate wiki, a document repository, a technical knowledge base, easily takes up hundreds of GB to terabytes of text, indexed and queryable with millisecond latency. The asymmetry is two or three orders of magnitude in favour of the external corpus.

The conclusion is not that the weights are useless, they are where reasoning lives, which is the expensive thing to replicate, but that competing with an external index on fact capacity is losing by construction. A 70B model has ~9× more memorisation budget than an 8B one, but it is still negligible against the corpus. That is why the large model also does RAG in production. The difference is that the SLM needs it: without retrieval, its factual knowledge is too sparse and, worse, prone to hallucinating exactly in the gaps it did not memorise.

The central tension: retrieving more is not putting in more

This is where most naive designs break. “Aggressive retrieval” sounds like a large top-k: if retrieval helps, retrieve 20 chunks instead of 5. But in an SLM that fails for two independent reasons, one of quality and one of cost.

(a) SLMs make worse use of long context

The lost in the middle effect (Liu et al., 2023) is well known: LLMs retrieve information placed at the beginning and end of the context better, and the information in the middle worse. What gets less emphasis is that the effect is more severe the smaller the model. An SLM has fewer attention heads, fewer layers and poorer internal representations for “tracking” a relevant fact buried at position 14 out of 20 chunks. Its nominal context window also tends to be shorter (4K–32K against the 128K+ of the large ones), and the effective window, the length beyond which quality collapses, is smaller still. Putting in 20 chunks does not mean the model reads all 20: it means it will probably ignore or misread the ones in the middle, while paying the cost of all of them.

(b) Prefill grows with the context and is compute-bound

This is the blow people underestimate. Prefill, processing the full prompt before emitting the first token, is the compute-bound phase of inference (unlike decode, which is memory-bound; the detail lives in The roofline inverts). Its cost grows with the context length $C$, and it determines TTFT (time to first token). More chunks → more prefill tokens → more TTFT and more compute cost per request. In an SLM, where prefill is proportionally more expensive relative to the model, this hurts especially.

The operational conclusion is uncomfortable but clear: you cannot compensate for fewer parameters simply by putting in more context. Each retrieved token is paid for twice, in degraded quality and in TTFT, and the SLM is the worst placed to absorb both costs. The way out is to retrieve less but better, and to compress what you retrieve.

The maths of prefill

Let us put numbers on “prefill grows with the context”. For a context of $C$ tokens, a transformer layer does two classes of work:

Linear projections (QKV, attention output, FFN): each token is multiplied by fixed-size weight matrices. The cost is $O(C)$ in FLOPs, linear in the number of tokens.
Attention ($QK^\top$ and the multiplication by $V$): each token attends to all the others. The cost is $O(C^2)$, quadratic in the number of tokens.

The total prefill cost per layer has the form:

$$\text{FLOPs}_{\text{prefill}} \approx \underbrace{a \cdot C}_{\text{projections}} + \underbrace{b \cdot C^2}_{\text{attention}}$$

with $a$ and $b$ constants that depend on the model dimension. For moderate contexts (a few thousand tokens) in an SLM, the linear term still dominates or is comparable to the quadratic one; the quadratic term becomes dominant at long contexts. What matters: if you compress the context $C \to C/k$, the linear term falls $\times k$ and the quadratic one falls $\times k^2$. Compression is the only lever that attacks both terms at once, and it attacks the worse one disproportionately.

Numerical example: TTFT before and after compressing, RTX 4090

Let us model TTFT as the time to process the prefill tokens at a given prefill throughput. Take an RTX 4090 (24 GB, Ada Lovelace) serving a quantised SLM, with a prefill throughput of ~5,000 tok/s (an illustrative figure; the real value depends on the model, the quantisation and the batch, so measure it, do not assume it).

Let there be a retrieved context of 4,000 tokens (8 chunks of ~500 tokens). Approximating TTFT as dominated by the context prefill:

$$\text{TTFT}_{\text{before}} \approx \frac{4000 \text{ tok}}{5000 \text{ tok/s}} = 0.80 \text{ s}$$

Now we compress that context to 1,000 tokens ($k = 4$). Prefill throughput is not constant with $C$, it drops a little at long contexts because of the quadratic term, but taking the conservative linear tokens/throughput approximation:

$$\text{TTFT}_{\text{after}} \approx \frac{1000 \text{ tok}}{5000 \text{ tok/s}} = 0.20 \text{ s}$$

TTFT falls from 0.80 s to 0.20 s, a $4\times$ reduction in the linear part. But the FLOPs arithmetic is even more favourable in the attention component: that part of the work falls $\sim k^2 = 16\times$. In practice total TTFT does not fall 16× because the cost is not purely quadratic at this scale, but the real reduction lies between 4× and something larger depending on how much attention weighed, and the aggregate compute saving (what pays the electricity bill and frees the GPU for another request) is substantially bigger than the plain 4× of the token count.

The argument generalises: compressing the context by a factor $k$ reduces TTFT by at least $\sim k\times$ and the attention cost by $\sim k^2\times$. For an SLM, where TTFT is often the SLA that matters, this is the difference between an assistant that answers instantly and one that feels slow.

The five levers to resolve the tension

The strategy is not “retrieve less and settle”. It is retrieve aggressively from the index and then distil aggressively what you retrieved before it reaches the SLM. Five levers, in order of application within the pipeline.

1. Aggressive reranking: precision over recall

The initial retriever (dense, sparse or hybrid) optimises recall: it brings back 50–100 candidates so as not to leave anything out. The reranker, a cross-encoder that sees the query and the document together, optimises precision: it reorders those candidates and you keep the 3–5 best. For an SLM this is not a luxury, it is structural: since the model uses long context badly, every chunk that goes in must earn its place. Better 4 chunks of very high relevance than 15 mediocre ones. The detail of hybrid retrieval and reranking is in Reranking and hybrid retrieval; here the rule is enough: maximise recall in the retriever, maximise precision in the reranker, and inject few.

2. Context compression: distilling the crib sheet

Once you have the best chunks, they still contain filler: padding sentences, redundancy, context irrelevant to the specific query. Context compression trims them before injection:

Extractive compression (LLMLingua / LongLLMLingua style, Jiang et al. 2023): a small model scores the perplexity or relevance of each token/sentence with respect to the query and removes the low-information ones, keeping the densest extractive subset. It reduces tokens without a second large generative model in the loop. LongLLMLingua adds position-aware reordering to mitigate lost in the middle.
Abstractive compression: a model summarises the retrieved chunks into shorter text. More aggressive in token reduction, but it introduces a generative step (cost and possible loss of fidelity).
Soft prompts / context distillation: compressing the retrieved context not into text, but into a handful of embeddings/soft tokens that the model consumes directly. It reduces the number of prefill tokens to a minimum, at the price of a trained, model-specific component.

The key point connects with the maths above: compressing what you retrieved by a factor $k$ reduces prefill tokens $\times k$, and therefore TTFT $\sim\times k$ and the attention cost $\sim\times k^2$. It is the lever with the best return when long context is the bottleneck.

3. Prefix caching of stable context

Not all the context changes between requests. System instructions, definitions, recurring reference documents, schemas: these are stable prefixes. Prefix caching stores the already-computed KV cache of those prefixes and reuses it, so that prefill only processes the new part (the query and the specific chunks). If 60 % of your context is stable, you save 60 % of that segment’s prefill on every hit. For it to work, the stable context must go at the start of the prompt (the KV cache is prefix-dependent) and it pays to maximise the hit rate; the hit-rate engineering detail is in Prefix cache hit rate. It combines especially well with RAG: retrieved documents that repeat across sessions get cached once.

4. Semantic caching of answers

A layer in front of the model: if a query is semantically equivalent to one answered before (embedding similarity above a threshold), return the cached answer and skip the model entirely, retrieval, prefill and decode included. In real workloads with long tails of repeated or near-repeated questions (FAQ, support), the saving is enormous because it eliminates the full cost, not just the prefill one. The trap is the threshold: too lax and you serve wrong answers to similar-but-different questions. The design is in Semantic caching for RAG.

5. Structured output and function calling: lean on tools, not on memory

The last lever changes what the SLM depends on. Instead of asking it to know a fact (its weak point), make it call a tool that knows it: a database query, an API, a calculator, a validator. Structured output (forcing JSON conforming to a schema) and function calling turn the SLM into an orchestrator that extracts arguments from the context and delegates the computation or the query. A reasonably capable SLM emits a well-formed tool call far more reliably than it recalls a specific fact. This reduces the pressure on parametric knowledge and on retrieval: for structured, fresh data (prices, inventory, states), querying beats retrieving text and beats memorising. The fundamentals are in Structured output and Function calling.

The complete pipeline

The five levers are not alternatives: they chain together. The flow, with the token counter falling at each step:

The order matters. Retrieving aggressively (high recall) before filtering guarantees that the right material is among the candidates; reranking and compressing afterwards guarantees that only the dense and relevant material pays the prefill toll; caching wraps everything so as not to repeat work. The SLM only sees the final crib sheet, short and ordered.

Implications for on-premise inference

The mental trap to avoid: treating the SLM as a large model with less quality. It is not. It is a different cost profile that rewards a different design. Three practical consequences:

The token budget is a first-class resource. With a large model with a 128K window, “putting in a bit more” is cheap relative to the model. With an SLM, every context token shows up in TTFT and in quality. Treat context size as a quantity to minimise subject to covering the answer, not to maximise.
The investment is worth it precisely because the model is cheap. Reranker, compressor and caches add complexity, but the model they serve is economical enough to run many replicas. The bottleneck shifts from the model to the data pipeline, which is exactly where you want it.
Retrieving does not replace adapting; they combine. For deep, recurring domain knowledge, adapting the SLM with LoRA (see the sibling QLoRA and aggressive multi-LoRA) can put part of the knowledge “into the weights” cheaply, reducing what has to be retrieved. Aggressive RAG and aggressive adaptation do not compete: the first gives freshness and citability, the second gives fluency and domain formatting. A good design uses both.

On the RTX 4090 (24 GB, Ada Lovelace)

The canonical scenario: a quantised SLM (4B–8B in INT4/FP8) fits with room to spare, leaving VRAM for a generous KV cache, essential for prefix caching, and for the reranker (a cross-encoder of a few hundred MB). The LLMLingua-style extractive compressor runs on a separate small model or on CPU. The TTFT calculation above (0.80 s → 0.20 s compressing 4× at ~5,000 tok/s) is representative of this card. The rule of thumb: if TTFT goes above your SLA, the first adjustment is to compress the context, not to change the model.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

With 320 GB and native FP8 prefill is much faster, so the temptation is to relax the token discipline. That is not quite advisable: the lever changes from TTFT to aggregate throughput. Compressing the context not only speeds up each request but frees prefill compute to serve more requests per GPU; compute-bound prefill is exactly the resource that saturates first under load. Here prefix caching and semantic caching, shared across replicas, are what pay off most: at high QPS, the prefill work you avoid by caching is pure throughput you gain. The SLM is still the cheap reasoning engine; the difference is that you now run many in parallel and the data pipeline is what decides how many requests fit.

What we have not covered

Evaluating compression: how to measure that compressing $k=4$ does not throw away correct answers (faithfulness, answer recall over a set of questions with ground truth).
Query-aware versus query-agnostic compression: compressing before or after knowing the question changes what can be cached and what can be discarded.
Chunking and granularity: chunk size interacts with reranking and compression; that is left for the corpus curation post.
Multi-hop and agents: when a question requires several retrieval rounds, the token budget is split across hops and compression discipline becomes critical.

References

Lewis, P., et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020. https://arxiv.org/abs/2005.11401
Liu, N.F., et al. Lost in the Middle: How Language Models Use Long Contexts. TACL 2024. https://arxiv.org/abs/2307.03172
Jiang, H., et al. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. EMNLP 2023. https://arxiv.org/abs/2310.05736
Jiang, H., et al. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression. ACL 2024. https://arxiv.org/abs/2310.06839

Aggressive quantisation (state of the art): from 4-bit to ternary

Tue, 09 Jun 2026 02:10:00 +0000

This post is the direct continuation of Quantization for LLM inference, which covers the “solved” regime (FP8, INT4 with GPTQ/AWQ). Read it first: here I assume the scale+zero-point maths, what GPTQ and AWQ do, and the PTQ/QAT distinction. What we add is the sub-4-bit frontier, where scalar post-hoc quantisation stops working and you have to change tools.

TL;DR

There is a sharp dividing line around 4 bits. Above it, quantising is a solved problem: INT8 is indistinguishable from BF16, and INT4 with a good method (AWQ, GPTQ) loses 1-2 MMLU points and little else. The method is still the usual one, take each weight, scale it, round it to a short integer, and it works. Below 4 bits, that method collapses: at 2 bits, naive scalar quantisation can double perplexity. The reason is geometric, each weight has only 4 possible values and the rounding error stops being negligible, and the way out is not “round better”, it is to change the representation. State-of-the-art 2-bit methods (AQLM, QuIP#, QTIP) stop quantising individual weights and quantise vectors of weights against dictionaries (codebooks), and they “whiten” the weight matrix to spread its energy and flatten outliers (incoherence processing). Ternary is something else again: BitNet b1.58, with weights in {-1, 0, +1} (~1.58 bits), is not PTQ, it is a model trained natively under that constraint, and it changes the arithmetic of the matmul from multiplications to additions and subtractions, hitting the compute ceiling and the memory ceiling at once. The mental rule: at ≥4-bit you compress the photo; below 4-bit you have to repaint it.

The analogy: the JPEG that cannot be compressed any further

In the quantization post we used the JPEG with an edge detector to explain INT4. The analogy still holds here, but it has to be pushed to its limit.

A JPEG at quality factor 90 is indistinguishable from the original. At quality 60 you notice it a little, but it is still “the same photo”. At quality 30 the 8×8 blocks appear, the haloes around edges, the banding in gradients. At quality 10 the image is destroyed: you can tell there was a face, but the details have disappeared under the artefacts. And here is the key: no JPEG encoder exists that can compress to quality 10 without those artefacts, because the JPEG algorithm (blockwise DCT + coefficient quantisation) has an information floor below which its own mechanism introduces the noise.

What do you do if you need the photo at that file size and you need it to still look good? You do not compress the original harder. You repaint the photo knowing in advance that it is going to live compressed: an illustrator redraws it with clean lines, a reduced palette, no subtle gradients, an image designed to survive brutal compression. The result at “10 KB” looks infinitely better than the original JPEG crushed to 10 KB, because it is not the same process: one destroys existing information, the other generates new information already adapted to the constraint.

That is exactly the frontier of this post:

Scalar PTQ (≥4-bit) = compressing the JPEG. Up to a certain ratio, it is still the same photo.
State-of-the-art vector PTQ (2-bit: AQLM, QuIP#, QTIP) = a far more sophisticated image codec (dictionaries, decorrelating transforms) that stretches the compressible ratio a bit further before the collapse.
Native ternary (BitNet b1.58) = repainting the photo. You are not compressing an existing BF16 model; you are training a new one that is born ternary.

The map of the frontier, bit by bit

Quantising a model is deciding how many distinct values each weight can take. With b bits per weight there are 2^b possible values. The central question is: past what b is the number of values so small that rounding destroys the model?

Bits	Values/weight	State of the art	Method required	Typical loss vs BF16
8	256	Solved	RTN, SmoothQuant, FP8	~0 (indistinguishable)
4	16	Solved	AWQ, GPTQ	1-2 pp MMLU, +0.1-0.3 PPL
3	8	Small degradation	tuned GPTQ/AWQ, GGUF Q3_K	3-5 pp MMLU
2	4	Serious unless SOTA	AQLM, QuIP#, QTIP (non-scalar)	scalar: collapse; SOTA: 4-8 pp
1.58	3 (ternary)	Native only	BitNet b1.58 (QAT/native training)	n/a (not PTQ)
1	2 (binary)	Research	native, dubious claims	large / no clear methodology

The three transitions that matter:

8 → 4 bits: nothing breaks. With 16 levels per weight and one scale per block of 128, the rounding error is small relative to the dynamics of the weights. GPTQ compensates for the error by propagating it to the neighbouring weights; AWQ protects the ~1 % of salient channels. The model barely notices. This is in the previous post.

4 → 2 bits: the elbow. Two things happen at once here. First, with only 4 levels, the scalar quantiser can no longer represent the weight distribution, which is roughly Gaussian with long tails, without a proportionally enormous rounding error. Second, and more subtly: the quantisation error stops being “small noise that the model absorbs” and becomes structured, systematically biasing the activations. Naive scalar PTQ at 2 bits on a Llama 8B typically doubles perplexity or worse. That is the elbow of the curve.

2 → 1.58 bits: a change of nature. You do not cross this with a better compression method. You cross it by training the model with the constraint from the start. It is a discontinuity: to the left you are doing PTQ, to the right you are doing training.

Why scalar PTQ collapses below 4 bits

The scalar quantiser has a fundamental limitation: it quantises each weight separately, ignoring the fact that the weights of a row or column are correlated and that the error of one could be compensated by another. At 4 bits this matters little; at 2 bits it is lethal. There are three possible attacks, and the state-of-the-art methods use all three.

1. Vector quantisation: dictionaries instead of scales

Instead of mapping each weight to one of 4 values, group the weights into vectors (say, of 8 weights) and map each vector to the nearest entry of a learned dictionary (codebook). If the dictionary has 256 entries, encoding a vector of 8 weights costs 8 bits (the index) → 1 bit/weight, but every “reconstructed value” is a point in an 8-dimensional space chosen to minimise the error over the real distribution of the weights.

The advantage is information-theoretic: a vector dictionary can place its reconstruction points where the weights actually are (in clusters), whereas the scalar quantiser is forced to put its 4 levels on a regular grid, wasting resolution on empty regions. It is the difference between a road map with a uniform grid and one that puts more detail where the cities are.

AQLM (Additive Quantization of Language Models, arXiv:2401.06118) takes this to the extreme with additive quantisation: each weight vector is reconstructed as the sum of several codes from several dictionaries (multi-codebook). It is more expressive than a single dictionary because the number of combinations is the product of the sizes, not the sum. AQLM was one of the first methods to make 2-bit “usable” (not collapsed) on large models, at the cost of an expensive calibration process and specialised inference kernels.

2. Incoherence processing: whitening the matrix

The second attack targets the outliers. The weight matrices of a transformer have a few entries (and a few channels) with a magnitude much larger than the rest. Those outliers dominate the quantiser’s range: if you have to represent a weight of magnitude 8 while the rest are of magnitude 0.5, your scale stretches to cover the 8 and you waste almost all the resolution.

Incoherence processing (the core idea of QuIP and QuIP#) attacks this by multiplying the weight matrix W by random orthogonal matrices on the left and on the right: W' = U W V^T. Because U and V are orthogonal, the operation is invertible and the maths of the product can be undone at inference time by absorbing it into the neighbouring layers (just as AWQ absorbs its scales). But the rotation spreads the energy: an “incoherent” matrix has its values distributed almost uniformly, with no concentrated outliers, because mixing coordinates with a random rotation flattens the distribution (it is, in essence, the central limit theorem acting on linear combinations). A matrix without outliers quantises far better at 2 bits. It is the equivalent of “whitening” a signal before digitising it.

QuIP# (arXiv:2402.04396) combines incoherence processing with E8 lattice codes: rather than an arbitrary dictionary, it uses the E8 lattice (an optimal sphere packing in 8 dimensions, the best known). Quantising vectors of 8 weights against the E8 lattice gives the smallest reconstruction error possible for a given bit density, because E8 is literally the most efficient way to place points in 8D. It is classical coding theory applied to LLM weights.

3. Coding with memory: trellis

QTIP (arXiv:2406.11235) adds the third attack: trellis-coded quantisation. Instead of quantising each vector independently, it models the sequence of weights as a path through a trellis (the same structure as the convolutional codes of telecommunications) and picks the optimal code sequence with the Viterbi algorithm. The intuition: introducing memory between successive quantisations allows correlated errors that cancel out, instead of independent errors that accumulate. QTIP, on top of incoherence processing, improves on QuIP# in quality at 2-3 bits while keeping fast inference kernels.

The idea common to all three: stop quantising scalars and start quantising vectors with dictionaries, and decorrelate the matrix before doing so. None of them is “rounding better”; all three change the representation at the root. That is why, below 4 bits, a flag in vLLM is no longer enough: you need co-design of the quantisation method and the inference kernel.

Native ternary: BitNet b1.58

Here we change continents. Everything above is PTQ: it starts from a trained BF16 model and compresses it. BitNet’s ternary compresses nothing.

BitNet b1.58 (arXiv:2402.17764) trains a transformer from scratch where every weight is constrained to {-1, 0, +1} throughout training. Three values ⇒ log₂(3) ≈ 1.58 bits/weight. Quantisation is not a later step: the linear layers (BitLinear) quantise their weights to ternary in the forward pass of every training step, and gradients flow through a straight-through estimator. The model learns to work with ternary weights. This is QAT taken to the extreme: not a short robustness fine-tune, but the constraint present from the first token of training.

That difference is what dodges the elbow of the curve. PTQ at 2 bits tries to find the best ternary or quaternary approximation of a model that was trained expecting full precision, and that model has “fragile” weights that depend on nuances 2 bits cannot capture. BitNet, by contrast, never had those nuances: its weights were born ternary, so the network distributed its representational capacity in a way compatible with the constraint. It is repainting the photo instead of compressing it.

What changes is not just the memory, it is the arithmetic

The most underrated point about BitNet: with weights in {-1, 0, +1}, multiplication disappears from the matmul. Multiplying an activation x by a ternary weight w is trivial: if w = +1 you add x, if w = -1 you subtract x, if w = 0 you do nothing. The dominant operation of a transformer, the matrix-vector product, goes from a sea of floating point multiply-accumulate (MAC) to integer additions and subtractions.

This matters because it connects to the roofline. As explained in The inverted roofline of small models, LLM inference has two ceilings: memory (HBM bandwidth for loading weights) and compute (tensor core FLOPs). Normal quantisation (INT4, FP8) attacks only the memory ceiling: the weight takes up less space, but to multiply it you dequantise to FP16 and do the same multiplication as always. Ternary attacks both ceilings at once: the weight occupies 1.58 bits (memory) and the operation is an addition instead of a multiplication (compute). That is why BitNet needs its own kernels, bitnet.cpp, that execute the ternary matmul without ever going through FP16; a kernel that dequantised to FP16 in order to multiply would throw away half the advantage.

The honest trade-off: BitNet b1.58 is training from scratch. You cannot “convert your Llama 8B to BitNet”. If you want ternary, you train (or use) a natively ternary model, with everything that implies in pre-training cost and weight availability. Today it is a research line with published models at modest scales, not a drop-in replacement for your current serving.

QAT as the bridge between PTQ and native

Between “compressing post-hoc” (PTQ) and “training natively ternary” (BitNet) there is a middle point: QAT (Quantization-Aware Training). You take an already trained model and do a short fine-tune with the quantisation operations inside the loop, so that it learns to be robust to low bit counts without paying for a full pre-training.

Gemma 3 publishes official QAT variants for exactly this: models that, after the QAT fine-tune, hold up at INT4 with far less quality loss than pure PTQ on the same model. The cost is a training cost (GPU hours to days on an existing model), not an inference cost. For INT4 with QAT you recover almost all the quality; for 2-bit, QAT helps but the terrain is still difficult; for ternary, QAT stops being a “short fine-tune” and becomes native training (BitNet).

The decision hierarchy:

PTQ = default at ≥4 bits. Minutes to hours, without touching training weights. It covers 90 % of production.
QAT = when PTQ loses too much and the difference matters. Low bits (2-3), or sensitive models. You pay for a fine-tune.
Native (ternary) = when you want to go below 2 bits and change the arithmetic. You pay for pre-training. It only makes sense if you control the model from its creation.

The maths that matter: footprint and how many fit

The weight footprint is direct: bytes = (bits/param / 8) × N, with N the number of parameters. For an 8B model:

Level	bits/param	8B footprint	Ratio vs BF16
BF16	16	16.0 GB	1.0×
INT8	8	8.0 GB	2.0×
INT4	4	4.0 GB	4.0×
3-bit	3	3.0 GB	5.3×
2-bit	2	2.0 GB	8.0×
1.58-bit (ternary)	~1.58	~1.6 GB	~10×

(Real ternary takes up slightly more than 1.58 bits/param because you have to pack 5 ternary values into 8 bits, 5 × log₂(3) ≈ 7.92 bits, and because norms and embeddings usually stay at higher precision. The ~1.6 GB figure for 8B is the right order of magnitude.)

How many 8B models fit on an RTX 4090?

An RTX 4090 (24 GB, Ada Lovelace) has 24 GB. Reserving ~4 GB for KV cache and activations leaves 20 GB for weights. How many distinct 8B models fit loaded at the same time:

Level	8B footprint	Models in 20 GB	Comment
BF16	16.0 GB	1	one, with scant headroom left
INT8	8.0 GB	2	two different models
INT4	4.0 GB	5	the solved regime; quality ~lossless with AWQ
3-bit	3.0 GB	6	small degradation already visible
2-bit	2.0 GB	10	only viable with AQLM/QuIP#/QTIP
1.58-bit	~1.6 GB	~12	natively ternary models only

The arithmetic is seductive, from 1 to 12 models on the same card, but it should be read sceptically. Jumping from INT4 (5 models, almost lossless) to 2-bit (10 models) doubles the capacity, but only if you use a state-of-the-art method and accept 4-8 MMLU points. And the jump from 2-bit to ternary (10 → 12) is marginal in memory terms: ternary is not justified by footprint against a state-of-the-art 2-bit, it is justified by the arithmetic (the compute ceiling) and because being native it avoids the quality elbow. If your only metric is “how many GB it takes”, state-of-the-art 2-bit already gives you nearly everything. Ternary is for when you also want the compute saving.

The conceptual curve: perplexity vs bits

Three readings of the curve. One: to the right of 4 bits, the three branches are almost stuck together and almost flat, the solved regime. Two: the red branch (naive scalar PTQ) has a brutal elbow between 3 and 2 bits; that is where it doubles perplexity. The blue branch (state-of-the-art vector PTQ) flattens that elbow, it does not remove it, but it makes it tolerable down to 2 bits. Three: the green point of native ternary is on neither of the two PTQ curves, because it is not obtained by compressing: it is obtained by training, and that is why it can fall below the elbow without paying the quality price that any PTQ pays at that bit density. It is the difference between the crushed JPEG and the repainted photo.

Mandatory scepticism: “lossless” 1-bit and benchmarks without methodology

Three warnings for reading the aggressive-quantisation literature:

“1-bit lossless” almost always comes with small print. Pure binary {-1, +1} (1 bit) loses the ability to represent zero, which matters in transformers (many weights are effectively null). That is why the true state of the art at low density is ternary (1.58 bits), not binary: the zero is worth its extra 0.58 of a bit. When a paper announces “1-bit”, it pays to check whether (a) it really is 1 bit or 1.58 rounded down in the headline, (b) “lossless” is measured in WikiText perplexity (easy) or on reasoning benchmarks (where the collapse shows up), and (c) it compares against a baseline of the same effective size or against a much larger model in order to inflate the advantage.

Flat perplexity ≠ preserved quality. Perplexity on a generic corpus is the metric most forgiving of aggressive quantisation. A 2-bit model can have a perplexity almost identical to BF16 and at the same time drop 10 points on GSM8K or on a code benchmark, because multi-step reasoning amplifies errors that average perplexity does not see. Distrust any sub-4-bit claim that only reports perplexity. As we already said in the quantization post, the loss has to be measured on the target task.

Hardware comparability. The “X times faster” numbers for ternary only apply with the specialised kernels (bitnet.cpp) and on hardware where addition/subtraction arithmetic genuinely wins. On a GPU with tensor cores designed for FP16/FP8, a naive ternary kernel can be slower than a well-optimised INT4, because it wastes the silicon. The ternary advantage is real, but it is a co-design advantage (model + kernel + sometimes hardware), not a flag you switch on over your current stack. Any benchmark that does not specify the exact kernel and hardware is noise.

Implications for on-premise inference

On the RTX 4090 (24 GB, Ada Lovelace): the practical regime today is still INT4 AWQ for 7-14B models, solved, almost lossless, natively supported. State-of-the-art 2-bit (AQLM/QuIP#/QTIP) is viable and lets you load larger models or more models at once, but it demands the specific kernels of each method and an expensive calibration, and it pays in quality. It makes sense when VRAM is the bottleneck and you accept the trade-off; not as a default. Ternary on a 4090 is experimental: without tensor cores designed for ternary addition and subtraction, the compute advantage is diluted, although the memory saving remains.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8): here the default is FP8 (almost indistinguishable quality, native throughput) or INT4 AWQ for models that do not fit in FP8. State-of-the-art sub-4-bit is for serving enormous models (200B+) when neither FP8 nor INT4 fits with the KV cache headroom you want, at the cost of quality and kernel complexity. Native ternary, today, is a research object rather than a production one: its promise, hitting both roofline ceilings, is greater on CPU/edge (where there are no FP8 tensor cores to exploit) than on an H100 cluster, which already has dedicated FP8 hardware.

The rule of thumb, June 2026: ≥4-bit is settled engineering; state-of-the-art 2-bit is a real lever but with a cost in method and in quality; ternary is an architectural bet, not a deployment tweak.

References

Ma, S. et al. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (BitNet b1.58). https://arxiv.org/abs/2402.17764
Egiazarian, V., Panferov, A., Kuznedelev, D. et al. Extreme Compression of Large Language Models via Additive Quantization (AQLM). https://arxiv.org/abs/2401.06118
Tseng, A., Chee, J., Sun, Q., Kuleshov, V., De Sa, C. QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks. https://arxiv.org/abs/2402.04396
Tseng, A., Sun, Q., Hou, D., De Sa, C. QTIP: Quantization with Trellises and Incoherence Processing. https://arxiv.org/abs/2406.11235
Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. https://arxiv.org/abs/2210.17323
Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Han, S. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. https://arxiv.org/abs/2306.00978
Google DeepMind. Gemma 3 QAT (Quantization-Aware Training) models — official blog: https://developers.googleblog.com/en/gemma-3-quantized-aware-trained-state-of-the-art-ai-to-consumer-gpus/
Microsoft. bitnet.cpp — 1-bit ternary inference kernels: https://github.com/microsoft/BitNet

Test-time quantization: quantising on the fly with no calibration dataset

Tue, 09 Jun 2026 02:00:00 +0000

This post is the natural continuation of Quantization for LLM inference, which is worth reading first: that is where GPTQ, AWQ, scale + zero-point and why activation outliers are the central problem all live. Here we are not discussing how many bits to use, but when and with what information the scales are computed: offline against a corpus (PTQ) or on the fly against real traffic (TTQ).

TL;DR

Activation-aware quantisation (AWQ, SmoothQuant) decides which channels to protect by measuring the magnitude of the activations over a calibration dataset in an offline pass, before deployment. The implicit assumption is that this corpus represents future traffic. But activation outliers, the channels with 10-100× the median magnitude that dominate the quantisation error, depend on the input: they change with the domain, the language and the client’s distribution. When real traffic drifts away from the calibration, the fixed scales stop being optimal and quality drops. Test-time quantization (TTQ) removes the corpus and the offline pass: it derives the activation-aware scales at inference time, from the activations that are actually observed, per token or per batch. The trade-off is honest and not minor: it introduces runtime overhead, computing statistics, detecting outliers, recomputing scales at every step, which competes directly with the saving from quantising. On small models that overhead weighs proportionally more, because the forward is short and the fixed per-step costs dominate (the framing is in the inverted roofline for SLMs). TTQ is orthogonal to the format: it is not a competitor to INT4 or FP8, it is a different way of deriving s. It pays off when there is no calibration pipeline, when the traffic distribution is shifting or unknown, and in multitenant settings where no representative corpus exists.

You are here: DEPLOY

The analogy: the tailor who takes measurements versus off-the-peg sizes

A clothes shop has two ways of dressing a customer.

The first is to sell off-the-peg sizes. At some point the factory measured an “average customer”, a mean mannequin built from a population sample, and cut the garments to those measurements. When a customer walks in, you hand them the size that comes closest. It is extremely fast: the garment is already sewn, it just gets handed over. The problem appears when the customer does not look like the average mannequin: if their shoulders are much broader than average, their particular outlier, the standard size pulls or leaves spare fabric, because it was cut to protect other areas. This is offline calibrated PTQ: AWQ measured the importance of each channel over a corpus and fixed the scales once and for all; fast at inference time, but blind to the specific customer.

The second is the tailor who takes measurements on the spot. When the customer walks in, the tailor gets out the tape, measures that customer, spots where their particular bulk is and adjusts the cut to their real anatomy. The result fits better, especially for customers who fall outside the mould. But every customer costs time: measuring, marking, deciding. This is TTQ: the scales are derived on the fly from the activations that the input actually generates.

The analogy holds up in three details:

The average mannequin = the calibration dataset. If the population walking into the shop resembles the mannequin, the sizes work; if not, they fail at the extremes.
Measuring every customer = computing activation statistics per token/batch. A better fit, but a fixed cost paid on every garment.
The broad shoulders = the outlier activation channels. They are precisely the areas where the fit matters and where the generic size gets it most wrong.

The tailor wins when the customers are varied or unknown. The tailor loses when you have a homogeneous population and a mannequin that represents it well: there, paying for measurement on every customer is a waste of time.

The problem TTQ solves: fixed calibration ages with the traffic

Let us recall from the quantization post what AWQ and SmoothQuant do exactly. They do not quantise all channels equally: they identify the ~1 % of channels whose activations have a large magnitude, the salient channels, and protect them by scaling them before quantising. To measure that importance they need to see activations, and they see them over a calibration dataset (128-512 samples, typically WikiText or a slice of the domain) in an offline pass prior to deployment.

The assumption is strong: that the activation distribution of the calibration corpus represents that of production traffic. Two reasons why that assumption breaks:

Activation outliers depend on the input. They are not a fixed property of the model the way weights are. The channel that is an outlier when processing C++ code may not be one when processing conversational Arabic or log JSON. The magnitude and position of the peaks change with the domain, the language and the input format.
Real traffic is rarely the corpus. You calibrate with English WikiText and the client sends you support tickets in Spanish with tables pasted in. The calibration protected the channels that WikiText activated, not the ones that real traffic activates. The scales are suboptimal exactly where the client lives.

The result is distribution-dependent degradation: the quantised model keeps its quality while the input resembles the calibration and loses it as the input drifts away. The most awkward case is multitenant: if you serve clients with different domains from the same quantised model, no single representative corpus exists; any fixed calibration favours some tenants and penalises others.

The TTQ mechanism: measure the real activations and scale on the fly

TTQ (arXiv:2603.19296, March 2026) proposes deriving activation-aware quantisation at inference time, with no offline pass and no calibration dataset. The idea, in its bare conceptual form:

Step 1 — Observe. When the activation tensor X arrives at a linear layer (per token or per batch), cheap statistics are computed over the channels: a measure of central tendency (median or mean magnitude) and one of dispersion per channel. This is the equivalent of AWQ looking at its corpus, but done over the activations that really are coming in right now.

Step 2 — Detect outliers on the fly. With those statistics, the channels whose magnitude shoots up relative to the tensor median are identified; the typical criterion is a threshold of the form “magnitude > k × median”. These are the channels that, if quantised with the same scale as the rest, blow up the error.

Step 3 — Derive scales and segregate. For normal channels a scale is computed that exploits the range; for the outliers a different treatment is applied, a scale of their own, or keeping them at higher precision, in the style of on-the-fly mixed precision. It is the same philosophy as LLM.int8() (segregating outliers to FP16) or AWQ (scaling salient channels), but with the threshold and the scales recomputed over the current input, not frozen since calibration.

Step 4 — Quantise and multiply. With the fresh scales, the tensor is quantised and the GEMM runs. The activations entering the next layer compensate for the rescaling, just as in AWQ, so that the maths cancels out.

The key difference with AWQ is not in what is done (protecting activation outliers) but in when and against what: AWQ decides once, offline, against a corpus; TTQ decides at every step, on the fly, against real traffic. It is the transfer to inference of the “test-time” idea: adapting the computation to the specific sample in front of you instead of to a precomputed average.

The maths that matter

The error of quantising an outlier with the wrong scale

Let us recall affine uniform quantisation from the base post: an integer code q = round(x/s) - z with scale s and zero-point z, and reconstruction x̂ = s·(q + z). For a b-bit quantiser with a symmetric range, the scale that covers a tensor of maximum magnitude M is approximately s = M / (2^{b-1} - 1). The rounding error of each element is bounded by half a scale: |x - x̂| ≤ s/2.

Here is the outlier problem. The scale s is chosen to cover the largest value in the group. If one channel has 30× the median magnitude and you share a single scale with the rest of the tensor, that magnitude rules: M is the outlier, so s is inflated 30× relative to what the majority would need. The absolute rounding error of the normal values rises proportionally.

A concrete calculation. Take a group where the median magnitude is 1.0 and an outlier channel is 30.0, quantised to INT4 (b = 4, levels ±7):

With a shared scale, s = 30 / 7 ≈ 4.29. The rounding error of a typical value (magnitude ~1) is up to s/2 ≈ 2.14. In other words, the error on the normal values is of the order of their own value: the outlier has destroyed the resolution of everything else. Relative error of a value of magnitude 1: up to ~214 %.
Segregating the outlier (pulling it out to FP16 or giving it its own scale) and quantising the rest with M = 1, s = 1/7 ≈ 0.143. The error of a typical value drops to s/2 ≈ 0.071, ~7 % relative. Thirty times less error on the majority of the group’s weights.

That is the entire reason for the existence of activation-aware quantisation: to detect and treat separately the ~1 % of channels that, if not segregated, hijack the scale. AWQ does it against the corpus; TTQ does it against the real input. And if the channel that is an outlier in production was not an outlier during calibration, AWQ did not protect it: it quantised real traffic with the inflated scale from the case above. That is where TTQ gains precision.

The overhead: the cost of measuring at every step

The price is symmetric. Computing the per-token statistics, per-channel magnitudes, median or percentile, outlier threshold, scales, means reductions over the activation tensor that did not exist in the forward pass with frozen scales. Let us call:

T = forward time per token with fixed scales (static PTQ), in µs.
Δ = extra per-token cost of deriving the statistics and scales on the fly, in µs.

The relative overhead is simply:

$$\text{overhead} = \frac{\Delta}{T}$$

The key point is that Δ is relatively fixed per step (it depends on the number of channels and layers, not on how much “useful” work the model does), while T scales with the size of the model. That is why the ratio behaves very differently depending on the model:

Large model (say, 70B): T is large, every forward moves tens of GB of weights from HBM. If Δ ≈ 8 µs and T ≈ 800 µs, the overhead is 8/800 = 1 %. Negligible against the saving from quantising.
SLM (say, 1B): T is small, the forward per token is short. With the same Δ ≈ 8 µs and T ≈ 60 µs, the overhead is 8/60 ≈ 13 %. No longer negligible: it eats a good part of what you gained by quantising.

This connects directly to the inverted roofline for small models: on an SLM the fixed per-step costs (kernel launches, synchronisations, overheads that do not scale with the model) weigh proportionally more, because there is less useful work to spread them over. TTQ’s Δ is exactly one of those fixed costs. Going per-batch instead of per-token amortises Δ across all the tokens of the batch and lowers the relative overhead, at the cost of coarser scales; it is the first parameter to touch.

The uncomfortable conclusion: TTQ gives away robustness to distribution but spends part of the acceleration budget on measuring, and in the regime where acceleration is scarcest, the SLMs, the ones most often deployed at the edge, is where that spending hurts most. It is not free; it is a change of currency.

A note of methodological scepticism: arXiv:2603.19296 is from March 2026, very recent, and as of the date of this post there are no broad independent reproductions. Whatever speedup and quality figures circulate should be taken with the same caution as any number without published methodology: what hardware, what batch size, what real measured Δ, against what baseline (well-calibrated or badly calibrated PTQ), in what domain? The conceptual argument, robustness to distribution in exchange for per-step overhead, is solid; the concrete multipliers are pending validation.

What TTQ is NOT: marking it off from the rest of the zoo

TTQ is easily confused with neighbouring techniques. The distinction that matters is that TTQ is the how of deriving the scales, not the format nor the moment in training.

Technique	When the scales are fixed	Needs a calibration corpus	Touches training	Is it a format
Static PTQ (GPTQ, AWQ)	Offline, before deployment	Yes	No	No (it uses INT4/INT8)
QAT	During training	No (training data)	Yes (retrains)	No
FP8 end to end	At runtime, but simple per-tensor scales	Minimal / none	No	Yes (E4M3/E5M2)
TTQ	At runtime, activation-aware per token/batch	No	No	No (orthogonal to the format)

The four distinctions, one by one:

Against static PTQ (GPTQ/AWQ). Same goal (protecting outliers), the same possible format (INT4), but PTQ freezes the decisions offline against a corpus and TTQ recomputes them on the fly. TTQ is, in a sense, “AWQ without the calibration phase, paid for at runtime”.
Against QAT. QAT puts quantisation inside the training loop so that the model learns to be robust to it; it costs a retrain. TTQ does not touch training: it operates on an already trained model, at inference. They are attacks at opposite moments of the pipeline.
Against FP8 end to end. FP8 is a format with its own logarithmic range; its “dynamic scaling” computes a simple per-tensor scalar at runtime, but it does not do activation-aware per-channel outlier detection. TTQ could, conceptually, derive scales on the fly for an FP8 or INT4 quantiser: it is orthogonal to the format.
TTQ is orthogonal to the format. It decides how to obtain s, not how many bits you store q in. You can imagine “TTQ over INT4” or “TTQ over FP8”. What defines TTQ is the source of the scale, real activations on the fly, not the width of the code.

When it pays off (and when it does not)

TTQ is not a universal replacement for AWQ. It is a tool for a specific deployment profile. It pays off when:

You have no calibration pipeline. You want to deploy a quantised model now, without assembling the calibration dataset, running the offline pass or validating that the corpus represents the traffic. TTQ cuts out that whole phase: you load the model and serve.
The traffic distribution is shifting or unknown. An assistant that one day receives code and the next legal contracts in another language. No fixed calibration covers both well; on-the-fly adaptation follows the distribution without recalibrating.
Multitenant with no representative corpus. You serve the same model to clients with disparate domains. No single corpus represents all of them; any fixed calibration creates winners and losers among tenants. TTQ adjusts to each input, whichever tenant it comes from.

It does not pay off when:

You have a stable domain and a good calibration corpus. If your traffic is homogeneous and representative, offline AWQ gives you the same quality with zero runtime overhead. Paying Δ on every token to relearn what a corpus already captured is waste.
You serve SLMs with a tight latency SLA. This is exactly the case where Δ/T is high. If the model is small and TPOT matters, the overhead of measuring can wipe out the gain from quantising. Measure your real Δ before assuming it adds up.
The batch is large and compute-bound. With high concurrency the forward is no longer memory-bound and the cost of the extra reductions competes worse; at the very least, amortise Δ per batch.

Implications for on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace)

The natural case for the 4090 is the SLM, Qwen 3 1.5B, Llama 3 8B AWQ-INT4, serving at low concurrency. That is precisely the regime where TTQ is riskiest: T per token is small and the 4090 has no accelerated native FP8 (we discussed this in the quantization post), so TTQ’s extra reductions run on CUDA cores competing for the same time. Here the question is not “does it improve quality?” but “does the overhead leave me an acceptable TPOT?”. If the traffic is homogeneous, offline AWQ wins on simplicity and latency. TTQ only justifies its Δ if the input distribution is genuinely unpredictable and the degradation from fixed calibration is measurable.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here the calculation partly flips. With large models T is high and Δ/T drops to the low single-digit percentage range, so TTQ’s overhead is more digestible. The strong use case is multitenant: a cluster serving a large model to clients with heterogeneous domains, where no calibration corpus satisfies everyone. There, TTQ’s robustness to distribution has real value and the overhead is diluted in a big forward. Even so, on an H100 with native FP8, the baseline to beat is demanding: static FP8 barely loses quality (see the table in the quantization post) and costs nothing at runtime. TTQ has to show that its robustness gain on the outlier tenants outweighs what it gives away in overhead. With a paper from March 2026 and no reproductions, that demonstration is pending.

What we have not covered

The memory cost of the on-the-fly statistics: per-channel buffers, their impact on the footprint and on cache pressure.
Interaction with continuous batching: how scales are derived when a batch mixes requests from different domains in the same step.
TTQ + speculative decoding: whether the draft and the target derive scales on the fly separately, and how that affects the acceptance rate.
Numerical stability: what happens when a batch contains a one-off extreme outlier that inflates the scale for every token of that step.

References

TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly (March 2026). https://arxiv.org/abs/2603.19296
Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Han, S. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (MLSys 2024). https://arxiv.org/abs/2306.00978
Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (ICLR 2023). https://arxiv.org/abs/2210.17323
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., Han, S. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (ICML 2023). https://arxiv.org/abs/2211.10438
Dettmers, T., Lewis, M., Belkada, Y., Zettlemoyer, L. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (NeurIPS 2022). https://arxiv.org/abs/2208.07339

Device-native architectures: fine-grained MoE and pre-attention router

Tue, 09 Jun 2026 01:50:00 +0000

This post belongs to the series on inference performance in small models. It is the architectural face of a problem we have already looked at from the compute-regime side (the SLM’s inverted roofline) and from the weight-loading side in From disk to HBM. Here the question is different: what if, instead of adapting a large model to the device, we design the model for the device from the first commit?

TL;DR

The default move to get an LLM onto a laptop, a phone or an edge box is to take a dense model built for cloud and compress it: distillation, pruning, quantisation. It is a gesture of reduction: you start from something big and take things away. SmallThinker (arXiv:2507.20984, SJTU IPADS + Zenergize AI) argues for the reverse gesture, designing from scratch, and articulates it in three pieces. First: fine-grained MoE, many small experts with very few activated per token, so that total parameters N (the capacity) decouple from activated parameters A (the compute cost per token). Second: sparse FFN, ReLU-style activation sparsity inside each block, which adds a second level of sparsity on top of the first. Third: a pre-attention router that predicts which experts will be needed before running the attention block and launches the prefetch of those weights from SSD/flash in parallel with attention compute, hiding the storage latency, which is the real bottleneck when the model does not fit entirely in RAM. The authors report SmallThinker-4B-A0.6B and SmallThinker-21B-A3B exceeding ~20 tok/s on consumer CPUs with Q4_0, using ~1 GB and ~8 GB of RAM. The numbers are interesting and the direction is right; the evaluation methodology and the quality cost of activating so little deserve scepticism, and that is what the last part is for.

The analogy: the librarian who anticipates your request

Picture a huge library with a small reading room. You are sitting in the room at a single desk: only a few books fit on it at a time (that is RAM). The bulk of the collection is in the back room, on long shelves that are slow to walk (that is the SSD/flash). And there is a librarian.

The naive method: you read, you reach a point where you need a specific book, you request it, and then the librarian gets up, walks to the back room, looks for it and comes back. Meanwhile, you wait with the page open, making no progress. Every time you need a new book, you pay the full trip to the back room. The reading room spends most of its time waiting, not reading.

The SmallThinker method: the librarian is clever and gets ahead of you. While you are still reading the chapter’s table of contents, working out what it is about and connecting ideas, which in the model is the attention block, they have already glanced over your shoulder, predicted which three or four books you are going to ask for and gone to the back room to fetch them. By the time you finish the contents and formulate the request, the books are already on your desk. You have not waited: the trip to the back room happened in parallel with your reading of the contents.

The analogy holds on four details:

The small desk is RAM; the slow back room is the SSD/flash.
The books are the MoE’s experts: only a few are on the desk at any moment.
Reading the contents is the attention block; requesting and using the books is the FFN/expert block.
The librarian who predicts and gets ahead is the pre-attention router: the prediction is made first, and the trip to fetch (the prefetch) overlaps with reading the contents (attention).

The quantitative question running through the whole post is: does the librarian get back in time? The wait is only hidden if the trip to the back room takes less than your reading of the contents. That is the condition t_{\text{attention}} \ge t_{\text{prefetch}}, and we will do it with numbers.

Compressing a dense model vs. designing for device

It is worth laying out the two approaches plainly, because they are not degrees of the same thing: they are different philosophies.

Approach A — compress a dense model built for cloud. You start from, say, a dense 7B–14B model trained to run on an RTX 4090 (24 GB, Ada Lovelace) or on a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). To fit it into a device you apply three levers, each with its own post: distillation (you train a small student that imitates the teacher), pruning (you remove weights or whole structures) and aggressive quantisation (you go down to 4 bits or fewer). The resulting model is still dense: all its parameters are activated on every token. You have reduced the number of parameters, but the compute pattern is the cloud’s, only smaller.

Approach B — design for device from scratch. Here the device’s constraints enter the architecture, not a later compression phase. The constraints are three and very concrete:

Weak compute. A laptop CPU or a mobile SoC does orders of magnitude fewer FLOPs than a datacenter GPU. This pushes you to minimise the activated parameters per token, not the total ones.
Little RAM. Tens of GB do not fit. This pushes you to keep resident only what is essential and to stream the rest.
Slow storage. The SSD or flash you are forced to stream from has far less bandwidth than a GPU’s HBM. This turns storage I/O into the bottleneck, and pushes you to hide it.

SmallThinker is approach B taken to the detail: each of those three constraints has an architectural answer. Weak compute is attacked with fine-grained MoE + sparse FFN (minimise A). Scarce RAM is attacked with streaming from SSD (resident ≈ A + cache, not N). Slow storage is attacked with the pre-attention router (hide the I/O behind attention). It is no accident that the three pieces fit together: each one solves a constraint, and together they reinforce each other.

An important nuance, so as not to fall into the hype: approach B is neither free nor universally superior. It requires training a new model (you do not reuse existing weights), and the quality ceiling of a model with a very small A is intrinsically bounded, as we will see. The argument is not “B always wins”, but “for the device regime, B attacks the right bottlenecks, and A only attacks them in passing”.

Two levels of sparsity

The central capacity idea is old and well understood in MoE: separate capacity from compute cost. In a MoE, the model has N total parameters spread across experts, but for each token only A parameters are activated (those of the top-k experts the router picks). Compute cost per token scales with A; knowledge capacity scales with N. SmallThinker applies this idea at two superimposed levels.

Level 1 — fine-grained MoE. “Fine-grained” means many small experts instead of few large ones, with very few activated per token. Instead of, say, 8 experts of which you activate 2, you have dozens of experts of which you activate a handful. With smaller experts, the same A is spread across more possible combinations, which gives the router fine granularity and keeps A very low relative to N. The result is an aggressive N/A ratio: a lot of capacity, very little compute per token.

Level 2 — sparse FFN (ReLU-style activation sparsity). This level is orthogonal and operates inside each FFN. With a ReLU-style non-linearity, a large fraction of the intermediate layer’s neurons produce exactly zero for a given token. A neuron that comes out at zero contributes nothing to the output: its matrix-vector multiplication can be skipped. This is activation sparsity: predictable token by token, and exploitable so as not to load or multiply the weight rows/columns corresponding to inactive neurons. It is the same phenomenon exploited by work such as Deja Vu or PowerInfer; SmallThinker builds it in from the factory by choosing activations that favour it.

The combined effect, in one sentence: large N (capacity), tiny A (compute cost per token ≈ proportional to A), and on top of that, inside that A, a fraction of the multiplications is saved through activation sparsity. It is sparsity on top of sparsity.

The pre-attention router: predict and prefetch

Here is the paper’s specific piece, and the one that gives the post its name. The problem it solves is one of I/O scheduling, not of quality.

When the model does not fit entirely in RAM, the expert weights live in SSD/flash and are loaded on demand. The naive flow of a MoE layer is sequential: you run attention, then the router decides which experts are needed, then you load those experts from SSD (waiting), then you run the FFN of those experts. The loading step is pure waiting: the CPU is blocked waiting for bytes from the SSD. In the device regime, where the SSD is slow, that waiting time dominates the decode step.

The pre-attention router breaks the sequentiality by inverting the order of the decision. The observation is that the router does not need this same layer’s attention output to make a reasonable prediction of which experts will be needed: it can predict it from the state it already has before running attention. So:

Before running the layer’s attention block, the router predicts the experts that will be needed.
It launches the prefetch of those experts from SSD/flash asynchronously.
In parallel, the CPU runs the attention block, which is pure compute and does not need the SSD.
When attention finishes, the prefetched experts are (ideally) already in RAM, and the FFN proceeds without waiting.

Storage I/O has been overlapped with attention compute. It is exactly the librarian who goes to the back room while you read the contents.

The hiding condition is the inequality above: the prefetch is completely hidden if and only if

$$t_{\text{attention}} \;\ge\; t_{\text{prefetch}}.$$

If attention takes longer than loading the experts, the load is free (it was already done). If the experts are too large or the SSD too slow, t_prefetch > t_att and a wait bubble equal to t_prefetch − t_att appears. That is why the design needs A to be small (small experts → fewer bytes to prefetch → low t_prefetch) and the grain to be fine: the two things level 1 of sparsity does are not only about saving FLOPs, they are about making the prefetch fit underneath attention.

The maths that matter

Memory footprint: resident N vs. A + cache

The parameter that decides whether the model fits is how much you have to keep resident in RAM at once.

Everything in RAM. If you require all the experts to be loaded, the footprint is \approx N (all parameters, multiplied by bytes/parameter according to the quantisation). For a 21B this is prohibitive on a device.
Streaming from SSD. If you only keep the active experts resident plus a cache of the recent/likely ones, the footprint drops to \approx A + \text{cache}. The weights that are not in RAM live on SSD and get prefetched when needed. This is where the real saving is: the resident set scales with A, not with N.

The non-expert part of the model (embeddings, attention, router, layernorms) is always resident, but in a fine-grained MoE the bulk of N is in the experts, so the approximation resident ≈ A + cache + dense_part is a good one.

The prefetch calculation, with numbers

Let us put numbers on the analogy. Suppose a consumer SSD at 5 GB/s of sequential read and a quantised expert of size X MB. The time to load one expert is

$$t_{\text{1 expert}} = \frac{X \text{ MB}}{5000 \text{ MB/s}} = \frac{X}{5000}\ \text{s} = \frac{X}{5}\ \text{ms}.$$

Let us make X concrete. In SmallThinker-4B-A0.6B with Q4_0 (~0.5 byte/param effective counting block overhead), a small expert of, say, 4M parameters weighs \approx 4\text{M} \times 0.5 = 2 MB. Loading it costs t_{\text{1 expert}} = 2/5 = 0.4 ms.

Now the scheduling question: if the layer’s attention block takes Y ms, how many experts can I prefetch while attention runs? The number is

$$n_{\text{prefetch}} = \left\lfloor \frac{Y}{t_{\text{1 expert}}} \right\rfloor = \left\lfloor \frac{Y \cdot 5}{X} \right\rfloor.$$

With Y = 2 ms of attention and X = 2 MB per expert: n_{\text{prefetch}} = \lfloor 2 \times 5 / 2 \rfloor = 5 experts. That is, within that layer’s attention window the SSD manages to bring in 5 experts. If the layer’s top-k activates ≤ 5 experts, the prefetch hides them all and t_prefetch ≤ t_att: zero load latency. If the layer needed 8 experts, you would bring 5 for free and pay the load of the remaining 3 as a bubble: (8-5) \times 0.4 = 1.2 ms of waiting per layer. Hence why the design wants fine grain with a small top-k: to fit underneath the attention window.

Two critical observations about this calculation:

The 5 GB/s is idealised sequential read. The experts are scattered across the disk; random 4K reads on a consumer SSD are far slower. The effective bandwidth can be a fraction of the nominal one, which reduces n_{\text{prefetch}}. Any methodology reporting tok/s ought to say whether it measures with experts pre-sorted on disk or with realistic access.
The attention window Y shrinks with short context and at the start of generation. With short prompts, attention is cheap and may not cover the prefetch; the advantage of the overlap grows with longer sequences. Another detail an honest benchmark should break down.

Weight footprint: why they report ~1 GB for a 4B

Let us do the 4B calculation in Q4_0. Quantisation to 4 bits ≈ 0.5 byte/param, plus a small overhead of per-block scales (Q4_0 adds one FP16 scale every 32 weights, ~0.56 effective byte/param). So:

$$4\text{B} \times 0.5\ \text{B/param} \approx 2\ \text{GB}.$$

That is, the full model in Q4_0 takes up ~2 GB on disk. But the authors report ~1 GB of RAM. A contradiction? No, and understanding why is understanding the design:

Not all experts are resident. Only the activated ones (A = 0.6B) plus a cache fit in RAM; the rest lives on SSD and is streamed. 0.6\text{B} \times 0.5 \approx 0.3 GB of active experts, plus the dense part (attention, embeddings, router) and a cache of hot experts.
Sparse FFN reduces the work and the useful resident set. The neurons that come out at zero do not need to be materialised for that token.

Adding up active experts + dense part + a reasonable cache, ~1 GB is plausible. But mind the nuance: ~1 GB is the resident set in RAM, not the total footprint in storage, which is still ~2 GB on SSD. Conflating the two, reporting a bare “1 GB”, is misleading if the reader understands “the model takes up 1 GB”. It takes up 2 GB; it keeps 1 GB in RAM. The distinction matters for a device with 2 GB of free storage: there it does not fit.

Analogously, SmallThinker-21B-A3B: 21\text{B} \times 0.5 \approx 10.5 GB on disk; 3\text{B} \times 0.5 \approx 1.5 GB of active experts, and the ~8 GB of RAM reported includes active experts + a generous cache + the dense part. The large cache is what takes it from 1.5 to ~8 GB: you keep many hot experts resident so as not to hit the SSD constantly.

The quality cost: the necessary scepticism

All the machinery above reduces the compute per token to \approx A. But A = 0.6B activated is very little. This is where the enthusiasm needs a brake:

Bounded reasoning capacity. A model that activates 0.6B parameters per token has, per token, the compute power of a 0.6B model, not of a 4B one. The total capacity N=4B helps to store more knowledge (more specialised experts), but the processing of each token is still limited by A. For tasks requiring composition and intensive multi-step reasoning, this is a real ceiling, not a detail.
The router is a single point of failure for quality. If the fine-grained router picks the wrong experts, and with fine grain there are more decisions to take, quality drops without any speed metric reflecting it. The pre-attention router makes this worse: it predicts the experts before seeing attention, with less information than a post-attention router. The authors ought to report how much quality is lost by predicting early (mismatch between the prefetched expert and the expert a post-attention router would have chosen).
The ~20 tok/s needs small print. On which CPU exactly? With what context and generation length (the advantage of the overlap depends on Y)? Cold start included or steady state? Did the SSD have the experts pre-sorted sequentially? An “exceeds 20 tok/s” without those conditions is a marketing number, not a methodological one.
A fair comparison. The right question is not “is it fast?”, but “at equal quality on an independent benchmark, is it faster or smaller than an equivalent compressed dense model?”. That requires evals the reader can reproduce, not just tok/s on the authors’ machine.

None of this invalidates the direction. Designing for device is, conceptually, the right approach: it attacks the real bottlenecks (compute, RAM, I/O) in the architecture instead of patching them afterwards. But “20 tok/s in ~1 GB” is a claim about efficiency, and efficiency only means something anchored to a level of quality measured honestly. Until that anchor is clear, the right amount of scepticism is high.

Implications for on-premise and edge inference

The SSD becomes part of the inference hierarchy. In cloud, the hierarchy is HBM → RAM. On device, the SSD/flash enters as one more level, and its bandwidth and random-access latency become first-order performance parameters. This connects with From disk to HBM: cold start and weight streaming stop being just a startup problem and become part of the steady state.
The heterogeneous edge box starts to make sense. In a mixed environments pattern, a device-native model like SmallThinker runs on the NUC/edge with CPU and SSD, serving locally, while the heavy work stays on the central cluster. The pre-attention router is what makes a GPU-less edge box viable.
Capacity planning changes axes. As discussed in Inference capacity planning, on device the resource to plan is not VRAM but the triad resident-RAM / SSD-bandwidth / CPU-FLOPs. A model with small A and overlapped prefetch moves the bottleneck from “does it fit in RAM?” to “does the SSD feed the prefetch in time?”.

Conclusion

SmallThinker is, above all, a change of question. Not “how do I shrink this cloud model so it fits on the device?” but “what would the model look like if I designed it for the device from the first parameter?”. The answer, fine-grained MoE to decouple N from A, sparse FFN to save inside A, and a pre-attention router that hides storage I/O underneath attention, attacks the device’s three constraints (compute, RAM, I/O) in the architecture, not in a later compression phase. The key condition, t_att ≥ t_prefetch, explains why the pieces fit: fine grain does not only save FLOPs, it makes the prefetch fit underneath attention. The reported numbers (~20 tok/s, ~1 GB / ~8 GB of RAM) are promising and the direction is solid; the cost of activating so little and the lack of methodological detail on quality call for caution. Designing for device is the right bet; measuring it honestly is the unfinished business.

References

Equipo SmallThinker (SJTU IPADS + Zenergize AI). SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment. arXiv:2507.20984. https://arxiv.org/abs/2507.20984
Repositorio oficial SmallThinker: https://github.com/SJTU-IPADS/SmallThinker
Self-Speculative Decoding for On-device MoE Acceleration. ACM The Web Conference (WWW) 2026. doi:10.1145/3774904.3792218. https://doi.org/10.1145/3774904.3792218
Liu, Z. et al. Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time. ICML 2023. https://arxiv.org/abs/2310.17157
Song, Y. et al. PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU (sparse activation + hot/cold experts). SJTU IPADS, 2023. https://arxiv.org/abs/2312.12456

Self-speculative decoding: the model that gets ahead of itself

Tue, 09 Jun 2026 01:40:00 +0000

This post is the direct companion to Speculative decoding: the secretary who types ahead. There, draft and target are two different models; here they are the same model at two depths. Read that one first: we take rejection sampling, the 1/(1-α) ceiling and the speedup formula as known, and all we change here is what the draft is.

TL;DR

Classic speculative decoding demands a pair: a cheap draft model proposes γ tokens and an expensive target verifies them in a single parallel forward pass. On large models the draft can be 1 % of the target and fit comfortably. On small models (SLMs, 1B–8B) that recipe breaks on two fronts: a draft that is 1/10 of a 3B is a 0.3B that barely gets anything right (α collapses), and loading a second model, however small, doubles the pieces to maintain and eats VRAM that is not spare on a 4090 or on device. Self-speculative decoding solves both: the draft is the model itself, run shallowly. A model with L layers produces draft tokens by exiting at an intermediate layer k < L (early-exit) or by skipping a subset of layers (layer-skip), and then verifies those tokens with the full forward pass through the L layers. Because draft and verify share weights and share the KV cache of the common layers, the extra memory cost is zero: there is no second model, no second KV cache, nothing new to load. The price is that the early-exit draft is more expensive than a tiny external draft (it walks k/L of the model instead of 1 %), so the relative cost c goes up. The honest trade-off: with a well-trained dedicated draft (EAGLE-3) that fits in memory, its α is usually higher and it wins; self-spec wins when there is no trained draft, when it does not fit, or when you are on device.

The analogy: the chess player who plays by eye and then calculates

A good chess player does two things with the same brain. First they look at the board and, by eye, in half a second, propose a move that “looks right”: pattern intuition, fast recognition, the shallow layers of judgement. Then, before moving, they calculate properly: three moves ahead, the opponent’s replies, the tactical lines. That deep calculation confirms the intuition or corrects it.

The decisive part is that it is the same person acting as drafter and as reviewer. They do not hire a second, weaker player to guess the move so they can validate it; that would be classic speculative with an external draft. Here the fast draft and the slow verification come out of the same brain, traversed at two depths.

The analogy holds point by point:

The glance by eye is the early-exit forward pass: the model walks only the first k layers and emits a draft token. Fast, approximate.
The deep calculation is the full forward pass through the L layers, which verifies the draft with exact rejection sampling.
That it is the same person is the reuse of weights and KV cache: the k shallow layers of the draft are literally the same as the first k layers of the verify; what has already been computed is not recomputed.
That the final move is identical to the one the player would have chosen by always calculating deeply is the rejection sampling guarantee: output quality does not degrade (the proof is in the speculative post).

Why the external draft does not fit small models

Let us revisit the cost of classic speculative with two numbers. The speedup depends on the acceptance rate α (how often the draft is right) and on the relative cost c = T_draft / T_target. A useful draft needs high α and low c at the same time. On large models that is reachable: a 1B draft for a 70B target has c ≈ 0.015 and, if well distilled (EAGLE), α > 0.8. The product pays off.

On a small model the balance breaks:

The proportional draft is useless. If you want c ≈ 0.1 for a 3B target, your draft is around 0.3B. A generic 0.3B has a distribution so different from the 3B that α falls into the 0.3–0.5 range. And 1/(1-α) with α = 0.4 is a ceiling of 1.67 tokens/step: not even infinite γ gets you more. The prize evaporates.
Loading a second model doubles the pieces. Even if the draft is small in VRAM, it is another checkpoint to version, quantise, validate and serve, and it has its own KV cache. On an RTX 4090 (24 GB, Ada Lovelace) with a quantised 8B and a long context, the KV cache is already tight; adding a second model and its cache can force you to lower concurrency or maximum context. On device (a phone, a NUC, an edge box) there is simply no room.
A trained draft does not always exist for your exotic or fine-tuned model. EAGLE needs the draft trained on-policy against that specific target (see knowledge distillation). If your SLM is your own fine-tune, there is no official published draft.

Self-speculative attacks all three at once with one idea: do not bring a second model; use the first one at half depth.

The mechanism: early-exit as draft, full forward as verify

A transformer with L layers transforms the hidden state layer by layer at each position: h_0 → h_1 → ... → h_L, and the LM head projects h_L to logits. The observation that enables everything: h_k for k < L is already a reasonable hidden state. If you push it through the same LM head (or through a dedicated lightweight head), you get an output distribution that is “premature” but often correct for easy tokens. That is the source of the draft.

The self-speculative iteration has the same structure as classic speculative (draft, verify, accept/reject) but both roles are the same model:

Step 1 — Shallow draft. To produce γ draft tokens, the model walks only the first k layers (or a subset of layers in the layer-skip case) and applies the LM head. Each draft token costs ≈ k/L of a full forward pass. We call c = k/L the relative cost of the draft. The γ drafts are generated autoregressively at this reduced cost.

Step 2 — Full verify. The model runs a single forward pass through the L layers over prompt + x_1...x_γ. Because attention is causal it obtains p(·|prompt, x_<i) for each position, exactly as in classic speculative.

Step 3 — Accept/reject. Rejection sampling identical to the previous post: tokens are accepted from left to right, the first divergence is corrected by sampling from the residual norm(max(0, p−q)), and if all γ are accepted the bonus token is appended. Output quality is exactly that of the full model.

The trick that makes `c` cheaper still: reusing the KV cache of shared layers

Here is the key difference against an external draft. When the model drafts by walking layers 0..k, it computes and stores the KV cache of those k layers for the prompt tokens and the drafts. When the full verify comes along, layers 0..k of the L-layer forward pass are bit for bit the same operations on the same weights that the draft already performed. There is no need to recompute them: the verify reuses the KV cache the draft left for layers 0..k directly, and only genuinely computes the missing layers k..L.

That has two consequences:

Zero extra memory. There is no second KV cache. The KV of the common layers is a single one, shared between draft and verify. Contrast that with vanilla SD, where the draft has its own complete cache (see KV cache).
Partially reused compute. The verify only pays for the “new” layers k..L for tokens that already went through the draft. The full forward pass is not as expensive as L suggests, because the first k layers come from the cache.

The families (state of play 2026)

There is no single way of doing self-speculative. They differ in which layers are skipped and in whether training is needed.

Family	Year / venue	How it picks what to skip	Training?	Distinctive idea
LayerSkip (Elhoushi et al.)	2024, arXiv:2404.16710	Early-exit at a fixed layer `k`; a single LM head serves every exit	Yes — layer dropout + early-exit loss in train/fine-tune	A single model trained to do both draft and verify; reuses partial compute
SWIFT	ICLR 2025 (OpenReview EKJhH5D5wA)	Selects which layers to skip on the fly, without touching weights	No — plug-and-play on the given model	Training-free self-spec: optimises the set of skipped layers at runtime
CLaSp	2025, arXiv:2505.24196	Dynamic in-context layer skip: the pattern of skipped layers adapts to the context	No (dynamic at inference)	The skip is not fixed; it changes with what is being generated
ConfLayers	2026, arXiv:2604.14612	Skips layers according to the confidence of the intermediate state (adaptive per token)	No (confidence criterion)	Variable depth: easy tokens exit earlier, hard ones go deeper
Saguaro	2025–26	Asynchronous formulation: the draft keeps speculating in parallel while verification runs	Depends on the variant	Overlaps draft and verify in time instead of alternating them
SSD for on-device MoE	ACM Web Conf. 2026, doi 10.1145/3774904.3792218	Self-spec exploiting MoE sparsity (few active experts per token)	MoE-specific variant	The shallow draft activates even fewer experts; fits MoE on device

Three operational readings of the table:

The axis that matters most is training yes/no. LayerSkip gives the best α because the model learns to be a good shallow draft (with early-exit loss the intermediate layers are explicitly trained to predict well). But it demands a fine-tune. SWIFT, CLaSp and ConfLayers are training-free: worse α, but they apply to any already-trained model without touching anything. For an SLM you do not control, training-free is the realistic option.
Adaptive skip (CLaSp, ConfLayers) raises α because it fits the draft’s depth to the token: it spends little on the easy ones and more on the hard ones, instead of a fixed k. In exchange, the effective c stops being constant.
Saguaro attacks something else: it does not raise α, it overlaps draft and verify time. It is orthogonal to the rest and can be combined.

The maths: same framework, different `c`

We reuse the apparatus of the speculative post without changing a letter. With α the acceptance rate and γ the number of drafts:

$$E[\text{tokens per step}] = \frac{1 - \alpha^{\gamma+1}}{1 - \alpha}, \qquad \text{Speedup} = \frac{1 - \alpha^{\gamma+1}}{(1 - \alpha)(\gamma c + 1)}$$

And the algorithmic ceiling is the same: lim_{γ→∞} = 1/(1-α). The only thing that changes in self-speculative is the value of c: it is no longer the size ratio of two models, but c = k/L, the fraction of layers the early-exit draft walks.

Numerical example: self-spec with L=32, exit at k=8

Take an SLM with L = 32 layers that exits at k = 8 for the draft: c = k/L = 8/32 = 0.25. Suppose α = 0.7 (reasonable for early-exit on conversational tokens) and γ = 4.

Expected tokens per step: (1 − 0.7⁵) / (1 − 0.7) = (1 − 0.168) / 0.3 = 0.832 / 0.3 = 2.77
Speedup: 2.77 / (4 × 0.25 + 1) = 2.77 / 2.0 = 1.39×

The denominator factor is γc + 1 = 4·0.25 + 1 = 2.0: the early-exit draft, costing a quarter of the model per token, eats part of the benefit. Exiting higher up helps: with k = 4 (c = 0.125), the denominator is 1.5 and the speedup 2.77/1.5 = 1.85×. But exiting higher up normally lowers α, so there is real tension between a small k (cheap) and a high α (accurate).

An honest comparison with an external draft

Put a tiny, well-distilled external draft alongside: c = 0.1 and α = 0.78 (what an EAGLE-style draft can deliver), same γ = 4.

Tokens/step: (1 − 0.78⁵)/(1 − 0.78) = (1 − 0.289)/0.22 = 0.711/0.22 = 3.23
Speedup: 3.23 / (4 × 0.1 + 1) = 3.23 / 1.4 = 2.31×

Configuration	c	α	tokens/step	speedup	extra VRAM	pieces to maintain
Self-spec early-exit (k=8)	0.25	0.70	2.77	1.39×	0	0
Self-spec early-exit (k=4)	0.125	0.65	2.50	1.67×	0	0
Distilled external draft	0.10	0.78	3.23	2.31×	yes (+model +KV)	1 extra model

The reading is exactly what you would expect, and it is worth not dressing it up: if you have a dedicated draft, trained against your target, and it fits in memory, its higher α and lower c give it more speedup. EAGLE-3 with a well-trained draft usually wins on raw speedup. Self-spec does not compete on raw speedup; it competes on total cost. Its winning columns are the two on the right: zero extra VRAM and zero pieces to maintain. Self-spec wins when:

there is no trained draft for your model (your own SLM, an unusual fine-tune),
the draft does not fit (a 4090 already full, a long context that needs the KV),
you are on device (phone, NUC, edge), where a second model and its KV simply do not go in.

It is the same pattern as with MTP in the previous post: sometimes the best draft is the one you do not have to load.

Why it fits small models and device precisely

The regime where self-spec shines is low concurrency, memory-bandwidth-bound, with a tight memory budget, exactly that of an SLM on a single GPU or on device (the reason for the regime is in the inverted roofline). Three reasons:

Zero extra memory is decisive where there is none to spare. On an RTX 4090 (24 GB, Ada Lovelace) serving a quantised 7B–8B with a long context, every GB counts. Self-spec asks for none: it reuses weights and KV. An external draft, however small, forces you to trim context or concurrency. On device the difference is binary: with self-spec you accelerate; with an external draft there is no room, full stop.
There is no second checkpoint to version. Operationally, an SLM at the edge deployed across hundreds of boxes becomes unsustainable if each one needs two synchronised models. A single binary that does draft and verify is far simpler to maintain.
It fits MoE on device. In a fine-grained MoE for device (see native architectures for device), the shallow draft activates even fewer experts, and the memory-bound regime persists even at medium batch, precisely what the work on SSD for on-device MoE (ACM WWW 2026) exploits.

The counterpoint, repeated so it does not get forgotten: on a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8), where memory is not the bottleneck, a dedicated EAGLE-3 draft does fit and its higher α gives it more speedup. There, self-spec is plan B: you use it if the model is exotic and there is no trained draft, not because memory is tight.

Pitfalls

α depends enormously on k. Exiting too high up (small k) makes the draft cheap but sinks α; exiting too low down (k close to L) raises α but the draft costs almost a full forward pass and c → 1, killing the speedup. The optimum is empirical and model-specific. Distrust any speedup number that does not say at which k it was measured.
Training-free is not free in draft quality. SWIFT and CLaSp give lower α than LayerSkip precisely because the model’s intermediate layers were not trained to be good premature exits. The number that matters is α measured on your distribution, not the paper’s.
Sampling temperature and creative outputs lower α just as in classic speculative. At high T, the self-spec speedup erodes faster still because it starts from a lower α.
Large batch neutralises it just as it does classic speculative. As soon as decode turns compute-bound, the drafts stop being “almost free”. Self-spec is for low concurrency.

References

Elhoushi, M., et al. LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding. 2024. https://arxiv.org/abs/2404.16710
SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration. ICLR 2025. https://openreview.net/forum?id=EKJhH5D5wA
CLaSp: In-Context Layer Skip for Self-Speculative Decoding. 2025. https://arxiv.org/abs/2505.24196
ConfLayers: Confidence-Adaptive Layer Skipping for Self-Speculative Decoding. 2026. https://arxiv.org/abs/2604.14612
Self-Speculative Decoding for MoE on Device. ACM Web Conference 2026. https://doi.org/10.1145/3774904.3792218
Hugging Face blog. Faster Text Generation with Self-Speculative Decoding. https://huggingface.co/blog/layerskip
Leviathan, Y., Kalman, M., Matias, Y. Fast Inference from Transformers via Speculative Decoding. ICML 2023. https://arxiv.org/abs/2211.17192

The roofline flips: why optimising small models is a different performance game

Tue, 09 Jun 2026 01:30:00 +0000

This post is the anchor of a mini-series on inference performance in small models (SLMs). Almost every optimisation post on the blog, KV cache, decode, quantisation, was written with a 70B in mind. Here I argue that when the model shrinks by an order of magnitude, the roofline changes regime and several of those intuitions flip. This is not a nuance: it is a different game.

TL;DR

Autoregressive decode in a large LLM is memory-bandwidth-bound: at every step you have to move all the model weights from HBM to the registers of the SMs, and that dominates over the arithmetic operations. The GPU spends its time waiting for bytes, not computing. That single sentence, that decode “waits on HBM”, is the root of half the optimisations on this blog. In a small model (SLM, say 0.5B–7B) the sentence stops being true in the simple form we used to tell it. At batch 1 you are still memory-bound with respect to the hardware, yes, but the forward pass is so cheap (moving 6 GB at 1 TB/s takes ~6 ms, not 70 ms) that the fixed per-step costs, kernel launches, Python scheduler overhead, the sampler, host↔device copies, the synchronize calls, stop being noise and start eating 20-30 % of the time. The bottleneck moves from HBM to orchestration. The concrete, quantitative consequences: (1) CUDA graphs and cutting scheduler overhead pay off more in SLMs than in large models; (2) weight quantisation gives less latency improvement at batch 1 in SLMs, because proportionally there are fewer weights to move relative to activations, KV cache and fixed overhead; (3) batching has more headroom because you cross the ridge point late; (4) the KV cache can dominate relative memory. All of this comes out of a single model, the roofline, applied with numerical honesty.

The analogy: the pantry and the waiter

A kitchen with two very different services.

Tasting menu service, one enormous slow dish (the 70B LLM). Every dish involves heavy ingredients that the assistant has to fetch from the pantry at the back, several times, carrying boxes. The chef, by contrast, assembles the dish in a moment: the slow part is fetching the ingredients, not cooking them. If you want the service to go faster, you do not hire a more skilful chef: you widen the corridor to the pantry, or make each trip bring more boxes. The pantry is HBM; the trip is memory bandwidth; cooking is compute. The big dish is bound by the pantry.

Tapas service, tiny plates (the SLM). Now each tapa has two ingredients and is assembled in a second. The trip to the pantry per tapa is very short. But a cost appears that was negligible for the big dish: the waiter. For every tapa, the waiter has to walk to the kitchen, pick up the plate, carry it to the bar, walk back, take the order, call it out. That back and forth is fixed: it costs the same for one tapa as for the enormous dish. When the tapa is assembled in a second, the waiter, not the pantry, is the bottleneck. Shortening the corridor to the pantry (widening HBM, quantising the weights) barely improves the service any more; what improves it is the waiter chaining several orders together without going back to the kitchen each time (CUDA graphs) or serving several tables in one pass (batching).

The roofline is the tool that says, with numbers, at what point the waiter dominates over the pantry. That boundary is the ridge point, and the joke in the title is that in SLMs we cross the regime much earlier than large-model intuition led us to believe.

The bare mechanism: what the roofline says

The roofline model (Williams, Waterman and Patterson, 2009) starts from a single quantity: arithmetic intensity, which is how many operations you do per byte moved from memory.

$$\text{AI} = \frac{\text{FLOPs}}{\text{bytes moved from memory}} \quad [\text{FLOP/byte}]$$

The hardware has two ceilings: compute (peak FLOPS) and memory (peak bandwidth × AI). Achievable performance is the minimum of the two:

$$\text{Perf} = \min\big(\text{peak FLOPS},\; \text{BW} \times \text{AI}\big)$$

Where the two lines cross is the ridge point, the AI above which you stop being limited by memory and start being limited by compute:

$$\text{AI}_{\text{ridge}} = \frac{\text{peak FLOPS}}{\text{peak BW}}$$

If your kernel has an AI below the ridge, you are memory-bound (the GPU waits for bytes). Above it, compute-bound (the GPU computes flat out and memory is in surplus). What matters is that the ridge point is a property of the hardware, not of the model. Let us look at the numbers, approximate, and I flag them as such because marketing figures mix dense and sparse, different dtypes and unrealistic thermal conditions.

Generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). Per GPU, ~989 TFLOPS BF16 dense (~1,979 TFLOPS FP8 dense; the figure with sparsity is double that and almost never applies to LLM inference). HBM3 ~3.35 TB/s. The ridge in BF16:

$$\text{AI}_{\text{ridge}}^{\text{H100,BF16}} \approx \frac{989 \times 10^{12}}{3.35 \times 10^{12}} \approx 295 \ \text{FLOP/byte}$$

In FP8 the ridge rises to ~590 FLOP/byte (twice the FLOPS against the same BW). Careful: these are datasheet peak figures; in practice a real kernel rarely gets past 70-80 % of either ceiling.

RTX 4090 (24 GB, Ada Lovelace). ~330 TFLOPS FP16 with FP16 accumulation via tensor cores (the “660 TOPS” figure that circulates is with sparsity), and ~1 TB/s of GDDR6X. The ridge:

$$\text{AI}_{\text{ridge}}^{\text{4090,FP16}} \approx \frac{330 \times 10^{12}}{1.0 \times 10^{12}} \approx 330 \ \text{FLOP/byte}$$

Curiously, the same order as the H100 in BF16: the 4090 has less BW but also fewer FLOPS, and the ratio comes out similar. The ridge sits around 300 FLOP/byte in both cases. Hold on to that number.

And where does decode fall? In decode at batch 1, each weight is loaded once from HBM and used for a single multiply-accumulate (one token, one row of activation). The AI of the decode GEMM at batch 1 is on the order of AI ≈ 1-2 FLOP/byte (each weight byte takes part in ~2 FLOP). With batch B, the same weight loaded once serves B rows of activation, so the AI scales roughly linearly:

$$\text{AI}_{\text{decode}}(B) \approx 2B \ \text{FLOP/byte} \quad (\text{for the weight GEMM part})$$

You cross the ridge when 2B ≈ 300, that is B ≈ 150 as an order of magnitude (in practice earlier, because of attention and overheads, but that is the frame). The clean conclusion: decode at low batch is always deeply memory-bound, nowhere near the ridge. That is why we say “decode waits on HBM”, and why quantising weights (moving fewer bytes) speeds up decode in a large model almost linearly. So far, this is the standard story from the large-model posts.

The point of the title: why it flips in SLMs

The classic roofline has a blind spot that does not matter in large models and is everything in small ones: it only models the work inside the kernel. It assumes the only time is bytes/BW or FLOPs/FLOPS. But a real decode step is not just the GEMM. It is a sequence of dozens of kernels (QKV projections, attention, the two MLP layers, normalisations, residuals, the logits head, sampling) and, around each one, there is a fixed orchestration cost:

Kernel launches (kernel launch): each cudaLaunchKernel costs on the order of 5-10 µs of CPU/driver overhead, regardless of kernel size. A decode forward with ~30-60 kernels launched sequentially drags ~0.3-0.6 ms just in launching.
Python scheduler overhead: the vLLM scheduler loop prepares metadata, decides which requests enter the step, builds the input tensors. In pure Python this is hundreds of µs to a couple of ms per step, especially at low concurrency where it is not amortised.
Sampling and post-processing: applying temperature, top-p, penalties, the argmax/multinomial, copying the token back. Another block of hundreds of µs.
Synchronisations and host↔device copies: every synchronize or small copy adds latency that is neither FLOPs nor HBM bytes.

Let us call the sum of all of that T_fixed, the per-step cost independent of model size, on the order of 1-3 ms in an unoptimised Python stack. The real time of a step is now:

$$T_{\text{step}} \approx \underbrace{\frac{\text{weight bytes}}{\text{BW}}}_{T_{\text{HBM}} \text{ (memory-bound)}} + \; T_{\text{fixed}}$$

In a 70B BF16, moving ~140 GB at 3.35 TB/s takes ~42 ms of T_HBM. Against that, a T_fixed of 1-3 ms is noise (2-7 %). The classic roofline is right: the model is memory-bound, full stop. But in a 3B BF16, T_HBM drops to a few ms, and suddenly T_fixed is the same order as T_HBM. The bottleneck stops being the pantry and becomes the waiter. This is the flip in the title, and four counterintuitive consequences follow from it:

(a) At batch 1 you are still memory-bound with respect to the hardware. The AI has not changed: it is still ~2 FLOP/byte, below the ridge. Anyone reading only the roofline will conclude “memory-bound, quantise the weights”. That is true but incomplete: the roofline does not see T_fixed.

(b) Fixed costs become a huge fraction of the step. This is the central point. In the 70B, T_fixed / T_step ≈ 5 %. In the 3B it can be 20-30 %. The effective bottleneck of the 3B is half HBM, half orchestration.

(c) That is why CUDA graphs and cutting scheduler overhead pay off MORE in SLMs. A CUDA graph captures the whole kernel sequence of the step and relaunches it with a single cudaGraphLaunch, removing almost all of the per-kernel launch overhead and much of the per-iteration Python scheduler work. In the 70B, shaving 0.5 ms off a 42 ms step is a +1 % you barely notice. In the 3B, shaving those same 0.5 ms off a ~7 ms step is +7 %, and if you take away nearly all of T_fixed you can gain 20-30 %. The same optimisation, a different prize, because the denominator changed.

(d) Weight quantisation gives LESS latency improvement at batch 1 in SLMs. This is the most counterintuitive one. In the 70B, T_HBM is almost the whole step; going from BF16 to INT4 quadruples the effective weight bandwidth and almost quadruples decode speed. In the 3B, T_HBM is only part of the step (the rest is T_fixed + attention + KV). By Amdahl’s law, if the weights are 60 % of the step and you speed them up 4×, the total step improves only 1/(0.4 + 0.6/4) = 1.8×, not 4×. And proportionally there are fewer weights to move relative to activations, KV cache and fixed overhead. Aggressive quantisation in SLMs does help, yes, but not through raw latency at batch 1, where it gives diminishing returns, rather through capacity and concurrency (we will see that at the end).

(e) The KV cache can dominate relative memory. With 6 GB of weights (3B BF16), a single long-context session can approach that order of magnitude in KV cache. In a 70B (140 GB of weights) the KV is proportionally small until high concurrency. In SLMs the VRAM balance tips towards KV much earlier (the detail is in KV cache), and that changes which memory optimisation is the lever.

The maths that matter: a 3B on a 4090

Let us do the whole calculation, which is where the flip shows up without rhetoric.

Model: 3B parameters, BF16 → 2 bytes/param → ~6 GB of weights. Hardware: RTX 4090, BW ≈ 1 TB/s.

Memory-bound ceiling for decode (batch 1). Each token requires loading the 6 GB once:

$$T_{\text{HBM}} = \frac{6 \times 10^{9} \ \text{bytes}}{1 \times 10^{12} \ \text{bytes/s}} = 6 \times 10^{-3}\ \text{s} = 6\ \text{ms/token}$$ $$\text{Ceiling} = \frac{1}{6\ \text{ms}} \approx 166\ \text{tok/s}$$

That is the theoretical memory-bound ceiling: 166 tok/s, assuming that moving the weights is the only cost. The classic roofline would stop here and say “166 tok/s, go get more BW or quantise”.

Now the fixed overhead. Let us put T_fixed ≈ 2 ms/step (a reasonable value for a Python scheduler + ~40 kernels launched + sampling, without CUDA graphs). The real step:

$$T_{\text{step}} = T_{\text{HBM}} + T_{\text{fixed}} = 6 + 2 = 8\ \text{ms} \;\Rightarrow\; \frac{1}{8\ \text{ms}} = 125\ \text{tok/s}$$

The overhead has eaten 41 tok/s out of the theoretical 166: T_fixed is 25 % of the step (2 of 8 ms). Compare with the 70B: T_HBM ≈ 42 ms, T_step ≈ 44 ms, T_fixed is 4.5 %. Same absolute overhead, relative impact 5-6× larger in the SLM.

What happens if you apply CUDA graphs and take away, say, 1.5 of the 2 ms of T_fixed:

$$T_{\text{step}}^{\text{graphs}} = 6 + 0.5 = 6.5\ \text{ms} \;\Rightarrow\; 154\ \text{tok/s}$$

From 125 to 154 tok/s: +23 % from orchestration alone, without touching the model or the memory hardware. In the 70B the same intervention would have gone from 44 to 42.5 ms, +3.5 %. There it is, in two numbers: “a different game”.

What happens if you quantise the weights to INT4 (1.5 GB instead of 6 GB), with T_fixed still at 2 ms:

$$T_{\text{HBM}}^{\text{INT4}} = \frac{1.5 \times 10^{9}}{1 \times 10^{12}} = 1.5\ \text{ms};\quad T_{\text{step}} = 1.5 + 2 = 3.5\ \text{ms} \;\Rightarrow\; 285\ \text{tok/s}$$

The 4× weight quantisation did not give 4× in latency: it went from 125 to 285 tok/s, a 2.3×, because the 2 ms of T_fixed now dominates (it is 57 % of the step). In the 70B, quantising to INT4 gives almost the full 4× because T_fixed is still noise. The same quantisation delivers twice the speedup on the large model as on the small one, at batch 1. And if you also apply CUDA graphs on top of INT4 (T_fixed → 0.5 ms): 1.5 + 0.5 = 2 ms → 500 tok/s. The order of the optimisations matters: in SLMs, attacking T_fixed first unlocks the rest.

Configuration (3B, 4090, batch 1)	T_HBM	T_fixed	T_step	tok/s	vs. baseline
BF16, no graphs (baseline)	6.0 ms	2.0 ms	8.0 ms	125	1.00×
BF16 + CUDA graphs	6.0 ms	0.5 ms	6.5 ms	154	1.23×
INT4, no graphs	1.5 ms	2.0 ms	3.5 ms	285	2.28×
INT4 + CUDA graphs	1.5 ms	0.5 ms	2.0 ms	500	4.00×

(Illustrative figures with T_fixed rounded; the point is the pattern, not the decimal. The real T_fixed depends on the stack, the PyTorch/CUDA version and whether there is tensor parallelism. Measure it in your own setup before believing any row.)

Implications per optimisation

With the model in hand, the blog’s levers reorder themselves once the regime changes.

Batching: far more headroom in SLMs. Remember that you cross the ridge at B ≈ ridge/2 ≈ 150 as an order of magnitude. In a large model, VRAM runs out long before you saturate compute (weights + KV do not let you reach batch 150). In an SLM the weights take up little space, so you can fit large batches in VRAM and stay memory-bound over a much wider range: the T_HBM of the weights is amortised across the B requests (you load them once, they serve B), so aggregate throughput per GPU rises almost linearly with B until very high values. This is exactly the opposite of the 70B’s fear of saturating compute. In SLMs, batching is the throughput lever par excellence because you saturate compute late; the batch grid search in vLLM has a much wider plateau of good behaviour. Careful: batching improves throughput, not per-request latency; for single-stream latency the prize is in T_fixed.

Speculative decoding: a different crossover point. Speculative wins when verifying γ tokens is “almost free” because you are memory-bound. In an SLM the target is already cheap, so the draft has to be tiny for c = T_draft/T_target to stay small, and the draft’s own T_fixed (launching its kernels) bites harder. The crossover to compute-bound with batching also arrives earlier in absolute terms of tok/s served. The variant that fits best here avoids a separate draft: self-speculative / early-exit reuses early layers of the model itself and saves the T_fixed of orchestrating two models.

Quantisation: it helps for capacity, not for latency at batch 1. As the table showed, INT4 in an SLM at batch 1 gives diminishing returns in latency. Its real prize in SLMs is capacity: weights 4× smaller free VRAM for more KV cache → more concurrency, and it is at high concurrency (aggregate throughput) that saving bytes pays off again. Aggressive sub-4-bit and ternary quantisation takes this to the extreme: in SLMs it makes sense above all for fitting more sessions per GPU, not for lowering the latency of a single one. And it is worth remembering (see quantisation) that at batch 1 the dequantize adds compute work which, in a regime already grazed by T_fixed, is not always free.

Architecture: fine-grained MoE changes which bytes you move. A fine-grained device-native MoE activates few parameters per token, so T_HBM drops relative to a dense model of the same total size, but the T_fixed fraction rises even further, and the router adds its own fixed overhead. It is the SLM regime taken to its limit: almost the whole game is played in orchestration.

Scheduler and CUDA graphs first. The operational conclusion, inverted with respect to the large-model posts: in SLMs, before touching the model, kill T_fixed. CUDA graphs (see SMs, streams and graphs), a vLLM scheduler with its Python part minimised or compiled, and kernel persistence are the first-order levers. In a 70B they would be marginal polish; in a 3B they are half the available speedup.

Applied to on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace). This is the scenario where the flip is most visible, because the 4090 has ~1 TB/s (a third of the H100) but T_fixed is the same in absolute terms. A 3B BF16 without CUDA graphs leaves ~125 tok/s on the table when the memory-bound ceiling is 166; enabling graphs and cleaning up the scheduler recovers most of that. The 4090 has plenty of VRAM room for an SLM, so the bottleneck is almost never total memory but orchestration and, at high concurrency, the KV cache. Rule of thumb: on a 4090 with an SLM, profile the per-step overhead first (Nsight Systems on the gap between kernels) before you quantise.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). The H100 has 3.35 TB/s, so the T_HBM of an SLM is even smaller (a 3B FP8 is ~3 GB → ~0.9 ms) and T_fixed dominates even earlier: a badly orchestrated SLM on an H100 can spend more time in the Python scheduler than moving weights. Serving a single single-stream SLM on an H100 is close to a waste; the right mode is aggressive batching (you saturate compute late, so you push large batches through and per-GPU throughput takes off) or multiplexing many SLMs/sessions per GPU via MPS/MIG. This connects with capacity planning: for SLMs the capacity calculation is governed by concurrency and KV cache, not by the weights. And with the dilemma of one big versus N small: replicating SLMs makes sense precisely because each replica saturates compute late and TP brings nothing (the model already fits; TP would only add communication T_fixed).

What we have not covered

The exact measured T_fixed, kernel by kernel, with Nsight Systems: how much is launch, how much scheduler, how much sampling. That is the content of the next post in the series.
torch.compile / partial captures: alternatives and complements to CUDA graphs when there is dynamic control flow.
The prefill regime in SLMs: prefill is compute-bound even in small models (it processes many tokens at once, high AI), so its roofline is the opposite of decode’s; see prefill.
Attention and KV as the second term of T_HBM: here we have folded them in implicitly; the fine-grained breakdown of attention (which scales with sequence length, not with the weights) deserves its own treatment.

References

Williams, S., Waterman, A., Patterson, D. Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM, 52(4), 2009. https://doi.org/10.1145/1498765.1498785
Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference. arXiv:2503.08311, 2025. https://arxiv.org/abs/2503.08311
Databricks. LLM Inference Performance Engineering: Best Practices. https://www.databricks.com/blog/llm-inference-performance-engineering-best-practices
NVIDIA. NVIDIA H100 Tensor Core GPU Datasheet. https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor-core-gpu-datasheet
NVIDIA. GeForce RTX 4090 — product specifications (Ada Lovelace tensor core figures; treat as approximate, they mix dense/sparse).
Yuan, Z. et al. LLM Inference Unveiled: Survey and Roofline Model Insights. arXiv:2402.16363, 2024 — application of the roofline specifically to LLM inference.

Instrumenting vLLM with OTel: measuring what the optimisations really do

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

vLLM exposes two independent observability signals: Prometheus metrics (pull, aggregated) and OTel traces (push, per request). To measure whether chunked prefill, prefix caching, speculative decoding, FP8 KV cache and concurrency are really working, you need both. The metrics tell you what is happening in the system; the traces tell you why one specific request was slow. This article configures the complete pipeline and maps each optimisation to its diagnostic metric.

The analogy

A Formula 1 driver and his telemetry engineers. The driver feels that the car “goes odd” in turn 3, but without the sensor data he does not know whether it is the rear tyre, the differential or the fuel. The engineers see exactly what happened in that turn, temperature per sensor, lateral load per millisecond.

vLLM without OTel is the driver on his own: you notice that the TTFT “looks high” but you do not know whether it is a long prefill, a prefix cache miss, or a KV cache preemption. With OTel you have the full picture: the metrics are the race summary (aggregated), the traces are the lap-by-lap telemetry (per-request).

Architecture of the two signals

vLLM deliberately separates its two observability channels:

 ┌─────────────────────────────┐
│ vLLM │
│ │
requests ────────────►│ SchedulerStats │
│ │ │
│ ├─► Prometheus /metrics │◄── scrape (pull)
│ │ (aggregated, ~15s) │
│ │ │
│ └─► OTLP exporter │──► push (per request)
│ (spans, immediate) │
└─────────────────────────────┘
│ OTLP gRPC/HTTP
▼
┌─────────────────────┐
│ OTel Collector │
│ │
│ receivers: │
│ otlp (traces) │
│ prometheus │
│ exporters: │
│ langfuse │
│ prometheus remote│
│ loki (logs) │
└─────────────────────┘

Prometheus pull exposes metrics with the vllm: prefix at :8000/metrics. They are histograms, gauges and counters updated on every scheduler iteration. Good for dashboards and alerts about the system as a whole.

OTLP push sends one span per request, immediately on completion. It contains attributes of the specific request: prompt tokens, generated tokens, TTFT, model. Good for debugging anomalous requests and for Langfuse.

Installation and basic configuration

# vLLM with OTel support
pip install "vllm[otel]"
# Installs: opentelemetry-sdk, opentelemetry-api,
# opentelemetry-exporter-otlp, opentelemetry-semantic-conventions-ai

# Start vLLM with OTel enabled
export OTEL_SERVICE_NAME="vllm-production"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://otel-collector:4317"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="grpc"
export OTEL_EXPORTER_OTLP_TRACES_INSECURE="true" # on an internal network without TLS

vllm serve Qwen/Qwen2.5-7B-Instruct \
 --otlp-traces-endpoint http://otel-collector:4317 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --speculative-model Qwen/Qwen2.5-0.5B-Instruct \
 --num-speculative-tokens 5

The Prometheus metrics need no extra configuration: they are always at :8000/metrics.

OTel Collector: minimum configuration

# otel-collector-config.yaml
receivers:
 otlp:
 protocols:
 grpc:
 endpoint: 0.0.0.0:4317
 http:
 endpoint: 0.0.0.0:4318

 prometheus:
 config:
 scrape_configs:
 - job_name: vllm
 scrape_interval: 15s
 static_configs:
 - targets: ["vllm:8000"]

processors:
 batch:
 timeout: 5s
 resource:
 attributes:
 - key: deployment.environment
 value: "production"
 action: upsert

exporters:
 otlphttp/langfuse:
 endpoint: "https://cloud.langfuse.com/api/public/otel"
 headers:
 Authorization: "Basic <base64(pk:sk)>"

 prometheusremotewrite:
 endpoint: "http://prometheus:9090/api/v1/write"

service:
 pipelines:
 traces:
 receivers: [otlp]
 processors: [batch, resource]
 exporters: [otlphttp/langfuse]
 metrics:
 receivers: [prometheus]
 processors: [batch]
 exporters: [prometheusremotewrite]

The five metrics that matter

Every optimisation has a primary diagnostic signal. If the metric does not move as expected after enabling the flag, there is a configuration or load problem.

1. Chunked prefill → `vllm:time_to_first_token_seconds`

Chunked prefill should reduce the variance of the TTFT, not necessarily the median. Its main goal is for the high percentiles (p99) to come down even if the p50 rises slightly.

# TTFT p50 and p99 — expect p99 to drop with chunked prefill enabled
histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))
histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))

Sign that it works: the p99/p50 ratio approaches 1. Without chunked prefill, one request’s long prefill blocks all the others and the p99 rises disproportionately.

Sign of a problem: p50 and p99 both rise. A --max-num-batched-tokens that is too low makes the chunks so small that the prefill takes many steps to complete even though the other requests are not blocked. Raise the budget.

It is also useful to look at the llm.usage.prompt_tokens attribute per span in the OTel traces: requests with many prompt tokens should have a proportional TTFT, not a blocking one.

2. Prefix caching → `vllm:gpu_prefix_cache_hit_rate`

# Prefix cache hit rate on the GPU (0.0–1.0)
vllm:gpu_prefix_cache_hit_rate

# Evolution over a 5-minute window
rate(vllm:gpu_prefix_cache_hit_rate[5m])

Sign that it works: a sustained hit rate > 0.5 in workloads with a shared system prompt. With a hit rate of 0.8, 80% of requests skip the prefix prefill; the TTFT of those requests falls to the cost of the variable suffix alone.

Sign of a problem: a hit rate close to zero despite system prompts that “look” identical. The usual causes:

# ❌ This breaks the prefix caching hash:
system_prompt = f"Today is {datetime.now()}. You are an assistant..."
# ^^^ a different timestamp on every request

# ✅ The system prompt has to be identical byte for byte:
system_prompt = "You are an assistant specialising in infrastructure..."

Any variation in the system prompt, timestamps, session IDs, interpolated prompt versions, produces a different hash and a cache miss. The OTel traces do not expose the hit/miss per request directly in the current implementation; use them to correlate a high llm.usage.prompt_tokens with a high TTFT in the same request.

3. Speculative decoding → `vllm:spec_decode_draft_acceptance_rate`

# Acceptance rate of the draft model (0.0–1.0)
vllm:spec_decode_draft_acceptance_rate

# Estimated effective speedup (with k=5 proposed tokens)
# speedup ≈ (1 + α·k) / (1 + overhead_draft)
# Simplified: if α=0.75 and k=5 → speedup ≈ 1 + 0.75×5×(1 - cost_ratio)

Sign that it works: a sustained acceptance rate > 0.70. Below 0.60, the draft model’s overhead outweighs the gain from the accepted tokens and speculative decoding is counterproductive.

Sign of a problem: an acceptance rate < 0.50. The usual causes:

A drafter from a different family than the verifier (for instance, Mistral 0.5B as the draft for Qwen 7B).
A high generation temperature (>0.9): the higher the temperature, the more the draft’s distribution diverges from the verifier’s.
A very large batch: at high concurrency, the draft can fall outside the domain of the current requests.

# Alert: inefficient speculative decoding
ALERT SpecDecodeInefficient
 IF vllm:spec_decode_draft_acceptance_rate < 0.60
 FOR 5m
 LABELS { severity = "warning" }
 ANNOTATIONS { summary = "Draft acceptance rate low: disable spec decode or change the drafter" }

In the OTel traces, the complete span of the request includes the total decode time. Without an acceptance rate per span, the way to detect that spec decode is working is to compare the total decode time divided by the tokens generated: if it is significantly lower than the baseline without spec decode, it is helping.

4. FP8 KV cache and concurrency → `vllm:gpu_cache_usage_perc` + `vllm:num_preemptions_total`

These two metrics are the two sides of KV cache management:

# KV cache utilisation (0.0–1.0)
# With FP8 enabled, the same hardware supports more requests before saturating
vllm:gpu_cache_usage_perc

# Cumulative preemptions (counter)
# Rises when vLLM cannot fit more requests and pauses one
rate(vllm:num_preemptions_total[5m])

Sign that FP8 works: with --kv-cache-dtype fp8 enabled, gpu_cache_usage_perc should saturate at concurrency levels around 2× higher than the BF16 baseline before num_preemptions_total starts to grow.

Sign of a problem: num_preemptions_total growing at rates > 1/minute with gpu_cache_usage_perc below 0.90. It indicates that max-num-seqs is too high for the available KV cache: requests enter the system but there are no free blocks to assign to them. Lower max-num-seqs or reduce max-model-len.

# Alert: KV cache saturated with preemptions
ALERT KVCacheSaturated
 IF rate(vllm:num_preemptions_total[2m]) > 0.5
 AND vllm:gpu_cache_usage_perc > 0.85
 FOR 3m
 LABELS { severity = "critical" }
 ANNOTATIONS { summary = "KV cache saturated: lower max-num-seqs or max-model-len" }

The impact of FP8 on capacity can be quantified:

$$\Delta\text{capacity} = \frac{\text{max tokens FP8}}{\text{max tokens BF16}} \approx 2\times$$

Measure before and after enabling --kv-cache-dtype fp8: the level of gpu_cache_usage_perc for a given concurrency should fall by half.

5. Effective concurrency → `vllm:num_running_seqs` + `vllm:num_waiting_seqs`

# Requests active in the engine (decode + prefill in progress)
vllm:num_running_seqs

# Requests queued waiting for a slot
vllm:num_waiting_seqs

# Waiting ratio: if > 0.2 sustained, there is a concurrency bottleneck
vllm:num_waiting_seqs / (vllm:num_running_seqs + vllm:num_waiting_seqs)

Healthy sign: num_running_seqs stable close to the configured --max-num-seqs value, num_waiting_seqs low (< 10% of running).

Sign of a problem: a high num_waiting_seqs with a low gpu_cache_usage_perc. It indicates that the scheduler is not filling the available slots because max-num-batched-tokens is too low: the token budget per step does not allow the pending prefills to be processed fast enough. Raise max-num-batched-tokens.

Reference dashboard: the 5 metrics in Grafana

{
 "panels": [
 {
 "title": "TTFT p50 / p99 (chunked prefill)",
 "targets": [
 {"expr": "histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))", "legendFormat": "p50"},
 {"expr": "histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))", "legendFormat": "p99"}
 ]
 },
 {
 "title": "Prefix cache hit rate",
 "targets": [{"expr": "vllm:gpu_prefix_cache_hit_rate", "legendFormat": "GPU hit rate"}]
 },
 {
 "title": "Spec decode acceptance rate",
 "targets": [{"expr": "vllm:spec_decode_draft_acceptance_rate", "legendFormat": "acceptance rate"}]
 },
 {
 "title": "KV cache usage + preemptions",
 "targets": [
 {"expr": "vllm:gpu_cache_usage_perc", "legendFormat": "cache usage"},
 {"expr": "rate(vllm:num_preemptions_total[2m]) * 60", "legendFormat": "preemptions/min"}
 ]
 },
 {
 "title": "Effective concurrency",
 "targets": [
 {"expr": "vllm:num_running_seqs", "legendFormat": "running"},
 {"expr": "vllm:num_waiting_seqs", "legendFormat": "waiting"}
 ]
 }
 ]
}

Connecting traces to Langfuse

vLLM’s OTel traces are GenAI semconv compatible spans. Langfuse accepts them directly via OTLP:

# In the OTel Collector (already configured above)
# The otlphttp/langfuse exporter sends traces to Langfuse Cloud or self-hosted

# For self-hosted Langfuse (ENS/sovereign):
exporters:
 otlphttp/langfuse:
 endpoint: "http://langfuse-internal:3000/api/public/otel"
 headers:
 Authorization: "Basic <base64(pk_xxx:sk_xxx)>"

In Langfuse, each vLLM request appears as a trace with:

gen_ai.system: the model served
gen_ai.usage.input_tokens: prompt tokens
gen_ai.usage.output_tokens: generated tokens
Span duration: end-to-end latency

What does not appear directly in the span: the speculative decoding acceptance rate, the prefix cache hit/miss, or the number of preemptions. That data is only in Prometheus. The correct workflow is:

Langfuse identifies an anomalous request by latency.
Prometheus/Grafana shows whether in that interval there were high preemptions, low spec decode, or a prefix cache miss.
The two are correlated by timestamp.

Quick diagnostic matrix

Observable symptom	Prometheus metric	Probable cause	Action
TTFT p99 very high	`ttft p99/p50 >> 2`	Long blocking prefills	Raise `--max-num-batched-tokens`
TTFT p50 high, p99 the same	`ttft p50 > 500ms`	Prefix cache not working	Check the system prompt hash
Slow decode with no improvement	`spec_decode_acceptance < 0.60`	Incompatible drafter	Change the drafter or disable it
Sporadic OOM / crash	`gpu_cache_usage_perc = 1.0` + preemptions	KV cache full	Lower `max-num-seqs` or enable FP8
High queue with free cache	`waiting >> 0` + `cache < 0.70`	Low token budget	Raise `--max-num-batched-tokens`

Implications for sovereign on-premise inference

In an ENS deployment where you cannot use Langfuse Cloud or DataDog, the complete self-hosted stack is:

# docker-compose.yml (or equivalent K8s manifests)
services:
 otel-collector:
 image: otel/opentelemetry-collector-contrib:latest
 volumes: [./otel-config.yaml:/etc/otel/config.yaml]

 langfuse:
 image: langfuse/langfuse:latest
 environment:
 DATABASE_URL: postgres://...

 prometheus:
 image: prom/prometheus:latest

 grafana:
 image: grafana/grafana:latest

The whole pipeline runs on-premise. The traces never leave the perimeter. ENS compliance does not depend on which observability you choose: it depends on the inference data not going out to third parties. With a local stack, both conditions are met.

References

Knowledge Distillation: teaching a small model to think like a big one

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

Knowledge Distillation is the technique of training a small model (student) using the output probabilities of a large model (teacher) as supervision, instead of using only the hard labels from the training dataset. The result is a small model that reasons better than its size suggests, because it learns the teacher’s uncertainty distributions rather than memorising binary answers. It is the reason Phi-4 (14B) beats most 70B models at reasoning, and why the Gemma 3 family models are surprisingly capable for their size. It is not a compression technique for an existing model: it is a training process that produces a smaller model from scratch or from a different starting point.

The analogy

A master surgeon with thirty years of experience and a first-year resident. If the resident only learns from the anatomy manual (binary correct answers: “cut here, not here”) it will take years to develop the master’s clinical judgement. But if they operate alongside them, watching their micro-decisions, their hesitations, the ambiguous cases where the master knows that two options are almost equally valid, they learn something the manual cannot teach: the structure of uncertainty.

Knowledge distillation is exactly that. The “anatomy manual” is the hard labels (the correct answer). The “master surgeon” is the teacher LLM. The probability distributions over the vocabulary are the materialisation of that uncertainty the student absorbs.

What it really is

When an LLM generates text, it does not produce a single word: it produces a probability distribution over its whole vocabulary at each position. For the next token, the model might say:

"Paris": 42%
"Lyon": 8%
"Marseille": 6%
"the city": 5%
...rest of the vocabulary: 39%

This distribution is dense information. It tells you not only what the correct answer is, but also which other answers were plausible and to what degree. A student trained only with the label “Paris” (probability 1.0 on the correct token, 0.0 on the rest) never sees this richness.

Distillation uses the teacher’s full distribution as the student’s training target. The loss function has two terms:

$$\mathcal{L}_{total} = (1 - \alpha) \cdot \mathcal{L}_{CE}(y, \hat{y}_S) + \alpha \cdot \mathcal{L}_{KD}(p_T, p_S, T)$$

Where:

$\mathcal{L}_{CE}$ is the standard cross-entropy with the hard labels (classic supervision).
$\mathcal{L}_{KD}$ is the KL divergence between the teacher’s and the student’s distributions.
$\alpha$ controls the relative weight of each term (typically 0.5–0.9 in favour of KD).
$T$ is the temperature, a parameter that softens the distributions to make the KD signal more informative.

The role of temperature

If the teacher assigns 99% to “Paris” and 0.001% to every other word, the distribution is almost as informative as a hard label. A temperature $T > 1$ softens that distribution:

$$p_T(k) = \frac{\exp(z_k / T)}{\sum_j \exp(z_j / T)}$$

With $T = 4$ and the original logits, the distribution that was previously [99%, 0.001%, 0.001%…] becomes something like [42%, 8%, 6%…]. The student sees the teacher’s real probability neighbourhood, not just its point answer.

Numerical example with temperature:

Teacher logits for “The capital of France is _____”:

Paris: 8.5
Lyon: 3.2
Europe: 2.1
a: 1.8

With T=1 (standard softmax):

$$p(\text{Paris}) = \frac{e^{8.5}}{e^{8.5} + e^{3.2} + e^{2.1} + e^{1.8}} \approx 99.3\%$$

With T=4:

$$p(\text{Paris}) = \frac{e^{8.5/4}}{e^{8.5/4} + e^{3.2/4} + e^{2.1/4} + e^{1.8/4}} = \frac{e^{2.125}}{e^{2.125} + e^{0.8} + e^{0.525} + e^{0.45}} \approx 54\%$$

The signal at T=4 is far more informative for the student: it learns that Lyon is more plausible than Europe, that Europe is more plausible than “a”, and so on.

The three modes of distillation

Offline (or “black-box”)

The teacher generates a synthetic dataset of answers before training. The student is trained on that dataset as if it were ordinary hard labels.

teacher → generates 100M (prompt, completion) pairs → dataset
student → is trained on that dataset

It is the cheapest way to scale: the teacher runs only once, the student is trained on the generated data with conventional hardware. Most open source instruction models (Alpaca, Vicuna, WizardLM in their early versions) used this strategy: GPT-4 as teacher, saved data, Llama-7B as student.

Limitation: the student never sees the teacher’s probability distributions, only its answers. It is distillation of “behaviour”, not of “knowledge” in the strict sense. If the teacher makes a mistake (and GPT-4 does make mistakes), the error is crystallised into the dataset.

Online (or “white-box”)

Teacher and student run together during training. The student processes each batch, the teacher processes the same batch in parallel, and the KD loss is computed in real time with the full probability distributions.

for batch in dataset:
logits_teacher = teacher(batch) # teacher forward pass
logits_student = student(batch) # student forward pass
loss = KL(softmax(logits_teacher/T), softmax(logits_student/T))
loss.backward() # only updates student

The teacher has gradients disabled (torch.no_grad()). The learning signal is richer than in offline, but the cost is high: you need to keep the teacher in VRAM throughout training. To distil a 405B teacher into an 8B student, you would need several H100s for the teacher alone.

On-policy

A recent variant (2024–2026) that combines the best of both: the teacher generates answers dynamically during training, but the student evaluates them with its own distribution. The cycle is:

The student generates a proposed answer (rollout).
The teacher scores that proposal with its probability distribution.
The student updates with the teacher’s signal.

This stops the student from learning from distributions outside its own domain (the distribution shift problem in offline). It is the basis of algorithms such as SimCT (2026) that use teachers from different families (Qwen, Phi, Gemma) to generate a cross-tokeniser signal.

Why the best small models use distillation

Phi-4 (Microsoft, 14B), Gemma 3 (Google, 9B/27B), and the compact Qwen3 family models are the clearest examples. Their benchmarks are anomalous with respect to their size: Phi-4-14B beats LLaMA-3-70B on MATH and GPQA-Diamond, two mathematical and scientific reasoning benchmarks where size is usually decisive.

Why? The key is what supervises the training:

A model trained on internet data learns the distribution of human text, which includes a lot of low-quality text, errors and ambiguities.
A student that learns from a frontier teacher (GPT-4o, Claude 3 Opus, Gemini 1.5 Pro) absorbs a distribution filtered towards high-quality text and correct reasoning.

The student with 14B parameters does not “know more” than an undistilled one of the same size, but it has learned to use them better because its training gradients were never contaminated by low-quality text.

Empirical data point: Phi-4 (14B distilled) vs LLaMA-3-70B (not distilled) on the MATH benchmark (2025):

Phi-4: 80.4%
LLaMA-3-70B: 68.0%

A model 5× smaller beats the large one because the training signal is better, not because it has more parameters.

Reasoning distillation: the case of thinking models

Reasoning models (DeepSeek-R1, Qwen3-thinking, QwQ) generate internal chains of thought before giving the final answer. Distilling reasoning is more complex because you do not only want to transfer the answer: you want to transfer the way of thinking.

The current strategy (2025–2026) is reasoning trace distillation:

The teacher (a large thinking model) generates answers with its full internal chain of thought.
The dataset includes those chains of thought as part of the output.
The student learns to imitate both the chain and the final answer.

This explains why Qwen3-7B-thinking can reason formally about mathematics while being 10× smaller than the models that preceded it without distillation: it learned the process, not just the result.

When to use distillation vs. the alternatives

Technique	What it does	Requires retraining	Result
Quantisation	Reduces weight precision	No	Same model, smaller
Pruning	Removes irrelevant weights	No (PTQ)	Same model, sparser
Distillation	Trains a new model	Yes	Different model, smaller

Distillation does not compress an existing model: it produces a new one. That is why it is complementary, not a substitute: you can distil a 405B into an 8B, and then quantise that 8B to INT4 to reduce its inference cost.

When it is the right option:

You need a model 5–10× smaller than the best available one.
You have access (API or local) to a quality teacher.
You have training data or the ability to generate it.
Latency or inference cost is a hard constraint.

When it is not:

You want to compress an existing model quickly: use quantisation + pruning.
You have no training budget (online distillation takes weeks of GPU).
The teacher is not significantly better than the base student: the KD signal will be weak.

Implications for on-premise inference

In a sovereign deployment, the teacher can be a large model running locally (no external API needed). The flow is:

generic 4×H100:
teacher: Llama-3.3-70B-Instruct (on the 4×H100, full load)
→ generates a dataset of 10M (prompt, completion with logits) pairs
→ 3-4 weeks of generation at batch 32
After the dataset:
student: Qwen2.5-7B (fine-tuned with KD loss over the dataset)
→ 2-3 days of training on the same H100s
→ result: a 7B that reasons like the 70B in the specific domain
Production:
RTX 4090: serves the 7B student quantised to INT4 (4 GB)

The teacher is only needed to generate the data. The student is what goes into production. The investment in training compute pays for itself in months of cheaper inference.

For ENS/NIS2: this flow is 100% on-premise, zero dependency on external APIs, and the resulting model is yours in every sense.

References

Distilling the Knowledge in a Neural Network — Hinton, Vinyals & Dean, 2015 (paper fundacional)
Phi-4 Technical Report — Microsoft Research, 2024
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — DeepSeek, 2025 (destilación de razonamiento)
Awesome LLM On-Policy Distillation — colección de papers de destilación en-policy, 2025–2026
Knowledge Distillation for LLMs: Survey — ICLR 2025

Optimising decode in vLLM: squeezing every token out of small hardware

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

Decode is the phase in which vLLM generates output tokens one at a time. It is memory-bound, not compute-bound: the GPU spends more time waiting for the weights to arrive from VRAM than doing arithmetic. On small hardware, an RTX 4090 (24 GB) or an L40 (48 GB), a badly configured decode wastes half the card’s capacity. Five vLLM parameters change the equation: gpu-memory-utilization, max-num-seqs, speculative decoding, FP8 KV cache and a swap-space correctly set to zero. Well calibrated, the difference is real: from 15 tokens/s to 35–50 tokens/s on the same hardware.

The analogy

An assembly-line worker putting cars together. Every car needs exactly the same process: walk to the store for the part, come back, bolt it on, repeat. The travel time to the store, the VRAM latency, is fixed and cannot be eliminated. But there are ways to make it less painful:

Have several cars in parallel on the line (more concurrency, the same travel time amortised).
Have an assistant prefabricate common parts (speculative decoding: the draft model proposes, the verifier confirms).
Keep only the most-used parts in the workshop (quantised KV cache: more contexts fit in the same space).

Those three strategies are exactly the three axes of decode optimisation in vLLM.

Why decode is memory-bound

During prefill, the GPU processes N tokens in parallel: the attention operation is a large matmul and the compute units are busy. During decode, it processes 1 token per step: the matmul becomes a vector-matrix product, an operation that underuses the tensor cores.

The typical compute utilisation ratio during decode on an RTX 4090:

$$\text{MFU}_{decode} \approx 5–15\% \quad \text{(vs 40–60\% in prefill)}$$

The bottleneck is not compute power, it is bandwidth. To generate each token, the model has to read its complete weights from VRAM:

$$\text{time per token} \approx \frac{\text{weight size in bytes}}{\text{VRAM bandwidth}}$$

For Qwen2.5-7B in BF16 (14 GB of weights) on an RTX 4090 (1,008 GB/s):

$$t \approx \frac{14 \times 10^9}{1.008 \times 10^{12}} \approx 13.9 \text{ ms/token} \approx 72 \text{ tokens/s theoretical maximum}$$

The real value is lower (~30–50 tok/s) because of scheduler overhead, attention over the growing KV cache and other latencies. But the theoretical limit sets the ceiling.

With Q4_K_M (weights ~4 GB):

$$t \approx \frac{4 \times 10^9}{1.008 \times 10^{12}} \approx 3.97 \text{ ms/token} \approx 252 \text{ tokens/s theoretical}$$

Quantising the model is the most direct way to improve decode throughput on memory-bound hardware. Everything else optimises on top of that ceiling.

The five levers

1. Give vLLM all the VRAM you can — `--gpu-memory-utilization`

--gpu-memory-utilization (short form --gpu-mem-util) defines the fraction of available VRAM that vLLM may use for the KV cache, once the model weights are loaded. The rest is reserved for activations during the forward pass and for the CUDA context.

vllm serve my-model \
 --gpu-memory-utilization 0.92

The default value is 0.90. On bare metal where no other process uses the GPU, 0.92–0.95 is safe. Do not go above 0.95: vLLM needs headroom for activations during batch peaks, and running out of VRAM in the middle of an inference results in a process crash, not a clean error.

Why it matters: more KV cache available = more simultaneous requests in flight = better GPU utilisation during decode. PagedAttention allocates the KV cache in fixed-size blocks (16 tokens per block by default), and vLLM manages them like virtual memory pages. The more blocks available, the more requests it can serve without any of them waiting for space.

RTX 4090, Qwen2.5-7B-BF16 (14 GB weights):
Total VRAM: 24 GB
Weights: 14 GB
Available for KV cache: 10 GB
gpu-memory-utilization 0.90 → 0.90 × 10 GB = 9 GB for KV cache
gpu-memory-utilization 0.94 → 0.94 × 10 GB = 9.4 GB → ~4% more tokens in flight

The impact is modest with models that fit comfortably, but it is amplified with models that push VRAM to its limit.

2. Real concurrency — `--max-num-seqs`

--max-num-seqs is the maximum number of requests vLLM can have in process simultaneously (counting prefill and decode together). It is the parameter that controls the system’s effective concurrency.

vllm serve my-model \
 --max-num-seqs 128

The effect is direct: more requests in simultaneous decode = better amortisation of the fixed cost of reading the weights. When the decode batch grows from 1 to 8, the time to generate 8 tokens is almost the same as generating 1 (the weights are read once for all of them). Aggregate throughput scales almost linearly until the KV cache or the activation VRAM becomes the bottleneck.

$$\text{aggregate throughput}(B) \approx B \times \text{throughput}(1) \quad \text{for } B \ll B_{max}$$

Common mistake: raising --max-num-seqs without making sure there is enough KV cache in VRAM for all the requests. If vLLM cannot hold the KV caches of 128 simultaneous requests, it does preemption (pauses some request and frees its KV cache) at a latency cost. Monitor vllm:num_preemptions_total.

Interaction with --max-num-batched-tokens: the vLLM scheduler processes up to max-num-batched-tokens tokens per step. If you have 128 requests in decode generating 1 token each, that is 128 decode tokens. The decode budget consumes 128 tokens of the total budget; the rest goes to prefill in chunks. Tune both values together.

# For an RTX 4090 serving ~50 concurrent users with responses of up to 512 tokens
vllm serve my-model \
 --max-num-seqs 64 \
 --max-num-batched-tokens 8192
# 64 decode tokens per step + up to 8128 chunked prefill tokens

3. Speculative decoding — `--speculative-model` + `--num-speculative-tokens`

Speculative decoding is the highest-impact change for decode on small hardware. The idea is simple: a small draft model proposes several tokens at once, and the verifier model validates or rejects them in a single forward pass.

vllm serve Qwen/Qwen2.5-7B-Instruct \
 --speculative-model Qwen/Qwen2.5-0.5B-Instruct \
 --num-speculative-tokens 5 \
 --speculative-draft-tensor-parallel-size 1

Why it works: the 7B verifier has to read 14 GB of weights per step. With 5 proposed tokens, if the acceptance rate is 80%, an average of 4–5 tokens are generated per verifier step instead of 1. Effective throughput goes up without the GPU working any harder.

The acceptance rate (α) depends on how well the draft predicts the verifier’s distribution. Within the same domain, models from the same family usually have α > 0.75:

$$\text{speedup} \approx \frac{1 + \alpha \cdot k}{1 + \alpha \cdot k / \text{cost ratio}}$$

Where $k$ is the number of proposed tokens and cost_ratio is the draft/verifier cost ratio. For a 0.5B draft and a 7B verifier (ratio ~14×):

$$\text{speedup} \approx 1 + 0.8 \times 5 \approx 5 \text{ (theoretical maximum, not reachable)}$$

In practice, with α = 0.75 and k = 5 on hardware without NVLink: 1.8–2.5× more tokens/s compared with decode alone.

EAGLE-3 in 2026: the best drafters today are not small versions of the same model, but networks specialised in predicting the verifier’s distribution. EAGLE-3 reports a 3–6.5× speedup over vanilla decode in public benchmarks. In production with mixed batches the real speedup is more conservative (1.5–3×). vLLM supports EAGLE/EAGLE-2 via --speculative-model:

# With an EAGLE drafter (requires a drafter trained specifically for the base model)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
 --speculative-model yuhuili/EAGLE3-LLaMA3.1-Instruct-8B \
 --num-speculative-tokens 6

When speculative decoding does NOT help:

Very large batches (>32 requests): the acceptance rate varies between requests and the batch spends more time re-drafting than accepting.
High-entropy tasks (brainstorming, very creative code): the draft predicts worse, α falls below 0.5 and the draft overhead outweighs the gain.
If the draft model does not fit in the available VRAM alongside the verifier.

On an RTX 4090 with a 7B verifier and a 0.5B draft (BF16): 14 + 1 GB = 15 GB. That leaves 9 GB for the KV cache. It works.

4. Quantised KV cache — `--kv-cache-dtype fp8`

This was already covered in the prefill article for its effect on context capacity. From the decode point of view, the benefit is different: more tokens fit in the KV cache → more simultaneous requests without preemption → better aggregate throughput.

vllm serve my-model \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales

Precision warning in decode: the KV cache is read on every attention step of the decode. Quantisation introduces noise into the attention activations. For long texts (>4K tokens of context) it can accumulate. In quality benchmarks (MMLU, HellaSwag) the degradation with FP8 KV and --calculate-kv-scales is <0.5% on modern models. Without --calculate-kv-scales, the degradation can be larger because the scales are fixed statically.

Optimal combination for RTX 4090:

vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \
 --quantization awq \  # weights in INT4: 4 GB model
 --kv-cache-dtype fp8 \  # KV cache at half the size
 --calculate-kv-scales \
 --gpu-memory-utilization 0.94

# Available VRAM: 24 - 4 = 20 GB for KV cache
# With FP8: ~40 KB/token (vs 80 KB BF16) → 20 GB / 40 KB = 500,000 tokens of total context
# With max-num-seqs 64 and a 4K ctx: 64 × 4096 × 40KB = 10 GB → fits with room to spare

5. Remove the swap — `--swap-space 0`

--swap-space defines how much system RAM (not VRAM) vLLM may use to preempt KV caches. When vLLM has more active requests than fit in VRAM, it can “pause” some by moving their KV cache to RAM and reactivating them later.

The problem: moving a 4K-token KV cache from VRAM to RAM and back has a latency of tens of milliseconds over PCIe. For a system where you want predictable latency, the swap introduces unacceptable jitter.

vllm serve my-model \
 --swap-space 0

With --swap-space 0, when vLLM cannot hold any more requests in VRAM, it simply queues them instead of doing preemption. The queue adds waiting latency, but it is predictable and it does not interrupt the requests already in flight.

When should you have swap? If your workload has short demand peaks and you can tolerate occasional jitter in exchange for not rejecting requests, a swap of 4–8 GB can be useful. In ENS deployments where latency is a contractual SLA, --swap-space 0 is the right option.

The reference configuration per hardware

RTX 4090 (24 GB) — 7B model, internal use

vllm serve Qwen/Qwen2.5-7B-Instruct \
 --gpu-memory-utilization 0.92 \
 --max-model-len 8192 \
 --max-num-seqs 64 \
 --max-num-batched-tokens 8192 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales \
 --swap-space 0 \
 --speculative-model Qwen/Qwen2.5-0.5B-Instruct \
 --num-speculative-tokens 5 \
 --speculative-draft-tensor-parallel-size 1 \
 --dtype bfloat16

Expected throughput: 35–55 tokens/s per user, up to 64 simultaneous, TTFT <500ms for prompts <1K tokens.

L40 (48 GB) — 14B model, multi-user

vllm serve Qwen/Qwen2.5-14B-Instruct \
 --gpu-memory-utilization 0.90 \
 --max-model-len 16384 \
 --max-num-seqs 128 \
 --max-num-batched-tokens 16384 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales \
 --swap-space 0 \
 --speculative-model Qwen/Qwen2.5-1.5B-Instruct \
 --num-speculative-tokens 5 \
 --dtype bfloat16

Expected throughput: 25–40 tokens/s per user, up to 128 simultaneous with speculative decoding active, TTFT <800ms for prompts <2K tokens.

How to measure that decode is optimised

# Key metrics at vllm:8000/metrics

vllm:generation_tokens_total # total tokens generated → trend
vllm:e2e_request_latency_seconds_* # end-to-end latency per percentile
vllm:time_per_output_token_seconds_* # ITL (inter-token latency)
vllm:num_preemptions_total # if it rises, the KV cache is filling up
vllm:spec_decode_draft_acceptance_rate # speculative decoding hit rate

If spec_decode_draft_acceptance_rate < 0.6, the drafter is not helping: turn speculative decoding off or find a drafter better trained for your model or domain.

If num_preemptions_total grows, you have too many simultaneous requests for the available KV cache. Options: lower max-num-seqs, enable FP8 KV cache, lower max-model-len, or quantise the model further.

Implications for on-premise inference

In a sovereign deployment with fixed hardware, you cannot buy more GPUs at will. Every well-calibrated tenth of gpu-memory-utilization, every point of speculative decoding acceptance rate and every MB of KV cache freed by FP8 is real capacity you do not have to provision with another node.

The combination of quantised weights (AWQ/GPTQ), FP8 KV cache and speculative decoding lets a 14B serve on one L40 what without optimisations would need two L40s in tensor parallel. That is the economic argument for investing time in these parameters.

Decode cannot be accelerated indefinitely on memory-bound hardware: the theoretical limit is set by VRAM bandwidth. But the difference between the minimum and the maximum achievable on that hardware can be 3–4× with the right levers.

References

Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023
EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees — Li et al., 2024
Fast Inference from Transformers via Speculative Decoding — Leviathan et al., 2022
vLLM Optimization and Tuning — documentación oficial
vLLM Speculative Decoding
FP8 KV Cache en vLLM

Optimising prefill in vLLM: the knobs your TTFT will not forgive

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

Prefill is the phase in which vLLM processes your input prompt and produces the initial KV cache. It is compute-bound (unlike decode, which is memory-bound), it takes longer the longer the prompt is, and it blocks the decode of every other queued request. There are four levers in vLLM that radically change its behaviour: chunked prefill, prefix caching, FP8 KV cache and the per-batch token budget. With modest hardware, a 24 GB RTX 4090 or a 48 GB L40, the difference between ignoring them and using them well can be a 3× lower TTFT and 40% more aggregate throughput.

The analogy

Picture a printing shop from the early twentieth century. Setting the lead type (preparing the forme) is slow and blocks the press. Printing the pages already set is fast, but it needs the forme ready before it can start.

Prefill is setting the type. Decode is printing. A press that can only do one thing at a time, either set or print, leaves the machinery idle half the time. The historical solution was to have one worker setting the next page while the previous one was already on the press. That is, exactly, chunked prefill.

What prefill really is

When a request arrives at vLLM, the engine has to process every token of the prompt at once before it can emit the first token of the response. During that processing it computes, for each prompt token, its attention Key and Value vectors. The result, the initial KV cache, is stored in VRAM and used throughout the subsequent decode.

Unlike decode, where the model processes one new token per step, in prefill it processes N tokens in one go. That makes it far more efficient in FLOPs/token (GPUs are good at large matmuls), but it carries a quadratic cost in attention:

$$\text{prefill attention FLOPs} \approx 4 \cdot N^2 \cdot d_{model}$$

With a prompt of 1,000 tokens and $d_{model} = 4096$ (Qwen2.5-7B): $4 \cdot 10^6 \cdot 4096 \approx 16 \times 10^9$ FLOPs in attention alone. With 4,000 tokens, 256× more, because of the quadratic nature.

Prefill (compute-bound):
prompt tokens → [attention O(N²)] → [FFN] → initial KV cache
Decode (memory-bound):
1 new token → [cross-attention over KV cache] → next token

Why prefill is a problem on small hardware

On an H100 with 3.35 TB/s of bandwidth, a long prefill amortises quickly. On an RTX 4090 (1.008 TB/s) or an L40 (864 GB/s), the bottleneck shows up sooner and has concrete consequences:

The head-of-line blocking problem. By default, vLLM processes a complete prefill before doing any decode. If you have 10 requests queued, 9 in decode and 1 with an 8,000-token prompt, those 9 requests stall while the GPU chews through the long prefill. Their users see the streaming freeze. This is called head-of-line blocking and it is the number one enemy of TTFT in production.

The four levers

1. Chunked prefill — `--enable-chunked-prefill` + `--max-num-batched-tokens`

Chunked prefill splits a long prefill into pieces (chunks) and interleaves them with decode steps in the same batch. In vLLM V1 (≥ 0.6) it is on by default.

vllm serve my-model \
 --enable-chunked-prefill \
 --max-num-batched-tokens 4096

--max-num-batched-tokens is the total token budget vLLM can process in a single engine step, counting prefill and decode together. It is the most important parameter for controlling the trade-off:

`max-num-batched-tokens`	Effect
Low (512–2048)	More decode steps per cycle → better ITL, worse TTFT
High (8192–32768)	Large prefill chunks → better TTFT and throughput, worse ITL

For an RTX 4090 serving 7B–13B models with mixed contexts (256–4096 tokens):

--max-num-batched-tokens 8192 # a reasonable balance point

For an L40 (48 GB) with larger models and longer prompts:

--max-num-batched-tokens 16384

How it works internally: with a budget of 4,096 tokens and a prefill of 10,000, vLLM splits it into 3 chunks (4,096 + 4,096 + 1,808). Between chunks, it processes the pending decode steps. The requests in decode keep advancing; the long prefill takes longer to finish, but it freezes nothing.

Without chunked prefill:
t=0 [prefill 10k tokens]─────────────────────────────┐
t=1 └─[decode r1,r2...r9]
With chunked prefill (budget 4096):
t=0 [prefill chunk 4096][decode r1..r9]
t=1 [prefill chunk 4096][decode r1..r9]
t=2 [prefill chunk 1808][decode r1..r9]
t=3 [decode all, including the new one]

The TTFT of the long request rises slightly (3 steps instead of 1), but the ITL of the other 9 requests is not interrupted.

2. Prefix caching — `--enable-prefix-caching`

If several requests share the same prefix, a system prompt, a few-shot block, a context document, vLLM can compute the KV cache of that prefix once and reuse it.

vllm serve my-model \
 --enable-prefix-caching

This is called Automatic Prefix Caching (APC). Internally, vLLM splits the KV cache into fixed-size blocks (16 tokens per block by default) and assigns them a content-based SHA hash. When a new request arrives, it checks whether any initial block is already in the cache. On a hit, that prefill is skipped.

The numerical impact: suppose a 512-token system prompt that appears in 80% of your requests, and a rate of 100 req/min:

Without APC: 80 req/min × 512 tokens × prefill_cost = 41,000 tokens/min of redundant prefill
With APC (80% hit rate): 20 req/min × 512 = 10,240 tokens/min of prefill → a 75% reduction

The TTFT of those 80 requests falls to whatever it costs to process only the new suffix.

Limitation with chunked prefill: when chunked prefill is active, only the first chunk of the prefill benefits from APC in the current vLLM implementation. For workloads where the cache hit rate is very high and the suffixes are short, consider lowering --max-num-batched-tokens so that the first chunk covers more of the shared prefix.

# Configuration optimised for a high prefix cache hit rate
vllm serve my-model \
 --enable-prefix-caching \
 --enable-chunked-prefill \
 --max-num-batched-tokens 4096 # smaller chunks = prefix fits in chunk 1

3. FP8 KV cache — `--kv-cache-dtype fp8`

The KV cache takes up VRAM. The more VRAM it consumes, the fewer concurrent requests you can keep in flight. On a 24 GB RTX 4090, the Qwen2.5-14B model in BF16 already takes ~28 GB, so it does not fit. In Q4 it takes ~9 GB, leaving ~14 GB for the KV cache.

How many context tokens fit in 14 GB of BF16 KV cache for a 14B with GQA?

$$\text{KV size per token} = 2 \cdot n_{kv\_heads} \cdot d_{head} \cdot n_{layers} \cdot 2 \text{ bytes}$$

For Qwen2.5-14B: $n_{kv\_heads}=8$, $d_{head}=128$, $n_{layers}=40$, BF16 → $2 \cdot 8 \cdot 128 \cdot 40 \cdot 2 = 163{,}840$ bytes ≈ 160 KB/token.

14 GB / 160 KB ≈ 87,500 tokens of total context. With 8 users in parallel and 4,096 tokens of context each: 32,768 tokens used out of 87,500. There is headroom, but it is finite.

Moving to FP8 (1 byte instead of 2):

$$\text{KV FP8} = 80 \text{ KB/token} \implies 14 \text{ GB} / 80 \text{ KB} = 175{,}000 \text{ tokens}$$

Twice the context capacity with the same VRAM. That buys either more concurrency or longer contexts.

vllm serve my-model \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales # calibrates the scales dynamically; without this there is degradation

Warning for RTX 4090 and L40: Ada Lovelace has FP8 instructions at the CUDA level but without Hopper’s (H100’s) dedicated scaling hardware. The memory reduction is real; the compute speedup is smaller than on an H100. Do not expect the same speedup as in a Hopper datacenter. On the L40S (the variant with optimised FP8 tensor cores) the benefit is larger than on an RTX 4090.

4. Context budget — `--max-model-len`

--max-model-len defines the maximum number of tokens vLLM can handle in a single request (prompt plus generation). It is the hard limit that determines how much VRAM is reserved for the KV cache in the worst case.

On small hardware, lowering it frees VRAM for more concurrency:

# 7B model on RTX 4090, typical context of 4K but the model supports 128K
vllm serve my-model \
 --max-model-len 8192 \  # instead of 131072
 --gpu-memory-utilization 0.92

With the context cut to 8,192 tokens, vLLM does not reserve KV cache for 131,072 potential tokens and can fit more simultaneous requests. The risk is obvious: requests above 8,192 tokens fail with an error. Tune it to the P99 of your real length distribution.

Interaction between parameters

The four parameters are not independent. A common mistake is enabling prefix caching without tuning the block size, or raising max-num-batched-tokens without checking that max-num-seqs allows it to be filled:

max-num-batched-tokens = 8192
max-num-seqs = 4
average prompt = 512 tokens → 4 × 512 = 2048 prefill tokens < 8192
The 8192 budget is never filled because max-num-seqs caps it first.
Fix: raise max-num-seqs or lower max-num-batched-tokens.

A balanced configuration for RTX 4090 + a 7B model:

vllm serve Qwen/Qwen2.5-7B-Instruct \
 --gpu-memory-utilization 0.92 \
 --max-model-len 8192 \
 --enable-chunked-prefill \
 --max-num-batched-tokens 8192 \
 --max-num-seqs 64 \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales

Configuration for an L40 (48 GB) + a 14B model:

vllm serve Qwen/Qwen2.5-14B-Instruct-AWQ \
 --gpu-memory-utilization 0.90 \
 --max-model-len 16384 \
 --enable-chunked-prefill \
 --max-num-batched-tokens 16384 \
 --max-num-seqs 128 \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales

How to measure that it is working

The metrics that confirm prefill is optimised:

# In vLLM's Prometheus metrics (port 8000/metrics):
vllm:time_to_first_token_seconds_bucket → TTFT distribution
vllm:gpu_cache_usage_perc → KV cache utilisation
vllm:prefix_cache_hit_rate → APC hit rate (if enabled)
vllm:num_running_seqs → simultaneous in-flight requests

A prefix_cache_hit_rate below 30% on workloads with a fixed system prompt means something in the hash is not working (a system prompt that varies by timestamp, a date format in the prompt, and so on).

Implications for on-premise inference

Chunked prefill and prefix caching are zero cost: they are enabled with flags and require no extra hardware. FP8 KV cache requires the model to be compatible (almost every modern transformer is) and that you are on Ada Lovelace or newer.

For sovereign ENS deployments where the hardware is fixed and you cannot scale horizontally on demand, a well-configured prefill is the difference between needing 4 nodes and needing 2 for the same load.

The second article in this series covers the decode optimisations: speculative decoding, KV cache tuning to maximise concurrency and how to configure gpu-memory-utilization without vLLM running out of VRAM at midnight.

References

Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023
Sarathi-Serve: Chunked Prefill and Stall-Free Scheduling — Agrawal et al., 2024
vLLM Optimization and Tuning — documentación oficial
Quantized KV Cache en vLLM

Pruning LLM models: cutting without amputating

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

A 7B-parameter model has tens of billions of neural connections. Many of them contribute so little that you could remove them without any reasonable benchmark noticing. That is pruning: identifying the irrelevant weights and deleting them to obtain a model that is smaller, faster or lighter on memory. Modern techniques (SparseGPT, Wanda, 2:4 structured sparsity) do this without retraining, in a few GPU hours, and with less than 1 point of perplexity penalty. It does not replace quantisation; it combines with it.

The analogy

An oak tree with a hundred branches. When winter comes, the tree prunes its weak branches: it redirects resources towards the main trunks. An expert pruner does not cut at random, they look at which branches have little foliage, which are dry, which grow in the wrong direction, and cut only those.

A language model is that tree. Its “branches” are the weights connecting neurons. After training, many of those connections are vestiges of the optimisation process: they existed so the gradient could descend smoothly, but in production they barely change the output. The pruner that removes them precisely is SparseGPT or Wanda. The one that cuts at random is magnitude pruning without calibration. Both give you a smaller tree; only the expert gives you one that still bears the same fruit.

What pruning really is

A transformer language model stores its knowledge in weight matrices. An attention layer has four matrices: $W_Q, W_K, W_V, W_O$. An FFN layer has at least two ($W_{up}, W_{down}$, plus $W_{gate}$ in SwiGLU). For a 7B model with 32 layers, the number of individual parameters exceeds 7 billion.

Pruning is the process of setting a subset of those parameters to zero so that:

The resulting model takes up less memory (if stored in a sparse format) or computes fewer operations.
Answer quality does not drop appreciably.

There are two classification dimensions that matter:

Granularity: which unit gets removed.

Unstructured pruning: individual weights, scattered across the whole matrix. High compression, hard to accelerate on conventional hardware.
Structured pruning: whole attention heads, entire FFN neurons, or complete layers. Less compression, but the resulting model is dense and compatible with any hardware.
Semi-structured N:M: for each group of M consecutive weights, exactly N are zero. The 2:4 case (2 zeros out of every 4) is the one supported by the Tensor Cores in NVIDIA Ampere and later.

Timing: when the removal happens.

Post-training (weight PTQ): requires no gradient, the standard in large LLMs.
During training (gradual/iterative): more precise, incompatible with 70B+ models on cost grounds.

Why so many redundant weights exist

The answer lies in how the models are trained. Stochastic gradient descent with millions of steps and a decaying learning rate produces networks that are over-parameterised by design: the extra parameters do not represent additional knowledge, but room for manoeuvre so that optimisation converges more easily.

The Lottery Ticket Hypothesis (Frankle & Carlin, ICLR 2019) formalised this intuition: inside any trained dense network there is a subnetwork that, trained from scratch in isolation, reaches the same quality. The original network is that subnetwork wrapped in parametric noise generated by the training process.

For LLMs, the empirical evidence is consistent: 7B–70B models tolerate up to 50% unstructured sparsity with no observable degradation on conversational tasks. In larger models, the tolerance threshold rises.

The maths that matter

Which weights are safe to remove?

Magnitude pruning: the naive criterion

$$\text{importance}(w_{ij}) = |w_{ij}|$$

The weights with the smallest absolute value are removed. Intuitive, but incomplete: a small weight connected to a very large activation still contributes significantly to the output.

Wanda: magnitude × activation

$$\text{importance}(w_{ij}) = |w_{ij}| \cdot \|x_j\|_2$$

Where $x_j$ is the input activation vector corresponding to weight $j$, computed over a calibration dataset of ~128 samples. The product captures both dimensions: a weight is safe to remove only if it is small and its input neuron is barely active.

Numerical example:

Weight A: $|w| = 0.001$, $\|x\|_2 = 500$ → importance = 0.5
Weight B: $|w| = 0.01$, $\|x\|_2 = 10$ → importance = 0.1

Magnitude pruning would remove A (smaller absolute value). Wanda removes B (lower importance). B is the safer one to delete.

Wanda requires neither gradients nor Hessian matrix inverses. It runs in minutes on a 70B model on a single GPU. On WikiText-2 perplexity benchmarks at 50% unstructured sparsity, Wanda reaches results comparable to SparseGPT at 10–100× less computational cost.

SparseGPT: Hessian compensation

SparseGPT applies the same mathematical framework as GPTQ (layer-by-layer quantisation), but for pruning. When it removes a weight $w_p$, it computes a correction $\delta w$ over the remaining weights in the same row to minimise the change in the layer’s output:

$$\min_{\delta w} \|W x - (W + \delta W) x\|_2^2 \quad \text{s.t.} \quad w_p + \delta w_p = 0$$

The solution uses the inverse of the second-order Hessian matrix $H = X X^T$. The extra cost is justified by the higher precision when target sparsity is high (>70%) or the model is small (<7B, where redundancy is lower).

Method	Criterion	Cost	50% sparsity (7B, WikiText-2 ppl)
Magnitude	\|w\|	Instant	+2–5 points
Wanda	\|w\| · \|x\|	Minutes	~+0.5 points
SparseGPT	Hessian	1–4h GPU	~+0.4 points

2:4 Structured Sparsity: NVIDIA’s special case

NVIDIA Ampere (A100) and later (H100, Ada Lovelace) include dedicated hardware for the 2:4 pattern: exactly 2 out of every 4 consecutive weights are zero. This lets the hardware skip the multiplications by zero efficiently, obtaining up to 2× speedup in matmul on models with 2:4 weights.

The constraint is that the sparsity has to be exactly 2:4, not an arbitrary pattern. NVIDIA’s tools (APEX Sparse, cuSPARSELt) and frameworks such as PyTorch 2.x support this natively:

from torch.sparse import to_sparse_semi_structured, SparseSemiStructuredTensor

# Convert dense weights to 2:4 sparse
sparse_weight = to_sparse_semi_structured(dense_weight)
# Forward pass automatically uses sparse tensor cores
output = F.linear(input, sparse_weight)

What to expect in practice with 2:4:

RTX 4090 (Ada Lovelace): supports 2:4 sparse tensor cores for FP16/BF16. Theoretical speedup 2×, real 1.3–1.6× depending on batch and sequence size.
H100 (Hopper): the same, with additional improvements in FP8 + 2:4 combined.
A100: supported, without FP8.
Consumer GPUs earlier than Ada (3090, etc.): no hardware support. 2:4 sparsity gives a smaller model on disk but does not accelerate inference.

Structured pruning: removing whole heads and layers

Attention head pruning

A 32-layer transformer with 32 heads per layer has 1,024 attention heads. Systematic studies on Llama-2 and Qwen models show that between 20–40% of the heads have marginal influence on the final output: their output can be set to zero without the benchmark changing within the margin of error.

The most used metric is Taylor importance: the product of the gradient of the loss with respect to the head’s output and the value of that output, summed over a calibration dataset:

$$\text{I}_{head} = \left| \sum_t \frac{\partial \mathcal{L}}{\partial o_t} \cdot o_t \right|$$

The heads with the lowest $I_{head}$ are removed first. After removing 25% of the heads in Llama-3-8B, MMLU degradation is <1% and attention inference time falls ~20% because the attention matmuls are smaller.

Layer dropping: the most aggressive shortcut

Removing a whole transformer layer deletes its attention block and its FFN. The most robust criterion is Block Influence (BI), introduced in ShortGPT (2024):

$$\text{BI}(l) = 1 - \cos(\text{input}_l, \text{output}_l)$$

A layer whose output is almost identical to its input (cosine close to 1, BI close to 0) acts as an identity function: removing it does not change the flow of information. The layers in the middle of the transformer tend to have lower BI than the initial and final layers.

Numerical example on LLaMA-2-70B:

Layers 0–5 (early): BI > 0.3 → do not remove
Layers 20–45 (mid): BI < 0.05 → candidates for removal
Layers 76–80 (final): BI > 0.2 → do not remove

Removing 8 layers out of 80 (10%): the model goes from ~140 GB to ~126 GB in BF16. Inference speedup: ~10% (proportional to the number of layers removed). Degradation on reasoning benchmarks: 1–3%.

Implications for on-premise inference

Unstructured pruning (50% sparsity) produces models with the same number of parameters but half of them at zero. Without specialised sparse kernels, that gives no speedup: the GPU still runs the multiplications, it just multiplies by zero very efficiently. The real benefit is storage and transfer (the model takes up less on disk and in system RAM).

With 2:4 structured sparsity on Ada/Hopper hardware, the speedup is real but moderate (1.3–1.7×) and requires additional tooling (cuSPARSELt or PyTorch sparse).

Structured pruning (heads, layers) does accelerate on any hardware because it reduces the model’s real size. It is the right option if the goal is throughput on hardware without sparse tensor cores.

Combining with quantisation: pruning and quantisation are orthogonal. A 50% sparse model at INT4 takes up roughly an eighth of the original in FP32. It is the end point of many aggressive compression pipelines for edge inference.

Applied to generic on-premise hardware

RTX 4090 (24 GB, Ada Lovelace)

Supports 2:4 sparse tensor cores for FP16/BF16. With Wanda + 2:4 sparsity on a Qwen2.5-14B:

# Pruning pipeline: Wanda 2:4 + INT4 quantisation
# 1. Run Wanda with calibration over 128 samples
python wanda/main.py \
 --model Qwen/Qwen2.5-14B \
 --sparsity_ratio 0.5 \
 --sparsity_type 2:4 \
 --save pruned_model/

# 2. Quantise the pruned model (optional but complementary)
python -m awq.entry --model_path pruned_model/ \
 --w_bit 4 --output_path pruned_awq_model/

Expected result: ~13 GB BF16 → ~6.5 GB after 2:4 pruning in sparse format → ~3.2 GB with AWQ INT4. The 14B model will fit in the RTX 4090 with room for KV cache.

4× H100 SXM (320 GB total, Hopper)

On this hardware structured pruning (layer dropping) makes more sense than 2:4 for high-concurrency inference: you reduce the number of FLOPs per token proportionally, which benefits throughput at large batch where the bottleneck is compute, not memory.

# Apply layer dropping with the ShortGPT BI metric
from shortgpt import compute_block_influence, drop_layers

bi_scores = compute_block_influence(model, calibration_data)
# Remove the 15% of layers with the lowest BI
model = drop_layers(model, bi_scores, drop_ratio=0.15)

A Llama-3-70B pruned by 15% of its layers fits in 3 H100s instead of 4, freeing a GPU for another task.

References

SparseGPT: Massive Language Models Can be Accurately Pruned in One Shot — Frantar & Alistarh, 2023
A Simple and Effective Pruning Approach for Large Language Models (Wanda) — Sun et al., ICLR 2024
The Lottery Ticket Hypothesis — Frankle & Carlin, ICLR 2019
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect — Men et al., 2024
NVIDIA 2:4 Sparsity in PyTorch — PyTorch Blog
SparseForge: Efficient Semi-Structured LLM Sparsification — 2025

Anatomy of an on-premise LLM inference stack: the seven layers that have to hold each other up

Sat, 30 May 2026 16:00:00 +0200

TL;DR

The on-premise LLM inference stack is not a model server: it is a building of seven layers that hold each other up. The inference layer (vLLM / SGLang) serves tokens, but without a dedicated embeddings layer RAG does not work; without the gateway layer the client couples its SDK to the specific engine; without an LLM-aware observability layer (Langfuse + OpenTelemetry GenAI) any quality degradation goes unnoticed; without a GitOps control plane layer (Flux + Forgejo) any manual change leaves invisible debt; and without a dependency tracking layer (Hubble + intent-based policies) decommissioning a Service silently breaks applications nobody remembers were using it. This post is born from a concrete incident: a pipeline that reported status: completed and matched_jobs: 0 for days because it kept calling an Ollama that had already been scaled to zero, while a badly written except labelled the generic ConnectError as “ChromaDB indexing error” and the vector store, innocent, took the blame. Each of the three symptoms was the cry of a layer that was missing or badly designed. The body of the post walks the seven layers with their canonical OSS piece, the design decisions that break them, the sizing maths on a generic 4×H100 SXM cluster (320 GB of VRAM, NVLink) and a final diagram of the connected stack. The thesis: a stack that passes the incident test is not measured by its peak throughput, it is measured by how long it takes to shout when something drifts from the design.

You are here: the seven layers seen from above

Before going into detail, the map. The seven layers are not seven servers: they are seven responsibilities the stack has to cover. A layer can collapse into one pod (gateway) or spread across several components (observability = traces + metrics + flow logs). What it cannot do is be missing.

Layers 1–4 sit on the request path: if they fall, the client notices in seconds. Layers 5–7 sit on the path of design and operations: if they fall, there is no immediately visible error, and that is why they are the ones that produce silent incidents, which is how most serious incidents end up. This post argues that the quality of the stack is measured in layers 5, 6 and 7, because 1–4 are commodities where everyone picks roughly the same pieces.

The analogy: the office building with shared services

Picture a twelve-storey office building with several tenants. It has a concierge desk (gateway: filters who comes in), it has lifts (LLM inference: they move the heavy load), it has stairs and a goods lift (embeddings: light but constant traffic, hardly anyone notices them until they break), it has plumbing and water tanks (vector store: what actually holds the state), it has a distribution board and sensors (observability: what warns you when something is drawing more than planned), it has a building manager (GitOps control plane: the only legitimate authority to move anything), and it has a tenant register (dependency tracking: who is connected to which shared service).

When a new client moves into the third floor and asks to install a server that needs more amperage than planned, the problem is not electrical: it is administrative. If the client can go straight to the board and plug in whatever they like, the building survives for a while and then a breaker trips at three in the morning. If the client has to go through the manager, the manager consults the tenant register (is anyone else hanging off that same circuit?), reviews the electrical planning (are we at the limit?) and either authorises or redirects. The building stays standing not because of its electrical installation but because of the discipline of going through the manager.

The LLM inference stack works identically. The physical layers (1–4) are the visible ones, the ones marketing people put on the slide. The governance layers (5–7) are the ones that distinguish a platform from a pile of pods with good luck.

Now to the incident that motivates the whole post.

The hook: the log that lied for six days

The application is called jobhunter internally. It is a cron pipeline that every six hours sweeps public job-ad sources, filters by EU geography, embeds the new ads, indexes them in a vector store and matches them against search profiles. The last step fires notifications.

For six days the pipeline reported the same thing on every run:

status: completed
total_found: 756
new: 23
matched_jobs: 0 ← zero, run after run

And in the logs:

[INFO] httpx: POST chromadb:8000/api/v2/.../collections/<id>/delete → 200 OK
[INFO] matcher: Purged 12 expired jobs from ChromaDB
[ERROR] pipeline: ChromaDB indexing error: All connection attempts failed
[INFO] httpx: POST chromadb:8000/api/v2/.../collections/<id>/get → 200 OK
[ERROR] pipeline: Matching error: All connection attempts failed

This is a log that invites you to blame ChromaDB. And in fact the first post-mortem written internally pointed at a version incompatibility between the Python client and the v2 server. A reasonable hypothesis, technically plausible, completely false.

The real cause: weeks earlier the general LLM on the platform had been migrated from Ollama to vLLM. The migration was clean for the two applications that depended directly on the large model, their manifests were pointed at the new endpoint. What nobody did was look at who else was calling the Service ollama.ollama.svc:11434. jobhunter called it to generate the embeddings for the ads. When the Ollama deployment was scaled to zero, the Service was left empty, and any outbound connection received a generic ConnectError("All connection attempts failed") from httpcore. The try/except wrapping the whole matching stage caught the exception and labelled it “ChromaDB indexing error”, because that was the lexical wrapper of the block, not because ChromaDB had anything to do with it. ChromaDB was answering 200 to delete and get in those same logs.

Three factors kept the incident alive for six days:

The except named the wrapper rather than the stage that failed. The log said “ChromaDB indexing error” when the error was an embeddings call against a nonexistent Service.
The pipeline returned status: completed even when there were errors. No alerting based on status fired. The metric that would have fired (matched_jobs stuck at zero) was not instrumented.
The pipeline image ran as :latest with no versioning, no SBOM, no reproducibility. When it started failing, it was impossible to know which Ollama client it had been built with.

Each of the three symptoms is the cry of a layer that was missing. The first asks for LLM-aware observability that distinguishes stages (layer 5). The second asks for the pipeline logic and the Prometheus metrics to behave like contracts (layer 5, SLI/SLO dimension). The third asks for GitOps with pinned images and an auditable SBOM (layer 6). And the whole incident, decommissioning a Service without knowing who consumes it, shouts dependency tracking (layer 7).

The rest of the post walks the seven layers through that lens: what each one solves, which OSS piece implements it in 2026, and which typical design decision breaks it.

Layer 1 — Gateway: the client SDK must not couple to the engine

What it has to solve. That the client sends POST /v1/chat/completions with the standard OpenAI SDK and never learns which engine (vLLM, SGLang, TensorRT-LLM), which specific model, which LoRA adapter or even which GPU pool is serving the request. Authentication, rate limit, routing by body.model and by tenant, header injection for tracing.

Canonical OSS piece in 2026. Envoy AI Gateway (Envoy with GenAI extensions: routing by model, token-based rate limit, fallback chains) or Cilium Gateway API with your own filters. For cases where the client wants multi-provider without distinguishing on-prem from SaaS, LiteLLM Proxy is the lightweight equivalent.

What breaks the layer. Exposing the inference engine endpoint directly. If clients call http://vllm-prod.svc:8000, any change of engine, model or pool forces you to touch the code of every app. The rule: the engine changes, the contract does not. The OpenAI SDK is standard; the routing behind the gateway is where design freedom lives.

Where jobhunter failed. There was no gateway. The app called ollama.ollama.svc:11434 directly. When Ollama died, there was no intermediate layer that could answer with a fallback, a retry against another pool, or at least a 503 error with a descriptive body.

Layer 2 — LLM inference: vLLM as the default choice, SGLang when prefix caching rules

What it has to solve. Serving tokens with throughput and latency under control: continuous batching, PagedAttention so the KV cache does not fragment the VRAM, FP8 so a 32B model fits with room to spare, multi-LoRA for per-tenant customisation without replicating the base, structured output for function calling with schema guarantees.

Canonical OSS piece in 2026. vLLM is the default choice: it covers the state of the art (PagedAttention, continuous batching, FlashAttention, FP8 quantisation, multi-LoRA, structured output, speculative decoding). SGLang comes in when the workload has high prefix caching (long chat with large system prompts, agents with repeated instructions), because its RadixAttention composes better than standard prefix caching. For very specialised inference with proprietary NVIDIA kernels, TensorRT-LLM, accepting the hardware lock-in.

What breaks the layer. Assuming a single model “does everything”. A 32B chat LLM does not serve /v1/embeddings; if you try, vLLM answers BadRequestError: "The model does not support Embeddings API". Assuming it did was one of the concrete wounds of the incident: the app expected an endpoint the model did not implement.

The design decision that costs more later. Serving the LLM with aggressive quantisation (INT4) without a quality eval calibrated for your corpus. INT4 with AWQ or GPTQ saves VRAM, but it degrades answers in technical or legal Spanish in a measurable way. The rule: any quantisation change goes through the same golden eval as a model change.

Layer 3 — Embeddings: separate from the LLM, fixed dimension, a life of its own

What it has to solve. Generating dense vectors for the RAG corpus and for retrieval queries. The embeddings model is a different thing from the chat LLM: different architecture (encoder, not decoder), different size (hundreds of millions of parameters, not tens of thousands of millions), different API (an /embeddings endpoint that takes text and returns a fixed-dimension vector).

Canonical OSS piece in 2026. Infinity or Hugging Face Text Embeddings Inference (TEI) to serve models of the bge-*, multilingual-e5-*, nomic-embed-* families with high throughput and multi-model support. OpenVINO Model Server when Intel hardware is available. sentence-transformers as a fallback embedded in the application itself when the corpus is small and the deployment is constrained.

What breaks the layer. Treating it as “the LLM does that too”. It does not if it is chat-only; and even when it does (some models have a dual endpoint), mixing chat and embeddings serving in the same process punishes the throughput of both. The physical separation is the design.

The technical fact people forget. The vector store and the embeddings model form an indivisible unit. Changing the model from multilingual-e5-large (1024 dim) to multilingual-e5-small (384 dim) is not a substitution: it is creating a new collection (my_corpus_v2) and re-embedding the whole corpus. If you upsert into the old collection, you hit a dim mismatch at runtime that takes the pod down. This seems obvious and is violated constantly, because the embeddings model is chosen once and then forgotten.

Layer 4 — Vector store + relational data + storage: what actually holds the state

What it has to solve. Persisting the vectors with efficient filters (tenant_id, created_at, source), persisting the relational metadata (users, configs, versioned prompts, traces), and persisting the model weights, the LoRA adapters, the datasets and the original corpora.

Canonical OSS pieces in 2026.

Qdrant for large collections (>200 k vectors), payload-aware filters, multi-tenancy via collection or via field.
pgvector for small collections with mandatory relational joins (wanting to do WHERE doc.author = ... AND vector <=> $1 in the same SQL).
PostgreSQL operated by CloudNativePG (CNPG) for the relational side: Barman Cloud backups, replication, connection via pooler.
MinIO for S3-compatible objects: bucket per tenant, cross-site replication, weights and adapters versioned by sha256.
Redis for queues, rate-limit counters and a cache of frequent retrievals.

What breaks the layer. Assuming the vector store is stateless ephemeral. It is exactly the opposite: it is the component where losing state costs the most. Without verified backups of the vector store, an index corruption forces you to re-embed the whole corpus, and on a corpus of millions of documents that is hours or days of GPU.

The design decision that pays later. Forgetting to version the collection by dimension and embeddings model. Suggested convention: my_corpus__embed-multie5l__1024d__v3. The name carries metadata; any change in any of the three attributes forces a new collection. It is ugly but it protects against an accidental upsert with the wrong dim.

Layer 5 — Observability: LLM-aware traces + infrastructure metrics + flow logs

What it has to solve. That any inference can be recovered from its trace_id, with all the gen_ai.* attributes (semantic conventions) plus your own attributes (tenant_id, adapter_id, priority_tier), a breakdown of latency (queue → prefill → decode → network), tokens consumed, tools invoked, exact model and adapter. And in parallel: Prometheus metrics from vLLM (vllm:num_requests_running, vllm:gpu_cache_usage_perc, vllm:prefix_cache_hit_rate), from the GPU (DCGM Exporter), from the network (Hubble flow logs with drops and NetworkPolicy enforcement).

Canonical OSS pieces in 2026.

OpenTelemetry Collector as the single transport for traces, metrics and logs, with OTLP receivers and separate exporters per destination.
Langfuse self-hosted for the LLM-aware side: tracing, prompt versioning, evals with LLM-as-judge (post and post).
VictoriaMetrics + Grafana for high-throughput TSDB metrics with long retention.
Hubble (Cilium) for L3/L4/L7 flow logs and NetworkPolicy visualisation.
DCGM Exporter for GPU metrics.

What breaks the layer. Wrapping the whole pipeline stage in a single try/except that labels the exception with the wrapper’s name. The operational rule: one try/except per stage, with the stage label in the message and a Prometheus metric with labels={"stage": "<name>"}. That way “ChromaDB indexing error” would never have been the log for an embeddings failure; it would have been “Embeddings call failed: ConnectError(ollama:11434)”.

Complementary rule. The pipeline returns status: completed if and only if there were no errors. With errors it returns completed_with_errors or failed, and the metric pipeline_errors_total{stage} is incremented. An alert based on increase(pipeline_errors_total[1h]) > 0 fires before the second failed run. Without this discipline, observability exists but does not warn.

Layer 6 — GitOps control plane: the only legitimate authority

What it has to solve. That the cluster state is the state declared in git. That any divergence between git and the cluster is visible and, in critical components, auto-reconciled or auto-alerted. That every deployed image has an immutable tag (sha digest or semver pin), an SBOM (Trivy) and traceability back to the commit that introduced it.

Canonical OSS pieces in 2026.

Flux (or ArgoCD) as the reconciler.
Forgejo (or Gitea, GitLab CE) as the self-hosted OSS forge.
cert-manager + Trust Manager for internal PKI.
External Secrets Operator + SOPS for encrypted versioned secrets.
Kyverno (or OPA Gatekeeper) for binding policies: deny :latest images, deny pods without a NetworkPolicy, deny Services without an owner label.
Trivy for SBOM and vulnerability scanning of images in the CI pipeline.

What breaks the layer. Images with a mutable tag (:latest, :main). Any kubectl edit in production that is not reflected in git. main branches with write permissions for humans without review. The rule: if a human can mutate the cluster without going through a signed commit, you do not have GitOps, you have somebody’s whiteboard.

How it applies to the incident. If the pipeline image had been pinned to a sha digest (registry.interno.local/jobhunter@sha256:9af2...), the team could have audited immediately which Ollama client it carried. With :latest, not even that.

Layer 7 — Dependency tracking: the layer the incident exposed

What it has to solve. Knowing who calls which Service, both declaratively (what the gitops repo says) and observed (what has been seen crossing the network over the last N days). And, on mature platforms, propagating that information as policy: if nobody declares and nobody observes traffic to the Service ollama.ollama.svc, decommissioning it is safe; if somebody declares it or uses it, decommissioning it opens a ticket.

Canonical OSS pieces in 2026.

Hubble (Cilium) for observed flow logs: hubble observe --to-namespace ollama --since 14d gives the list of source namespaces that have talked to Ollama in the last two weeks.
Otterize for intent-based policy: each Deployment declares “I need to talk to ollama-svc”, and the operator generates the corresponding NetworkPolicy and maintains a browsable catalogue of who is trying to talk to what.
manual kubectl-grep as a fallback: kubectl get deployments,cronjobs,statefulsets -A -o yaml | grep -E 'ollama[.-]' produces the declarative list.
NetworkPolicy as code reviewed in CI: every PR that touches a Service requires the associated policy to be kept or explicitly updated.

A pre-decom checklist that the incident suggests codifying as a CI hook:

SVC="ollama.ollama.svc.cluster.local"

# (a) declarative grep in the gitops repo
git -C $GITOPS_REPO grep -l "$SVC" || echo "OK declarative"

# (b) observed grep in Hubble (last 14 days)
hubble observe --to-fqdn "$SVC" --since 336h --output json \
 | jq -r '.source.namespace' | sort -u || echo "OK observed"

# (c) live grep in the cluster
kubectl get all -A -o yaml | grep -E "$SVC" || echo "OK live"

If all three return empty, the decom is safe. If any of them has content, there is downstream debt still open. The jobhunter incident is exactly what happens when this check does not exist: the team that decommissioned Ollama looked at the list of applications it knew depended on it directly; nobody looked at the list of those that depended on it silently.

The maths that matter: sizing the stack on 4×H100 SXM (320 GB)

The generic reference cluster for everything that follows: 4×H100 SXM 80 GB, NVLink between the four, 640 GB of system RAM, 2×NVMe NVMe-oF for local storage of weights and caches, redundant 25/100 GbE to the switch. Aggregate VRAM: 320 GB.

The VRAM budget is not free. A first reasonable split for a stack that serves one large general LLM, a medium code-specialised model, embeddings and a reranker, with room for multi-LoRA and for the KV cache:

Component	Reference model	Quant	Model weights	Reserved KV cache	Total VRAM
General LLM (TP=4)	70B-instruct	FP8 W8A8	70 GB	60 GB	130 GB
Code LLM (TP=2)	32B-coder	FP8 W8A8	32 GB	28 GB	60 GB
Embeddings	multilingual-e5-large	FP16	1.3 GB	n/a	8 GB (×2 replicas)
Reranker	bge-reranker-v2-m3	FP16	0.6 GB	n/a	4 GB
Multi-LoRA pool (over the general LLM)	up to 16 adapters	bf16	16 × 0.4 GB ≈ 6 GB	reuses the LLM’s KV	6 GB
Reserved for fragmentation + overhead					~30 GB
Total committed					~238 GB / 320 GB
Free headroom					~82 GB

The 26 % of free headroom is not waste: it is what lets the vLLM scheduler avoid preempting requests under moderate pressure, lets continuous batching group large batches without aborting, and lets a failover from the other site promote a standby without OOM.

Expected throughput, with the 70B general LLM in FP8 and tensor parallel 4, on an H100 SXM with continuous batching active and prefix caching at 35–55 % (typical in multi-turn chat with shared system prompts):

$$ \text{aggregate tokens/second} \approx 1500 \text{ to } 2500 $$

for concurrency between 32 and 64 requests, with sub-second TTFT P95 on short prompts (<2k tokens) and TPOT P95 around 40–60 ms/token as perceived by the client. These numbers are reasonable orders of magnitude, not guarantees: real throughput depends on the prompt mix, on whether speculative decoding (EAGLE-3) is active and burnt-in, and on the network cost between gateway and inference pods.

Embeddings throughput on two replicas of multilingual-e5-large with dynamic batching:

$$ \text{embeddings/second} \approx 3000 \text{ to } 6000 \quad (\text{optimal batch} \sim 64) $$

Enough to reindex a corpus of 1 million documents in roughly an hour, assuming chunks of 512 tokens and two chunks per document on average. For corpora of tens of millions of documents, re-embedding is done by delta via CDC over the source (covered in RAG corpus curation), not by full sweep.

Retrieval latency on Qdrant with HNSW (M=16, ef_construct=200) over a collection of 5 million 1024-dim vectors filtered by tenant_id:

$$ \text{P95 latency retrieve top-50} \approx 8 \text{ to } 25 \text{ ms} $$

Below the cost of cross-encoder reranking (bge-reranker-v2-m3 over top-50 = roughly 30–60 ms more), and below any call to the LLM. The bottleneck in a well sized RAG pipeline is never the vector store: it is the LLM’s decoding.

Final diagram: the complete connected stack

The solid lines are the request path: client → gateway → inference engine → vector store/embeddings → response. The dashed blue lines are telemetry: each component emits OTel to the collector, which routes traces to Langfuse, metrics to VictoriaMetrics and logs to Loki. The dashed red lines are reconciliation: the GitOps control plane keeps any layer in its declared state and warns of divergence.

The diagram is not decorative: every arrow is a stable contract between two layers. If one layer changes (vLLM → SGLang, multilingual-e5 → bge-m3, Qdrant → pgvector), the arrows hold. That stability of contracts is the architectural property that lets a team migrate components without breaking downstream apps.

Typical design decisions that break the stack

A short list of mistakes seen repeatedly in stacks that looked well designed on paper:

1. Coupling the client SDK to the inference engine. Dropping the gateway because “vLLM already speaks OpenAI-compatible” works on day one and hurts the day you have to add a fallback, a canary or a second model.

2. Sharing the LLM and embeddings endpoint. A qwen2.5-32b-Instruct is chat-only; BadRequestError: "The model does not support Embeddings API" is the cry of a design that confused the two layers.

3. Reusing the vector store collection when changing the embeddings model. Different dimensions do not allow upsert. Versioning the collection by (model, dim, version) is ugly but it saves the day of the change.

4. A try/except wrapping an entire pipeline with the wrapper’s label. The log lies because the label is lexical, not causal. Every stage in its own try/except with its own label and its own metric.

5. status: completed with errors. The pipeline has to distinguish completed, completed_with_errors and failed, and the alerting has to fire on the last two. Without this, observability exists in theory and does not warn in practice.

6. Images with a mutable tag. :latest and :main are not tags, they are aliases. Without a sha digest there is no reproducibility and no auditable SBOM.

7. Decommissioning a Service without a pre-decom check. The three-grep check (declarative + observed + live) takes two minutes and costs six days of incident when it is skipped.

8. Default limits.memory on pods that load models. A sidecar loading sentence-transformers + torch + tokenizer needs 2–4 GB; with limits.memory: 1Gi you hit OOM on the first pod restart, and sometimes with no alert because the liveness probe answers through another route.

All of them are variants of the same principle: the stack does not fail in its most expensive layer (inference, where nobody underestimates the cost), it fails in the cheap, boring layers (gateway, observability, GitOps, dependency tracking) where it is tempting to save.

Applied to typical on-premise hardware: a 4×H100 SXM cluster

On the generic reference cluster (4×H100 SXM 80 GB, NVLink, 640 GB RAM), the suggested split into pods:

gpu-node-01 (4×H100 SXM, intra-node NVLink)
├── vllm-llm-general (TP=4) ~130 GB VRAM (4 GPUs)
└── (shares GPUs with the multi-LoRA pool on the same deployment)

gpu-node-02 (4×H100 SXM, second node)
├── vllm-llm-code (TP=2) ~60 GB VRAM (2 GPUs)
├── infinity-embeddings (×2) ~16 GB VRAM (shared on 1 GPU with optional MIG)
├── tei-reranker ~4 GB VRAM (cohabitant)
└── failover reserve ~120 GB VRAM free for canary / standby

Not everything fits comfortably on a single node; two nodes with two H100 SXM each would be enough for the conservative setup, and the rest of the cluster (CPU-bound: gateway, vector store, observability, control plane) runs on GPU-free nodes.

The operational rule: inference concentrates, the rest of the stack distributes. Concentrating inference maximises the use of NVLink (cross-GPU tensor parallel without going through PCIe); distributing the rest prevents an event on the GPU node from taking the control plane down with it.

An even more conservative configuration, for SMEs with a single 4×H100 SXM node as a starting point, serves the general LLM (TP=4) and embeddings/reranker cohabiting with MIG (Multi-Instance GPU to partition an H100 into hardware-isolated slices). The code LLM is deferred to a second phase. It is viable and cost-aware; what is not viable is doing without layers 5, 6 and 7.

What we have not covered (upcoming posts)

This post focuses on the static design of the stack. Some pieces still deserve an article of their own:

The multi-site active/standby plane: Cilium Cluster Mesh, cross-cluster Qdrant replication, realistic RTO/RPO, when active-active pays off and when it does not.
The continuous fine-tuning plane: how the LoRA pipeline closes the loop from production feedback to a promoted adapter, in the spirit of the retrain post.
The safety/guardrails plane: where Llama Guard fits, Presidio for PII and XGrammar for guaranteed structured output, connected to layer 1, the gateway.
The cost plane: gen_ai.usage.* instrumentation at tenant and model level, tokens/euro dashboards, GPU elasticity decisions via KEDA.
The compliance plane: how the stack maps to ENS High, NIS2 and ISO/IEC 42001 without turning the deployment into a compliance exercise that paralyses delivery.
Below the engine: the mini-series that opens the stack’s basement — the interconnect (NVLink/NCCL), the host (NUMA, hugepages, CPU isolation) and the RKE2 resource managers that pin each pod to the right NUMA node.

Each one falls into a different series on the blog and will be covered with the same discipline: a concrete piece, a justified decision, the typical mistake seen in practice.

References

The six-stage LLMOps pipeline — the general framework this stack operates in.
The OSS catalogue for LLMOps in six stages — a card per tool in the stack.
Anatomy of an LLM request in production — the same architecture seen from the perspective of an individual request.
OSS vs hyperscalers in LLMOps — a comparison with the AWS/Azure/GCP stacks.
LLM tracing with OpenTelemetry GenAI — the backbone of layer 5.
Multi-LoRA serving — the piece that gives per-tenant customisation without replicating the base.
RAG corpus curation and Reranker and hybrid retrieval — everything that goes into layers 3-4 so that RAG does not degrade.
Quantisation for LLM inference — the reason for the FP8 W8A8 in the sizing.

Relevant official documentation

vLLM Production Stack — docs.vllm.ai
SGLang RadixAttention — github.com/sgl-project/sglang
Envoy AI Gateway — aigateway.envoyproxy.io
Langfuse self-hosted — langfuse.com/docs/self-hosting
OpenTelemetry Semantic Conventions for GenAI — opentelemetry.io/docs/specs/semconv/gen-ai
Hubble flow observability — docs.cilium.io/en/stable/observability/hubble
Otterize intent-based access — docs.otterize.com
Flux GitOps toolkit — fluxcd.io

Continuous batching: the salon with 8 chairs that does not wait for the slow customer — Orca, vLLM, chunked prefill and goodput

Sat, 30 May 2026 16:00:00 +0200

This post complements those on KV cache (the artefact continuous batching manages), PagedAttention (the memory piece that makes it viable), Disaggregated serving (the next layer of optimisation), Speculative decoding, Multi-LoRA and MoE (the three extensions that coexist with the scheduler in production).

You are here: DEPLOY

TL;DR

The static batching of the original HuggingFace Transformers (the pre-2022 era) systematically under-used the GPU for two structural reasons. First: the scheduling unit was the complete request; the longest one in the batch blocked all the others until it finished (severe head-of-line blocking, P95 TTFT five to ten times worse than reasonable). Second: each batch slot reserved memory for max_seq_len even though the real output was much shorter; documented padding waste sat between 60 % and 80 % and sustained GPU SM utilisation on real workloads fell to 20-40 %. Orca (Yu et al., OSDI 2022, FriendliAI + Seoul National University) introduced the idea that unjammed everything: the scheduling unit stops being the request and becomes one decoder iteration, a single token. After each iteration the scheduler can add new requests to the batch and remove the finished ones. vLLM (Kwon et al., SOSP 2023, UC Berkeley) materialised it open-source and production-grade thanks to PagedAttention, which solves the KV cache fragmentation that theoretical continuous batching would cause with dynamic allocation. SARATHI / Sarathi-Serve (Microsoft Research India, OSDI 2024) closed the last gap: the prefill stalls that paused active decodes when a new request arrived, via chunked prefill (splitting a long prefill into small chunks and mixing them with decodes in the same step) and stall-free batching. DistServe (Zhong et al., OSDI 2024) reformulated the key metric: what matters is goodput (requests/s meeting the TTFT and TPOT SLOs), not raw throughput. In May 2026, vLLM v1 brings a unified scheduler with chunked prefill always-on; SGLang adds RadixAttention, which gives cross-request prefix-cache hits; TensorRT-LLM calls it in-flight batching; llama.cpp supports it natively. The three operational tensions are speculative decoding (nested raggedness), multi-LoRA (each request with its adapter) and MoE (each expert sees very few tokens per step at a typical batch size). This post takes apart the mechanism, the maths (GPU utilisation, goodput vs throughput), the three variants (Orca → vLLM → Sarathi-Serve), the pitfalls (preempt-on-OOM, starvation, inverse HoL) and the real numbers with production configurations.

The analogy: the salon with 8 chairs

A hair salon with 8 chairs and a single brilliant stylist who moves between them. Customers arrive with very different needs: some want a 15-minute cut, others a 2-hour colour with a base, others a 90-minute straightening. The question is how to organise the flow.

The traditional strategy (what HuggingFace Transformers did in its original generate()) is to seat 8 customers at once, all at the same time, and accept nobody new until the last one finishes. If one of those 8 is a 2-hour job, the 7 who wanted the 15-minute cut sit there doing nothing for 1 hour and 45 minutes. The stylist finishes with the quick ones and is left staring at empty chairs until the colour is done. When everyone is ready, another 8 come in. That is static batching, and the only thing that stops it from being worse is that the GPU does not complain like a human customer.

The continuous strategy (Orca, vLLM) changes the scheduling unit. The stylist does not think “I am going to do one whole customer and then the next”; he thinks “on each tick I take one step of work at every occupied chair and, every time a chair frees up, I call the next customer in the queue without waiting for the others to finish”. The quick-cut customer leaves after 15 minutes, his chair is filled immediately with the next one, and the slow jobs proceed at their own pace without delaying anyone. The stylist is never idle.

The continuous strategy with chunked prefill (SARATHI / Sarathi-Serve) adds a subtler distinction. Some customers need a long initial phase (a 10-minute hair analysis before the cut; in LLM terms, the prefill of the prompt). Without chunked prefill, the stylist had to stop all the other chairs to do the new customer’s analysis in one go, and that was a stall visible in the TPOT of the active ones. With chunked prefill, the analysis is split into 2-minute pieces interleaved between the other customers’ active cuts. The customers already under way no longer notice pauses; the new customer takes slightly longer to start his cut proper, but the whole salon does not freeze.

And the metric that matters: the owner does not want to maximise “customers served per hour” at the cost of some leaving furious. He wants to maximise “customers served per hour within the time SLA”, and that is goodput, DistServe’s contribution.

The problem continuous batching solves

There are two structural pathologies of static batching that deserve to be explained with concrete numbers.

Padding waste. Each batch slot reserved memory for max_seq_len (prompt + max output), even though the real output ended in far fewer tokens. For a batch of 32 with heterogeneously distributed output lengths (half of them ≤50 tokens, a long tail up to 4,000), the typical memory waste was 60-80 %. That translated directly into wasted concurrency: with the same VRAM, instead of serving 32 requests with smart allocation, you served 8.

HoL blocking (Head-of-Line). The scheduling unit was the complete request. A batch containing one request of 500 tokens and 31 of ≤50 tokens ran 450 extra “empty” iterations (the GPU executed forward passes for all 32 slots, even though 31 had already finished). Wasted compute cost: ~84 % of the tail time in the example.

Measurable result. Sustained GPU SM utilisation on real workloads under static batching: 20-40 %. That is, ~70 % of the datacenter’s compute went unused. When the first Orca and vLLM benchmarks came out showing 10-24× throughput improvement, it was not marketing exaggeration; it was recovering all that wasted compute.

Orca (OSDI ‘22): the idea that changed everything

The paper by Yu, Jeong, Kim, Kim and Chun at OSDI 2022 (“Orca: A Distributed Serving System for Transformer-Based Generative Models”, from Seoul National University + FriendliAI) introduced two contributions that have remained the basis of everything that followed.

Iteration-level scheduling. Instead of scheduling at the level of the complete request, it schedules at the level of one decoder iteration: the step that generates ONE token. After each iteration the scheduler can (a) add new requests to the batch, (b) remove requests that have generated EOS or reached max_tokens, (c) reorder priorities. The compute engine executes exactly one iteration over the current batch.

Selective batching. Here is the non-obvious subtlety. The technical problem with batching requests of different lengths and different KV cache states is that some operations (the GEMMs of the Q, K, V projections and the FFN) are insensitive to position and can be batched by concatenating tokens, whereas attention is sensitive to per-request state (each request has its own KV cache of a different length). Orca’s solution: batch the GEMMs (concatenate all the step’s tokens into a [total_tokens, hidden] tensor) and run attention sequentially per request.

Paper result: up to 36.9× throughput vs FasterTransformer on GPT-3 175B at the same latency level. Orca is not open-source, it is only documented in the paper. FriendliAI commercialises it as the Friendli Engine. But the idea was published and everyone adopted it.

vLLM (SOSP ‘23): the open-source materialisation

What Orca described as a concept, vLLM materialised in production. The paper by Kwon, Li, Zhuang, Sheng et al. (UC Berkeley Sky Computing Lab, SOSP 2023) introduces PagedAttention, covered in detail in the PagedAttention deep dive, but it also consolidates continuous batching as universal practice.

The reason PagedAttention is a prerequisite for practical continuous batching is fragmentation. If you are going to insert and remove requests from the batch dynamically, and each request has a KV cache that grows on every iteration, traditional contiguous allocation fragments the HBM until it is unusable. PagedAttention splits the KV cache into fixed-size blocks (16 tokens by default) allocated on demand from a global pool. Memory waste falls from ~60-80 % to less than 4 % (only the last partially filled block per sequence).

vLLM paper metrics (2023):

vs HuggingFace Transformers (static, no continuous batching): up to 24× throughput.
vs HuggingFace TGI (which already had primitive continuous batching): ~3.5×.
vs FasterTransformer: 2-4× at the same latency.

What operationally distinguishes vLLM from Orca: vLLM runs attention in a single fused CUDA kernel (paged_attention_kernel) over non-contiguous blocks; Orca described attention as sequential, request by request. And vLLM exposes OpenAI-compatible APIs that let you drop the engine into existing stacks without touching the client.

Chunked prefill (SARATHI / Sarathi-Serve, OSDI ‘24)

There is a detail the original Orca/vLLM continuous batching did not solve: when a new request enters the batch, its prefill (processing the whole prompt at once) can take hundreds of milliseconds. During that time, the active decodes of the other requests are essentially paused, the GPU being dedicated to the new prefill. This showed up as spikes in TPOT (“inter-token latency”) every time a long request came in, which broke strict SLAs.

SARATHI (Agrawal et al., arXiv 2308.16369, August 2023) and later Sarathi-Serve (same group at Microsoft Research India, OSDI 2024, arXiv 2403.02310) introduce two combined ideas:

Chunked prefill. A long prefill (8,192 tokens, say) is split into chunks (2,048 tokens, say) processed one per iteration. Instead of one 200 ms step processing 8K tokens, four 50 ms steps processing 2K each.

Decode-maximal batching (“stall-free”). On each iteration, the scheduler first fills the batch with the active decodes (each costing 1 token), and only the leftover space goes to new prefill chunks. The result: active decodes keep advancing 1 token per iteration without pausing, while the new request completes its prefill in small bites.

The observation that justifies it: prefill is compute-bound (it processes N tokens at once and saturates FLOPs) while decode is memory-bound (1 token per step, under-using compute, with the GPU waiting on HBM). Mixing prefill chunks with decodes in the same step exploits the arithmetic-intensity slack: the decodes piggyback on the free compute of the prefill chunk.

Numbers:

Original SARATHI (LLaMA-13B on an A6000): decode throughput +10×, end-to-end +1.33×.
Sarathi-Serve (Mistral-7B on an A100): 2.6× serving capacity vs plain vLLM. Yi-34B on 2×A100: 3.7×. Falcon-180B with pipeline parallel: 5.6×.

Adoption in May 2026: always-on in vLLM v1 (default since v0.8.0, January 2025), SGLang, TensorRT-LLM. The key setting in vLLM is --max-num-batched-tokens (the token budget per step; default 2048). Raising it prioritises throughput, lowering it prioritises low TPOT.

Goodput: the metric that matters (DistServe, OSDI ‘24)

The classic “throughput” metric (requests/s or tokens/s) has a problem when there are SLOs. A server can report 1,000 req/s while P99 TTFT is 30 seconds and the SLO is 1 second, so only about 200 req/s actually meet the contract.

DistServe (Zhong et al., OSDI 2024) formalises goodput as the right metric: goodput = max sustained request rate meeting the SLOs (TTFT bound AND TPOT bound). The practical definition is usually: maximum rate with ≥90 % of requests within both SLOs.

Why it matters for the scheduler:

Optimising raw throughput leads to maximising batch size, which inflates P99 TPOT.
Optimising goodput limits the batch size when TPOT starts violating the SLO, prefers small requests if the batch already has a tail, and leaves resources available for new requests.

DistServe result: up to 7.4× more requests served or a 12.6× tighter SLO vs vLLM at the same SLO attainment. The gain comes from disaggregating prefill and decode onto different GPUs (removing the interference between phases), but the idea of optimising for goodput is independent and applicable to any scheduler.

Operationally this translates into monitoring:

goodput_proxy = histogram_quantile(0.95, vllm:time_to_first_token_seconds_bucket) < SLO_TTFT
AND histogram_quantile(0.95, vllm:time_per_output_token_seconds_bucket) < SLO_TPOT

The iterative scheduler in action

The maths that matter

Three formulas explain much of the operational behaviour.

GPU utilisation under static batching. With a batch of size B whose seq_len_i are the real lengths and max(seq_len_i) is the length that defines the padding:

$$U_{\text{static}} = \frac{\sum_i \text{seq len}_i}{B \cdot \max_i \text{seq len}_i}$$

For B=32, 30 sequences of 50 tokens and 2 of 500: U = (30·50 + 2·500) / (32·500) = 2500/16000 = 15.6 %. Four out of every five GPU cycles wasted.

GPU utilisation under continuous batching (idealised).

$$U_{\text{continuous}} \approx 1 - \frac{T_{\text{scheduler}}}{T_{\text{iteration}}}$$

With scheduler overhead of ~50-200 µs and an iteration time of ~10-30 ms: U > 95 %. The unit of loss is no longer padding, it is scheduling overhead, and that is negligible compared with the forward pass.

Goodput vs throughput.

$$\text{Goodput}(R) = R \cdot P(\text{latency} < \text{SLO})$$

where R is the offered request rate. Typical curve: goodput grows linearly with R up to the saturation knee, then falls because P(SLO) collapses when the system congests. The optimal point is just before the knee, not at peak throughput.

Example: at R=100 req/s with P(SLO)=0.99, goodput = 99. At R=200 req/s with P(SLO)=0.4, goodput = 80. More offered load, less useful goodput.

Chunked prefill token budget. On each vLLM step with chunked prefill active:

$$\text{prefill tokens this step} = \text{max num batched tokens} - \text{num decodes active}$$

Each active decode costs 1 token of the budget; the rest is filled with new prefill chunks. If max_num_batched_tokens = 2048 and num_decodes_active = 200, there are 1,848 tokens for prefill (one chunk of 1,848 or several small chunks).

Real implementations in May 2026

Engine	Current scheduler V	Chunked prefill default	Relevant notes
vLLM v1 (default ≥0.8.0)	unified V1	always-on	EngineCore isolated in a separate process; prefix caching with O(1) eviction; preempt-mode `recompute` by default; xgrammar/outlines backends.
SGLang	own (PyTorch ecosystem)	yes	RadixAttention gives cross-request prefix-cache hits; non-blocking CPU scheduler; leader in stable latency at high concurrency.
TensorRT-LLM	proprietary “in-flight batching”	yes	Policies `GUARANTEED_NO_EVICT` (conservative, default) and `MAX_UTILIZATION` (aggressive, risk of a pause when the KV is full). Compile-time vs runtime.
Triton + tensorrtllm_backend	`gpt_model_type: inflight_fused_batching`	yes	`max_queue_delay_microseconds` to group newly arrived requests. Decoupled mode for SSE streaming.
llama.cpp (llama-server)	own	`--cont-batching` ON since 2024	`-np N` parallel slots; no PagedAttention (contiguous KV per slot), so less flexible but simpler. Endpoint `:8080/metrics`.

A typical production-ready vLLM v1 configuration:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
 --tensor-parallel-size 4 \
 --max-num-batched-tokens 4096 \
 --max-num-seqs 256 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --preemption-mode recompute \
 --scheduling-policy fcfs \
 --gpu-memory-utilization 0.92

SGLang equivalent:

python -m sglang.launch_server \
 --model meta-llama/Llama-3.1-70B-Instruct \
 --tp 4 \
 --chunked-prefill-size 4096 \
 --max-running-requests 256 \
 --enable-radix-cache

The three operational tensions

Continuous batching + speculative decoding. Speculative decoding produces 1 to γ+1 tokens per step depending on the acceptance rate. The batch stops being uniform in tokens produced per iteration: nested raggedness. PagedAttention absorbs it (the KV cache can grow at different rates per request in the same step), but the scheduler loses symmetry. At low QPS (a conversational assistant) the combination is excellent: vLLM reports up to 2.8× speedup. At high QPS, the draft consumes slots from the decode pool and can reduce aggregate goodput. Rule of thumb: disable speculative decoding when gpu_cache_usage > 0.85. Full detail in Speculative decoding.

Continuous batching + multi-LoRA. Each request in the batch can use a different adapter (via SGMV, see Multi-LoRA serving). Worst case: every request in the batch with a different adapter and a different rank, and throughput falls by up to 50 % vs the base with no LoRA. Best case: all requests on the same adapter, equivalent to the base with no LoRA. Practical mitigation: group adapters by rank in the routing ahead of the engine; set --max-lora-rank to the maximum actually served, not generously above it.

Continuous batching + MoE. Each expert sees batch · k / N tokens per step. With DeepSeek-V3 (256 experts, k=8) and batch=32 in decode, each expert processes only 1 token on average: total compute starvation. To match the per-GPU throughput of a dense model, MoE needs batches »10× larger, which puts pressure on the KV cache. Wide-EP (see MoE inference) distributes the experts across many GPUs and allows larger effective batches per expert, at the cost of all-to-all comms that add milliseconds per step.

Metrics to monitor

The Prometheus metrics exposed by vLLM (prefix vllm:, vllm_ when scraped):

vllm:time_to_first_token_seconds (Histogram) — TTFT including queue time.
vllm:time_per_output_token_seconds (Histogram) — TPOT.
vllm:e2e_request_latency_seconds (Histogram) — end-to-end.
vllm:num_requests_running (Gauge) — active batch.
vllm:num_requests_waiting (Gauge) — queue depth.
vllm:num_requests_swapped (Gauge) — preempted to CPU.
vllm:gpu_cache_usage_perc (Gauge) — fraction of KV cache occupied.
vllm:gpu_prefix_cache_hit_rate (Gauge) — prefix cache hits.
vllm:num_preemptions_total (Counter) — preemptions. Any sustained value is a red flag.

Practical operational rules:

Stable zone under sustained load: gpu_cache_usage_perc ∈ [0.7, 0.9].
Warning at >0.95 (preemption imminent).
Critical if num_requests_waiting grows faster than num_requests_running: the server is not absorbing the load; scale out.

Operational pitfalls

Preempt-on-OOM. When gpu_cache_usage reaches ~1.0 with pending requests that need to grow their KV, vLLM preempts. V1 does RECOMPUTE by default (discards the KV, regenerates it on return); V0 did SWAP (moves it to CPU). RECOMPUTE is better for short sequences (cheap to regenerate); SWAP is better for long ones. The vllm:num_preemptions_total metric should be zero or near zero in steady state.

Inverse HoL blocking (memory monopoly). A very long request occupies many KV blocks, so small requests do not fit in the batch even though compute is free. Chunked prefill mitigates the compute blocking during a new prefill, but it does not solve the memory monopoly. Partial solution: per-request limit policies (an aggressive max_tokens) or priorities.

Starvation. FCFS can leave requests pending for a long time if the active ones do not finish. vLLM supports --scheduling-policy priority with an x-priority header. Recent work (NeurIPS 2024 Efficient LLM Scheduling by Learning to Rank, arXiv:2501.14312 Locality-aware Fair Scheduling) proposes schedulers with quantum-based starvation prevention; not yet integrated into vLLM mainline.

Badly calibrated chunk size. A small chunk (512) gives low TPOT, high TTFT and memory overhead from more KV accesses. A large chunk (8192+) gives low TTFT and TPOT spikes during the chunk. Rule: start at 2,048, measure P95 TPOT, adjust.

Batch size cap. A high --max-num-seqs gives more concurrency but P99 TPOT explodes. A low one wastes throughput. Rule of thumb: max_num_seqs ≈ HBM_for_KV / (avg_seq_len × bytes_per_token_KV).

Implications for on-premise hardware

On an RTX 4090 (24 GB). llama.cpp with --cont-batching -np 4-8 is the natural pattern. Typical models: Llama 3 8B Q4_K_M with ~8 parallel slots, aggregate throughput on the order of hundreds of tok/s. vLLM also works if the weights fit (Llama 3 8B BF16 does; the 70B does not fit whole), although PagedAttention on consumer hardware gives less return than in the datacenter.

On a generic 4×H100 SXM cluster (320 GB, NVLink). Here vLLM v1 / SGLang are the de facto standard. Typical configurations:

Llama 3 70B FP8 + TP=4: dozens of concurrent sessions with P95 TPOT under 50 ms, tens of thousands of aggregate tok/s at a moderate batch size.
Llama 3 70B AWQ-INT4 + TP=2 plus the rest of the cluster for additional concurrency or multi-LoRA with SGMV.
DeepSeek-V3 needs larger setups (8-16 H100) to fit whole in FP8; with Wide-EP, continuous batching starts operating on much larger batches and the economics change (see MoE).

The rule of thumb for May 2026: vLLM v1 with chunked prefill always-on and prefix caching enabled is the sensible default configuration for any dense model that fits comfortably; SGLang offers better stable latency at high concurrency thanks to the overlap between the CPU scheduler and the GPU step; TensorRT-LLM gives peak throughput at high concurrency with the rigidity of compile-time.

What we have not covered

Locality-aware fair scheduling (arXiv:2501.14312) and learning-to-rank schedulers (NeurIPS 2024): the next generation of algorithms closing the fairness vs prefix-cache locality trade-off.
Smooth goodput (arXiv:2410.14257): a refinement of the DistServe metric using max slowdown instead of a binary SLO.
Triton tensorrtllm_backend in production: decoupled mode for streaming, ensembles with pre/post-processing, autoscaling with KServe.
vLLM speculators v0.3.0 and the vLLM-compatible drafter training framework.

References

Yu, G.-I., Jeong, J., Kim, G.-W., Kim, S., Chun, B.-G. Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022. https://www.usenix.org/system/files/osdi22-yu.pdf
Kwon, W. et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. https://arxiv.org/abs/2309.06180
Agrawal, A. et al. SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. 2023. https://arxiv.org/abs/2308.16369
Agrawal, A. et al. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. OSDI 2024. https://arxiv.org/abs/2403.02310
Zhong, Y. et al. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024. https://arxiv.org/abs/2401.09670
Sheng, Y. et al. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. MLSys 2024. https://arxiv.org/abs/2311.03285
Efficient LLM Scheduling by Learning to Rank. NeurIPS 2024.
Locality-aware Fair Scheduling. 2025. https://arxiv.org/abs/2501.14312
vLLM V1 alpha release (Jan 2025): https://blog.vllm.ai/2025/01/27/v1-alpha-release.html
vLLM Anatomy (Sep 2025): https://blog.vllm.ai/2025/09/05/anatomy-of-vllm.html
vLLM Large-Scale Serving (Dec 2025): https://blog.vllm.ai/2025/12/17/large-scale-serving.html
vLLM speculators v0.3.0 (Dec 2025): https://blog.vllm.ai/2025/12/13/speculators-v030.html
Anyscale, Continuous Batching for LLM Inference (2023): https://www.anyscale.com/blog/continuous-batching-llm-inference
vLLM metrics docs: https://docs.vllm.ai/en/latest/design/metrics/
vLLM optimization: https://docs.vllm.ai/en/stable/configuration/optimization/
SGLang: https://github.com/sgl-project/sglang
TensorRT-LLM performance tuning: https://nvidia.github.io/TensorRT-LLM/performance/performance-tuning-guide/tuning-max-batch-size-and-max-num-tokens.html
Red Hat, 5 steps to triage vLLM performance (Mar 2026): https://developers.redhat.com/articles/2026/03/09/5-steps-triage-vllm-performance
NVIDIA blog, Chunked Prefill with TensorRT-LLM: https://developer.nvidia.com/blog/streamlining-ai-inference-performance-and-deployment-with-nvidia-tensorrt-llm-chunked-prefill/

MoE inference: the call centre with 256 specialists and 8 taking each call — fundamentals, expert parallel and the economics of DeepSeek-V3

Sat, 30 May 2026 08:00:00 +0200

This post goes into an architectural decision in the Deploy stage of the six-stage LLMOps pipeline. It complements those on KV cache (which does not change with MoE), FlashAttention (still the attention kernel, also dense in every current MoE), Quantization (FP8/NVFP4 over expert weights is what lets DeepSeek-V3 fit into a modest cluster) and Speculative decoding (MTP in DeepSeek-V3 is native speculative decoding).

You are here: DEPLOY

TL;DR

A Mixture of Experts (MoE) replaces the transformer’s dense FFN layer with N parallel experts plus a router that picks the k experts relevant to each token. The operational consequence is that the model’s total parameters (how many were trained and how many must be loaded into VRAM) decouple from the active parameters per token (how many take part in each forward). DeepSeek-V3 has 671 B total but only 37 B active per token; Qwen3-235B-A22B is 235 B total and 22 B active; Mixtral 8x22B is 141 B and 39 B; Llama 4 Maverick is 400 B and 17 B. Compute per token falls to 2·N_active FLOPs, half that of a dense Llama-3-70B in DeepSeek-V3’s case, but memory stays proportional to N_total: MoE sparsity is compute sparsity, not memory sparsity. Since each expert sees only batch·k/N tokens per step, the memory-bound regime persists at a much higher batch than in dense. Serving DeepSeek-V3 671 B FP8 at low latency requires expert parallel (EP) at scale, EP=32 in prefill and EP=144 in decode in DeepSeek’s production deployment, where the dominant bottleneck stops being HBM and becomes the all-to-all of dispatch and combine between GPUs. As of May 2026, the pieces that make this viable are DeepEP (FP8-native all-to-all CUDA kernels, intra-node NVLink plus inter-node RDMA), EPLB (balancing by replicating hot experts) and Wide-EP on NVL72 (72 GPUs in one coherent NVLink domain). This post takes apart the mechanism, the maths (FLOPs/token, all-to-all bytes), the updated table of MoE models as of May 2026, the parallelisms (TP / EP / DP+EP / Wide-EP), the operational pitfalls (memory wall, cold experts, imbalance) and the real reproducible numbers on H100, B200 and GB200 NVL72.

The analogy: the call centre with 256 specialists

A serious call centre handling complex technical queries runs on specialists, not generalists. You have 256 operators on the payroll, each an expert in their sub-domain: employment law, mortgages, income tax, probate, and so on. But each incoming call only needs the attention of eight specific specialists (top-8 routing): a client’s query about tax relief on a mortgage for an inherited second home involves the income tax person, the mortgage person, the capital gains person, and so on. The receptionist (the router) listens to the first second of the call, decides which eight specialists are relevant, conferences them in and combines their answers with weights.

Three immediate consequences, identical to those of MoE in production:

You pay 256 salaries (every expert is loaded in VRAM) but you only spend the time of 8 (only k experts take part in the forward of each token). Your payroll cost scales with N_total; your cost per minute of call scales with N_active.
If all the operators fit in one office (a single GPU), fine: paper just moves between desks. But if they do not fit, you have to open branches: the conference call between operators in different branches goes over a telephone line between buildings (intra-node NVLink, inter-node InfiniBand) and that line becomes the dominant bottleneck once dozens of branches are coordinating.
If one week every client asks about income tax, the three income tax operators saturate while the probate ones sit idle. You need a system that replicates the hot operators across several branches to absorb peaks, and that pairs hot with cold so that no branch goes idle.

The receptionist is the router. The branches are the GPUs. The conference call between branches is the all-to-all (dispatch plus combine). The system for replicating hot operators is EPLB. The good telephone line is DeepEP. And when someone designs a building where all 72 branches fit together with very fast internal cabling, that is NVL72.

The bare mechanism

A classic MoE layer replaces the dense FFN (y = FFN(x) = down(act(up(x) · gate(x))) in SwiGLU) with:

Router: a linear projection g(x) = W_router · x produces N scores. A softmax over those scores gives affinity probabilities.
Top-k selection: the k experts with the highest score are chosen. Their softmax_topk weights are normalised to sum to 1.
Parallel compute: each of the k selected experts runs its own FFN over x. The N - k not selected sit idle for this token.
Weighted combination: the output is y = Σ_{i ∈ topk} w_i · Expert_i(x).

Variants the market consolidated in 2025-2026:

Shared experts: on top of the k routed ones, there are s experts that always run (1 or 2). They capture general knowledge every token needs. DeepSeek-V3 uses 1 shared + 8 routed; Mixtral uses 0 shared.
Fine-grained experts: more small experts instead of a few large ones (DeepSeek: 256 vs Mixtral: 8). This allows better specialisation because the top-k combination covers more sub-domains.
Auxiliary-loss-free routing (DeepSeek-V3): instead of adding a balancing loss during training, it dynamically adjusts a bias in the routing. It keeps balance without polluting the main loss.

Attention is still dense in every current MoE model (Mixtral, DeepSeek, Qwen3-MoE, Llama 4, Kimi K2). Only the FFN is replaced. That is why FlashAttention v3/v4 is still used as-is; DeepSeek’s MLA is an optimisation orthogonal to MoE.

The relevant MoE models in May 2026

Model	Date	Total	Active	Experts	Top-k	Shared	Notes
Mixtral 8x7B	Dec 2023	47 B	13 B	8	2	0	The one that normalised MoE in open weights
Mixtral 8x22B	Apr 2024	141 B	39 B	8	2	0	—
Grok-1 (open)	Mar 2024	314 B	~86 B	8	2	0	Weights opened by xAI
DeepSeekMoE 16B	Jan 2024	16 B	2.8 B	64+2	6	2	Introduces fine-grained + shared
DeepSeek-V2	May 2024	236 B	21 B	160+2	6	2	+ MLA
Snowflake Arctic	Apr 2024	480 B	17 B	128	2	10 B dense residual	Dense-MoE hybrid
Hunyuan-Large	Nov 2024	389 B	52 B	16+1	1+shared	1	Cross-layer KV cache
DeepSeek-V3	Dec 2024	671 B	37 B	256+1	8	1	MLA + MTP, aux-loss-free, FP8 training
Llama 4 Scout	Apr 2025	109 B	17 B	16	1+shared	1	Designed for 1 H100 node
Llama 4 Maverick	Apr 2025	400 B	17 B	128	1+shared	1	—
Qwen3-235B-A22B	Apr 2025	235 B	22 B	128	8	0	—
Kimi K2 (Moonshot)	2025	1 T	32 B	384	8	(MLA)	Trained with MuonClip
DeepSeek-V3.2-Exp	Sep 2025	671 B	37 B	256+1	8	1	+ DeepSeek Sparse Attention

Three operational observations:

The active/total ratio falls with every generation: Mixtral 8x7B was 28 %, DeepSeek-V3 is 5.5 %, Llama 4 Maverick is 4.3 %. What the industry discovered is that sparsity can be very aggressive without losing quality, provided N is large enough and the routing is well learned. Fine specialisation is worth more than per-expert capacity.
The flagships of the large providers (OpenAI, Anthropic, Google) are almost certainly MoE, but the architectures are not published. The only semi-credible leak (Hotz, Jul 2023) claimed GPT-4 was an 8×~220 B MoE; no official confirmation.
DeepSeek-V3 marks the inflection point: open weights, frontier-class, MoE with aggressive sparsity and compatible with FP8 quantisation. It is the model that forced the ecosystem (vLLM, SGLang, TensorRT-LLM) to optimise wide-EP in 2025.

The maths that matters

Three numbers drive the whole architectural decision with MoE.

Compute per token. A dense transformer spends roughly 2N FLOPs per token in the forward pass (with N = total parameters). A MoE spends 2·N_active FLOPs per token (the N_total - N_active experts do not take part). For DeepSeek-V3: 2 × 37 B = 74 GFLOPs/token. For dense Llama-3-70B: 2 × 70 B = 140 GFLOPs/token. DeepSeek does half the compute per token of Llama-3-70B while offering capacity equivalent to a model almost 10 × larger.

Total memory. But all the experts must be loaded in VRAM at some point in the forward pass. The sparsity is in compute, not memory. Weight memory:

$$\text{Memory}_{\text{weights}} \approx N_{\text{total}} \cdot \text{bytes per param}$$

For DeepSeek-V3 FP8 (1 byte): 685 GB. It does not fit in 8×H100 SXM (640 GB). You need either extra quantisation or 16+ GPUs. For Mixtral 8x22B FP8: 141 GB, comfortably inside 2×H100. For Qwen3-235B-A22B FP8: 235 GB, fits in 4×H100.

All-to-all bytes per MoE layer. When experts are spread across GPUs (EP), each token has to travel to the GPUs where its k experts live. The volume per MoE layer is roughly:

$$\text{bytes a2a/layer} \approx 2 \cdot \text{batch} \cdot \text{seq len} \cdot d_{\text{model}} \cdot k \cdot \text{bytes}$$

(the factor 2 is dispatch plus combine). For DeepSeek-V3 with d_model = 7168, k = 8, batch = 4096, FP8 (1 byte): each MoE layer moves ~470 MB; multiplied by 61 MoE layers in DeepSeek-V3, the complete step adds ~29 GB of inter-GPU traffic in MoE comms alone. That is of the order of a GPU’s HBM bandwidth in a single step. This is why bandwidth and NVLink/InfiniBand topology determine how large EP can be before comms saturate.

Persistent memory-bound regime. Here is the least intuitive consequence. In dense, raising the batch moves the operation from memory-bound to compute-bound: with enough tokens per batch, the weights are amortised over more operations. In MoE, each expert sees only batch · k / N tokens. For Qwen3-235B (k=8, N=128) with batch=128, each expert processes 8 tokens per step. To reach the compute-bound regime of the tensor cores (arithmetic intensity ~100-200 FLOPs/byte) you need batches in the thousands, around 1,600 according to recent estimates for Qwen3 (Memory-Bound MoE Serving, arXiv:2512.09277). The operational consequence: MoE scales throughput with batch more linearly than dense, but it also needs far larger batches to get close to its compute ceiling.

Expert parallel and the all-to-all bottleneck

There are three main ways to parallelise a MoE across GPUs:

Tensor Parallel (TP, Megatron-style). Splits each weight matrix across GPUs by width or height. Every GPU processes all tokens and all layers. Comms per layer: 2 all-reduces. It works well for dense; for MoE it leaves the routing sparsity unexploited (each GPU holds fragments of every expert).

Expert Parallel (EP). Splits whole experts across GPUs: 256 experts / 32 GPUs = 8 experts per GPU. Each token, after routing, travels to the GPUs where its top-k k experts live. Comms per MoE layer: an all-to-all dispatch (tokens towards experts) plus an all-to-all combine (outputs back). The cost grows with EP size and becomes the bottleneck when EP > 8.

DP + EP / Wide-EP. Replicates the model across DP groups; inside each group, EP shards the experts. Wide-EP takes EP to the full dimension of a rack (NVL72, 72 GPUs in a coherent NVLink domain). Each expert receives more tokens per step, so arithmetic intensity in each expert’s GEMMs improves and per-GPU throughput rises. vLLM reports 1.8× throughput per GPU with wide-EP against smaller setups on DeepSeek-V3 (vLLM blog, Dec 2025).

Quick decision table:

Case	Recommendation
Small MoE that fits in 1 node (Mixtral 8x7B, Llama 4 Scout)	TP only or EP=2 intra-node
Medium MoE (Mixtral 8x22B, Qwen3-235B) on 1-2 H100 nodes	EP=8 intra-node, optionally TP=2
Large MoE (DeepSeek-V3) on a multi-node cluster	TP × EP cross-node (e.g. TP=4 × EP=8 on 32 GPUs)
Large MoE with NVL72 available	Wide-EP=72 (decode), EP=32 (prefill)

DeepSeek’s production deployment combines Prefill EP=32 (4 nodes × 8 GPUs) with Decode EP=144 (18 nodes × 8 GPUs). The prefill/decode split fits conceptually with Disaggregated serving: prefill is compute-bound and benefits from moderate EP; decode is memory-bound and benefits from Wide-EP, which raises the tokens per expert per step.

DeepEP, EPLB and Wide-EP: the pieces that unblocked 2025

DeepEP (DeepSeek, opened in February 2025) is a library of all-to-all CUDA kernels optimised specifically for MoE EP. Four important properties:

Native FP8 dispatch: tokens travel in FP8, halving the bandwidth consumed.
Mixing intra-node NVLink with inter-node RDMA: the kernel picks the optimal route according to topology.
CPU bypass: the GPU→GPU dispatch goes over RDMA without touching the host.
Aligned with DeepSeek-V3’s group-limited gating: the routing prefers experts in the same node group where possible, minimising inter-node traffic.

In 2025 Tencent contributed optimisations that added +30 % on the normal kernels. There is a ROCm port for AMD.

EPLB (Expert Parallelism Load Balancer) solves the runtime imbalance problem: even when the model is well balanced in global distribution, a specific batch can heavily activate certain experts (prompts in a single language, a single domain). EPLB replicates the hot experts across several GPUs (redundant experts) and packs them heuristically with cold experts to minimise load variance. It has two modes: hierarchical (for prefill with medium EP) and global (for decode with Wide-EP). It is integrated into SGLang and vLLM.

Wide-EP on NVL72 is the peak setup of 2026: 72 Blackwell GPUs in a coherent NVLink domain (1.8 TB/s bidirectional per GPU, intra-rack). Each expert receives many more tokens per step, so arithmetic intensity improves and so does throughput per GPU. Combined with NVFP4 over expert weights and FP8 over attention, SGLang reported on GB200 NVL72 26,156 tok/s/GPU in prefill and 13,386 tok/s/GPU in decode for DeepSeek-V3 (LMSYS, Sep 2025), a 3.8× prefill and 4.8× decode versus H100.

Operational pitfalls

HBM fragmentation from expert weights. With EP=8 in 1 node, each GPU holds N/8 experts, each a tuple of matrices (gate, up, down). If N=256, that is 32 FFNs per GPU. The allocator can fragment; in some cases you need a more conservative --gpu-memory-utilization than usual.

Cold experts. In real distributions some experts are activated by <1 % of tokens. EPLB compensates by replicating hot ones, but the cold ones sit occupying HBM without taking part, “dead” memory. When real demand diverges a lot from the training distribution, this shows.

Continuous batching. It works with MoE, but the effective batch size per expert is batch_total · k / N. With large N and a moderate batch, each expert sees very few tokens per step. You need a batch_total far larger than in dense to match per-GPU throughput.

Speculative decoding + MoE. They interact well. MoE sparsity keeps the memory-bound regime at high batch, which is exactly the regime where speculative decoding wins most. DeepSeek-V3 integrates MTP (Multi-Token Prediction), speculative decoding native to the model, with no external draft, with acceptance around 85-90 % on the second token (see Speculative decoding for the mechanism).

MLA and FlashMLA are not MoE. DeepSeek-V3 combines MoE with Multi-head Latent Attention (MLA), an attention variant that reduces the KV cache ~10× against standard MHA. MLA and MoE are orthogonal: MLA optimises attention, MoE optimises the FFN. For serving DeepSeek you use the specific FlashMLA kernel (not FA3 directly).

DSA (DeepSeek Sparse Attention) in DeepSeek-V3.2-Exp (Sep 2025) introduces sparsity in attention too. The first time at the frontier that sparsity is applied in both attention and FFN.

Implications for on-premise hardware

On an RTX 4090 (24 GB). Large MoEs are out of reach. What does fit: Mixtral 8x7B AWQ-INT4 (~24 GB of weights, no headroom for KV cache, so it really requires TP=2 across two 4090s); DeepSeekMoE 16B BF16 (~33 GB, does not fit whole; INT4 ~8 GB, fits with room to spare). The interesting case is llama.cpp with --n-cpu-moe: expert weights offloaded to RAM (CPU), attention and shared experts on the GPU. It lets you run DeepSeek-V3 IQ4 (~400 GB) on a workstation with 1-2 GPUs plus 512 GB of RAM at modest throughput, but it opens the door to a low-budget setup.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). These fit comfortably:

Mixtral 8x22B FP8 (~141 GB) with TP=2 or EP=4. Low latency, excellent throughput.
Qwen3-235B-A22B FP8 (~235 GB) with TP=4 or EP=4. Plenty of room.
Llama 4 Scout 109B FP8 (~110 GB) with TP=2 or TP=4. Designed by Meta to serve well on one H100 node.

What does not fit in 4×H100: DeepSeek-V3 FP8 (~685 GB) without aggressive quantisation. Options: (1) scale to 8-16 H100; (2) use AWQ INT4 over routed experts plus FP16 over activated ones (~400 GB), which fits in 8×H100; (3) wait for Blackwell or use AMD MI300X (192 GB/GPU, so DeepSeek-V3 FP8 on 4×MI300X).

The rule of thumb as of May 2026: one 4×H100 node comfortably serves MoEs up to ~200 B total in FP8; for DeepSeek-V3 you have to scale to 2 nodes or wait for B200 NVL72 (where the whole model fits with room for enormous batches).

What we have not covered

MLA and FlashMLA in detail: DeepSeek’s KV cache optimisation, orthogonal to MoE.
DeepSeek Sparse Attention (DSA) introduced in V3.2-Exp: the first production implementation of attention sparsity for frontier models.
MoE during training: load balancing losses, drop policies, auxiliary terms vs aux-loss-free, Kimi K2’s MuonClip.
MoE + LoRA: how adapter fine-tuning is done on a MoE, and what happens to the routing.
LLM-d and other open-source platforms that materialise Wide-EP on Kubernetes.

References

Shazeer, N. et al. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR 2017. https://arxiv.org/abs/1701.06538
Lepikhin, D. et al. GShard: Scaling Giant Models with Conditional Computation. ICLR 2021. https://arxiv.org/abs/2006.16668
Fedus, W., Zoph, B., Shazeer, N. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. JMLR 2022. https://arxiv.org/abs/2101.03961
Gale, T., Narayanan, D., Young, C., Zaharia, M. MegaBlocks: Efficient Sparse Training with Mixture-of-Experts. MLSys 2023. https://arxiv.org/abs/2211.15841
Mistral AI. Mixtral of Experts. 2024. https://arxiv.org/abs/2401.04088
DeepSeek-AI. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. 2024. https://arxiv.org/abs/2401.06066
DeepSeek-AI. DeepSeek-V2. 2024. https://arxiv.org/abs/2405.04434
DeepSeek-AI. DeepSeek-V3 Technical Report. 2024. https://arxiv.org/abs/2412.19437
Tencent. Hunyuan-Large. 2024. https://arxiv.org/abs/2411.02265
DeepEP repo: https://github.com/deepseek-ai/DeepEP
EPLB repo: https://github.com/deepseek-ai/EPLB
DeepSeek open-infra-index, V3/R1 Inference System Overview: https://github.com/deepseek-ai/open-infra-index
vLLM blog, Large-Scale Serving DeepSeek (Dec 2025): https://blog.vllm.ai/2025/12/17/large-scale-serving.html
vLLM blog, WideEP en GB200 (Feb 2026): https://blog.vllm.ai/2026/02/03/dsr1-gb200-part1.html
LMSYS blog, DeepSeek 96 H100 PD+EP (May 2025): https://www.lmsys.org/blog/2025-05-05-large-scale-ep/
LMSYS blog, DeepSeek GB200 NVL72 part II (Sep 2025): https://www.lmsys.org/blog/2025-09-25-gb200-part-2/
NVIDIA Dynamo + GB200 NVL72 for MoE: https://developer.nvidia.com/blog/how-nvidia-gb200-nvl72-and-nvidia-dynamo-boost-inference-performance-for-moe-models/
Tensor Economics, MoE Inference Economics from First Principles: https://www.tensoreconomics.com/p/moe-inference-economics-from-first
Cohere, Why MoE models get more from speculative decoding: https://cohere.com/blog/mixture-of-experts-models-get-more-from-speculative-decoding

Speculative decoding: the secretary who types ahead of the boss — fundamentals, maths and state of play in May 2026

Fri, 29 May 2026 11:00:00 +0200

This post complements KV cache: the working memory and Quantisation for LLM inference. Both are acceleration levers inside a single forward pass. Speculative decoding is the complementary lever: instead of making each forward pass cheaper, it tries to produce more tokens per forward pass. It is orthogonal to quantisation and to the KV cache, and it stacks multiplicatively with both.

You are here: DEPLOY

TL;DR

Autoregressive LLM inference has a structural problem: every new token needs a complete forward pass of the model, and decode forward passes are memory-bandwidth-bound, not compute-bound (the detail is in KV cache). The GPU spends most of its time waiting for the next weight to arrive from HBM. Speculative decoding uses that dead time by doing two things in parallel: a small, cheap model (draft) generates γ tokens autoregressively, fast but imprecise, and the large model (target) verifies those γ tokens in a single parallel forward pass, at almost the same cost as a single-token forward pass. An acceptance rule based on rejection sampling decides how many draft tokens are accepted, and the maths proves, it does not approximate, it proves, that the output distribution matches sampling directly from the target exactly. The technique has a ceiling: you never generate more than 1/(1-α) tokens per step, with α the acceptance rate. The five dominant families in May 2026 are vanilla SD (Leviathan 2023), Medusa (extra parallel heads), EAGLE-1/2/3 (a draft that operates at the level of hidden states), MTP (multi-token prediction native to DeepSeek-V3) and P-EAGLE (a draft that produces the γ tokens in a single forward pass, integrated in vLLM 0.16+). On low-concurrency workloads with short prompts and long outputs, the typical case of the on-premise conversational assistant, the real speedup sits between 2× and 4× on modern hardware.

The analogy: the secretary who types ahead and the boss who validates

Picture a crowded press conference with a real-time transcription setup. Two people are working:

The secretary sits at the keyboard. They know the subject, they have read the briefings and they know the boss’s verbal tics. As soon as the boss opens their mouth, the secretary starts typing what they think is coming. They are fast, three or four words ahead of the boss, but they sometimes get it wrong, especially when the boss takes an unexpected turn.

The boss periodically reads the secretary’s screen. They read a whole block at once, three or four words, and mentally compare what is written with what they would have said. If it matches their intent, they leave the text and carry on. If the secretary drifted at some point, they correct right there, and whatever the secretary typed after the divergence is discarded (it could be wrong because of the earlier error). If the whole block was fine, the boss takes the chance to add one more word while correcting.

The result: the final text is identical to what the boss would have dictated alone, but it comes out faster because the secretary works ahead.

The analogy holds on four details:

The secretary is the draft model. Small, cheap, fast, approximate.
The boss is the target model. Slow, expensive, but the only authority on output quality.
The block the boss reads at once is the target’s parallel forward pass over prompt + γ draft tokens. Verifying γ tokens costs almost the same as generating one, because the bottleneck is loading the weights from HBM, not the operations.
The “correct right where it diverged” rule is the rejection sampling that preserves the target distribution.

From here we go into the mechanism and into why quality is preserved exactly, not approximately.

The bare mechanism

Call p the target distribution (what it would produce if it were the only one speaking) and q the draft’s. The speculative decoding iteration has three steps:

Step 1 — Draft. The draft generates γ tokens autoregressively: x_1, x_2, ..., x_γ. Each one through its own forward pass. Since the draft is small, the γ steps cost little. Typically γ ∈ [4, 8].

Step 2 — Verify. The target runs a single forward pass over the complete sequence prompt + x_1 ... x_γ. Because attention is causal, that forward pass simultaneously produces the distributions p(·|prompt, x_<i) for each position i. In other words, in a single step it obtains the verification of the γ tokens and, on top of that, an extra distribution p(·|prompt, x_1...x_γ) for the next token.

Step 3 — Accept/reject token by token. For each x_i from left to right it applies:

If p(x_i) ≥ q(x_i): always accept.
If p(x_i) < q(x_i): accept with probability p(x_i)/q(x_i).
If it rejects: stop, sample a replacement token from the normalised residual distribution norm(max(0, p − q)), and discard the rest.
If it reaches the end having accepted the γ tokens: append a bonus token sampled directly from p(·|prompt, x_1...x_γ), which is already in the logits of the target’s forward pass.

Result per iteration: between 1 and γ+1 new tokens. In the best case (all accepted + bonus) γ+1 tokens are generated at the cost of a single target forward pass plus γ draft forward passes, against γ+1 target forward passes in the version without speculative.

Why quality does not degrade (the proof)

This is the part of the technique that confuses people most the first time. It looks like magic: how can sampling from an arbitrary draft and then “validating” produce exactly the same distribution as sampling from the original target?

The proof fits in two lines. For any token x, the final probability of emitting it is the sum of two disjoint events: the draft proposes it and it is accepted, or the draft proposes something else, it is rejected, and sampling from the residual yields x. Formally:

$$P(\text{emit } x) = q(x) \cdot \min\!\left(1, \frac{p(x)}{q(x)}\right) + P(\text{reject}) \cdot \frac{\max(0, p(x)-q(x))}{\sum_y \max(0, p(y)-q(y))}$$

The first term is min(p(x), q(x)). The second is max(0, p(x) − q(x)) (the total rejected mass Σ max(0, p−q) cancels the denominator exactly). Adding them: min(p(x), q(x)) + max(0, p(x) − q(x)) = p(x). The output is statistically indistinguishable from sampling directly from the target, down to floating-point numerical differences.

This matters operationally: it means speculative decoding is not perceptual compression, it is not a quality-speed trade-off, it does not require extra validation with evals. If the target would have passed a given eval, the system with speculative passes it too.

The maths that matters: ceiling and speedup

Call α the acceptance rate: the expected probability that an individual draft token is accepted. Assuming that acceptances of consecutive tokens are independent (a reasonable approximation in practice), the expected number of tokens generated per iteration is:

$$E[\text{tokens per step}] = \sum_{k=0}^{\gamma} \alpha^k + \alpha^{\gamma+1} = \frac{1 - \alpha^{\gamma+1}}{1 - \alpha}$$

And the theoretical speedup relative to generating one token per target iteration, with c = T_draft / T_target the relative cost of the draft, is:

$$\text{Speedup} = \frac{1 - \alpha^{\gamma+1}}{(1 - \alpha)(\gamma c + 1)}$$

There is a ceiling that many implementations try to beat and cannot: as γ → ∞, the speedup converges to:

$$\lim_{\gamma \to \infty} E[\text{tokens per step}] = \frac{1}{1 - \alpha}$$

It is an algorithmic ceiling, not a hardware one. With α = 0.7 you never generate more than 3.33 tokens per iteration no matter how much the hardware improves. With α = 0.8 → 5. With α = 0.9 → 10. That is why EAGLE-3, which targets α > 0.8 on many benchmarks, is not an incremental improvement over vanilla SD (α ≈ 0.5-0.6): it is a change of regime, because it raises the ceiling instead of merely getting closer to it.

A concrete numerical example, Llama 3 70B as target with a draft giving α = 0.75, γ = 5, c = 0.1:

Expected tokens per step: (1 − 0.75⁶) / (1 − 0.75) = (1 − 0.178) / 0.25 = 3.29
Speedup: 3.29 / (5 × 0.1 + 1) = 3.29 / 1.5 = 2.19×

If we raise α to 0.85 with the same configuration: tokens per step = (1 − 0.85⁶) / 0.15 = (1 − 0.377) / 0.15 = 4.16, speedup = 4.16 / 1.5 = 2.77×. A change of α = 0.75 → 0.85 (10 absolute points) raises the speedup by 27 %. That is why the research of the last two years focuses on pushing α: that is where the prize is.

The five modern families (May 2026)

Family	Year	Core idea	Draft size	Typical α / τ	Speedup
Vanilla SD	2023	Same-family target + draft pair, rejection sampling	1/10 – 1/100 of target	0.5 – 0.8	2-3×
Medusa	2024	N extra heads in parallel predict t+1, t+2, …; tree attention verifies several candidates	no draft, +1-2 % params	top-5 > 0.8	2.18-2.83×
EAGLE-1/2/3	2024-25	Autoregressive draft at the level of features (hidden states), not tokens. Reuses the target’s embedding	1 transformer block (~0.5-2 % params)	EAGLE-3: α > 0.8, τ up to 7.5	up to 6.5× peak
MTP (DeepSeek-V3)	2024	Multi-token heads trained from scratch as part of the model; at inference they act as a “free” draft	14B params in V3 671B	α > 0.8 (MTP1)	1.5-1.8×
P-EAGLE	2026	EAGLE but producing the γ drafts in a single forward pass (parallel, not autoregressive)	same as EAGLE	+30 % over EAGLE-3	4-5× vs AR

Three operational observations:

EAGLE dominates in production because its overhead is minimal (one transformer block, ~1 % of the target in parameters) and its α is high. The draft does not need its own “complete” KV cache because it shares features with the target.
Medusa mattered historically, it showed that speculative was possible without a separate draft, but EAGLE beat it on every published benchmark through 2024-2025.
MTP is special: it is not something you add to an existing model. It is something the model trained natively. If you buy DeepSeek-V3, MTP comes free and gives around 1.8× without touching anything. If you buy Llama 3, there is no MTP to be had; use EAGLE-3.

There are also two related techniques that deserve a brief mention, both without a draft model: Lookahead decoding (Fu et al. 2024), which formulates decoding as Jacobi iteration and extracts n-grams from the trajectory; and REST (He et al. 2024), which keeps an n-gram datastore and proposes drafts by longest-prefix match against the last generated tokens. Both give 1.5-2× with no extra VRAM. Useful when you have no trained draft model and do not want to maintain one.

Real implementations in May 2026

vLLM v0.16+: unified support for EAGLE-1/2/3, P-EAGLE, Medusa, native MTP (DeepSeek-V3 and variants), n-gram/suffix decoding without a draft, an arbitrary draft model and MLP speculators. The canonical flag is --speculative-config '{"method":"eagle3", "model":"...", "num_speculative_tokens": 5}'.
SGLang: native EAGLE-3 support with --speculative-algorithm EAGLE3. For DeepSeek-V3 it uses MTP through an EAGLE adapter. It has its own draft training framework (SpecForge).
TensorRT-LLM: Medusa, EAGLE (a simplified variant without a tree), ReDrafter, Lookahead. They report around 2.2× with EAGLE.
llama.cpp: basic speculative only with a draft model (--model-draft). No native EAGLE/Medusa/MTP as far as I have verified. Typical speedup 1.5-2.5× single-user.

When speculative does NOT help

The technique has three important blind spots:

Large batch. The GPU moves from memory-bound (decode at low concurrency) to compute-bound (decode at high concurrency). In the compute-bound regime the “almost free” target forward passes stop being free: γ tokens start costing γ times more instead of nearly 1. The typical crossover is at batch 16-32 for dense models; for MoE with few active parameters per token the crossover happens later. At load peaks, speculative can worsen aggregate throughput per GPU.

Prefill / TTFT. Speculative decoding does not touch prefill. The phase that processes the whole prompt stays identical. TTFT does not improve and can get marginally worse because of the draft setup. If the SLA is TTFT-bound (assistants with short outputs, search, RAG with brief answers), this is not the right tool.

Short outputs. If the model generates 10-20 tokens and the answer is over, the fixed setup overhead is not amortised. Speculative shines with long outputs (300+ tokens): generative assistants, extended code completion, drafting.

Operational pitfalls

Extra VRAM for the draft. In vanilla SD, loading a complete draft plus its KV cache is expensive. Llama 3 8B in BF16 as the draft for a 70B is about 16 GB of weights plus KV cache. On an H100 80 GB with the target already nearly filling it, that can force you to shrink the target’s KV cache and lower maximum concurrency. EAGLE solves this: the draft is one transformer block (~1 GB for a 70B) and reuses the target’s features.

Draft quantisation. Quantising the draft to INT4 with GPTQ degrades α substantially (errors in the logits accumulate in the comparison against p). AWQ holds up better but also lowers α. Common practice in 2026: target in FP8 or INT4, draft in FP16/BF16. The draft is small enough that the VRAM saved by quantising it does not compensate for the drop in α and therefore in speedup. The detail of each format is in Quantisation.

Interaction with continuous batching. It does not break it, but it creates nested raggedness: each request in the batch can accept a different number of tokens on each iteration. vLLM’s PagedAttention absorbs it, but the scheduler loses efficiency. At low QPS (conversational assistant, low simultaneous concurrency) the combination is excellent. At high QPS there is real tension, and work such as Goodput-optimized speculative decoding (Liu et al., 2024) optimises γ dynamically according to the state of the batch.

Sampling temperature. α falls with high temperatures. At T = 1.0 with creative outputs (free-form writing), α can drop 10-15 points relative to T = 0 with the same model-draft pair. The speedup scales accordingly.

Implications on on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace)

The classic use is vanilla SD with two quantised models: for instance, Llama 3 70B AWQ-INT4 as target (~35 GB → needs TP=2 across two 4090s) and Llama 3 8B AWQ-INT4 as draft. In practice, the most realistic single-card case is Llama 3 8B target + a 1B model as draft or Qwen 3 14B target + Qwen 3 0.5B as draft: they fit comfortably and give 1.8-2.5× on conversational tasks. For EAGLE on consumer cards, the official drafts for popular families (Llama 3, Qwen 3) are published on Hugging Face and take up 0.5-2 GB extra.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here EAGLE-3 shines:

Llama 3 70B FP8 + EAGLE-3 draft in FP16: the draft takes about 0.7 GB; the target about 70 GB with TP=2 (35 GB per GPU). The speedup observed in reproducible benchmarks is between 2.5× and 4× at batch 1-4, falling to almost break-even at batch 32.
DeepSeek-V3 671B FP8 + native MTP: the model ships with MTP trained in; there is nothing to add. The speedup is 1.5-1.8× with zero extra VRAM. It is the most efficient option operationally: zero additional pieces.
Combining with disaggregated serving: as Disaggregated serving explains, prefill and decode can live in separate pods. Speculative applies only in the decode pods, which fits the separation perfectly (prefill is compute-bound and would not benefit).

The rule of thumb on an H100 cluster in May 2026: if the model is DeepSeek-V3 / V4 → native MTP, nothing else; if it is Llama 3 / Qwen 3 → EAGLE-3 with the official draft; if it is exotic → vanilla SD with a draft from the same family.

What we have not covered

Training custom EAGLE drafts with SpecForge: how to collect target trajectories and train the draft on-policy.
Speculative Prefill (arXiv:2502.02789): a variant for accelerating TTFT, a different mechanism from the decode one described here.
Tree attention in detail: how Medusa and EAGLE-2 verify several candidates at once with specific attention masks.
MoE + speculative: the combination has non-trivial interactions with the expert router. Low activated params keep the memory-bound regime in place even at high batch, which changes the rules.

References

Leviathan, Y., Kalman, M., Matias, Y. Fast Inference from Transformers via Speculative Decoding. ICML 2023. https://arxiv.org/abs/2211.17192
Chen, C., Borgeaud, S., Irving, G., Lespiau, J.B., Sifre, L., Jumper, J. Accelerating Large Language Model Decoding with Speculative Sampling. DeepMind 2023. https://arxiv.org/abs/2302.01318
Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J.D., Chen, D., Dao, T. Medusa: Simple LLM Inference Acceleration with Multiple Decoding Heads. ICML 2024. https://arxiv.org/abs/2401.10774
Li, Y., Wei, F., Zhang, C., Zhang, H. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. ICML 2024. https://arxiv.org/abs/2401.15077
Li, Y., Wei, F., Zhang, C., Zhang, H. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. EMNLP 2024. https://arxiv.org/abs/2406.16858
Li, Y., Wei, F., Zhang, C., Zhang, H. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. NeurIPS 2025. https://arxiv.org/abs/2503.01840
DeepSeek-AI. DeepSeek-V3 Technical Report — Multi-Token Prediction. https://arxiv.org/abs/2412.19437
Fu, Y., Bailis, P., Stoica, I., Zhang, H. Break the Sequential Dependency of LLM Inference Using Lookahead Decoding. ICML 2024. https://arxiv.org/abs/2402.02057
He, Z., Zhong, Z., Cai, T., Lee, J.D., He, D. REST: Retrieval-Based Speculative Decoding. NAACL 2024. https://arxiv.org/abs/2311.08252
vLLM speculative decoding docs: https://docs.vllm.ai/en/latest/features/speculative_decoding/
SGLang speculative decoding docs: https://docs.sglang.ai/advanced_features/speculative_decoding.html
TensorRT-LLM speculative sampling: https://nvidia.github.io/TensorRT-LLM/advanced/speculative-decoding.html
vLLM blog Speculative Decoding in vLLM (oct 2024): https://blog.vllm.ai/2024/10/17/spec-decode.html
vLLM blog P-EAGLE: Parallel Speculative Decoding in vLLM (mar 2026): https://vllm.ai/blog/2026-03-13-p-eagle
Official EAGLE repo: https://github.com/SafeAILab/EAGLE
Official Medusa repo: https://github.com/FasterDecoding/Medusa

Quantization for LLM inference: FP8, INT4 (GPTQ, AWQ) and GGUF — the model's accounting zoom

Wed, 27 May 2026 11:30:00 +0200

This post complements KV cache: the working memory of LLM inference, where cache quantisation is mentioned as a savings lever; here we go into the whole method, model weights and cache, and into why each format does what it does.

TL;DR

Quantising is a change of representation: instead of storing every model weight as a float16 or bfloat16 (2 bytes), it is stored as a short integer (1 byte INT8, half a byte INT4) with a scale factor that reconstructs a value close to the original. The price is loss of numerical precision; the reward is 2-4× less VRAM, 2-3× more throughput and, on Hopper and Blackwell, a radically lower compute cost because FP8/FP4 units execute in fewer cycles than BF16 ones. The four dominant formats in May 2026 are FP8 (E4M3/E5M2, datacenter), INT4 GPTQ (Hessian-aware reconstruction), INT4 AWQ (activation-aware) and GGUF (the llama.cpp family). Each has a sweet spot: FP8 when the datacenter is Hopper/Blackwell and quality matters; GPTQ and AWQ when serving runs on Ampere/Ada and 4 bits are mandatory; GGUF when the target is edge or a consumer GPU. This post explains the minimum maths, the algorithms behind each format, what each one loses measured in perplexity and MMLU, and how it all applies on a 4090 versus an H100 cluster.

You are here: DEPLOY

The analogy: JPEG with an edge detector

A JPEG compresses an image by reducing the precision with which pixels are stored, but it does not reduce it uniformly. Where there is flat sky, thousands of very similar pixels, it throws away detail without anyone noticing. Where there is a sharp edge, the outline of a face, it keeps the fidelity. The trick is to detect which parts are sensitive before compressing.

Quantization of an LLM works the same way. You do not take every model weight and say “all of them in 4 bits”. Some weights are very important, projection weights that move the output a lot when they change, and others are less so. Modern techniques (GPTQ, AWQ) are basically edge detectors: they identify which weights can be quantised aggressively and which need more bits or special treatment, and they apply the quantisation with that information.

The analogy holds up in three details:

Calibration with a small dataset = the phase in which the JPEG encoder analyses the image before choosing blocks.
A block of 128 weights with a shared scale = the JPEG 8×8 block with its DCT.
Outliers preserved with more precision = the high frequencies of an edge are preserved more than the flat areas.

From there on, what follows is the maths and the operational detail.

The minimum maths: scale and zero-point

Quantising a weight vector w ∈ ℝ^n (in BF16) to INT4 means finding two things:

A scale s ∈ ℝ (in BF16 or FP16).
For each weight, an integer code q ∈ {0, 1, ..., 15} that fits in 4 bits.

And an approximate reconstruction formula:

$$\hat{w}_i \approx s \cdot (q_i - z)$$

where z is the zero-point (the integer that defines which code represents the original zero). The zero-point exists in asymmetric INT4/INT8 so that half the range is not wasted on negative values when the weight distribution is not symmetric.

The choice of s and z for a block of weights w_block:

$$s = \frac{\max(w_\text{block}) - \min(w_\text{block})}{2^{b} - 1}, \quad z = -\frac{\min(w_\text{block})}{s},$$

with b = number of bits (4 in INT4). Encoding:

$$q_i = \text{clip}\!\left(\text{round}\!\left(\frac{w_i}{s} + z\right),\, 0,\, 2^b - 1\right).$$

And decoding at inference time:

$$\hat{w}_i = s \cdot (q_i - z).$$

Numerical example

Take 8 real weights from a linear layer: w = [0.31, -0.12, 0.78, -0.05, 1.42, -0.91, 0.23, 0.66]. We want to quantise to INT4 (16 levels).

max = 1.42, min = -0.91. Range = 2.33.

$$s = \frac{2.33}{15} \approx 0.1553, \quad z = -\frac{-0.91}{0.1553} \approx 5.86 \to 6.$$

Encoding of each weight:

`w_i`	`w_i/s + z`	`round`	`q_i`	`ŵ_i = s·(q-z)`	error
0.31	8.00	8	8	0.311	+0.001
-0.12	5.23	5	5	-0.155	-0.035
0.78	11.02	11	11	0.776	-0.004
-0.05	5.68	6	6	0.000	+0.050
1.42	15.14	15	15	1.398	-0.022
-0.91	-0.06	0	0	-0.932	-0.022
0.23	7.48	7	7	0.155	-0.075
0.66	10.25	10	10	0.621	-0.039

Mean squared error: MSE ≈ 0.0015. For a single layer with millions of weights, the aggregate effect is what calibration tries to minimise.

Storage: instead of 8 values × 2 bytes = 16 bytes (BF16), we have 8 × 4 bits = 4 bytes + 2 bytes (BF16 scale) + 0.5 byte (INT4 zero-point) ≈ 6.5 bytes. 2.5× less, yet the useful data is still recoverable with a small error.

PTQ vs QAT: when quantisation happens

Two different operational regimes.

Post-Training Quantization (PTQ) is applied after training, on a model already trained in BF16/FP16. It reads a small dataset (typically 128-512 examples) to calibrate the scales, runs the quantisation algorithm (GPTQ, AWQ, and so on) and produces the quantised weights. Cost: minutes to a few hours. Typical loss: 0.05-0.3 PPL in perplexity (~0.5-2 % in MMLU) for INT4 with modern methods.

Quantization-Aware Training (QAT) introduces the quantisation operations inside the training loop. During training the model “sees” that its weights get quantised and learns to be robust to it. Cost: retraining the model (expensive), but little is needed, a short fine-tune on top of the already quantised PTQ model. Typical loss: ~0 (quantisation becomes indistinguishable from the original model).

When to use which:

PTQ = default. 90 % of production cases. The model arrives in BF16, you quantise it in 1-2 hours on one GPU, you deploy it.
QAT = when PTQ loses too much and the difference matters (typical case: INT2/INT3, or sensitive models such as specific reasoning ones).

The dominant formats in 2026

FP8: the Hopper/Blackwell datacenter format

FP8 is not “INT8 + sign”: these are two 8-bit floating point formats.

E4M3 (4 exponent bits, 3 mantissa bits): range ±448, reasonable precision around ±1.0. Typically used for weights and for activations in most layers.
E5M2 (5 exponent, 2 mantissa): range ±57,344, lower precision. Used for gradients during training or for activations with large outliers at inference time.

Why FP8 left INT8 behind in the datacenter: the tensor cores on H100/H200/B200 execute FP8 operations natively with 2× the throughput of BF16 and 4× that of FP16. And because FP8 preserves logarithmic dynamics (just like FP16), matrices with values spread out in magnitude, typical of transformers, quantise with less error than with INT8.

The loss measured in production is minimal: for a Llama 3.1 70B FP8 versus BF16, perplexity rises by ~0.03 and MMLU drops by ~0.5 points. It is the default option in any modern deployment on H100/B200.

Microscaling: NVFP4 and MXFP4

Blackwell (B100/B200, 2025) introduces NVFP4 and MXFP4, 4-bit formats with small-block scaling (typically 16 or 32 elements per scale, against 128 in INT4 GPTQ). The scale is FP8 instead of FP16/BF16, which reduces storage further.

The result: 4 bits with quality close to FP8. In 2026 NVFP4 is becoming the default option for very large models (200B+) on Blackwell clusters. For 4×H100 SXM, Hopper rather than Blackwell, FP8 remains the sweet spot.

INT4: GPTQ vs AWQ

The two algorithms that dominate 4-bit quantisation solve the same problem with different strategies.

GPTQ (Frantar et al. 2022)

The idea: layer-by-layer quantisation, explicitly minimising the error at the output of each linear layer using information from the Hessian matrix (the second derivative of the loss). For each layer:

Estimate the Hessian H = X^T X where X are the calibration activations.
Quantise one weight at a time in order (typically the most sensitive one first).
Update the remaining weights to compensate for the error of the weight just quantised.

Step 3 is what makes GPTQ better than naive round-to-nearest: weights compensate for their neighbours’ errors. The official implementation quantises a Llama 3 70B in ~3-4 hours on one H100 with 128 calibration samples.

AWQ (Lin et al. 2023)

The AWQ observation: within a layer, not all channels (weight columns) are equally important. Roughly 1 % of the channels account for most of the impact on the activations. AWQ identifies them by measuring the mean magnitude of the activations that multiply them, and scales them before quantising so they are better preserved.

Concretely: if a channel c has large mean activations, AWQ multiplies that channel’s weights by a factor s_c before quantising, and when quantising the input to the next layer it divides by s_c. The maths cancels out, but the important weights end up with more resolution inside the INT4 range. No retraining, no Hessian, faster than GPTQ (~1-2 h for 70B on an H100).

Which one to choose

Criterion	GPTQ	AWQ
Quantisation speed	Slower	Faster
Quality preserved in INT4	Good	Slightly better (~0.05-0.1 PPL)
Supported hardware	Broad (Ampere+)	Broad (Ampere+)
Ecosystem	Mature, widely integrated	More recent, gaining ground
Default in 2026	When GPTQ artefacts already exist	Default for new quantisations

The practical rule in May 2026: AWQ by default for new INT4. GPTQ when a GPTQ-Int4 artefact published by the community already meets your requirements.

GGUF: the llama.cpp ecosystem

GGUF is not a quantisation algorithm, it is a file format, and a whole tooling ecosystem, around the llama.cpp runtime.

Its value: universal compatibility. The same GGUF file runs on:

Pure CPU (Intel/AMD x86, ARM).
Apple Silicon (M1/M2/M3/M4) with Metal acceleration.
Consumer GPUs (RTX, AMD Radeon) with layer offload to VRAM.
Edge devices (Jetson, ARM phones).

That is what llama.cpp allows and vLLM/TensorRT-LLM do not. The trade-off: lower peak throughput on datacenter GPUs than vLLM.

The most used GGUF sub-formats in 2026:

Sub-format	Effective bits	Relative quality	Typical use
`Q8_0`	8.5	Almost lossless	Baseline validation
`Q6_K`	6.6	Very small loss	High quality + savings
`Q5_K_M`	5.7	Small loss	Quality/size sweet spot
`Q4_K_M`	4.8	Moderate loss	Consumer default
`Q4_K_S`	4.5	Moderate-to-high loss	When Q4_K_M does not fit
`Q3_K_M`	3.9	Noticeable loss	Very constrained hardware
`Q2_K`	3.3	Large loss	Last resort

The _K_M suffix indicates the degree of mixing: inside the file, certain layers (typically attention.wv, feed_forward.w2) are stored with more bits than others. It is the equivalent of the JPEG “edge detector” applied layer by layer with a pre-established heuristic.

KV cache quantization

Quantising the model weights is half the problem. The KV cache, covered in detail in KV cache: the working memory, typically consumes 20-50 % of VRAM in production with concurrency. Quantising the cache is a lever too:

--kv-cache-dtype=auto (BF16/FP16, the default). 2 bytes per dimension × num_heads × head_dim × 2 (K and V).
--kv-cache-dtype=fp8 (E4M3 or E5M2 depending on hardware). 1 byte. Halves the cache with a typical loss of < 0.5 % on quality benchmarks.
--kv-cache-dtype=int4 (with blocks of 128). 0.5 bytes plus scale overhead. Divides the cache by ~3.5. Measurable loss (1-2 %) but acceptable with long contexts.

KV cache quantisation is orthogonal to weight quantisation: you can have BF16 weights and an FP8 cache, or INT4 weights and an FP8 cache, and so on. The dominant combination in 2026 on H100: FP8 weights + FP8 cache, which is almost indistinguishable from BF16 in quality and doubles concurrency capacity.

Measured quality loss (Llama 3.1 70B Instruct, reference)

A representative table for Llama 3.1 70B Instruct with a WikiText-2 calibration dataset (128 samples). Figures aggregated from public sources; they may vary by ±0.05 PPL and ±0.5 MMLU depending on implementation and seed.

Format	Model VRAM	Perplexity (WikiText-2)	MMLU (5-shot)	Relative speed (H100)
BF16 (baseline)	140 GB	4.85	82.1	1.00×
FP8 (E4M3)	70 GB	4.87 (+0.02)	81.6 (-0.5)	1.85×
INT8 SmoothQuant	70 GB	4.92 (+0.07)	81.0 (-1.1)	1.65×
INT4 AWQ	35 GB	4.99 (+0.14)	80.4 (-1.7)	2.50×
INT4 GPTQ	35 GB	5.05 (+0.20)	80.0 (-2.1)	2.40×
GGUF Q5_K_M	49 GB	4.94 (+0.09)	81.1 (-1.0)	n/a (llama.cpp)
GGUF Q4_K_M	42 GB	5.08 (+0.23)	79.8 (-2.3)	n/a (llama.cpp)
GGUF Q3_K_M	33 GB	5.45 (+0.60)	77.5 (-4.6)	n/a (llama.cpp)

Three lessons worth keeping:

FP8 is almost free in quality terms. If your hardware supports it, there is no debate.
INT4 AWQ is noticeably better than INT4 GPTQ in preserved quality, at comparable speed.
Q3 is already in measurable-loss territory; Q2 should no longer be used except for experiments or extreme demos.

Implications for on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace, no native FP8)

Llama 3.1 8B Instruct fits comfortably in BF16 (16 GB), but that leaves little headroom for a KV cache under concurrency. The usual sweet spot:

Llama 3.1 8B AWQ-INT4: ~5 GB of weights, 19 GB free for the KV cache → 4-8 concurrent sessions with a moderate context.
Llama 3 70B GGUF Q4_K_M: ~42 GB. It does not fit on a whole 4090; it requires CPU offload with llama.cpp (slow decode but workable for a single user).
Llama 3 70B AWQ-INT4 with TP=2 (two 4090s): ~17 GB per GPU → it fits and leaves headroom.

The 4090 does not support native FP8 (Ada Lovelace has the instruction but not Hopper’s accelerated throughput). In practice, FP8 on a 4090 works but without the speed gain: the sensible choice is INT4 AWQ.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here FP8 shines:

Llama 3.1 70B FP8 with TP=2: ~35 GB/GPU. Comfortable, leaving a huge amount of room for the KV cache → dozens of concurrent sessions.
Llama 3.1 405B FP8 with TP=4: ~200 GB/GPU. It just fits, with prefill+decode in the same pool.
Llama 3.1 405B INT4 AWQ with TP=2: ~100 GB/GPU. It allows serving the large model without saturating the cluster; there is headroom left for the cache and for serving another model at the same time.

The rule of thumb on an H100 cluster in 2026: FP8 if quality matters and the model fits; INT4 AWQ if the model does not fit in FP8 or if you want more concurrency at the cost of 1-2 MMLU points.

What we have not covered (upcoming articles)

Speculative decoding: the other big acceleration lever in inference. Orthogonal to quantization, it multiplies the speedup.
MoE quantization: Mixture-of-Experts models (Mixtral, DeepSeek V3, Qwen3-235B-A22B) have different quantisation patterns, experts are not quantised uniformly and there is dynamic routing.
Calibration dataset matters: how to choose the 128-512 calibration samples. The common mistake of grabbing a random dataset off the internet, and how to avoid it.
Multimodal quantization: vision-language models have heterogeneous layers (a CNN vision encoder, a transformer language model) that need separate treatment.

References

Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (ICLR 2023).
Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Han, S. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (MLSys 2024).
Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs (NeurIPS 2023). Introduces NF4 and double quantization.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., Han, S. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (ICML 2023).
NVIDIA. FP8 Formats for Deep Learning — E4M3/E5M2 white paper: https://arxiv.org/abs/2209.05433.
Rouhani, B. et al. Microscaling Data Formats for Deep Learning — MXFP4/MXFP8: https://arxiv.org/abs/2310.10537.
llama.cpp GGUF spec: https://github.com/ggerganov/llama.cpp/blob/master/docs/gguf.md.
vLLM quantization docs: https://docs.vllm.ai/en/latest/quantization/.

Disaggregated serving: prefill and decode in specialised pods

Fri, 22 May 2026 01:00:00 +0200

TL;DR

LLM inference has two phases with opposite profiles: prefill (processing the whole prompt in one go) is compute-bound, decode (generating token by token) is memory-bandwidth-bound. Running them on the same GPU forces a choice between two incompatible optimal hardware profiles, and leaves between 60 % and 80 % of peak capacity unused. The industry consolidated the pattern in 2026: disaggregated serving, separate pods for each phase, connected by a KV cache transfer channel (NIXL over UCX, RDMA, or NCCL as a fallback). DistServe demonstrated 7.4× more request rate at the same SLO; NVIDIA Dynamo 1.0 (GA at GTC 2026) takes the pattern to production at datacenter scale. Mixing heterogeneous hardware, H100 for prefill and commodity GPUs for decode, cuts up to 48 % off the cost per token. This article explains the why, the how, and the numbers that matter for a typical on-premise infrastructure.

You are here: Deploy

Disaggregated serving is an architectural decision in the Deploy stage of the six-stage LLMOps pipeline. It does not change the model, it does not change the data, it does not change the evals. It only changes how inference pods are spread over the GPU hardware. But that change moves aggregate throughput between 2× and 7×.

The analogy: the kitchen with two brigades

A serious industrial kitchen, any kitchen serving more than 50 covers a night, runs with two distinct brigades and two physically separate spaces.

The prep brigade starts at dawn. Its job is the mise en place: chopping, marinating, blanching, boiling stocks, preparing complex components. Equipment: good knives, big burners, convection ovens, 40-litre pots. It is capacity-intensive work and it is done in one batch. When it finishes, everything is left in labelled trays ready to use.

The service brigade comes in mid-afternoon. Its job is service: taking the trays from prep, heating portions, plating, running the pass. Equipment: salamanders, small griddles, thin spatulas, plenty of crockery. It is wrist work, rhythm work, not-failing-the-customer-who-has-the-plate-in-front-of-them work. Capacity per hour matters less than latency per plate.

If you make the same person do prep and service, both suffer. The cook is idle while doing mise en place in the middle of the afternoon. They have to stop and plate up when five orders land at once. Their working kit is designed for one or the other, not for both.

Serious kitchens solved this decades ago: separate brigades, separate spaces, separate equipment. The only thing that crosses between them is the trays of mise en place.

The trays are the KV cache. The separation is disaggregated serving. The handover from prep to service is the KV cache transfer, solved today with NIXL over RDMA. And the specialised pods are the two brigades with their optimal kit.

Quick recap: prefill and decode

A request to an LLM always goes through two phases:

Prefill. Take the complete prompt (say, 4,000 tokens) and process it in a single pass through all the layers of the model. The result is the KV cache of those 4,000 tokens (see the previous article on KV cache if you want to recall exactly what it holds). This step is massively parallel: all the tokens go through the attention matrices at once, which translates into huge, dense matrix multiplications. The GPU sits at 90-95 % compute utilisation. TTFT (time to first token) is determined by this phase.

Decode. Once the KV cache is ready, the model generates tokens one by one. Each new token is a pass through all the layers with a single query vector, reading the whole accumulated KV cache to compute attention. There is no parallelism between tokens (each one depends on the previous). What limits you here is not compute but bandwidth: every step has to read the model’s full weights from HBM. The GPU sits at 20-40 % compute utilisation, but at 90 % HBM utilisation. TBT (time between tokens) is determined by this phase.

Phase	Characteristic	Bottleneck	Key metric
Prefill	Massive parallel compute over N tokens at once	TFLOPS (compute)	TTFT
Decode	Streaming weights from HBM, 1 token at a time	HBM bandwidth	TBT (inter-token latency)

The asymmetry is structural: prefill burns compute and leaves memory half idle, decode does the opposite. A GPU designed to be excellent at both at once is a GPU designed to be badly used all the time.

Why putting them on the same GPU is a bad deal

Until 2023, the universal assumption was to run prefill and decode in the same inference process, on the same GPU. The engine scheduler (vLLM, TGI, Triton) decided on each cycle whether to prefill a new request or decode the ones already in flight. The intuition was that sharing hardware saves money.

The intuition is wrong. The problem has three faces:

Latency interference. When the engine decides to prefill a new request, it interrupts every decode in progress. That pushes up the TBT of the other requests. The user who was watching tokens flow smoothly down the screen notices a stall of several hundred milliseconds. This is known as prefill-decode interference and it degrades the experience visibly as concurrency rises.

Sub-optimal hardware for each phase. An H100 SXM has 989 TFLOPS BF16 of compute and 3.35 TB/s of HBM3. It is excellent for prefill, where compute is the limit. For decode, where the only thing that matters is bandwidth, those 989 TFLOPS are 60-70 % wasted. Conversely, a GPU with less compute but similar relative bandwidth (RTX 4090, L40S) would handle decode just as well for a fraction of the price.

Low aggregate utilisation. In real workloads with Llama 3 70B and 512-token outputs, around 80 % of the wall-clock is spent in decode. That means 80 % of your H100 cluster budget is doing memory reads, not calculations. It is like paying for a Ferrari to use it in a car park queue.

The idea: specialised pods, KV cache as the deliverable

Disaggregated serving breaks the inference cycle into two distinct services:

Prefill pod. Receives the prompt, runs the prefill, produces the KV cache. Hardware: high-compute GPUs (H100, H200, B200). Optimised for aggressive batching and throughput, not for individual latency: if 32 prompts arrive within 100 ms, it processes them together.

Decode pod. Receives the already-built KV cache, runs the token-by-token generation, streams to the client. Hardware: GPUs with good bandwidth but ideally cheaper per TFLOPS (RTX 4090, L40S, A100, even A30 depending on the case). Optimised for per-token latency (low TBT).

Between the two: a KV cache transfer over the network, which can be node-local (shared memory, NVLink), intra-rack (RDMA with InfiniBand or RoCE) or cross-rack (NIXL over UCX). The cost of this transfer scales linearly with context length, and it is the economic crux of the scheme.

The transfer protocol: the economics of movement

The KV cache transferred for a Llama 3 70B with 4K of context weighs roughly 2.6 GB (80 layers × 8 KV heads × 128 dim × 4,096 tokens × 2 (K and V) × 2 bytes in BF16). Moving 2.6 GB between two GPUs is not trivial:

Channel	Effective bandwidth	Time for 2.6 GB
Intra-node NVLink (NVSwitch)	~450 GB/s	~6 ms
Shared memory (same node, PCIe 5)	~60 GB/s	~45 ms
RDMA InfiniBand 400 Gbps	~50 GB/s	~55 ms
RDMA RoCE 200 Gbps	~25 GB/s	~105 ms
TCP/IP 10 GbE	~1 GB/s	~2.6 s

The immediate reading: above InfiniBand grade, the transfer is comfortable. Below it, it ruins the very TTFT we are trying to improve. Disaggregated serving is only viable with a decent interconnect. It is not a pattern for clusters built out of consumer Ethernet switches.

NVIDIA answered this with NIXL (NVIDIA Inference Transfer Library), released in mid-2025: a library that abstracts the transport (UCX, NCCL, direct RDMA verbs, shared memory) and picks the best available path automatically. vLLM has integrated it since late 2025 through the NixlConnector. It is now the de facto default for new deployments.

Real implementations in May 2026

The pattern’s path over two years:

2024 Jan · DistServe (HKU + UCSD): 7.4× requests at the same SLO
2024 May · SplitWise (Microsoft): variant with heterogeneous hardware
2024 Dec · experimental vLLM disagg (SharedStorage + PyNcclConnector)
2025 Mar · NIXL release (NVIDIA): unified transfer library
2025 Jul · stable vLLM NixlConnector
2025 Nov · SGLang, llm-d, MoonCake adopt the pattern
2026 Mar · NVIDIA Dynamo 1.0 GA (GTC 2026): production-ready at datacenter scale

As of today, the pattern is the default in any serious serving framework. The ones still monolithic are the small ones or the educational ones.

Three realistic options for an on-premise infrastructure:

vLLM disagg with NixlConnector. The most open route, requiring two sets of vLLM pods to be deployed (one with --kv-transfer-config '{"kv_role":"producer"}', another with "kv_role":"consumer") plus a proxy router. Enough for clusters of 4-16 GPUs.
SGLang with disagg. Conceptually equivalent, better performance on some MoE workloads.
NVIDIA Dynamo 1.0. The one taking over at datacenter scale. It covers routing, KV cache management, monitoring and scheduling in a single control plane. Heavier, but the reference solution if your cluster grows beyond 32 GPUs.

The numbers that matter

What disaggregation unlocks, in direct terms:

Metric	Aggregated (monolithic)	Disaggregated	Improvement
Goodput (req/s at the SLO)	baseline	1.4 – 2×	up to 2×
TTFT under high load	rises sharply from QPS 4	stable up to QPS 7+	~2×
Request rate at the same SLO (DistServe paper)	baseline	7.4×	7.4×
MoE throughput on Blackwell (Dynamo, GB300 NVL72)	baseline (Hopper)	up to 50×	depends on the model
Cost per token (heterogeneous H100 + commodity)	baseline (all H100)	-48 %	almost half

These numbers have to be read carefully: the most spectacular ones (7× and 50×) need specific hardware (Blackwell GB200/GB300 NVL72) and specific models (large MoE). The realistic range for a typical on-premise setup is 1.4-2× in goodput and -30 to -50 % in cost per token, depending on how heterogeneous the GPU mix is and how optimised the KV cache transfer is.

Heterogeneity: the radical version

The logical next step, proposed by SplitWise in 2024 and matured in 2025-2026 (Cronus, Tessera and others), is to mix GPU types: expensive high-compute GPUs for prefill, commodity GPUs with good bandwidth for decode.

Indicative cost (typical market prices in mid-2026):

H100 SXM: ~30-40 kUSD capex, ~3-4 USD/h amortised. Compute-heavy profile.
L40S: ~8-10 kUSD capex, ~1.5 USD/h. Intermediate profile, 864 GB/s of bandwidth.
RTX 4090: ~1.5 kUSD capex, ~0.30 USD/h. Modest compute profile but 1 TB/s of GDDR6X bandwidth, enough for decode of models up to ~30B parameters.

A realistic mixed cluster for serving an 8B model:

2× RTX 4090 (prefill batch) → ~3,000 USD capex, ~0.60 USD/h
4× RTX 4090 (decode pool) → ~6,000 USD capex, ~1.20 USD/h
TOTAL → ~9,000 USD capex, ~1.80 USD/h

Against the monolithic alternative with equivalent throughput:

2× H100 SXM (all in one) → ~70,000 USD capex, ~7 USD/h

The same throughput at a fraction of the capex and a quarter of the hourly cost, at the price of operational complexity: you now have two pools to coordinate, a transfer network to look after, and a scheduler that is not trivial.

For larger models (Llama 3 70B), the decode pool no longer fits on a single 4090 (the model does not fit in 24 GB, not even quantised to INT4 with headroom). There the sensible mix is H100 for prefill plus L40S or A100 80GB for decode, with a typical saving of 30-40 % over the all-H100 option.

Applied to typical on-premise hardware

Case 1 — One or two RTX 4090s: monolithic still wins

With a single GPU there is no disaggregation to speak of: the pattern needs at least two GPUs in separate pods. With two 4090s you can technically try it (one for prefill, one for decode with the KV cache transferred over PCIe 5 or basic RDMA), but the transfer overhead eats the gain for small models where prefill is already fast.

Recommendation: stay monolithic (traditional vLLM, properly configured with a quantised KV cache). The next justifiable level of complexity is a cluster with a fast interconnect.

Case 2 — 4×H100 SXM cluster (320 GB, NVLink): the sweet spot

The minimum realistic configuration for serious disaggregation, serving a 70B model in production:

2× H100 (TP=2) → 2 prefill pods
2× H100 (TP=2) → decode pods with several instances sharing TP
NIXL over NVLink → KV cache transfer <6 ms
Router (vLLM or Dynamo) → prompt distribution and streaming

Realistic expected result: goodput 1.6-1.9× relative to the same cluster running monolithic, with TTFT stable up to loads of QPS 7-8 (against the QPS 4 at which the monolithic setup starts to degrade).

If the heterogeneous mix is possible (adding 4-8 L40S to the cluster to make the decode pool), the cost per token drops a further 25 % to 35 %, while still serving the 70B model whole.

Position within the architecture

Disaggregated serving is a cross-cutting layer over almost everything discussed in previous articles. It touches:

The KV cache because it is the artefact transferred between pods. Without a solid grasp of how much the cache weighs and how it grows with context, you cannot size the transfer.
Continuous fine-tuning because multi-LoRA hot-swap keeps its semantics: each pod (prefill or decode) loads the adapters separately, and the router decides which adapter to apply in each phase.
Cluster topology: it changes the recommended HW, the networking required and the cost model.

If you are designing an inference infrastructure for 2026 from scratch, disaggregation stops being optional for any cluster beyond 4 GPUs of capacity. If you are modernising an existing one, it is the upgrade with the best return per euro invested, provided the networking between pods is decent (intra-node NVLink or intra-rack RDMA as a minimum).

What we have not covered (upcoming articles)

NIXL in detail: how it picks the optimal transport, how UCX is configured, what happens when RDMA fails and you have to fall back to TCP.
Routing scheduler: how the orchestrator decides which pod gets which request, dynamic batching, priority handling.
Multi-tenant disagg: KV cache isolation between tenants, per-adapter ACLs, multi-LoRA over specialised pods.
Disagg + prefix caching: how it combines with the KV cache reuse pattern when several prompts share a prefix (a common system prompt).
Disagg at the edge / local inference: viability on home hardware (4090 + Mac Studio, for example), where the transfer depends on Thunderbolt or residential Ethernet.

References

Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (OSDI 2024).
Patel et al., SplitWise: Efficient Generative LLM Inference Using Phase Splitting (ISCA 2024).
NVIDIA, NVIDIA Dynamo 1.0: Production-Ready Disaggregated Inference (GTC 2026, March): https://developer.nvidia.com/blog/nvidia-dynamo-1-production-ready/.
NVIDIA, NIXL: NVIDIA Inference Transfer Library — official documentation.
vLLM, Disaggregated Prefilling: https://docs.vllm.ai/en/stable/features/disagg_prefill/.
vLLM, NixlConnector Usage Guide: https://docs.vllm.ai/en/stable/features/nixl_connector_usage/.
Hao AI Lab, Disaggregated Inference: 18 Months Later (UCSD, 2025) — technical retrospective on the DistServe paper.

LLM inference operators on Kubernetes: OME, vLLM Production Stack, NVIDIA Dynamo and llm-d

Mon, 18 May 2026 17:00:00 +0200

TL;DR

Serving an LLM in production is not running a binary: it is coordinating a model (tens of gigabytes that take minutes to load), a runtime (vLLM, SGLang, TensorRT-LLM with a hundred flags), heterogeneous GPUs (NVLink, MIG, PCIe), prefill and decode that live better apart, a KV cache that wants offloading to colder tiers, intelligent routing that exploits prefix caching, and autoscaling that reacts to metrics other than CPU%. A flat Kubernetes Deployment only covers the first 20% of that. The other 80% is covered by LLM inference operators, of which four matter in 2026: OME (LMSYS, July 2025, multi-engine with a focus on SGLang), vLLM Production Stack (a curated Helm chart from vLLM itself with LMCache for tiered KV), NVIDIA Dynamo (the official successor to Triton, multi-engine, with its own Grove scheduler) and llm-d (a CNCF donation of March 2026 by Red Hat + Google + IBM + CoreWeave + NVIDIA, built on vLLM, focused on distributed scale). Behind all four sits KServe, the CNCF parent operator that normalised the InferenceService concept and that several of them lean on. This article walks through the full hierarchy, gives a decision map and teaches you not to get lost when someone drops seven acronyms in the first meeting.

This article closes the LLM inference series. The previous ones were KV cache: the working memory that holds up LLM inference, vLLM on Kubernetes: the LLM inference piece that actually scales and PagedAttention from the inside and the state of the art of the KV cache in 2026. There we explained what happens inside one inference process. Here we explain how many inference processes are coordinated through Kubernetes.

The analogy: from `init.d` to systemd to operators

Anyone with 20 years of sysadmin behind them will recognise the pattern. Decades ago, starting a service on Linux was a shell script in /etc/init.d/: start, stop, status, reloaded by hand. When services got more complex, with dependencies between them, monitoring, restart on failure, per-user slots, it became obvious that a script was not enough. Along came systemd, which turned “a service” into a declarative unit with dependencies, resources, restart policy, sockets and timers. The script did not disappear; it moved up one level of abstraction.

Kubernetes made the same move for distributed services. A Deployment declares “I want N replicas of this container”; a Service declares “these replicas are exposed like this”; an Ingress declares “this HTTP traffic comes in here”. The controller translates the declaration into real state and keeps the system converged.

Serving LLMs in 2024 was the equivalent of /etc/init.d/: every team wrote its own Deployment/Service/HPA with custom scripts for model loading, session draining and GPU handling. We covered it in the vLLM on Kubernetes article: it can be done, and it does work, but it is repetitive, fragile and nobody is extracting the right abstractions. Serving LLMs in 2026 has gone through the same transition as services: the equivalent of systemd has appeared, the inference operators, which normalise the abstractions and let the engineer declare what matters: “this model, with this runtime, this scalable, with this routing policy”.

There are four relevant operators in 2026 and a fifth common ancestor. Let us take them in order.

Why an operator, and not just a Deployment

Listing what an inference operator adds over a flat Deployment is the best way to understand what problem it solves:

The model as a first-class citizen. In a Deployment, the model is “the thing you download in an initContainer and mount as a volume”. In an operator, the model is a CustomResource with metadata (origin, fingerprint, licence, GPU requirements). It can be shared between InferenceServices, versioned, replicated to multiple nodes. It is the difference between “a file” and “a managed artifact”.

The runtime as a first-class citizen. The same goes for the runtime (vLLM/SGLang/TRT-LLM): it is not “a Docker image with flags”; it is a ServingRuntime that declares which args it accepts, which metrics it exports, which deployment types it supports (single-node, multi-node TP, PD-disag). Changing runtime is changing a reference, not rewriting every manifest.

Declarative composition. An InferenceService (the core CRD of KServe and its descendants) references a model and a runtime, declares the scaling policy, wires up observability and configures routing. The controller composes all the pieces: Deployment(s), Service, HPA, possibly a LeaderWorkerSet, a KEDA ScaledObject, a Gateway API HTTPRoute. You declare intent; the operator emits the 8 derived resources.

Operational prefill–decode disaggregation. As we saw in the PagedAttention article, separating prefill and decode into distinct pools can give 7× goodput. Modelling that with flat Deployments is feasible, but it requires coordinating two sets of pods, a transport to move the KV cache and conditional routing. An operator models it as a single InferenceService with two sub-pools.

Autoscaling on LLM metrics. The standard HPA does not understand vllm:num_requests_waiting. An operator integrates KEDA or Prometheus Adapter automatically and exposes the right metrics as knobs on the CRD.

Multi-tenancy. Multiple models on the same cluster, with quotas, priorities and fairness. One Deployment per model scaling independently is fine up to the fifth model; beyond that, coordinating GPUs between tenants becomes operationally hostile.

Model lifecycle. Weights on a shared PVC, warm-up of the first pod, rolling updates with maxUnavailable: 0, draining of active sessions, integrated observability. Things that with a flat Deployment have to be reinvented by every team.

If your workload is one model, one node, up to three replicas, a flat Deployment is enough and an operator is overkill. If your workload is two or more models, serious scaling, disaggregation or multi-tenancy, an operator stops being optional.

KServe: the common ancestor

Before the four new ones, we have to mention KServe, the parent operator from which all the rest conceptually descend. It was born as KFServing inside the Kubeflow project in 2019, was renamed KServe when it became independent in 2021, and in 2025 was accepted into the CNCF as a project incubating towards graduation.

KServe’s conceptual contribution is the InferenceService CRD, which has become the field’s common vocabulary: a declarative K8s object that joins a model (origin + metadata) with a predictor (runtime + resources) and produces a ready HTTP service. Under the hood, the controller emits Deployments, Services, HorizontalPodAutoscalers, Knative Services if you go serverless, Istio VirtualServices if you do traffic splitting.

KServe was designed in a pre-LLM era: its first use cases were traditional scikit-learn, TensorFlow and PyTorch models served as simple REST APIs. That gives it strengths (it is mature, with 6 years in production at Bloomberg, JPMorgan and others) and weaknesses (it was not designed to manage multi-node tensor parallel, prefill–decode disaggregation, or the patterns specific to LLMs).

The way the ecosystem has reacted is elegant: the new LLM operators inherit from or take inspiration from InferenceService but extend the API with LLM-specific primitives. OME is the clearest example: it uses the name InferenceService and the idea of “model + runtime → service”, but adds BaseModel, a ServingRuntime with LLM-aware flags, and deployment modes (PD-disag, multi-node) that KServe does not cover natively.

OME (Open Model Engine)

OME was published by the LMSYS team in July 2025 (announced on their blog). It is an operator that understands SGLang deeply (that is its first-class runtime) but also supports vLLM, TensorRT-LLM and Triton.

The CRD hierarchy

OME models the domain with four main CRDs:

BaseModel and ClusterBaseModel: the model itself. It defines origin (Hugging Face, S3, URL), fingerprint and metadata. The Cluster* version is global; BaseModel is namespaced. It lets multiple InferenceService objects reference the same model without duplicating the download.
FineTunedWeight: LoRA adapters or finetuned weights served on top of a BaseModel. Critical for multi-tenant setups where each customer has their own finetune.
ServingRuntime and ClusterServingRuntime: the runtime (vLLM, SGLang, and so on) with its configuration. It declares which args it accepts, which metrics it exports, which deployment modes it supports.
InferenceService: the central declarative piece, joining BaseModel + ServingRuntime + infrastructure.

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
 name: llama3-70b-prod
 namespace: inference
spec:
 model:
 name: meta-llama-3-70b-instruct  # reference to a BaseModel
 runtime:
 name: sglang-h100  # reference to a ServingRuntime
 deploymentMode: PrefillDecodeDisaggregated  # standard | PD | MultiNode | Serverless
 prefill:
 minReplicas: 2
 maxReplicas: 8
 resources:
 requests:
 nvidia.com/gpu: 4
 decode:
 minReplicas: 4
 maxReplicas: 16
 resources:
 requests:
 nvidia.com/gpu: 1
 router:
 type: cache-aware  # SGLang router with cache awareness
 autoscaling:
 metricSource: keda
 metrics:
 - type: prometheus
 metricName: vllm_requests_waiting
 threshold: "10"

That is what the operator takes as input. The output is roughly 8 derived resources that would be horrible to declare by hand: two LeaderWorkerSets (one per prefill/decode pool), two Services, a Deployment for the router, KEDA ScaledObjects for each pool, a Gateway API HTTPRoute, and a PriorityClass that hooks into Kueue for gang scheduling.

The four deployment modes

OME materialises the InferenceService differently depending on deploymentMode:

Standard: one Deployment with N replicas; the classic. For small or single-GPU models.
PrefillDecodeDisaggregated: two coordinated pools; the SGLang router routes between them.
MultiNode: tensor parallel across multiple nodes via LeaderWorkerSet, with NCCL/InfiniBand. For models above 70B where a single node is not enough.
Serverless: Knative-style scale-to-zero. For sporadic workloads where the cost of keeping GPUs powered up does not pay off. The trade-off: the first request pays the model’s cold start cost (minutes).

Integration with the K8s ecosystem

OME does not invent primitives where they already exist. It leans on:

Kueue for gang scheduling: all the pods of a tensor parallel group have to start at once or not at all; Kueue guarantees it.
LeaderWorkerSet (LWS) for multi-node: workers join the leader’s Ray cluster, with an atomic lifecycle (one going down restarts the group).
KEDA for autoscaling on LLM-specific Prometheus metrics (queue depth, GPU cache usage, TTFT p95).
Gateway API and its Inference Extension for advanced routing (model-aware, prefix-aware, weighted canary).

The consequence: OME feels “idiomatically Kubernetes”. It does not introduce new concepts where they are not needed; it uses standard primitives and concentrates on what is specific to the LLM domain.

When to choose it

OME is the natural choice if SGLang is your main runtime and/or if you come from the KServe ecosystem and want an idiomatic evolution of it. It is mature but relatively young (a year old at the time of writing); expect rough edges on advanced features.

vLLM Production Stack

vLLM Production Stack is the official project from vLLM itself for production on Kubernetes. Its philosophy is the opposite of OME’s: instead of an operator with new CRDs, it is a curated Helm chart that deploys a coherent set of pieces.

The three pieces

The stack has three components:

Serving engines: vLLM pods, configured with the flags we have been seeing throughout the series (--enable-prefix-caching, --kv-cache-dtype fp8, and so on). The Helm chart lets you declare them as a list; it deploys the underlying Deployments and Services.
Request router: a proxy in front of the engines that decides which one each request goes to. It supports several policies:
- Round-robin: trivial, for a baseline.
- Session-based: pins each session to one replica to keep its KV cache.
- Prefix-aware: detects shared prefixes between requests and routes them to the replica that already has them cached.
- KV-aware: sees each replica’s gpu_cache_usage_perc and avoids the saturated ones.
- Disaggregated-prefill with native LMCache: separates prefill and decode, with LMCache as the transport for the KV cache between the two.
Observability stack: Prometheus + Grafana with ready-made dashboards. It measures TTFT, TBT (Time-Between-Tokens), throughput, queue depth and GPU memory.

LMCache and tiered KV

One of the most interesting pieces the stack brings is LMCache, which adds a KV cache with multiple tiers: GPU HBM as L1, CPU RAM as L2, local disk as L3, and optionally remote storage as L4. When a KV cache block does not fit in HBM, instead of evicting and recomputing it, LMCache drops it to a lower tier. For workloads with shared prefixes and multi-turn conversations, the saving is huge.

LMCache integrates as a sidecar to the engines and as part of the transport in disaggregated-prefill. Production Stack ships it enabled by default in its Helm chart.

A typical manifest (values.yaml)

servingEngineSpec:
 modelSpec:
 - name: llama3-8b
 repository: vllm/vllm-openai
 tag: v0.6.3
 modelURL: meta-llama/Meta-Llama-3-8B-Instruct
 replicaCount: 3
 requestCPU: 4
 requestMemory: 16Gi
 requestGPU: 1
 vllmConfig:
 enablePrefixCaching: true
 kvCacheDtype: fp8
 maxModelLen: 32768
 enableChunkedPrefill: true

routerSpec:
 routingLogic: prefix-aware  # round-robin | session | prefix-aware | kv-aware
 sessionKey: x-user-id  # when routingLogic=session

cacheserverSpec:
 enabled: true # LMCache for tiered KV
 storageBackends:
 - cpu
 - disk  # offload to local disk

observabilitySpec:
 prometheus:
 enabled: true
 grafana:
 enabled: true
 dashboards:
 - vllm-engine-metrics
 - lmcache-metrics

This is declarative but these are not CRDs: they are Helm chart values. The difference with OME is not semantic (both start from declaration) but operational: with Helm, changes go through helm upgrade; with CRDs, they go through kubectl apply. For teams already living in GitOps with Argo CD or Flux, both approaches integrate cleanly, but the flows are different.

When to choose it

If vLLM is your only runtime and you want the closest thing to “the happy path the project recommends”, this one. It is the productionised version maintained by the same people who write the engine. The downsides: it ties you to vLLM (it is not generic) and it does not solve some advanced cases such as multi-tenancy with strict quotas or gang scheduling, where OME or full-fledged operators are better.

NVIDIA Dynamo

NVIDIA Dynamo is the official successor to Triton Inference Server, announced at GTC 2025 and merged into the brand as Dynamo-Triton in March of that year. Triton had spent years as the most-used inference engine in “serious” NVIDIA infrastructures; Dynamo is what NVIDIA believes the new generation needs.

What it is exactly

Dynamo is a distributed inference framework, not exactly a Kubernetes operator. It has its own runtime (it can run engines), a scheduler (Grove), intelligent routing, multi-tier KV cache management and disaggregation. It supports SGLang, TensorRT-LLM and vLLM as engines, but the engines are executed by Dynamo, not the other way round: the model is “Dynamo manages, the engine executes”.

On Kubernetes, Dynamo is deployed via its own operator plus CRDs, normalised with the K8s integration NVIDIA formalised at the end of 2025 (covered by this InfoQ piece). The CRDs are product-specific: they define a DynamoCluster, a topology of prefill/decode workers and a routing policy.

The four contributions

Dynamo is sold on four pillars, with figures reported by NVIDIA:

Disaggregated serving built in, with its own scheduler.
Smart routing based on cache state: if a worker already has most of a prompt cached, the request goes there.
Multi-tier KV cache: analogous to LMCache, with HBM/RAM/SSD/NVMe.
Autoscaling integrated with Dynamo’s scheduler.

The marketing figure: up to 30× more throughput than legacy Triton on the same hardware. With all the caution a vendor benchmark deserves.

Grove: its own scheduler

A controversial decision in Dynamo is not to rely 100% on the Kubernetes scheduler and instead bring its own scheduler, called Grove, that understands GPU topologies. Grove decides which worker runs on which physical GPU, which interconnects (NVLink/InfiniBand) are relevant, and how to distribute tensor parallel across nodes. That gives it more control than the standard kube-scheduler.

Operationally: if your cluster is “pure Kubernetes” with kube-scheduler and heterogeneous workloads (not only LLMs), Grove adds one more component to operate. If your cluster is dedicated to LLM inference and there is already a team dedicated to running it, Grove gives you more levers.

When to choose it

Dynamo makes sense if:

Your infrastructure is NVIDIA-heavy (Hopper, Blackwell, GB200) and you want to exploit the latest in TensorRT-LLM with the familiar Triton integration, modernised.
You were already a Triton user for legacy inference (vision, recommendation) and want to keep the ecosystem.
You have a dedicated SRE team for inference and Grove’s extra operational complexity is not a problem.

It is the vendor-specific option of the quartet. In exchange it gives you NVIDIA’s support and first-rate integration with their hardware. If your organisation is already fighting NVIDIA for GPUs, they may even call you offering help with Dynamo.

llm-d

llm-d is the youngest and the most “political” of the four. In March 2026, at KubeCon Europe Amsterdam, Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA announced the joint donation of the project to the CNCF as a Sandbox project, with backing from AMD, Cisco, Hugging Face, Intel, Lambda, Mistral AI, UC Berkeley and the University of Chicago. An explicit coalition for vendor neutrality.

Philosophy

llm-d positions itself as the vendor-neutral “Kubernetes blueprint” for distributed inference. It is not a runtime; it is a system that sits on top of vLLM (the default engine) and orchestrates the control plane.

The primitives the project puts on the table:

Intelligent routing with prefix-cache awareness and load-aware balancing.
Tiered KV cache with offload to CPU and disk for multi-turn.
Prefill/decode disaggregation over fast interconnects.
Wide expert parallelism for serving very large Mixture-of-Experts (MoE) models, a critical pattern popularised by DeepSeek-V3 and Mixtral, where the experts live on different GPUs and tokens have to be routed to the right expert.

Numbers

The v0.5 release validates ~3.1k tok/s per decode GPU on B200, and up to 50k output tok/s on a 16×16 B200 prefill/decode topology. The most interesting benchmark: an order-of-magnitude reduction in TTFT against a round-robin baseline. In other words, intelligent routing is worth what they say it is.

CNCF and the future

Donating to the CNCF as a Sandbox project means neutral governance: no vendor is in charge. For an organisation wary of being tied to a single supplier, llm-d is probably the safest medium-term bet. The price: like any Sandbox project, it is not yet “boring” in the way vLLM is. There is API churn, features that move around, documentation that lags the code.

When to choose it

llm-d makes sense if:

You want multi-vendor portability without ties to NVIDIA, Red Hat or Google.
Your workload includes large MoE models (DeepSeek-V3, Mixtral 8x22B, Llama 4 Behemoth if the size is confirmed), where wide expert parallelism is decisive.
Your organisation is already comfortable with CNCF Sandbox (projects under active evolution, not yet a stable 1.0).
You want to bet on the project that will probably be the de facto standard in 2-3 years.

The common ancestor is still there: KServe

It is worth reconnecting before the comparison: KServe is still alive and heavily used in organisations that serve both LLMs and traditional models (scikit-learn, XGBoost, PyTorch CV). Its InferenceService is generic enough to serve any model, including vLLM or SGLang as a ServingRuntime. What it does not do well is the LLM-specific part: disaggregation, multi-node tensor parallel, routing with KV cache awareness. If your organisation already has KServe in production for other models, adding an LLM-specific operator alongside it (OME, vLLM Stack or llm-d) is reasonable. Fighting it all out from pure KServe is not.

Decision map

Dimension	OME	vLLM Prod Stack	NVIDIA Dynamo	llm-d
Philosophy	Classic K8s-idiomatic operator	Curated Helm chart	Framework with its own scheduler	Vendor-neutral CNCF blueprint
Own CRDs	Yes (BaseModel, ServingRuntime, InferenceService…)	No (Helm values)	Yes (DynamoCluster)	Yes (KServe-derived + extensions)
Primary runtime	SGLang (first class), also vLLM/TRT-LLM/Triton	vLLM only	TensorRT-LLM (first class), also SGLang/vLLM	vLLM (first class)
PD-disaggregation	Yes, declarative	Yes, with LMCache	Yes, own scheduler	Yes, native
Multi-node TP	Yes, via LWS	Limited	Yes, via Grove	Yes, via LWS and MoE EP
Multi-model in a cluster	Yes, mature multi-tenant	Yes (list of models in values)	Yes	Yes
Multi-LoRA	Yes, first class (FineTunedWeight CRD)	Limited	Yes	On the roadmap
Tiered KV cache	Via LMCache (external integration)	Native LMCache	Own multi-tier	Yes, native
Intelligent routing	Cache-aware via the SGLang router	Prefix-aware / KV-aware / session-based	Own smart routing	Prefix-cache + load-aware
GPU scheduler	kube-scheduler + Kueue	kube-scheduler	Grove (its own)	kube-scheduler + Kueue
Hardware	NVIDIA, AMD ROCm, Intel	NVIDIA, AMD ROCm	NVIDIA only (with emphasis)	NVIDIA, AMD, Intel — neutral
Maturity (mid-2026)	Young, evolving	Stable	Stable, vendor-driven	CNCF Sandbox, evolving fast
Governance	LMSYS (academic-industrial)	vLLM project (academic)	NVIDIA (vendor)	CNCF (neutral)
Learning curve	Medium (4 new CRDs)	Low (familiar Helm values)	Medium-high (Grove + own CRDs)	Medium (similar to extended KServe)

When to choose each one

Choose OME if:

SGLang is your main engine.
You need multi-LoRA serving in production.
The hierarchical abstraction (BaseModel → ServingRuntime → InferenceService) fits you and you come from, or live alongside, KServe.
You have an appetite for a young and very active project.

Choose vLLM Production Stack if:

vLLM is your only engine and you want to align with what the project recommends.
Your team already lives in Helm and does not want to learn new CRDs.
LMCache + advanced routing inside a single Helm chart is exactly what you need.
Your scale is medium (tens of replicas), not extreme.

Choose NVIDIA Dynamo if:

Your infrastructure is NVIDIA-heavy and you want the most optimised path for Hopper/Blackwell.
You already ran Triton for legacy inference and the transition is natural.
You accept vendor lock-in in exchange for direct NVIDIA support.
Your organisation has a dedicated SRE team for inference.

Choose llm-d if:

You want to bet on the future CNCF standard, neutral between vendors.
Your workload includes large MoE models with wide expert parallelism.
You operate multi-cloud or multi-hardware and portability is valuable.
You accept the immaturity of a Sandbox project in exchange for the bet on the future.

Choose pure KServe if:

You already serve non-LLM models and want to unify; LLMs are a minority of your workload.
You need the most conservative and mature use case.
You accept that advanced LLM features (disaggregation, MoE EP, smart routing) are yours to add with external pieces.

Concrete scenarios

Scenario A — Small startup, 1-2 models, 1-3 GPU nodes. You probably do not need an operator. Deployment + Service + HPA with KEDA metrics, as in the vLLM on Kubernetes article. When you grow to 5+ models, reassess.

Scenario B — Mid-sized company, 5-15 models, internal multi-tenant. vLLM Production Stack or OME are the reasonable options. Production Stack if vLLM is all you are going to use; OME if you want runtime flexibility and idiomatic CRDs.

Scenario C — Internal corporate platform or an external service for end customers. llm-d or Dynamo. llm-d if you value vendor neutrality; Dynamo if you live on NVIDIA infrastructure and want the path they recommend.

Scenario D — Mixed cluster of LLMs plus traditional models. KServe as the base, with an LLM operator alongside (OME is the most natural given its conceptual kinship).

Common traps

“I’ll start with pure KServe because it’s mature”. For mid-sized LLMs and up, pure KServe leaves a lot of optimisation on the table. The sensible route is KServe as the base if you live alongside other models, but with an LLM-specific operator next to it.

“I’ll wire everything by hand to understand it”. Reasonable in a PoC, suicidal in production. There are 8 derived resources per model. Multiply by 10 models. You are writing 80 YAMLs and maintaining them. Use an operator.

“I’ll pick the one I like best and pivot if I’m wrong”. Pivoting between operators is not free: although the InferenceService abstraction is converging, the details (how LoRA is modelled, how routing is configured, how metrics are exposed) vary. Migrating from OME to Dynamo is a project of weeks, not days.

“I’ll go with Dynamo because it’s NVIDIA’s and therefore better”. Only if your organisation is already aligned with its operational philosophy (its own scheduler, acceptable vendor lock-in). For many cases, vLLM Production Stack or llm-d give 95% of the value with less friction.

“Helm chart vs operator is a technical decision”. It is a cultural and operational decision. If your team ships via Argo CD with Helm values in Git, Production Stack fits without friction. If your team lives in direct kubectl apply -f and the idea of operators feels natural, OME or llm-d.

What we have not covered

Mooncake: the KV cache system shared between instances that Kimi/Moonshot runs in production across hundreds of millions of queries. It is a primitive (not a complete operator), but it integrates as a cache tier with several of the above.
Ray Serve LLM: Anyscale’s offering, on Kubernetes through KubeRay. More tied to the Ray ecosystem than to native K8s CRDs. Useful if Ray is already part of your infrastructure.
Fireworks AI, Modular MAX: commercial platforms with similar primitives, but hosted. They are not K8s operators; they are competitors at another layer.
Gateway API Inference Extension: the sigwg proposal to extend Gateway API with LLM primitives (model-aware routing, sticky sessions, fairness). In 2026 it is in alpha; the operators above are already starting to support it. When it matures, routing will stop being each operator’s problem and become part of the Kubernetes standard.
A generic inference observability stack: Prometheus + Grafana is standardising around the vllm:* metrics we covered in the vLLM article. There is an OpenTelemetry effort for LLMs (gen-ai semantic conventions) that is probably the next link in the chain.

Closing the series

This series of four articles has walked through LLM inference in production from the bottom up:

KV cache: the working memory that holds up LLM inference — why each token consumes VRAM, and how much.
vLLM on Kubernetes: the LLM inference piece that actually scales — how a model is served in production with a serious Deployment.
PagedAttention from the inside and the state of the art of the KV cache in 2026 — what happens inside the engine at block level, and what has arrived since.
This one — how many models are orchestrated in a cluster.

If you have got this far, you have the vocabulary and the map to sit in a meeting where five people throw acronyms around and place each one correctly. And, more importantly, to start making reasoned decisions about where to begin.

References

Operators and projects covered:

OME — Open Model Engine (GitHub) — LMSYS operator for LLM serving with SGLang/vLLM/TRT-LLM/Triton.
Introducing OME (LMSYS Blog, jul 2025) — announcement and architecture.
vLLM Production Stack (GitHub) — official vLLM Helm chart for K8s.
vLLM Production Stack docs — installation and configuration.
LMCache (GitHub) — KV cache with tiers.
NVIDIA Dynamo — successor to Triton.
NVIDIA Dynamo Addresses Multi-Node LLM Inference Challenges (InfoQ, dic 2025) — K8s integration.
llm-d (GitHub) — CNCF Sandbox project.
IBM, Red Hat, and Google donated llm-d to CNCF (The New Stack) — KubeCon EU 2026 announcement.
Red Hat bets big on Kubernetes inference with llm-d (SiliconANGLE, mar 2026) — coverage of the announcement.

Ancestors and primitives:

KServe (site) and KServe joins CNCF (The New Stack).
Kueue — gang scheduling.
LeaderWorkerSet — coordinated workloads such as multi-pod tensor parallel.
KEDA — autoscaling on external metrics.
Gateway API — the successor to Ingress.

Analysis and perspectives:

Building Efficient LLM Inference with the Cloud Native Quartet: KServe, vLLM, llm-d, and WG Serving (Jimmy Song) — an integrating view.
Complete Guide to llm-d CNCF Sandbox (DEV Community) — an operational walkthrough.
Previous articles on this blog: KV cache, vLLM on Kubernetes, PagedAttention deep dive.

vLLM on Kubernetes: the LLM inference piece that actually scales

Mon, 18 May 2026 13:00:00 +0200

TL;DR

vLLM is the inference engine that turns a general-purpose GPU into a productive LLM server. Its value is not in running a model, since any transformers.pipeline does that in three lines of Python, but in squeezing the GPU down to the last gigabyte and the last cycle: PagedAttention for the KV cache, continuous batching to mix requests, its own scheduler to share GPU time between sessions. Kubernetes is its natural habitat because vLLM behaves like a modern UNIX process, with a health endpoint, Prometheus metrics, orderly draining and declarable resources, and K8s already knows how to manage those. But there are traps: the standard HPA does not scale vLLM well, the model takes minutes to load, and naive rolling updates cut sessions halfway through decoding. This article takes the engine apart and then fits it, with real manifests, into a cluster that can actually serve it.

This article is the natural continuation of KV cache: the working memory that holds up LLM inference. There we explained why every token consumes VRAM. Here we look at what you do with that VRAM when you want to offer it as a service.

The analogy: a multiprocess kernel for your GPU

Imagine you have a single processor and need to serve a hundred concurrent processes without any of them blocking the rest. Nobody in their right mind would write a while-true loop dispatching processes one by one: they would install an operating system. The kernel takes care of scheduling, memory paging, isolation, priorities and cleanup on exit. The “process” becomes a convenient abstraction and the kernel does the dirty work.

vLLM is, for your GPU, what the kernel is for your CPU. As far as the GPU is concerned, a conversation with an LLM is a process that lives across many decoding steps, occupies a slice of VRAM (its KV cache) and demands compute time every time a token has to be generated. You have a hundred of those processes at once. You need to:

Share GPU time between them without pausing any of them entirely (it would be disastrous if one long conversation monopolised the GPU).
Manage memory with paging because, just as in RAM, contiguous allocation is inefficient.
Queue new requests when the GPU is saturated and serve them in a reasonable order.
Reclaim resources when a session ends.

PagedAttention is the virtual memory of the KV cache. Continuous batching is the time-slicing scheduler that shares the GPU token by token. The OpenAI-compatible server is the uniform syscall interface. Calling it a “kernel” for the GPU is marketing, but it is marketing that captures the idea well.

What vLLM does inside

Continuous batching: stop waiting for the slowest

The naive inference engine does static batching: it groups N requests, processes them until all of them are finished, returns and starts another round. The problem is obvious: if one request asks for 8 tokens and another asks for 800, the other seven wait for the slow one. GPU utilisation collapses.

Continuous batching (Yu et al., 2022, popularised by vLLM) changes the model. At every decode step, which produces one token for each active session, the engine composes the batch out of the active tokens of ALL sessions alive at that instant. When a session finishes its generation it releases its slot immediately and another request from the queue takes it. The batch never waits for the slowest session because nobody is blocked: everyone advances at one token per step.

The original paper measured 5–23× more throughput than the equivalent static batching. The exact number depends on how variable the response lengths are, but the order of magnitude holds up in practice.

The consequence for the operator is counterintuitive: a single vLLM replica performs like three naive replicas. There is no point adding pods without justifying it with real metrics.

PagedAttention: virtual memory for the KV cache

We already flagged it in the KV cache article: the naive engine reserves one contiguous block per session, sized for the worst case (max_context_len), and wastes 60–80% of the VRAM because real sessions come nowhere near their ceiling.

PagedAttention borrows the solution operating systems have been using for half a century: split the VRAM into small blocks (16 tokens in the default implementation) and keep a logical → physical page table per session. A session with 273 tokens of context occupies 18 blocks (not necessarily contiguous), and grows block by block as it generates. The paper measured <4% waste, an order of magnitude better than contiguous allocation, and that translates into 2–4× more aggregate throughput on the same hardware, because more sessions fit at once.

There is a cost: every attention operation has to be indirected through the page table. But vLLM’s CUDA kernels are written so that this indirection is cheap, and the net result is massively positive.

Prefill vs decode: two phases with opposite profiles

An LLM request has two phases with radically different GPU profiles:

Prefill: processes the entire prompt in one go. It is compute-bound: it uses the tensor cores intensively, the GPU sits above 90%, and it lasts between hundreds of ms and a few seconds depending on prompt size.
Decode: generates token by token. It is memory-bound: the compute is modest but the whole KV cache has to be read for every token, and it lasts from tens of ms per token up to minutes for long responses.

A naive server treats each request as a unit and serves the two phases in series. vLLM decouples them: it mixes requests in prefill with requests in decode in the same step (a technique called chunked prefill when it also splits long prefills into chunks). The result: the GPU is always busy doing something, the tensor cores with prefills, the HBM bandwidth with decodes, instead of oscillating between phases.

Operational implication: the “% GPU utilisation” metric from nvidia-smi is misleading. A GPU at 100% doing prefills may have its HBM bandwidth idle. A GPU at 40% doing decodes may have its HBM saturated. For LLM serving, the useful metric is effective HBM bandwidth, not the compute percentage.

Tensor parallel: when the model does not fit on one GPU

Llama 3 70B in BF16 is ~140 GB. There is no single GPU on the market that can hold it. The solution is tensor parallel: split each model layer by columns and run the partitions on N GPUs in parallel, synchronising with an all-reduce after each layer.

For N=5 GPUs and a 70B model, each GPU sees roughly 28 GB of weights. That sounds fine until you remember that the all-reduce on every layer means reading and writing large tensors between GPUs. If the GPUs share NVLink/NVSwitch (300–900 GB/s), the all-reduce is cheap. If they only share PCIe (~32 GB/s gen4 x16), the all-reduce eats half the time and throughput collapses.

The implication for K8s, which comes next: the scheduler has to guarantee that the N GPUs are physically close. That translates into NodeAffinity to the right product (NVIDIA-H100-80GB-HBM3), a single pod with nvidia.com/gpu: N (not N pods sharing) and, if multi-node is unavoidable, InfiniBand with NCCL as transport.

The OpenAI-compatible server

On top of all of the above, vLLM exposes an HTTP server with endpoints identical to OpenAI’s: /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models. It supports Server-Sent Events streaming. It supports tool calling. It supports logprobs.

The value of this is enormous and underrated: any client using the OpenAI SDK works without changes. Your application points at https://vllm.your-cluster.local/v1 instead of https://api.openai.com/v1, and everything else, the LangChain, LlamaIndex, OpenAI Python and OpenAI JS SDKs, just works. It is the main reason vLLM has gained traction over technically comparable alternatives: it is the boring option that works.

Why Kubernetes is the natural habitat

vLLM is a well-behaved process: it starts, exposes metrics, serves a health endpoint, takes SIGTERM with dignity and declares the resources it needs. Kubernetes has spent ten years perfecting the management of processes like that. The only thing K8s took a while to absorb properly was the GPU, and that is now solved.

The GPU as a primitive resource

The plumbing goes like this:

The node has the NVIDIA driver installed (or the GPU Operator installs it).
A DaemonSet, nvidia-device-plugin, registers the physical GPUs as nvidia.com/gpu resources with kubelet.
The Kubernetes scheduler sees those resources the way it sees CPU and memory, puts them in its accounting and assigns them to Pods that ask for them.
The nvidia-container-toolkit makes sure containerd injects the right devices into the container at startup.

For the pod, asking for a GPU looks like this:

resources:
 requests:
 nvidia.com/gpu: 1
 limits:
 nvidia.com/gpu: 1

Without MIG, MPS or time-slicing configured, a GPU is not shared between pods: you ask for it whole or you do not ask for it. For vLLM, which wants the entire GPU to itself, that is exactly what you want.

The vLLM Pod lifecycle

Differences from a typical webapp Pod:

Long startup. Loading 16 GB of weights into VRAM over the network takes 30 seconds in the best case and 5 minutes in the worst. A readinessProbe with initialDelaySeconds: 30 and failureThreshold: 3 kills the pod before it starts. Solution: a startupProbe with a high threshold before the livenessProbe starts evaluating.
Warm-up matters. The first prefill compiles CUDA kernels specific to the input shape. The first 2–3 requests are noticeably slower. If latency matters from second one, it is worth firing a warm-up POST after ready.
Draining is not instantaneous. SIGTERM must not kill in-flight sessions. vLLM, configured with --disable-graceful-shutdown false (the default), finishes active requests before closing. That can take 30–180 seconds. terminationGracePeriodSeconds has to accommodate it.
Hostile rollouts. A naive rolling update (maxUnavailable: 1) can leave you with no replicas serving if the new one is slow to load. Set maxSurge: 1, maxUnavailable: 0 so the new pod is Ready before the old one is drained.

Anatomy of a serious deployment

First of all: GPU Operator

Without the GPU Operator (or an equivalent manual installation), a Pod with nvidia.com/gpu: 1 stays Pending forever. What the operator installs as DaemonSets on every GPU node:

nvidia-driver-daemonset — the kernel-mode driver (if you do not have it installed at host level).
nvidia-device-plugin-daemonset — registers the GPUs as a kubelet resource.
nvidia-container-toolkit-daemonset — the containerd integration.
nvidia-dcgm-exporter — Prometheus metrics for the GPU (utilisation, temperature, ECC errors, memory).
gpu-feature-discovery — node labels: nvidia.com/gpu.product, nvidia.com/gpu.memory, and so on, essential for NodeAffinity.

The recommended installation is the official Helm chart. The delicate part is aligning the driver with the host kernel version: if the nodes run kernel 6.x, the operator needs a compatible driver branch.

A complete, annotated vLLM Deployment

The following deploys Llama 3 8B with an FP8-quantised KV cache, up to 32K of context, on an RTX 4090. It is the reference manifest; the comments explain the non-obvious decisions.

apiVersion: apps/v1
kind: Deployment
metadata:
 name: vllm-llama3-8b
 namespace: inference
spec:
 replicas: 1
 strategy:
 type: RollingUpdate
 rollingUpdate:
 maxSurge: 1
 maxUnavailable: 0 # never run out of replicas during the rollout
 selector:
 matchLabels:
 app: vllm-llama3-8b
 template:
 metadata:
 labels:
 app: vllm-llama3-8b
 annotations:
 prometheus.io/scrape: "true"
 prometheus.io/port: "8000"
 prometheus.io/path: "/metrics"
 spec:
 # Only nodes with the GPU we expect
 nodeSelector:
 nvidia.com/gpu.product: NVIDIA-GeForce-RTX-4090
 tolerations:
 - key: nvidia.com/gpu
 operator: Exists
 # Pre-download the weights if they are not on the shared PVC
 initContainers:
 - name: model-download
 image: ghcr.io/huggingface/huggingface-cli:latest
 command: ["sh", "-c"]
 args:
 - |
 if [ ! -f /models/llama-3-8b/config.json ]; then
 huggingface-cli download meta-llama/Meta-Llama-3-8B-Instruct \
 --local-dir /models/llama-3-8b --local-dir-use-symlinks False
 fi
 env:
 - name: HF_TOKEN
 valueFrom:
 secretKeyRef:
 name: huggingface
 key: token
 volumeMounts:
 - name: models
 mountPath: /models
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.6.3
 args:
 - --model=/models/llama-3-8b
 - --served-model-name=llama-3-8b
 - --tensor-parallel-size=1
 - --max-model-len=32768
 - --kv-cache-dtype=fp8
 - --enable-chunked-prefill
 - --enable-prefix-caching
 - --gpu-memory-utilization=0.92
 - --port=8000
 ports:
 - name: http
 containerPort: 8000
 - name: metrics
 containerPort: 8000 # same port as http; /metrics
 resources:
 requests:
 cpu: "4"
 memory: 8Gi
 nvidia.com/gpu: 1
 limits:
 cpu: "8"
 memory: 16Gi
 nvidia.com/gpu: 1
 startupProbe:
 httpGet:
 path: /health
 port: 8000
 periodSeconds: 10
 failureThreshold: 60 # 10 min of grace to load the model
 readinessProbe:
 httpGet:
 path: /health
 port: 8000
 periodSeconds: 5
 livenessProbe:
 httpGet:
 path: /health
 port: 8000
 periodSeconds: 20
 failureThreshold: 3
 volumeMounts:
 - name: models
 mountPath: /models
 readOnly: true # no process should write here at runtime
 - name: shm
 mountPath: /dev/shm  # vLLM uses shared memory for IPC between workers
 volumes:
 - name: models
 persistentVolumeClaim:
 claimName: model-cache
 - name: shm
 emptyDir:
 medium: Memory
 sizeLimit: 4Gi
 terminationGracePeriodSeconds: 120 # accommodates draining of active sessions
---
apiVersion: v1
kind: Service
metadata:
 name: vllm-llama3-8b
 namespace: inference
spec:
 selector:
 app: vllm-llama3-8b
 ports:
 - name: http
 port: 80
 targetPort: 8000

Five things that are not visible on a first read:

/dev/shm in memory, 4 GB. vLLM launches worker processes (one per GPU in tensor parallel, plus the driver) that communicate over shared memory. Docker’s default (64 MB) blows up as soon as the model is mid-sized. Without this, the pod starts but fails the moment it serves the first complex request.
--enable-prefix-caching. If the prompts in your workload share structure (a common system prompt, few-shot examples), vLLM reuses the KV cache of the shared part. A free 30–60% gain in TTFT.
--gpu-memory-utilization=0.92. vLLM reserves the given percentage of the VRAM for itself. The remaining 8% leaves room for activations, CUDA kernels and the overhead that does not get counted. Lowering it buys safety; raising it above 0.95 invites OOM.
A ReadOnlyMany PVC is ideal. The model does not change at runtime. Several pods can mount the same PVC without contention.
No livenessProbe shorter than the terminationGracePeriodSeconds. If a drain takes 90s and liveness kills at 60s, rollouts lose sessions.

Multi-pod tensor parallel: LeaderWorkerSet

When the model needs more GPUs than a single node has, the pattern is a group of coordinated pods, one per GPU, that behave as a single replica. For years this was modelled with a StatefulSet plus init scripts; since Kubernetes 1.32, the idiomatic primitive is LeaderWorkerSet (LWS):

apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
 name: vllm-llama3-70b
 namespace: inference
spec:
 replicas: 1
 leaderWorkerTemplate:
 size: 5 # 1 leader + 4 workers = 5 pods, 5 GPUs
 restartPolicy: RecreateGroupOnPodRestart
 leaderTemplate:
 spec:
 nodeSelector:
 nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
 containers:
 - name: vllm-leader
 image: vllm/vllm-openai:v0.6.3
 args:
 - --model=/models/llama-3-70b
 - --tensor-parallel-size=5
 - --distributed-executor-backend=ray
 # ...
 workerTemplate:
 spec:
 nodeSelector:
 nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
 containers:
 - name: vllm-worker
 image: vllm/vllm-openai:v0.6.3
 # the workers join the leader's Ray cluster

LWS guarantees startup order (workers first, leader afterwards) and an atomic lifecycle (if a worker goes down, the whole group restarts, not a single pod). Without it, the coordination is manually fragile.

A simpler alternative, if all the tensor parallel GPUs fit on a single node (the case with HGX H100 boxes with 8 GPUs and an internal NVSwitch): one single Pod with nvidia.com/gpu: 5, --tensor-parallel-size=5, and vLLM handles everything internally. No Ray, no LWS, far simpler. It is the recommended path whenever it is available.

Autoscaling: the standard HPA is no use

An HPA on CPU% is useless for vLLM. The GPU does the work; the pod’s CPU sits at 5–10% even under maximum load. Nor is the GPU utilisation percentage from dcgm-exporter any use: a pod at 100% GPU% with gpu_cache_usage_perc=15% is serving one long session without being saturated, while a pod at 60% GPU% with gpu_cache_usage_perc=95% is on the edge of evicting sessions.

The right metrics are exported by vLLM itself at /metrics (Prometheus format):

Metric	What it says	When to scale
`vllm:num_requests_waiting`	Requests queued and not yet in the batch.	If it stays above 5–10.
`vllm:num_requests_running`	Active requests in the batch.	For capacity planning, not for scaling.
`vllm:gpu_cache_usage_perc`	% of the KV cache occupied.	If sustained above 80%, there is a risk of preemption.
`vllm:time_to_first_token_seconds`	Prefill latency (histogram).	If p95 exceeds your SLA.
`vllm:e2e_request_latency_seconds`	Total latency per request.	An output metric.

For the HPA to consume them there are two routes: Prometheus Adapter (exposes custom metrics to the K8s API) or KEDA (scales on Prometheus queries directly, far more convenient). With KEDA:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
 name: vllm-scaler
 namespace: inference
spec:
 scaleTargetRef:
 name: vllm-llama3-8b
 minReplicaCount: 1
 maxReplicaCount: 8
 pollingInterval: 10
 cooldownPeriod: 120 # 2 min before scale-down (long sessions)
 triggers:
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.monitoring:9090
 threshold: '5'
 query: |
 sum(vllm:num_requests_waiting{app="vllm-llama3-8b"})

The long cooldownPeriod matters: if you drop replicas while there are sessions decoding, you kill them. Better to leave 2 minutes of slack.

Observability: the four metrics that matter

Of everything /metrics exports, a minimal dashboard needs these four:

TTFT p50/p95 (time to first token) — what the user perceives when they hit send.
TPOT p50/p95 (time per output token) — the “speed” of the streaming.
Aggregate throughput (tokens generated per second across the cluster) — for capacity planning.
Queue depth (vllm:num_requests_waiting) — the leading indicator: if it grows, everything is about to degrade.

To that, add HBM utilisation and free memory per GPU (from dcgm-exporter) to spot bandwidth saturation and fragmentation problems. A decent Grafana dashboard with those 6 charts anticipates 90% of incidents.

Two concrete scenarios

We reuse the same hardware as the previous article for continuity. The same cache arithmetic, now with the engine on top.

Scenario A — 1×RTX 4090 (workstation or small K8s node)

Topology: 1 Pod, --tensor-parallel-size=1, 1 GPU, 1 node.
Model: up to 8B BF16 (Llama 3 8B, Qwen3 8B, Mistral 7B) or up to 14B in FP8/AWQ.
PVC: the node’s local SSD. The 4090 reads 1 TB/s from HBM; an NVMe SSD at 5 GB/s takes 5 seconds to feed 25 GB of weights into VRAM, negligible next to initialisation.
HPA: irrelevant inside the 4090 (always 1 vLLM replica per GPU), but useful across nodes: 3 replicas on 3 nodes with a 4090 each, and the K8s Service spreads round-robin.
Useful concurrency: 4–8 simultaneous sessions with 8K of context, 1–2 with 32K.
Natural use case: PoC, small teams, departmental environments, edge.

The manifest above is sized for this scenario. Changing only the model and the args, the same Deployment serves Qwen, Mistral or whatever is next.

Scenario B — 5×H100 SXM (cluster with NVLink/NVSwitch)

Topology: 1 Pod with nvidia.com/gpu: 5 on an HGX node, --tensor-parallel-size=5. If the platform does not allow grouping 5 GPUs into a single Pod, a LeaderWorkerSet with 5 pods coordinated by Ray.
Model: up to 70B BF16 (Llama 3 70B) or up to 200B+ in FP8 with cache quantisation.
PVC: NVMe attached directly to the node, or fast network storage (Ceph with a 25/100 GbE network, Lustre, GPFS). Loading 140 GB of weights over a slow network takes 5 minutes per start.
HPA: irrelevant inside the 5-GPU cluster (the 5 are an indivisible unit), but useful for adding whole HGX nodes when the load passes a given threshold. This combines with Cluster Autoscaler if the underlying infrastructure allows it.
Useful concurrency: 32–128 simultaneous sessions with mid-sized contexts, 4–16 with huge contexts.
Natural use case: internal corporate service, public exposure with an SLA, multi-tenant.

A and B, side by side

Aspect	A (1×4090)	B (5×H100 SXM)
Pod topology	1 pod, 1 GPU	1 pod with 5 GPUs (or an LWS of 5)
Maximum BF16 model	8 B	70 B
TTFT @ 8K context, idle	~250 ms	~80 ms
TPOT, idle	~30 ms/tok	~15 ms/tok
Throughput @ concurrency 16	~50 tok/s/session	~200 tok/s/session
Session drain	30–60 s	60–180 s
Useful autoscaling	Replicas on peer nodes	Whole nodes via Cluster Autoscaler
Reasonable multi-tenancy	Limited: 4–8 sessions	Comfortable: 32–128 sessions
Indicative cost (hardware)	~2 K €	~250 K € (≈ 125×)

The asymmetry is the same one as in the previous article: 125× more expensive, only ~4× more throughput per session and ~10× more concurrency. What the cluster buys is not proportional; it buys access to models an order of magnitude larger and latencies low enough for interactive use at scale. If your workload is batch or asynchronous agents where latency is not critical, several 4090s come surprisingly close.

vLLM against TensorRT-LLM and SGLang

Honestly, all three are good engines. The choice depends on practical criteria, not technical ones. A decision map, not a benchmark:

Criterion	vLLM	TensorRT-LLM	SGLang
Supported hardware	NVIDIA, AMD ROCm, Intel Gaudi	NVIDIA only	NVIDIA, AMD ROCm
Pure latency (TTFT)	Good	Best: kernels compiled for the exact hardware	Good
Aggregate throughput	Excellent	Excellent	Excellent (RadixAttention)
Deployment	Trivial: Docker image + args	Complex: build an engine per model and per GPU	Moderate
OpenAI-compatible API	Native, complete	Yes, through Triton Inference Server	Yes
Support for new models	Days after release	Weeks (recompile the engine)	Days
Quantization	AWQ, GPTQ, FP8 cache	INT4/INT8/FP8, very mature	AWQ, FP8
Multi-modal	Yes (Llava, Pixtral, Qwen-VL)	Yes	Excellent, a priority
Function calling / tool use	Good	Limited	First class
Community / release cadence	Very active, weekly	Active, NVIDIA-driven	Very active, academic
Licence	Apache 2.0	Apache 2.0	Apache 2.0

When to choose each one:

vLLM: the “boring choice” that works. The path of least friction to production. If your team has no dedicated inference serving specialist, this one. It supports varied hardware, up-to-date models, a stable API and a huge community.
TensorRT-LLM: when per-request latency is the only metric that matters and your model is stable (trained in-house, not swapped every fortnight). The price of the performance is that every model plus every GPU plus every TRT version requires an engine rebuild, and that blocks fast iteration.
SGLang: for agent-dominated workloads (heavy tool calling) or complex multi-modal. Its RadixAttention, structural caching of prompts with shared prefixes, shines in ReAct-style patterns where the same system prompt is repeated thousands of times.

For most teams starting out with on-prem LLM serving, vLLM is the right answer until you have production data pushing you elsewhere.

Common operational traps

A list of gotchas that come up again and again:

The model is downloaded on every rolling update

Symptom: every deploy takes 5+ minutes to become available. Cause: there is no shared PVC. Every new pod downloads the model from Hugging Face from scratch. Fix: a ReadOnlyMany PVC on fast storage, or a local mirror of the registry (a Pod with huggingface-cli serving a directory over HTTP). In CI/CD, hydrating the PVC before the rollout is 1 line of bash.

Short readiness timeouts that kill pods while they load

Symptom: new pods go into CrashLoopBackOff during the first model load. Cause: a readinessProbe with too low a timeout fires before vLLM finishes loading; the livenessProbe finishes the job. Fix: a startupProbe with failureThreshold: 60 or more (10 minutes of grace) before liveness starts evaluating.

An unquantised KV cache, then OOM

Symptom: the pod starts fine, serves for five minutes, then gets OOMKilled when session number five arrives with a long context. Cause: a BF16 KV cache (the default) consumes twice what FP8 does. Fix: --kv-cache-dtype=fp8. Negligible quality loss in the vast majority of cases, double the capacity.

Confusing replicas with concurrency

Symptom: the HPA scales to 8 replicas under light real load and the cloud bill goes up. Latency does not improve. Cause: someone configured targetAverageUtilization: 50% on CPU, thinking that is “load”. The reality: a single vLLM replica serves dozens of simultaneous sessions. Fix: an HPA on vllm:num_requests_waiting. If the queue is empty, one replica is enough even if the GPU is at 90%.

Tensor parallel on GPUs without NVLink

Symptom: throughput 3× worse than expected, GPUs at 30%, a lot of PCIe traffic. Cause: tensor_parallel=4 on 4 GPUs connected only by PCIe; the all-reduce saturates the bus on every layer. Fix: either the GPUs share NVLink/NVSwitch (SXM/HGX models), or you use pipeline parallel (worse latency but less all-reduce), or you reduce TP and accept that the whole model does not fit.

Sessions cut off during a rolling update

Symptom: users see truncated responses during the deploy. Cause: terminationGracePeriodSeconds: 30 (the default) is not enough to drain long generations. Fix: terminationGracePeriodSeconds: 120–180. Combined with maxUnavailable: 0, rollouts are invisible to active users.

What we have not covered (upcoming articles)

vLLM with hot LoRA adapters: serving a base model plus N tenant-specific adapters without reloading weights.
Disaggregated serving: separating prefill and decode into specialised pods, each optimised for its GPU profile.
Quantization deep-dive: AWQ vs GPTQ vs dynamic FP8 vs FP4, the real trade-offs, and when to use each.
Gateway API + AI Inference Extensions: the sigwg proposal to make LLMs first-class citizens in K8s (routing by model, sticky sessions per conversation, multi-tenant fairness).
Multi-modal serving: the same runtime, a different kind of request — images, audio, embeddings.

References

Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023) — the original vLLM paper.
Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022) — the paper that popularised continuous batching.
Official vLLM documentation — operational and well maintained.
NVIDIA GPU Operator — installation and troubleshooting of the GPU layer on Kubernetes.
LeaderWorkerSet — the primitive for coordinated workloads such as multi-pod tensor parallel.
KEDA — event-driven autoscaling, ideal for scaling on queue metrics.
TensorRT-LLM and SGLang — the two most serious comparables.
LMSYS Chatbot Arena — periodic benchmarks comparing the three engines.
Previous article on this blog: KV cache: the working memory that holds up LLM inference.

KV cache: the working memory that holds up LLM inference

Mon, 18 May 2026 10:00:00 +0200

TL;DR

The KV cache is the working memory a language model keeps during a conversation. Without it, every new token would force a recomputation of the whole conversation from the start, at a cost that is quadratic in the length of the text. With it, the cost is linear, but in exchange the cache lives in VRAM and grows with every token. In practice, it is not the model that limits how much context you can serve: it is the KV cache. On an RTX 4090 running Llama 3 8B, the model fits in 16 GB and barely leaves room for ~64 K tokens of cache in total, adding up all the simultaneous sessions. Understanding that number is the difference between promising a client “128 K of context” and actually delivering it.

You are here: Deploy

This post opens the series on LLM inference fundamentals. Within the six-stage LLMOps pipeline that articulates the whole system, the KV cache lives in the Deploy stage: it is the piece that dictates how much traffic fits in your inference engine and, therefore, how much platform you can offer on top of it.

The analogy: the speaker with amnesia

Imagine you are at a two-hour technical conference. Every time the speaker is about to say a new sentence, he mentally rewinds the whole talk from the beginning, reassembles the thread, and only then carries on. His next sentence requires recalling the previous one; the one after that, the two before it; an hour in, every new word costs him an hour of recap. A conference like that would be materially impossible.

Now imagine the same speaker with a notebook in which he jots down, as he goes, the two or three key ideas of each sentence: subject, object, link to what came before. Before each new sentence he glances at the notebook and continues. His next word only costs a glance at the notebook, not a rewind of the entire talk.

That notebook, in a transformer, is called the KV cache. Without it, conversational language models would be unworkable. With it, they are commercial products. But the notebook weighs something, and understanding how much, where and why is what separates an inference infrastructure that works from one that falls over at the third concurrent client.

The mechanism itself (in plain terms)

A transformer generates text one token at a time. To decide the next token, the model applies a mechanism called attention over all previous tokens: it asks “which parts of the earlier context are relevant for predicting what comes next?”.

Internally, each input token is projected into three vectors:

Q (Query): “what I am looking for”
K (Key): “what this token offers”
V (Value): “what information this token carries”

The attention of the current token against the context is computed by multiplying its Q against the K of all previous tokens, normalising with softmax, and weighting the corresponding V. The result is a contextualised representation of the current token.

Q·Kᵀ → softmax × V

representation of token N

Here is the key point: to predict token N, I only need the new Q (that of token N) and the K, V of all previous tokens. The K and V of tokens 1..N-1 have not changed since the previous iteration. Recomputing them would be throwing work away.

The KV cache is exactly that: the memory that holds the K and V of every token already processed, in every layer of the model, so they never have to be recomputed.

Why it exists: the quadratic cost without it

Generating a text of N tokens involves N steps. At step i, attention is computed over i previous tokens. Without a cache, at every step you recompute the K, V of the i-1 previous tokens plus those of the new one. The total count of attention computations grows as:

$$\sum_{i=1}^{N} i = \frac{N(N+1)}{2} \approx \frac{N^2}{2}$$

With a KV cache, you only process the new token at each step: cost linear in N.

Cumulative compute to generate N tokens (schematic scale — exact data is in the table)

tokens generated (N) attention operations

0 1K 2K 3K 4K

with KV cache O(N) — linear

without KV cache O(N²) — quadratic

The concrete numbers are devastating:

Tokens generated	Without KV cache (operations)	With KV cache	Ratio
128	8,256	128	64×
1,024	524,800	1,024	512×
4,096	8,390,656	4,096	2,048×
32,768	536,887,296	32,768	16,384×

At 32 K tokens, the cache saves you four orders of magnitude of compute. This is not an optimisation: it is what makes conversational inference possible.

The price: how heavy the backpack is

The KV cache is paid for in VRAM. The formula, per sequence, is:

KV_size = 2 · n_layers · n_kv_heads · head_dim · context_len · bytes_per_param
↑
K and V

Per token (dropping context_len), it is a constant of the model itself. Let us look at real numbers:

Model	n_layers	n_kv_heads	head_dim	Bytes/token (BF16)	GB at 8 K ctx	GB at 32 K	GB at 128 K
Llama 3 8B (hypothetical MHA)	32	32	128	524,288	4.00	16.00	64.00
Llama 3 8B (real GQA)	32	8	128	131,072	1.00	4.00	16.00
Llama 3 70B (GQA)	80	8	128	327,680	2.50	10.00	40.00
Qwen3 8B (GQA)	36	8	128	147,456	1.12	4.50	18.00
Mistral 7B (GQA)	32	8	128	131,072	1.00	4.00	16.00

Two immediate readings:

Without GQA there is no 128 K worth talking about. A Llama 3 8B with classic multi-head attention would need 64 GB of KV cache alone for a single sequence with 128 K tokens. In other words, it does not fit on any consumer GPU. That is why Meta, Mistral and the rest adopted Grouped Query Attention.
The KV cache can be larger than the model. Llama 3 8B in BF16 takes ~16 GB. With 128 K of context, its cache is another 16 GB. A single session ties with the model in VRAM.

0 10 20 30 40 GB

0 8K 32K 64K 128K

≈ VRAM free after loading an 8B on a 4090

Llama 3 8B Qwen3 8B Llama 3 70B

The dashed red line marks the realistic VRAM available on an RTX 4090 after loading the model. Any model whose curve crosses that line will not be able to serve that context without additional strategies: cache quantisation, offload, partitioning.

Inference is memory-bound, not compute-bound

There is a common misconception: thinking that “fast GPU = fast inference”. In the regime where inference services with a KV cache actually operate, what gets measured is memory bandwidth. Every new token requires reading the K and V of all previous tokens from HBM. The compute is modest; the data movement is massive.

That is why an H100 SXM (3.35 TB/s of HBM3) can be 2-3× faster than an A100 (1.55-2 TB/s) without the clock frequency or the core count fully explaining the difference. Bandwidth explains it.

And that is also why offers of “cheap GPUs with plenty of VRAM but slow HBM” (some variants with GDDR6 or LPDDR5) disappoint in inference with long contexts: they have room to store the cache but take forever to re-read it.

Tricks to make the notebook thinner

Three techniques, in chronological order, have progressively flattened the size of the KV cache:

Multi-Head Attention (MHA). The original transformer design (Vaswani et al., 2017). Every attention head has its own K and V. Expensive in cache but theoretically maximal in expressiveness. It is what models had until around 2023.

Multi-Query Attention (MQA). A single K and V shared by all heads. It reduces the cache by a factor of n_heads. It works reasonably well but degrades generation quality on some benchmarks.

Grouped Query Attention (GQA). The middle ground that won. Heads are grouped: in Llama 3 8B, 32 query heads share K, V in groups of 4, giving 8 KV groups. It cuts the cache 4× relative to MHA with almost identical quality. It has been the de facto standard since 2024.

Multi-Head Latent Attention (MLA). DeepSeek-V2/V3’s innovation: instead of storing K, V per head, it compresses the state into a smaller latent vector and projects out to K, V on the fly. The cache can come down to 70 bytes/token, two orders of magnitude less than GQA. It is the main reason DeepSeek-V3 (671 B parameters, 37 B active) is servable on affordable infrastructure.

Note: the MLA bar is illustrative, using typical values published by DeepSeek; the exact implementation depends on the latent size. What matters is the order of magnitude.

On top of this comes a fourth, orthogonal technique: quantising the cache to FP8, INT8 or even INT4. vLLM and TensorRT-LLM already support it in production. Going from BF16 (2 bytes) to FP8 (1 byte) halves the cache at a small cost in quality. Going to INT4 divides it by four, at a somewhat higher cost.

The next dragon: fragmentation

So far we have talked about the cache as if it were a contiguous block. In practice, an inference server handles dozens of simultaneous sessions, each with its own cache growing at a different rate. Naive allocation, reserving the maximum possible per session, wastes between 60 % and 80 % of the VRAM according to the original PagedAttention paper.

session A

session B

session C

session D

→ ~70 % of VRAM reserved and empty

 <rect x="0" y="22" width="30" height="20" class="used blk"/>
<rect x="30" y="22" width="30" height="20" class="used blk"/>
<rect x="60" y="22" width="30" height="20" class="used blk"/>
<rect x="90" y="22" width="30" height="20" class="used blk"/>
<rect x="120" y="22" width="30" height="20" class="used blk"/>
<rect x="150" y="22" width="30" height="20" class="used blk"/>
<rect x="180" y="22" width="30" height="20" class="free blk"/>
<rect x="210" y="22" width="30" height="20" class="free blk"/>
<rect x="0" y="44" width="30" height="20" class="used blk"/>
<rect x="30" y="44" width="30" height="20" class="used blk"/>
<rect x="60" y="44" width="30" height="20" class="used blk"/>
<rect x="90" y="44" width="30" height="20" class="free blk"/>
<rect x="120" y="44" width="30" height="20" class="free blk"/>
<rect x="150" y="44" width="30" height="20" class="free blk"/>
<rect x="180" y="44" width="30" height="20" class="free blk"/>
<rect x="210" y="44" width="30" height="20" class="free blk"/>
</g>

→ < 4 % waste (vLLM paper)

PagedAttention, the idea from Kwon et al. (2023) that gave rise to vLLM, solves this by borrowing a technique from operating systems: divide the VRAM into small blocks (typically 16 tokens) and keep a page table mapping logical to physical blocks per session. A session no longer reserves one huge contiguous block: it grows one block at a time, and the blocks can be scattered across the VRAM. The result is an effective occupancy of 90 % instead of 30 %, and therefore 2-4× more aggregate throughput on the same hardware.

PagedAttention deserves an article of its own. I am noting it down for the next one.

Applied to generic on-premise hardware

Let us come down to concrete cases.

Case 1 — RTX 4090 (24 GB, Ada Lovelace)

A typical configuration with Qwen3-8B BF16:

Model in BF16: ~16 GB
Activations + overhead: ~2 GB
VRAM available for KV cache: ~6 GB (with margin)

At 144 KB/token (Qwen3-8B GQA), that comes to ~43 K total tokens of cache distributed across all simultaneous sessions. In practice:

Concurrency	Maximum context per session
1	32,768
4	8,192
8	4,096
16	2,048

If you need to advertise “we support 32 K of context” with concurrency of 4 or more, you have to quantise the cache (FP8 brings it down to 72 KB/token, doubling capacity) or move up the model range (a 4B with GQA and a quantised cache would have room to spare).

Case 2 — Cluster of 4×H100 (320 GB total, NVLink)

With tensor parallel = 4 and Llama 3 70B BF16:

Model in BF16: ~140 GB (35 GB/GPU)
vLLM overhead per GPU: ~2 GB
VRAM free for KV per GPU: ~43 GB → ~172 GB aggregate

At 320 KB/token (Llama 3 70B GQA), that comes to ~537 K total tokens of cache. Ample margin for long contexts with high concurrency:

Concurrency	Maximum context per session
4	134,000
16	33,500
64	8,375

For DeepSeek-V3 671 B with MLA the economics change radically, because the cache is ~100× thinner. What limits you is no longer the cache but the VRAM of the model itself (quantised to FP8 it is ~335 GB, so it fits on 4×H100 with room for the KV cache).

Operational implications

Three observations we repeat in every consulting engagement:

First, the maximum context a model advertises is not the one you can serve on your hardware. Llama 3 8B “supports” 128 K, but on a 4090 with 4 simultaneous sessions your effective context is ~8 K. It is trivial to check before promising it to a client.

Second, quantising the KV cache is one of the optimisations with the best cost/benefit ratio in the ENS context. It does not touch the weights, it does not affect audit reproducibility, and it doubles capacity. vLLM supports it via --kv-cache-dtype fp8.

Third, if the SLAs dictate long contexts with many concurrent users, GQA is necessary but not sufficient. In the medium term you need to look at models with MLA or attention variants with compression.

What we have not covered (upcoming articles)

PagedAttention and its implementation in vLLM: blocks, page table, eviction.
Prefix caching: when several requests share the system prompt, there is no need to recompute the K, V of the common part.
Speculative decoding and its interaction with the cache.
Cache offloading: moving cold blocks to RAM or NVMe, a key technique for contexts above 1 M.

References

Vaswani et al., Attention Is All You Need (NeurIPS 2017) — foundational transformer paper.
Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (EMNLP 2023).
Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023) — original vLLM paper.
DeepSeek-AI, DeepSeek-V2 Technical Report (2024) — introduction of Multi-Head Latent Attention.
Official vLLM documentation: https://docs.vllm.ai/.
Llama 3 model card (Meta): GQA specs, n_layers, n_kv_heads.

Inference on lo0 — Tech Blog

The blog map: from the utility feed to the token served

The map

Layer 0 · Physical infrastructure

Layer 1 · The node

Layer 2 · Storage

Layer 3 · The cluster

Layer 4 · The inference engine

Layer 5 · The model

Layer 6 · Data and RAG

Layer 7 · Control plane and delivery

Layer 8 · Observability and evaluation

Layer 9 · Security, trust and compliance

Layer 10 · Cost, performance and energy

The master maps

Four routes

What is missing

See also

MLPerf Inference: how to read it, what LoadGen enforces and what comparability it offers

TL;DR

What it is and how it is organised

The four scenarios

LoadGen: what it measures and what it does not

Duration, query count and early stopping

The compliance tests

Divisions, categories and what can be touched

The three divisions

The availability categories

What Closed allows and forbids

The SLOs almost nobody cites: TTFT and TPOT

Quality: the 99 % and the token count

The current Datacenter suite

How to read a results row

Per-accelerator is not an official metric

The comparison rules

How much audit sits behind a row

The recent rounds, in figures

What MLPerf Inference does not measure

Usage flow for an on-premise platform

Cross-links from the benchmarking track

See also

Sources

Cost, performance and energy: the three axes of LLM inference and the identity that ties them together

What this series is

The three axes and their metrics

Cost axis in detail

Performance axis in detail

Energy axis in detail

The identity that ties the three axes together: throughput

Worked example: unit economics of an 8×H100 node in Europe

Energy is inside the cost

Sensitivity analysis: what moves the cost per token

Simplified TCO model (on-premise)

Reference: the price of a GPU-hour in European sovereign cloud

Energy and carbon reference: France, Germany, Spain

Sovereignty and regulation: the axis that does not appear in the CPM

Methodology: how to read (and not fool yourself with) these figures

The combined scorecard

Where each axis is measured: from design to production

Glossary of units and conversions

How the rest of the series fits together

See also

Sources

Serving reasoning models: the invisible scratchpad that decides your latency and your bill

TL;DR

The analogy

What changes when the model “thinks”

The 2026 landscape: from the model that “always reasons” to the switchable hybrid

The economics: why the scratch paper drives up the bill

Controlling the thinking budget

1. Switching think / non-think

2. Setting a cap: thinking_token_budget

3. Budget forcing (state of the art)

The impact almost nobody measures: the KV cache

Reasoning + structured output: order matters

Reference architecture

Operational pitfalls (and honest scepticism)

Closing

Sources

Long context and KV offloading: when the notebook does not fit on the desk

2. Setting a cap: `thinking_token_budget`

Embedder (`bge-m3`) on GPU

Reranker (`bge-reranker-v2-m3`) on GPU