Throughput on lo0 — Tech Blog

Cost, performance and energy: the three axes of LLM inference and the identity that ties them together

Sat, 13 Jun 2026 03:30:00 +0200

Notation: amounts in euros (N €), decimals with a point. Cost and energy references are European (France, Germany, Spain), because the aim is to defend a sovereign European architecture; when a source quotes prices in dollars it is flagged as “USD”. The dollar symbol is not used (on this site it is a formula delimiter).

What this series is

This is the first instalment of a series of 28 data articles (not opinion pieces) about the three dimensions that decide whether a sovereign European LLM platform is sustainable: cost (FinOps), performance (benchmarking) and energy. The aim is to have a body of verifiable figures, with their methodology, to support architecture decisions with numbers rather than intuition. This first article sets the frame: what each axis measures, how they link to one another, an end-to-end worked example with European data and the 2026 references that will be used as a base.

The quantitative thesis, in one sentence: the three axes are not independent. Throughput (tokens per second) is the common denominator of cost per token and energy per token, and energy is, on top of that, between 30 % and 50 % of the cost itself. Optimising one moves the other two. And in Europe, where the cluster sits (France, Germany, Spain) changes both the electricity cost and the carbon per token at once.

The three axes and their metrics

Axis	Question	Key metrics	Unit	Tool family
Cost (FinOps)	How much does each unit of work cost?	cost/1M tokens (CPM), cost/request, €/GPU-hour, utilisation	€/1M tok, €/h	OpenCost, Kubecost, CloudZero
Performance	How much work per second and at what latency?	TTFT, TPOT/ITL, throughput, goodput, P50/P99	tok/s, ms	GuideLLM, AIPerf, LLMPerf, MLPerf Inference
Energy	How many watts per unit of work?	energy/token, average power, PUE, carbon	J/tok, Wh, kWh, gCO₂	Zeus, CodeCarbon, MLPerf Power, Kepler

Each axis has its own in-depth introductory article in this same batch; here only how they fit together matters. It is worth breaking each one down a little more before crossing them.

Cost axis in detail

Metric	Definition	Why it matters
CPM (cost/1M tokens)	cluster cost ÷ tokens produced	the comparable unit on-prem vs cloud
Cost per request	cost attributed to one complete request	billing by real usage
€/GPU-hour	hourly cost of a GPU (amortised or rented)	the basis of any calculation
Utilisation	fraction of the GPU actually used	the lever that moves everything else
Fixed vs variable cost	amortised capex vs energy/operations	defines the break-even point

Performance axis in detail

Metric	Definition	Dominant phase
TTFT	latency to the first token	prefill
TPOT / ITL	time between output tokens	decode
Throughput	tokens or requests per second of the system	both
Goodput	throughput that meets the SLO (not the raw figure)	the honest number
P95 / P99	latency percentiles (not the mean)	the real experience

Energy axis in detail

Metric	Definition	Unit
Average power	average instantaneous consumption	W
Energy per token	energy per generated token	J/token
Total energy	integral of power over time	Wh, kWh
PUE	datacenter overhead (cooling, losses)	ratio ≥ 1
Carbon	energy × grid intensity	gCO₂eq

The identity that ties the three axes together: throughput

The cost per million tokens (CPM) is the cluster’s cost divided by what it produces:

$$\text{CPM} = \frac{\text{cluster cost per hour}}{\text{throughput (tok/s)} \times 3600 / 10^6}$$

Energy per token has exactly the same form, with power in the numerator:

$$\text{energy per token (J)} = \frac{\text{average power (W)}}{\text{throughput (tok/s)}}$$

The two share the same denominator: throughput. Hence the key operational consequence: every performance improvement that raises tokens/s lowers both cost per token and energy per token at the same time. These are not three separate optimisations; throughput moves them together. That is why performance benchmarking (axis 2) is, at bottom, also a FinOps tool and an energy efficiency tool.

The CPM↔throughput relationship is direct: the higher the throughput, the lower the CPM. It is the identity on which the whole cost analysis of the series is built.

Worked example: unit economics of an 8×H100 node in Europe

To see the identity in action, an end-to-end calculation on an example node (8×H100 SXM, Llama 3.1 70B FP16 served with vLLM), with European prices. Order-of-magnitude figures; the methodology is what matters.

Starting data:

Parameter	Value	Source
European cloud rental (Scaleway, per H100)	2.73 €/h	Scaleway
→ 8×H100 node on-demand	~21.84 €/h	8 × 2.73
Amortised on-prem cost (estimate)	~10–13 €/h	capex 24–36 months + opex
Throughput	2,800 tok/s	reference benchmark
Power per H100 (TDP)	up to 700 W	NVIDIA
Reference PUE	1.4	typical of a datacenter

Cost per million tokens. At 2,800 tok/s, hourly throughput is 2,800 × 3,600 / 10⁶ = 10.08 million tokens/hour. Therefore:

$$\text{CPM (cloud)} = \frac{21{.}84}{10{.}08} \approx 2{.}17 \text{ €/1M tok}$$

With the amortised on-prem cost (~11 €/h) it drops to ~1.09 €/1M tok, the difference between renting from a provider and amortising your own iron.

Energy per token. The power of the 8 cards is 8 × 700 = 5,600 W (board only). Over 2,800 tok/s:

$$\text{energy per token} = \frac{5,600}{2,800} = 2 \text{ J/token (board)}$$

Over a million tokens that is 2 × 10⁶ J = 0.56 kWh (GPU only). Applying PUE 1.4: ~0.78 kWh per million tokens. At French wholesale prices (~0.058 €/kWh) that is ~0.045 € of electricity per million tokens; in Germany (~0.091 €/kWh), ~0.071 €, some 57 % more, because of the grid, not the model.

The link. If a serving optimisation raised throughput from 2,800 to 4,200 tok/s (+50 %), the CPM would fall to ~1.45 €/1M tok and energy per token to ~1.33 J, both at once, through the same denominator. That is the identity, with numbers.

Energy is inside the cost

The second link: energy is not an axis separate from cost, it is a large component of it. 2026 reference data:

Figure	Value	Source
Electricity as a share of TCO (5 years)	30–40 % (>50 % at high density 24/7)	Spheron
Day-ahead wholesale France (Mar-2026)	~58 €/MWh (0.058 €/kWh)	TradingEconomics
Day-ahead wholesale Spain	~54 €/MWh (0.054 €/kWh)	TradingEconomics
Day-ahead wholesale Germany	~91 €/MWh (0.091 €/kWh)	TradingEconomics
Electricity cost/token at 80 % util	¼ of that at 20 % util	Spheron

A methodological warning: the wholesale price (day-ahead) is not what a company pays; industrial retail adds grid charges and taxes (roughly ×1.5–2). But the relative ordering holds: France and Spain, cheap; Germany, expensive.

The third link, utilisation, closes the triangle: power is consumed even when the GPU is idle, so spreading the fixed cost (energy included) over more tokens (more throughput, more occupancy) lowers all three axes at once. The electricity cost per token at 80 % utilisation is a quarter of what it is at 20 %.

Sensitivity analysis: what moves the cost per token

Four levers dominate the CPM, and it is worth knowing their elasticity before defending any figure:

Lever	Change	Effect on CPM	Mechanism
Throughput	+50 % tok/s	−33 % CPM	direct denominator
Utilisation	20 % → 80 %	up to −75 % on the electricity portion	spreads fixed cost
Location (energy)	Germany → France	lowers the electricity portion (~37 % less €/kWh)	wholesale FR < DE
Amortise vs rent	21.84 → 11 €/h	−50 % CPM	removes the provider’s margin
Quantisation (FP8/INT4)	+throughput, −VRAM/KV	lowers CPM and J/token	same identity

The reading: the CPM is not a number, it is a function of utilisation, throughput, location (energy price) and ownership model. Any on-prem vs cloud comparison that does not fix those assumptions is not comparable, and that is exactly the job of the synthesis article.

Simplified TCO model (on-premise)

The real on-premise cost is not the GPU alone: it is amortised capex + opex, over a hardware cycle of 24–36 months. Reference structure of an inference node:

Component	Type	Typical weight	Notes
GPUs	capex	the bulk	amortise over 24–36 months
Server, network, BMC	capex	medium	NICs, switching, out-of-band management
Electricity	opex	30–50 % of TCO	depending on country (FR/ES cheap, DE expensive) and PUE
Cooling	opex	inside the PUE	PUE 1.2–1.6 typical
Maintenance and support	opex	medium	spares, contracts, RMA
Staff and operations	opex	variable	the hidden cost that cloud rental includes

The difference between cloud rental (21.84 €/h in the example) and the real amortised cost (~10–13 €/h) is this breakdown: the provider charges a margin for absorbing capex, cooling and operations. The break-even point appears when sustained volume exceeds ~2M tokens/day, with typical amortisation in 6–12 months (Spheron). Below that threshold, renting usually wins; above it, amortising. Article A8 develops the full model and synthesis S3 connects it with sizing.

Reference: the price of a GPU-hour in European sovereign cloud

The other side of the on-prem comparison is the price of renting the GPU. For a sovereign European platform, what matters is not AWS/GCP/Azure (subject to the US CLOUD Act), but the European providers:

Provider	H100 (€/hour)	Location	Sovereignty
Scaleway	from 2.73	Paris, Warsaw	EU jurisdiction, exempt from US CLOUD Act
OVHcloud	+40–45 % vs Scaleway	France, Germany	EU jurisdiction, exempt from US CLOUD Act
Hyperscalers (reference)	3–7 USD	global	subject to US CLOUD Act

The data: Scaleway offers the H100 from 2.73 €/h in European datacenters (Paris, Warsaw), with models and data under EU jurisdiction, immune to extraterritorial regulation; OVHcloud serves it from France and Germany, ~40–45 % more expensive than Scaleway as of March 2026. Both are exempt from the US CLOUD Act, the foundational sovereignty requirement for data subject to GDPR (Scaleway, DeployBase).

The implication for the proposal: the comparison is not only cost, it is cost + sovereignty. An on-prem cluster of your own or a European cloud keep the data under EU jurisdiction; a US hyperscaler does not, however cheap the GPU-hour comes out. That is the leg that the cost axis on its own does not capture, and that synthesis S2 quantifies.

Energy and carbon reference: France, Germany, Spain

Here the choice of European country changes both the electricity cost and the carbon per token. It is the most important table for a sovereign proposal:

Country	Wholesale (€/MWh, Mar-2026)	Grid carbon (gCO₂/kWh)	Profile
France	~58	~20–60	nuclear: cheap and very clean
Spain	~54	~150–170	renewables + gas: cheap, medium carbon
Germany	~91	~363 (2024 average)	coal/gas + renewables: expensive and dirty

Sources: day-ahead wholesale prices (TradingEconomics), German carbon ~363 gCO₂/kWh (2024 average, Statista via Wikipedia), nuclear ~17–35 gCO₂e/kWh (EEA), Spain ~167 gCO₂/kWh (2020). The PUE multiplies all of that energy: at 1.4, for every kWh of GPU the datacenter spends 1.4 kWh with cooling included.

The calculation, on the example node (0.78 kWh per million tokens, with PUE): in France (~40 gCO₂/kWh) it is ~31 gCO₂ per million tokens; in Germany (~363 gCO₂/kWh), ~283 gCO₂ per million, a factor of ~9×. That is, the same load served in France emits of the order of 9× less carbon per token than in Germany, and costs ~37 % less in electricity, without touching a single line of the stack, purely because of the country’s grid. Siting the cluster in France or Spain is at once a cost lever, a carbon lever and a compliance lever. Hourly grid intensity is obtained from ElectricityMaps, the reference source for these calculations.

Scale context: global datacenters consumed ~460 TWh in 2025 (≈1.8 % of the world’s electricity), and the IEA projects passing 800 TWh in 2028, which is why energy efficiency per token stops being a detail and becomes a design constraint, and why carbon enters the European regulatory equation.

Sovereignty and regulation: the axis that does not appear in the CPM

For a European platform there is a fourth factor that the three numerical axes do not capture, and that can invalidate the lowest cost figure: jurisdiction. Three fronts that turn “the cheapest GPU-hour” into “the cheapest GPU-hour that complies”:

Front	What it requires	Architectural implication
GDPR + US CLOUD Act	personal data under a jurisdiction accessible to US authorities is exposed	European cloud (Scaleway/OVHcloud) or on-prem keep the data in the EU
EU AI Act	obligations for high-risk systems (traceability, control, documentation)	on-prem control makes auditability and compliance easier
CSRD / emissions reporting	reporting the carbon footprint (scope 2)	a low-carbon grid (France) improves the reported figure

The key point for the proposal: a US hyperscaler can offer a competitive GPU-hour, but it puts the data under the US CLOUD Act, which for data subject to GDPR is a compliance risk, not a saving. That is why this series’ comparison is not made in USD against US providers: it is made in euros, against options that keep data sovereignty, on-prem of your own or European cloud. Cost, energy and carbon are measurable axes; sovereignty is the constraint that decides which options even enter the table. This links with the mapping of the EU AI Act onto the on-premise architecture and with the technical controls ENS × ISO 42001 × EU AI Act.

The operational conclusion: France and Spain not only come out cheaper and cleaner than Germany; being EU jurisdiction, they meet the sovereignty constraint that a US hyperscaler does not meet at any price. Three advantages that reinforce each other, and that the proposal can defend with the same table.

Methodology: how to read (and not fool yourself with) these figures

Five warnings that apply to all the data in this series:

Board vs node. The GPU’s power (700 W per H100) is the floor; the complete node (CPU, NICs, power supplies, fans) consumes more. Always compare the same boundary.
Wholesale vs retail. The day-ahead price is not what the company pays; industrial retail adds grid charges and taxes. Use whichever matches your contract.
Measured vs estimated. A wattmeter figure and an nvidia-smi estimate are not the same thing; the difference is quantified in the energy articles.
Amortisation assumptions. The on-prem cost/hour depends on the hardware cycle (24 vs 36 months) and on the assumed utilisation; changing those assumptions moves the CPM more than any serving optimisation.
Comparability. Two CPM or J/token figures are only comparable if model, precision, context length, load and country match. Most public comparisons (and in USD) do not fix those assumptions, which is why the series reproduces the ones that matter with its own harness and European data.

The combined scorecard

The destination of the series is a single board where each model/configuration/location is scored on all the axes at once, to decide on a Pareto frontier rather than on a single number. An illustrative example (order-of-magnitude figures) on the same 8×H100 node:

Candidate	CPM (€/1M)	Goodput (tok/s)	Energy (J/tok)	Carbon (gCO₂/1M)	Sovereignty
A · 70B FP16, on-prem France	~1.09	~2,800	~2.0	low (grid ~40 gCO₂/kWh)	EU
B · 70B FP8, on-prem France	~0.75	~4,000	~1.4	lower	EU
C · Scaleway on-demand	~2.17	~2,800	n/a	low (FR)	EU
D · US hyperscaler	~3.0	~2,800	n/a	depends on region	not EU

The reading: B dominates A on the three intrinsic axes thanks to FP8. C (European cloud) is more expensive per token but shifts capex and operations, while keeping sovereignty. D (hyperscaler) may be competitive on cost but loses sovereignty, an axis that for GDPR data is not negotiable. No isolated number decides: the choice comes from seeing the whole row.

The trap the board avoids: optimising a single axis. Each cell will be filled in by an article in the series with measured data.

Where each axis is measured: from design to production

Each axis is measured at three different moments, with different tools and increasing precision. Confusing them is a common source of figures that do not add up:

Axis	Design (estimate)	Benchmark (controlled)	Production (continuous)
Cost	datasheet, GPU-hour price	CPM measured over synthetic load	OpenCost/Kubecost per team and token
Performance	theoretical throughput (roofline)	GuideLLM/AIPerf with SLO	OTel/Langfuse over real traffic
Energy	TDP × hours	Zeus/MLPerf Power on the bench	Kepler/DCGM per pod in the cluster

The rule: a design figure serves for sizing, but not for defending a proposal; for that you need a reproducible benchmark figure, and to operate, a continuous production one. When someone disputes a number in the proposal, the answer is not “a blog says so”, but “here is the bench, here is the methodology, reproduce it”.

Glossary of units and conversions

Quantity	Unit	Conversion / reference
Energy	joule (J), watt-hour (Wh)	1 Wh = 3,600 J; 1 kWh = 3.6 MJ = 3,600,000 J
Power	watt (W)	1 W = 1 J/s; H100 ≈ 700 W TDP
Energy per token	J/token	= average power (W) ÷ throughput (tok/s)
Throughput	tok/s	tokens generated per second by the system
Cost per token	€/1M tok (CPM)	= cost/hour ÷ (tok/s × 3,600 / 10⁶)
Latency	ms	TTFT (first token), TPOT/ITL (between tokens)
PUE	ratio (≥1)	total DC energy ÷ compute energy
Carbon	gCO₂eq	= energy (kWh) × PUE × grid intensity (gCO₂/kWh)

A useful conversion to keep in your head: at a constant 700 W, an H100 consumes ~0.7 kWh per hour, ~16.8 kWh a day and ~6,130 kWh a year (board only, no PUE). Multiplied by the PUE and your country’s energy price (FR ~0.058, ES ~0.054, DE ~0.091 €/kWh wholesale), that number is the basis of the electricity portion of the TCO.

How the rest of the series fits together

The 28 articles group into four tracks; this is the map:

Track	What it contributes	Closes with
FinOps (8)	measuring and allocating cost, TCO model	cost/token comparable on-prem vs European cloud
LLM benchmarking (8)	measuring performance reproducibly	Pareto frontier of engines
Energy (8)	measuring watts and carbon per token	energy in the TCO and in European regulation
Synthesis (4)	crossing the three axes	scorecard, on-prem vs cloud, sizing, harness

The three articles that follow this one in the batch introduce each axis in depth: their frameworks and the state of the art. From there, each article adds a layer of data to the final scorecard.

The thread running through all 28 instalments is the one in this article: cost, performance and energy are a single problem with three faces, tied together by throughput and by the energy that lives inside the cost, and in Europe, by the country where the cluster sits. Anyone who optimises one face while ignoring the others will end up with a pretty number and a platform that does not hold up. The work of the series is to produce the figures, measured, dated, European and reproducible, that allow a sovereign architecture to be defended before any committee with the only answer that closes a technical debate: the data and the bench to reproduce it.

Sources

Scaleway · H100 GPU instance (€/h price, EU sovereignty) — https://www.scaleway.com/en/h100/
DeployBase · OVHcloud GPU Pricing (European sovereignty) — https://deploybase.ai/articles/ovhcloud-gpu-pricing
TradingEconomics · electricity price France / Germany / Spain — https://tradingeconomics.com/france/electricity-price
EEA · emission intensity of electricity in Europe — https://www.eea.europa.eu/en/analysis/indicators/greenhouse-gas-emission-intensity-of-1
Electricity sector in Germany (carbon ~363 gCO₂/kWh) — https://en.wikipedia.org/wiki/Electricity_sector_in_Germany
Spheron · AI Inference Power & Cost Economics (2026) — https://www.spheron.network/blog/ai-inference-cost-economics-2026/
NVIDIA · H100 (TDP/specs) — https://www.nvidia.com/en-us/data-center/h100/

LLM inference benchmarking: frameworks, metrics and the state of the art (tool by tool)

Sat, 13 Jun 2026 02:30:00 +0200

Notation: amounts in euros (N €), decimals with a point. Performance is not very sensitive to the country, but its associated cost (cost per token) is expressed in € and links back to the opening article.

What this introduction covers

Third article in the data series, a deep dive into the performance axis. Measuring the performance of an inference engine looks trivial (“how many tokens per second?”) and it is exactly where people fool themselves most: two tools can report results that differ by a factor of 7 for the same system. This article inventories the metrics that matter and how they are defined, why the architecture of the tool biases the figure, how the saturation point is found with a concurrency sweep, and the profile of each framework. No recommendations: the choice of engine is settled in the Pareto article (B8); here there are only the facts and the methodology, because a benchmark without a published methodology is not comparable.

The performance metrics

There is no single performance metric, there are five, and mixing them is the first source of error:

Metric	Definition	Unit	Dominant phase
TTFT (Time To First Token)	time from sending the prompt to the first token	ms	prefill
TPOT / ITL (Time Per Output Token / Inter-Token Latency)	average time between output tokens once generation has started	ms/token	decode
Request throughput	complete request-response cycles per second at the concurrency tested	req/s	both
Token throughput	total tokens (input + output) per second across all concurrent requests	tok/s	both
Goodput	share of requests that meet the defined SLO	useful tok/s	both
P50 / P95 / P99	latency percentiles (not the mean)	ms	—

Precise definitions, because every tool computes them its own way (Anyscale · latency and throughput metrics): TTFT is what a user waits before seeing the first character, dominated by prefill compute; ITL is the average time between successive output tokens and sets the perceived “typing speed” of the answer; request throughput is complete cycles per second; token throughput is total tokens (input plus output) per second across all concurrent requests.

The latency decomposition

The total latency of a request with $N$ output tokens decomposes as:

$$\text{latency} \approx \text{TTFT} + (N-1)\times \text{TPOT}$$

That is why TTFT and TPOT are reported separately: a single mean latency hides very different profiles. A system with high TTFT and low TPOT (expensive prefill, fast decode) and another one the other way round can have the same mean latency for one particular length, yet behave in opposite ways when the response size changes. For an interactive chat experience TTFT and TPOT rule; for a batch of long summaries, token throughput does. Measuring the mean hides both realities.

Goodput: the honest metric

Raw throughput (TPS, RPS) says how much work the system does; goodput says how much of that work meets your quality-of-service standards (SLO) (Anyscale). An engine can boast 10,000 aggregate tok/s, but if half the requests violate the P99 TTFT SLO, its goodput is 5,000. The figure you defend in a proposal is the goodput, not the catalogue throughput: it is the only one that translates into satisfied users and an honest cost per token.

How each metric is instrumented (and where the error creeps in)

Before comparing numbers it helps to know where each clock starts and stops, because two tools can call different things “TTFT”:

Client-side versus server-side TTFT. The TTFT measured by the client includes network latency and gateway queueing; the one measured by the server does not. For an engine comparison the server-side figure is the one that matters; for user experience, the client-side one. Mixing the two invalidates the comparison.
Streaming is required. TTFT and ITL can only be measured if the response arrives as a stream (token by token). If the tool measures complete responses, there is no real TTFT: there is total latency in disguise.
Token counting. Throughput in tok/s depends on which tokenizer counts the tokens. If the tool uses a tokenizer different from the model’s, the token count (and therefore the tok/s and the cost per token) is biased. You have to count with the tokenizer of the model being served.
Warm-up and prefix cache. The first requests of a benchmark benefit from a warm prefix cache and give artificially low TTFT; the warm-up has to be discarded or the result inflates reality.

These four instrumentation decisions explain a good share of the discrepancies between tools. A performance figure that does not say where the clock is measured and with which tokenizer is not comparable, however precise it may look.

The architecture of the tool biases the figure

Here is the trap that invalidates half the published benchmarks. Tools split into two classes by the architecture of the client that generates the load, and that architecture determines whether the measurement is reliable at high concurrency:

Single-process micro-bench (vLLM bench, SGLang bench, genai-perf): a Python client with asyncio in a single process. Useful for quick experiments on one engine, but the single-process architecture introduces a bottleneck on the client side that biases the data at high concurrency (genAI-perf and vLLM): the client cannot generate enough load and you measure the limit of the client, not the limit of the engine.
Multi-process load (GuideLLM, AIPerf): they spread load generation across several processes, avoiding that limit. This is the class that has emerged for measuring at real scale.

The size of the bias is enormous: at 1,000 QPS, a single-process benchmark processed 75,574 tokens against the 545,733 tokens of a distributed architecture, a discrepancy of 7.2× in measurement capacity for the same system ([search]). Anyone comparing two engines with tools from different classes is not comparing the engines: they are comparing the benchmark clients.

Frameworks, one by one

vLLM bench and SGLang bench — engine micro-bench

What they measure: TTFT, TPOT and throughput of the engine itself (vLLM or SGLang). Class: single-process micro-bench. Use: quick experiments to tune one engine and see the effect of its optimisations (see decode and prefill). Limit: they saturate on the client at high concurrency; they are no good for measuring real capacity at scale.

AIPerf — the NVIDIA one (formerly genai-perf), multi-process

What it measures: TTFT, ITL, throughput and latency against vLLM, NIM, TGI and any compatible endpoint. Class: multi-process load. State-of-the-art datum: NVIDIA retired genai-perf and replaced it with AIPerf on 15 April 2026. During the sweep, AIPerf detects GPU saturation and identifies the previous iteration, returning it as estimatedCapacity; if it detects no saturation, estimatedCapacity is the last iteration tested, which is why the sweep has to extend beyond the knee (AIPerf).

GuideLLM — from the vLLM project, SLO-oriented

What it measures: full distributions of TTFT, ITL and end-to-end behaviour, for SLO-driven evaluation. Class: multi-process load. Differentiator: it generates realistic, configurable traffic patterns in synchronous, concurrent and rate-based modes, including reproducible sweeps to identify safe operating ranges (Red Hat · GuideLLM, GuideLLM · GitHub). It is the tool for answering “how hard can I load this engine without breaking the SLO?”.

LLMPerf — the Anyscale/Ray classic

What it measures: throughput and latency at the inference level. Use: endpoint validation, historically very widespread. Class: load generator. Limit: less focused on distributions and sweeps than GuideLLM/AIPerf.

MLPerf Inference — the industry standard

What it measures: performance under standardised scenarios with strict rules, for comparability across vendors. Maintainer: MLCommons. It is the gold standard of cross-vendor comparability; it is expanded on below.

Comparison table

Tool	Class	What it measures	Maintainer	When to use it
vLLM bench	single-process micro	TTFT, TPOT, throughput of vLLM	vLLM (OSS)	tune vLLM, quick experiments
SGLang bench	single-process micro	metrics of the SGLang engine	SGLang (OSS)	tune SGLang
AIPerf (formerly genai-perf)	multi-process load	TTFT, ITL, throughput; estimatedCapacity	NVIDIA (OSS)	real capacity, multi-endpoint
GuideLLM	multi-process load	distributions, SLO, sweeps	vLLM (OSS)	validate SLO, find the knee
LLMPerf	load	throughput and latency	Anyscale/Ray (OSS)	endpoint validation
MLPerf Inference	standard suite	server/offline/interactive scenarios	MLCommons	cross-vendor comparability

The concurrency sweep: finding the knee

The most useful measurement for sizing is not a number, it is a curve: how latency and throughput change as concurrency rises. Raising concurrency keeps the GPU busier and raises RPS, but past a certain point it sends TTFT, ITL and end-to-end latency through the roof ([search]). The goal of the sweep is to find the knee: the maximum concurrency where throughput still rises without latency breaking the SLO.

That is why AIPerf extends the sweep beyond the knee: only by seeing where latency takes off can it return the safe capacity (estimatedCapacity). This curve is the raw material of capacity planning: the knee yields the replica count and the cost per token at the target load.

Worked example: reading a sweep

An illustrative sweep on an example node (a 70B on 8×H100, SLO of P99 TTFT < 500 ms), to see how the knee is read:

Concurrency	RPS	TTFT P50 (ms)	TTFT P99 (ms)	Token tput (tok/s)	Goodput
1	2	80	110	350	100 %
8	14	110	240	2,100	100 %
16	22	180	460	3,400	98 %
24	26	320	980	3,900	62 %
32	27	540	1,800	4,000	20 %

Reading: up to concurrency ~16 throughput grows and P99 stays under the SLO (goodput ~100 %). Between 16 and 24 sits the knee: throughput barely rises any more (3,400 → 3,900 tok/s) but P99 takes off (460 → 980 ms) and goodput collapses (98 % → 62 %). At concurrency 32 raw throughput is at its maximum (4,000 tok/s) but goodput is 20 %: the system “performs well” while serving mostly requests that violate the SLO. The defensible safe capacity is the one at concurrency ~16, not the one at maximum throughput. This is the number that goes into capacity planning and into cost per token: at 3,400 useful tok/s, not 4,000 raw tok/s.

MLPerf Inference: the comparability standard

To compare across vendors and engines with identical rules there is MLPerf Inference (MLCommons). The datacenter category focuses on two scenarios plus an optional one (MLCommons · datacenter):

Scenario	What it simulates	Metric
Offline	raw throughput processing the whole dataset in batch	maximum throughput
Server	interactive environment: requests one at a time following Poisson at a mean RPS	RPS under TTFT and TPOT limits
Interactive (optional)	like server but with stricter latency limits	RPS under a hard SLO

The Server scenario is the realistic one for online inference: the load generator sends requests following a Poisson distribution and demands that specific TTFT and TPOT bounds be met. MLPerf Inference v5.0 (April 2025) introduced a large-scale 405B benchmark and a low-latency interactive 70B one, offering language benchmarks at every scale (7B to 405B), architectural diversity (including MoE) and scenarios (MLCommons · v5.0); v5.1 (September 2025) widened the results with record participation (MLCommons · v5.1).

The value of MLPerf is comparability: everyone measures the same thing under the same rules. Its limit is that those rules may not match your workload (your length distribution, your specific SLO), so it serves to compare hardware and engines against each other, not necessarily to size your case. For that, your own sweep.

Measurement bias and reproducibility

That two benchmarks give very different results for the same system is no accident: systematic measurement bias in production benchmarks is characterised in the literature (arXiv 2605.24217), and there is work dedicated to the meta-metrics and good practices of system-level performance benchmarking (arXiv 2508.10251). The most common sources of bias:

Source of bias	Effect	Mitigation
Single-process client	underestimates throughput at high concurrency	use multi-process load
Unrealistic length distribution	results that do not apply to your traffic	use realistic traces
Measuring the mean instead of percentiles	hides the latency tail	report P95/P99
Uncontrolled warm-up	the prefix cache inflates the first results	discard the warm-up
Not pinning engine/model version	irreproducible	pin everything and publish it

The methodological conclusion of this article: a benchmark without a published methodology is not comparable. For a performance figure to support a proposal it has to come with the tool, its version, the model and precision, the load distribution and the SLO. Synthesis article S4 builds a reproducible harness that pins all of that.

Checklist for a reproducible benchmark

For a performance figure to be defensible before a committee, it has to come with everything that allows it to be reproduced. The minimum published alongside the result:

What to pin	Why
Tool + version	each one measures differently; the version changes behaviour
Model + precision (FP16/FP8/INT4)	precision changes throughput and quality
Hardware (GPU, count, interconnect)	an 8×H100 NVLink is not an 8×H100 PCIe
Engine + version + flags	vLLM/SGLang/TRT-LLM and their configuration
Length distribution (in/out)	real traffic is not fixed length
Concurrency levels of the sweep	you have to go past the knee
SLO (which percentile, which threshold)	it defines the goodput
Warm-up handling	discard it or it biases the result
Tokenizer used for counting	it affects tok/s and cost per token

The practical rule: if you cannot hand over this table alongside the figure, the figure is not a datum, it is an anecdote. The reproducible harness of article S4 automates the recording of all these parameters so that anyone, including whoever challenges the proposal, can reproduce the exact number.

Performance ≠ quality

One warning that avoids the most expensive mistake: these tools measure speed and throughput, not accuracy. An engine can be blisteringly fast while serving a model that answers badly. Quality is measured with another family of tools (lm-evaluation-harness, HELM, task leaderboards) which is another axis of the scorecard (article B7). Confusing “fast” with “good” is how you build a platform that serves bad answers very quickly. On the final Pareto frontier, performance and quality are two distinct axes that must be looked at together, never one instead of the other.

The other family of tools, for reference (expanded in B7):

Tool	What it measures	Common trap
lm-evaluation-harness	accuracy on hundreds of standardised tasks	contamination of the test dataset
HELM	holistic evaluation (accuracy, robustness, bias, efficiency)	heavy to run
LiveBench / dynamic leaderboards	tasks that rotate to avoid contamination	comparability over time

The data point: contamination, the model having seen the test during training, inflates quality metrics just as warm-up inflates performance ones. That is why dynamic leaderboards rotate their questions. Quality and performance share that lesson: the measurement method biases the result as much as the system being measured.

The latency versus throughput trade-off

A property the sweep reveals and which is worth keeping in mind: latency and throughput pull in opposite directions. Batching groups requests to amortise the cost of moving the weights out of VRAM, which raises throughput, but each request waits for the batch to form, which raises individual latency. There are two operating regimes, and the benchmark serves to place you in the right one:

Regime	Optimises	Configuration	Case
Latency	low TTFT/TPOT	small batch, low concurrency	interactive chat, copilots
Throughput	maximum tok/s	large batch, high concurrency	overnight batch, ingestion

There is no single “best” point: there is the best point for your SLO. A benchmark that reports only maximum throughput is describing the throughput regime and ignoring whether that point meets the latency your case needs. That is why goodput, throughput under the SLO, is the metric that reconciles the two regimes: it measures how much throughput you get without stepping outside acceptable latency. The sweep traverses the curve between both regimes; your SLO marks where on that curve your system sits.

The connection with cost and energy

Performance is not an isolated axis: by the identity of the opening article, throughput is the denominator of cost per token and of energy per token. A sweep that finds a knee at 4,000 tok/s rather than 2,800 is not just “faster”: it lowers the CPM from ~1.09 to ~0.76 €/1M tok and the energy per token in the same proportion, on the same iron. That is why performance benchmarking is the tool that, indirectly, moves cost the most: every goodput improvement translates into euros and into watts per token. The number that connects the three axes is goodput, the throughput that meets the SLO, not catalogue throughput.

State of the art 2026

genai-perf → AIPerf migration (15 Apr 2026): NVIDIA consolidates its benchmarking into a multi-process tool with saturation detection.
GuideLLM as the OSS standard for SLO-driven evaluation with reproducible sweeps.
MLPerf Inference v5.0/v5.1 extends to 405B, interactive 70B and MoE, with record participation: cross-vendor comparability is maturing.
Measurement bias characterised: the community accepts that the measurement method matters as much as the system measured; the emphasis on reproducibility and meta-metrics is growing.

Limits and traps (data-driven)

Comparing tools from different classes. A single-process micro-bench and a multi-process load generator are not comparable; the difference can be 7×. Pin the class.
Catalogue throughput instead of goodput. The honest number is the one that meets the SLO.
Means instead of percentiles. The mean hides the tail; report P95/P99.
Not extending the sweep beyond the knee. Without seeing where latency takes off you do not know the safe capacity.
Confusing performance with quality. They are distinct axes; fast is not good.
Not pinning versions. Engine, model, precision and load left unpinned = irreproducible = not defensible.

The next article in the track (B2) goes into the tool catalogue in depth; this one pins the metrics and the methodology. With performance measured reproducibly, the scorecard can cross it with cost (in €) and energy for the final decision.

Closing

Performance benchmarking looks like the most “objective” of the three axes (they are tokens per second, after all) and it is exactly where the most manipulation happens, almost always without bad intent: a single-process tool here, a mean instead of a P99 there, a catalogue throughput instead of the goodput. The difference between a marketing number and a defensible datum lies not in the engine measured but in the methodology: the class of tool, where the clock is measured, with which tokenizer, how far the sweep goes and which SLO defines the goodput. For a sovereign architecture proposal, performance only counts if it is delivered with that reproducibility record, and, crossed with cost in euros and energy per token, it becomes the column of the Pareto frontier that decides which engine and which configuration hold up the platform. The number you defend is not the highest one: it is the reproducible goodput.

Sources

Anyscale · métricas de latencia y throughput de LLM — https://docs.anyscale.com/llm/serving/benchmarking/metrics
Red Hat · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
NVIDIA AIPerf · guía de benchmarking (ex genai-perf) — https://lucaberton.com/blog/nvidia-aiperf-llm-inference-benchmarking-guide/
MLCommons · MLPerf Inference datacenter (escenarios) — https://mlcommons.org/benchmarks/inference-datacenter/
MLCommons · MLPerf Inference v5.0 (405B + 70B interactivo) — https://mlcommons.org/2025/04/llm-inference-v5/
MLCommons · MLPerf Inference v5.1 — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
arXiv 2605.24217 · sesgo sistemático de medición en benchmarks de inferencia LLM — https://arxiv.org/html/2605.24217
arXiv 2508.10251 · meta-métricas y buenas prácticas de benchmarking de rendimiento — https://arxiv.org/pdf/2508.10251
Medium · LLM Inference Benchmarking (genAI-perf y vLLM) — https://kchandan.medium.com/llm-inference-benchmarking-genai-perf-and-vllm-5dd06b57428e

Batch sizing in vLLM: the two-hour grid search worth weeks of hardware

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

max-num-seqs and max-num-batched-tokens are the two dials that control how much work vLLM processes on each scheduler iteration. Their default values are calibrated to be safe on any hardware, not to maximise throughput on yours. A systematic grid search over 25 configurations, runnable in two hours, identifies the combination that, for your specific workload and hardware, can double throughput without changing a single line of the model or adding a GPU. The OTel metrics that confirm you found the optimum are vllm:num_waiting_seqs, vllm:num_preemptions_total and vllm:time_per_output_token_seconds.

The analogy

An industrial kitchen with one chef and ten stoves. If the maître d’ only sends one order at a time, the chef works at 10% of capacity. If he sends a hundred simultaneous orders but there are ingredients for only twenty, the chef spends half his time waiting for restocking. The optimum is the point where every stove is lit and the resupply never runs dry.

max-num-seqs is how many orders the chef can have in preparation at once. max-num-batched-tokens is how many ingredients he can process in a single movement of the wok. Getting either of them wrong leaves stoves empty.

The problem: the defaults are not for your hardware

In vLLM V1 (≥ 0.6), the defaults are:

max-num-seqs = 1024 (V1) / 256 (V0)
max-num-batched-tokens = 8192

These values guarantee that vLLM starts on any GPU without OOM. They do not guarantee optimal throughput. The reason: the optimal point depends on three variables vLLM does not know at start-up:

The length distribution of your real workload — a RAG system with 2K-token prompts needs a different budget from a chat with 50-token messages.
VRAM available for KV cache — determined by the model, the quantisation and --gpu-memory-utilization.
Real expected concurrency — how many simultaneous users arrive at the 95th percentile.

The interaction between these three factors makes it impossible for a universal default to be optimal in specific cases.

The arithmetic that matters

On each iteration, vLLM’s scheduler decides which tokens to process. The total budget available per step is max-num-batched-tokens. That budget is split between:

Decode tokens: 1 for each active request in the generation phase. With 64 requests in decode, 64 tokens of budget are consumed.
Prefill tokens (in chunks): the rest of the budget goes to processing new prompts.

$$\text{prefill tokens per step} = \text{max num batched tokens} - \text{num requests decode}$$

If max-num-batched-tokens = 8192 and you have 512 requests in decode, each step can only process 8192 - 512 = 7680 prefill tokens. With 2000-token prompts, that is ~3.8 new prompts per iteration.

The problem appears when max-num-seqs is very high relative to the available KV cache. Each active request in decode occupies KV cache blocks. If the blocks run out, vLLM performs preemption: it pauses a request, frees its KV cache and re-queues it. Each preemption costs additional latency for the paused request and complexity for the scheduler.

$$\text{KV budget} = \frac{\text{free VRAM} \times \text{gpu memory utilization}}{\text{bytes per token} \times \text{max model len}}$$

For a Qwen2.5-14B on an RTX 4090 with Q4_K_M (9 GB of model, 15 GB free):

$$\text{KV budget} = \frac{15 \times 0.92 \times 10^9}{40\,000} \approx 345\,000 \text{ tokens}$$

With max-model-len = 8192, the maximum number of simultaneous requests with a full context is:

$$\text{real max seqs} = \frac{345\,000}{8192} \approx 42 \text{ requests}$$

Setting max-num-seqs = 1024 with those numbers guarantees constant preemptions. The optimum is at 40-50.

The grid search: methodology

Step 1: measure the real workload

Before hunting for the optimum, you need to know your traffic’s percentiles. From Langfuse or the vLLM logs:

# Extract the length distribution from Langfuse
import langfuse
client = langfuse.Langfuse()

traces = client.fetch_traces(limit=1000).data
prompt_lens = [t.input_tokens for t in traces if t.input_tokens]
output_lens = [t.output_tokens for t in traces if t.output_tokens]

import numpy as np
print(f"Prompt p50={np.percentile(prompt_lens,50):.0f} p95={np.percentile(prompt_lens,95):.0f} p99={np.percentile(prompt_lens,99):.0f}")
print(f"Output p50={np.percentile(output_lens,50):.0f} p95={np.percentile(output_lens,95):.0f} p99={np.percentile(output_lens,99):.0f}")

Step 2: work out the KV budget

Run once with --dry-run, or read vLLM’s start-up log:

INFO: # GPU blocks: 4521, # CPU blocks: 512

Each block is 16 tokens. 4521 × 16 = 72,336 tokens of total KV budget.

Step 3: the grid

With the KV budget known and the p95 of prompt/output length:

# grid_search_batch.py
import subprocess, json, time

MODEL = "Qwen/Qwen2.5-14B-Instruct-AWQ"
PROMPT_LEN = 512 # p50 of your workload
OUTPUT_LEN = 256
CONCURRENCY = 32 # simultaneous users expected at peak

seqs_values = [32, 64, 128, 256, 512]
tokens_values = [4096, 8192, 16384, 32768, 65536]

results = []
for seqs in seqs_values:
 for tokens in tokens_values:
 cmd = [
 "python", "-m", "vllm.entrypoints.benchmark_throughput",
 "--model", MODEL,
 "--max-num-seqs", str(seqs),
 "--max-num-batched-tokens", str(tokens),
 "--num-prompts", "200",
 "--input-len", str(PROMPT_LEN),
 "--output-len", str(OUTPUT_LEN),
 ]
 out = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
 # Parse throughput from the output
 for line in out.stdout.splitlines():
 if "Throughput" in line:
 tps = float(line.split(":")[1].strip().split()[0])
 results.append({"seqs": seqs, "tokens": tokens, "tps": tps})
 print(f"seqs={seqs} tokens={tokens} → {tps:.1f} tok/s")

# Save for analysis
with open("grid_results.json", "w") as f:
 json.dump(results, f, indent=2)

25 configurations × ~5 min = ~2 hours. Real execution time, not waiting time.

Step 4: read the surface

The result is a 5×5 throughput matrix. The typical shape:

max-num-batched-tokens → 4K 8K 16K 32K 64K
max-num-seqs ↓
32 180 310 380 390 385 ← max-num-seqs too low
64 185 350 480 510 508 ← optimal point for this workload
128 182 340 450 480 475
256 178 320 400 410 402 ← KV cache runs out, preemptions
512 170 290 360 370 368 ← high preemptions

The optimum in this example: max-num-seqs=64, max-num-batched-tokens=32768. Above that, preemptions cancel out the concurrency gain.

Confirmation with OTel

Once the optimal configuration is deployed, three Prometheus metrics confirm it is well calibrated:

# 1. Queued requests — should stay close to 0
# If it grows steadily: max-num-seqs too low or max-num-batched-tokens insufficient
vllm:num_waiting_seqs

# 2. Preemptions — should be 0 or very occasional (<1/min)
# If it grows: max-num-seqs too high for the available KV cache
rate(vllm:num_preemptions_total[5m]) * 60

# 3. ITL (inter-token latency) — should be stable, with no spikes
# Bimodality = badly calibrated batch size (some requests outside the CUDA graph bucket)
histogram_quantile(0.99, rate(vllm:time_per_output_token_seconds_bucket[5m]))

The optimal configuration produces:

num_waiting_seqs ≈ 0 under normal operation
num_preemptions_total stable (not growing)
time_per_output_token unimodal

If num_waiting_seqs is high with gpu_cache_usage_perc low: raise max-num-batched-tokens to process prefills faster. If num_preemptions_total grows: lower max-num-seqs or enable FP8 KV cache to free up blocks.

Reference configurations by profile

Based on the grid search for mid-range hardware (generic 4×H100, 14B-70B model):

Profile	Prompt p50	Output p50	max-num-seqs	max-num-batched-tokens
Conversational chat	150 tok	300 tok	256	16384
Enterprise RAG	1500 tok	200 tok	64	32768
Coding (completion)	800 tok	500 tok	128	32768
Summarisation	2500 tok	400 tok	32	65536
Batch processing	4000 tok	800 tok	16	65536

None of these is universal. They are starting points for the grid search on your real hardware and workload.

When not to touch the defaults

If your system sits below 50% KV cache utilisation (vllm:gpu_cache_usage_perc < 0.50) under real demand and with no num_waiting_seqs, the defaults are enough for your current load. The grid search contributes more when you are close to maximum capacity or when you want to extract the full performance of fixed hardware.

References

One big replica or many small ones: the decision that defines your platform

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

With 4 GPUs available you have two basic options: one vLLM instance using all 4 (TP=4) or two independent instances using 2 each (TP=2 × 2 replicas). The first gives lower latency per individual request. The second gives higher aggregate throughput at high concurrency, better fault tolerance and finer-grained scaling. The crossover point, where the second overtakes the first, is typically between 16 and 64 concurrent requests for 70B models, much earlier than most people assume. The metric that decides it: goodput, the tokens generated within the latency SLO divided by the total.

The analogy

Two ways to organise a translation service: one senior translator with access to four specialised dictionaries at once (able to resolve any complex query in 30 seconds), or two junior translators with two dictionaries each (they take 45 seconds per complex query, but they can handle two at the same time).

For a client who arrives alone and expects a fast answer, the senior wins. For a queue of twenty clients arriving at once, the two juniors process twice as many queries per hour even though each one takes longer. The question is not who is better, but what kind of traffic you have.

The two architectures in vLLM

Architecture A: TP=4, one replica

# A single instance uses all 4 GPUs via tensor parallelism
vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
 --tensor-parallel-size 4 \
 --gpu-memory-utilization 0.92 \
 --port 8000

GPU-0 ─┐
GPU-1 ─┤─ vLLM instance 0 ──► port 8000
GPU-2 ─┤ (TP=4, the model is
GPU-3 ─┘ split across 4 GPUs)

Every attention and FFN operation is split across 4 GPUs. They require an all-reduce after each layer (on NVLink: ~50-200 µs; on PCIe: ~2-8 ms). The complete model is available in the aggregate VRAM.

Architecture B: TP=2 × 2 replicas

# Two independent instances, each with 2 GPUs
# Instance 0 on GPU 0-1
CUDA_VISIBLE_DEVICES=0,1 vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
 --tensor-parallel-size 2 --port 8000

# Instance 1 on GPU 2-3
CUDA_VISIBLE_DEVICES=2,3 vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
 --tensor-parallel-size 2 --port 8001

GPU-0 ─┐ ┌─► port 8000
GPU-1 ─┘─ vLLM instance 0 ─┘
← load balancer
GPU-2 ─┐ ┌─► port 8001
GPU-3 ─┘─ vLLM instance 1 ─┘

Each instance holds half the model. Requests are distributed across instances. No communication between instances (they are completely independent).

Why TP=4 has higher individual latency than TP=2

Tensor parallelism splits every transformer layer. After computing its fraction, each GPU needs to synchronise with the others via all-reduce before moving on to the next layer. The cost of that synchronisation:

$$\text{overhead TP} = n\_layers \times 2 \times \text{allreduce latency}$$

For Llama 3 70B (80 layers) on 4×H100 NVLink:

$$\text{overhead TP4} = 80 \times 2 \times 100\,\mu s = 16\,ms$$

On PCIe (without direct NVLink between GPUs):

$$\text{overhead TP4 PCIe} = 80 \times 2 \times 3\,ms = 480\,ms$$

That overhead is added to every decode step. With TP=2:

$$\text{overhead TP2} = 80 \times 2 \times 60\,\mu s = 9.6\,ms \text{ (NVLink)}$$

The difference between TP=2 and TP=4 on NVLink is ~6 ms per decode step, which matters for TPOT (inter-token latency) in streaming applications.

On PCIe without direct NVLink, TP=4 can be 400 ms slower per step than TP=2. For an output of 200 tokens, that is 80 extra seconds. In that scenario TP=4 over PCIe should never be used unless the model does not fit in 2 GPUs.

The crossover point: when TP=2×2 beats TP=4×1

For a 70B model on 4×H100 SXM (NVLink), aggregate throughput in tokens per second:

Concurrency | TP=4 × 1 instance | TP=2 × 2 instances | Winner
──────────────┼──────────────────────┼───────────────────────┼────────
1 | 200 tok/s | 170 tok/s | TP=4 (latency)
4 | 650 tok/s | 620 tok/s | TP=4 (slight)
16 | 1,800 tok/s | 2,100 tok/s | TP=2×2
32 | 2,400 tok/s | 3,600 tok/s | TP=2×2 (+50%)
64 | 2,800 tok/s | 5,200 tok/s | TP=2×2 (+86%)
128 | 2,900 tok/s | 5,800 tok/s | TP=2×2 (+100%)

Why they diverge at high concurrency: with TP=4, the scheduler of a single instance manages every request but the KV cache is shared. With TP=2×2, each instance has its own scheduler and KV cache: less contention, more real parallelism.

The crossover point on NVLink sits around 16-32 simultaneous requests for 70B. For smaller models (14B, 7B) the crossover happens earlier, because the TP communication overhead weighs more in relative terms.

The three implications nobody mentions

1. Fault tolerance

With TP=4 × 1 replica: if one GPU fails, the whole instance goes down. The service drops to 0% until the GPU recovers or the pod restarts on another node.

With TP=2 × 2 replicas: if one GPU fails, one instance goes down. The service keeps running at 50% capacity. For ENS/NIS2, where availability is a contractual requirement, this difference is decisive.

2. Autoscaling granularity

With KEDA or HPA based on vllm:num_waiting_seqs, autoscaling has to provision in multiples of the deploy unit:

TP=4 × 1: each new node requires 4 GPUs. The minimum scaling granularity is 4 GPUs.
TP=2 × 2: each new pod requires 2 GPUs. The minimum granularity is 2 GPUs, which is finer and more cost-efficient.

3. Quality degradation under load

TP=4 with many concurrent requests starts to suffer preemptions when the KV cache fills up. TP=2×2 spreads that pressure across two independent KV cache pools, so the probability of preemption is lower under the same total load.

Measuring the crossover point with OTel

Goodput is the right metric for comparing the two architectures. Not raw throughput (which ignores the SLO), but the tokens generated within the agreed TPOT SLO:

# Goodput: tokens generated with TPOT inside the SLO (e.g. <50ms/token)
# For TP=4:
rate(vllm:generation_tokens_total{instance="tp4"}[5m])
 * (1 - histogram_quantile(0.95, rate(vllm:time_per_output_token_seconds_bucket{instance="tp4"}[5m])) > 0.050)

# For TP=2×2 (sum of the two instances):
sum(rate(vllm:generation_tokens_total{instance=~"tp2-.*"}[5m]))
 * (1 - histogram_quantile(0.95, sum(rate(vllm:time_per_output_token_seconds_bucket{instance=~"tp2-.*"}[5m]))) > 0.050)

A direct comparison on the same dashboard, with synthetic traffic at different concurrency levels, determines the exact crossover point for your hardware and model.

The decision by workload profile

Profile	Recommended architecture	Reason
Single-user chatbot / low concurrency (<10 simultaneous)	TP=4 × 1	Lower p50 latency, better streaming experience
Enterprise API (20-100 concurrent)	TP=2 × 2	Higher goodput, fault tolerance, finer autoscaling
Batch processing (throughput > latency)	TP=2 × 2 (or more replicas)	Maximum throughput always lies in replicas
Very large model (>80B, does not fit in 2 GPUs)	TP=4 × 1	No structural alternative
ENS / contractual availability	TP=2 × 2 minimum	Losing one GPU is not catastrophic

Kubernetes configuration for both architectures

# Parallel deployments for an A/B test or different topologies

# TP=2 instances (2 replicas per deployment)
apiVersion: apps/v1
kind: Deployment
metadata:
 name: vllm-tp2
spec:
 replicas: 2
 template:
 spec:
 containers:
 - name: vllm
 args: ["serve", "meta-llama/Meta-Llama-3.1-70B-Instruct",
 "--tensor-parallel-size", "2",
 "--gpu-memory-utilization", "0.92"]
 resources:
 limits:
 nvidia.com/gpu: "2" # 2 GPUs per pod
---
# Service with load balancing across the 2 replicas
apiVersion: v1
kind: Service
metadata:
 name: vllm-tp2
spec:
 selector:
 app: vllm-tp2
 ports:
 - port: 8000
 sessionAffinity: ClientIP  # for prefix cache awareness
 sessionAffinityConfig:
 clientIP:
 timeoutSeconds: 10800 # 3 hours of affinity per session

The sessionAffinity: ClientIP setting in the Kubernetes Service is the simplest way to implement session-affinity routing: requests from the same client always go to the same replica, maximising the prefix cache hit rate on the conversation history.

References

Prefix cache: hit rate engineering to go from 15% to 75%

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

vLLM’s prefix cache stores the KV cache blocks of shared prefixes and reuses them in later requests. A hit avoids recomputing that prefix: TTFT drops to the cost of the variable suffix only. In enterprise workloads with fixed system prompts (RAG, domain chatbots, assistants with long instructions) the hit rate should be >70%. In practice it is 10-20% for entirely avoidable reasons. This article identifies them, fixes them, and gives the OTel queries to confirm the result.

The analogy

A simultaneous conference interpreter who has to translate the speeches of twenty speakers. All of them open with the same two-page protocol preamble: the conference declaration, the rules of conduct, the day’s programme. An interpreter with no memory rereads the two pages for each speaker before starting on their specific speech. An interpreter with good notes reads them once, files them, and when the second speaker starts goes straight to the speech.

The prefix cache is that file. The prefix hash is the reference that lets you skip to the new part. But if the preamble changes by even one word, because someone drops in the day’s date, the interpreter has to reread everything from the beginning.

How the prefix cache hash works

vLLM splits the KV cache into blocks of 16 tokens. Each block has a hash computed over its exact content. When a new request arrives, vLLM checks whether any initial block of the prompt is already in cache by comparing hashes.

The hash is computed over the byte-by-byte content of the tokens. Any difference, a space, a different character, one extra token, produces a completely different hash. There is no partial matching inside a block.

Direct consequence: if your system prompt has 512 tokens and token number 3 changes between requests (because you interpolate a date, an ID, a version number), no block hits even though 99% of the text is identical.

Block 0 (tokens 0-15): hash = a3f7... ← in cache?
Block 1 (tokens 16-31): hash = 9d2c... ← in cache?
...
Block 31 (tokens 496-511): hash = 7e1a... ← in cache?

If block 0 does not hit (because its content changed), blocks 1-31 are not even checked although they are identical. The prefix cache is sequential.

Audit: why your real hit rate is low

Before changing anything, you need to know what is breaking the hash. The most direct method: pull the last 1000 production prompts and work out what fraction of the prefix varies.

# audit_prefix_cache.py
import langfuse, hashlib, collections
from transformers import AutoTokenizer

client = langfuse.Langfuse()
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-14B-Instruct")

traces = client.fetch_traces(limit=1000).data
prompts = [t.input for t in traces if t.input]

# Tokenise and extract the first 512 tokens (the typical system prompt)
prefixes = []
for prompt in prompts:
 tokens = tokenizer.encode(prompt, add_special_tokens=False)
 prefix_tokens = tuple(tokens[:512])
 prefixes.append(prefix_tokens)

# How many unique prefixes are there?
unique = len(set(prefixes))
total = len(prefixes)
print(f"Unique prefixes: {unique}/{total} ({unique/total*100:.1f}%)")
print(f"Theoretical hit rate if they were all equal: {(1 - unique/total)*100:.1f}%")

# Find which token differs between the most common prefix and the rest
from collections import Counter
most_common_prefix = Counter(prefixes).most_common(1)[0][0]

divergence_positions = []
for prefix in prefixes:
 if prefix == most_common_prefix:
 continue
 for i, (a, b) in enumerate(zip(most_common_prefix, prefix)):
 if a != b:
 divergence_positions.append(i)
 break

if divergence_positions:
 pos = Counter(divergence_positions).most_common(1)[0][0]
 token_text = tokenizer.decode([most_common_prefix[pos]])
 print(f"\nMost frequent divergence at position {pos}: '{token_text}'")
 print("→ The token at that position varies between requests")

The most common culprits, in order of frequency:

1. Timestamps and dates:

# ❌ Breaks the hash on every request
system = f"Current date: {datetime.now().strftime('%Y-%m-%d %H:%M')}. You are an assistant..."

# ✅ Take the date out of the system prompt
system = "You are an assistant specialising in cloud infrastructure."
# Pass the date as part of the user message if it is needed

2. Session and user IDs:

# ❌
system = f"User ID: {user_id}. Preferences: {user_prefs}. You are an assistant..."

# ✅ Separate the static from the contextual
system = "You are a specialised assistant." # always the same
# Add the user context as the first message of the history

3. Interpolated prompt versions:

# ❌
system = f"[v{PROMPT_VERSION}] You are an assistant..." # changes with every deploy

# ✅ Do not version in the text, version in the prompt name in Langfuse
system = "You are an assistant..."

4. Dynamic few-shots:

# ❌ Examples retrieved at random from a pool
examples = random.sample(example_pool, k=3)
system = f"Examples:\n{format_examples(examples)}\n\nYou are an assistant..."

# ✅ Fixed few-shots always in the same order
FIXED_EXAMPLES = [example_pool[0], example_pool[1], example_pool[2]]
system = f"Examples:\n{format_examples(FIXED_EXAMPLES)}\n\nYou are an assistant..."

Template engineering: the structure that maximises hits

The principle is simple: everything static goes first, everything dynamic goes after. The prefix cache is sequential. Once a block fails to hit, the rest are not looked up either.

OPTIMAL STRUCTURE for maximising prefix cache:
┌──────────────────────────────────────────────┐
│ STATIC BLOCK (tokens 0-511) │ ← hit rate ~100%
│ Invariant system prompt │
│ Fixed instructions │
│ Few-shots always in the same order │
├──────────────────────────────────────────────┤
│ SEMI-STATIC BLOCK (tokens 512-1023) │ ← hit rate ~60-80%
│ RAG documents for this session │
│ Conversation history so far │
├──────────────────────────────────────────────┤
│ DYNAMIC BLOCK (tokens 1024+) │ ← hit rate ~0% (expected)
│ Current user message │
│ Context specific to this request │
└──────────────────────────────────────────────┘

For RAG specifically: if the retrieved documents are the same across a set of similar queries (very frequent in RAG over fixed corporate documents), ordering them always in the same order (by ID, by a fixed score, not by a variable score) multiplies the hit rate of the semi-static block.

Prefix-aware routing: the next level

With a single vLLM instance, the prefix cache works automatically. The problem shows up with multiple replicas: the load balancer distributes requests round-robin, and the prefix cached on replica A is no use at all when the request lands on replica B.

The solution is prefix-aware routing: send requests with the same prefix to the same node.

With Ray Serve (native integration):

# ray_serve_prefix_router.py
from ray import serve
from ray.serve.llm import LLMConfig, build_llm_deployment

@serve.deployment
class PrefixAwareRouter:
 def __init__(self, replicas):
 self.replicas = replicas # list of vLLM handles

 async def __call__(self, request):
 body = await request.json()
 messages = body.get("messages", [])

 # Compute the hash of the system prompt (static prefix)
 system_content = ""
 for msg in messages:
 if msg["role"] == "system":
 system_content = msg["content"]
 break

 prefix_hash = hash(system_content)
 # Deterministic routing: same hash → same node
 replica_idx = prefix_hash % len(self.replicas)
 return await self.replicas[replica_idx].remote(request)

With an L7 gateway (Nginx/Traefik):

# nginx.conf — routing by X-Prefix-Hash header
upstream vllm_backends {
 hash $http_x_prefix_hash consistent;
 server vllm-0:8000;
 server vllm-1:8000;
 server vllm-2:8000;
 server vllm-3:8000;
}

The client computes the hash of the static prefix and includes it as a header:

import hashlib, requests

def llm_request(messages, base_url):
 system_msg = next((m["content"] for m in messages if m["role"] == "system"), "")
 prefix_hash = hashlib.sha256(system_msg.encode()).hexdigest()[:16]

 return requests.post(
 f"{base_url}/v1/chat/completions",
 json={"messages": messages, "model": "my-model"},
 headers={"X-Prefix-Hash": prefix_hash}
 )

Measuring the impact with OTel

# Current hit rate (0.0 to 1.0) — target > 0.70 with enterprise workloads
vllm:gpu_prefix_cache_hit_rate

# TTFT by percentile — should fall when the hit rate rises
histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))
histogram_quantile(0.95, rate(vllm:time_to_first_token_seconds_bucket[5m]))

The inverse correlation between hit rate and TTFT is the proof that the cache is working. If the hit rate goes from 15% to 70% and the p50 TTFT does not change, there is a configuration problem: the cache may be disabled, or the routing may not be sending requests to the right node.

Correlation query in Grafana (two-axis panel):

# Left Y axis: hit rate
vllm:gpu_prefix_cache_hit_rate

# Right Y axis: TTFT p50 (inverted)
histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))

The inverse slope should be visible: when the hit rate drops (a spike of requests with new prompts), TTFT rises. When the hit rate stabilises (users repeating the same flow), TTFT falls.

The impact in numbers

For a system with 100 req/min, a 512-token system prompt and a before/after hit rate:

Metric	Hit rate 15%	Hit rate 75%	Difference
Prefill tokens per minute	5,100	12,800 — 50% cached → 6,400 effective	−37% load
TTFT p50 (512 prompt + 100 suffix)	~820 ms	~180 ms (suffix only)	−78%
Prefill capacity freed	—	+1,200 tok/min	available for more requests

The 75% hit rate in this example is equivalent to being able to serve 37% more requests on the same hardware, because the prefill work of 3 out of every 4 requests is already done.

When the prefix cache does not help

The prefix cache is ineffective in workloads where every request has a completely unique prompt: translating a different document every time, code analysis with always-different context, creative generation with no system prompt. In these cases the hit rate structurally cannot exceed 5-10% and the template engineering effort does not pay off.

The signal: if your p99 input length is greater than the p50, you have high prompt variance and the prefix cache contributes little. If the p50 and the p99 are similar (consistent prompts), the prefix cache is the cheapest lever available.

References

Capacity planning for on-premise LLM inference: how to size GPUs from an SLO

Mon, 01 Jun 2026 15:00:00 +0200

This post complements the ones on KV cache (the piece that dominates the VRAM budget), Continuous batching (what defines the effective utilisation of the compute) and Seven layers of the stack (the pieces the sizing takes for granted). Before reading this one, make sure your team has written down the SLOs it is going to chase; without that input the calculation is not defensible.

TL;DR

LLM inference capacity planning does not answer “how many tokens per second does a GPU give” — that question has no universal answer, because throughput depends on concurrency, the prefill/decode split, context length, the inference engine and quantisation. The right question has three inputs (SLO: TTFT P95, TPOT P95, sustained RPS), a hardware reference (GPU model, VRAM, HBM bandwidth, effective FLOPs) and a model (parameters, GQA/MHA/MoE architecture, weight format). The calculation resolves into two coupled budgets that cross each other. VRAM budget: from the GPU’s total you subtract model weights and activations, what remains is the KV cache budget, and from there you derive the maximum concurrency possible at the average context you expect. Time budget: the engine (vLLM, SGLang, TensorRT-LLM) has a tokens/s ceiling in decode given by HBM bandwidth and another in prefill given by useful FLOPs; from there you derive the expected TPOT and, dividing prefill_tokens by prefill throughput, the expected TTFT. Both budgets must meet the SLO simultaneously: whichever is tighter dictates the sizing. On the Llama 70B BF16 example with tensor parallel 4 on 4×H100 SXM, a single replica saturates at ~28 concurrent requests and delivers ~3,200 tokens/s of aggregate decode with a median TPOT of 35 ms; for 200 sustained RPS at a profile of 800 prompt tokens + 250 of output, you need between 4 and 5 replicas with a 25 % cushion over the observed peak. Quantisation (FP8 → INT4) divides VRAM cost and decode time by between 1.5 and 4×, but it degrades quality measurably — it is not assumed free, it is validated with evals. The five habitual traps: confusing the mean with P95, ignoring the prefill/decode split of the real workload, sizing without head-room for retraining or rollback, forgetting that a GPU at 100 % SM util means nothing if the HBM is saturated, and not documenting the assumptions behind the calculation (a sizing with no written assumptions is a throwaway calculation).

You are here: DEPLOY (with one foot in OBSERVE)

Capacity planning is a piece with dual residency. It lives in DEPLOY because without a valid sizing you do not buy hardware or configure the inference engine. But its input is real observations: the distribution of prompt and output lengths, the prefill/decode mix of the workload, the real P95s already being seen in pre-production. Without that data the calculation is a napkin sketch, defensible only until the first customer arrives who does not fit the assumed average.

The analogy: the hotel with variable-sized rooms

Imagine a hotel where the rooms have no fixed size: each guest pays for the square metres they need, and the building’s floor plan reorganises itself dynamically to accommodate whoever arrives. Management wants to maximise occupancy, but it has two real constraints and one quality metric.

Constraint 1 — physical space. The floor has 1,000 m² in total. If a family comes in needing 200 m², that family occupies that surface and it cannot be handed to the next guest. The largest room limits how many simultaneous guests fit.

Constraint 2 — service staff. There are 10 receptionists. Each can handle the check-in of one guest every two minutes. When 60 guests arrive within an hour, the last ones wait in a queue; the time from walking into reception to receiving their key depends on how many are ahead of them.

Quality metric — a promise about time. The brochure says “check-in in under 15 minutes”. If too many guests arrive at once, that promise breaks even if there is physical space free.

The physical space is the GPU’s VRAM. Each room is a request with its KV cache (larger the longer the conversation). The receptionists are the compute units (Streaming Multiprocessors + Tensor Cores). The check-in is the prefill phase; the nights the guest spends afterwards are the decode steps. The 15-minute promise is the TTFT P95 SLO.

The hotel’s capacity planning is exactly this: given the expected guest profile (how many arrive per hour, how much space they ask for on average, how many minutes of waiting they tolerate), work out how many floors and how many receptionists are needed. It is not done by estimating “rooms per hour” in the abstract; it is done by crossing the two budgets with the promise about time. The analogy holds up the calculation all the way to the end.

The three SLO inputs

Before putting a single number on the sheet, you have to write down the three dimensions of the SLO. Without this, the calculation is aesthetics, not engineering.

TTFT P95 (Time-To-First-Token). The time from the client sending the request until it receives the first token. It is dominated by the prefill phase (processing the whole prompt in one go) plus the scheduler queue. For conversational chat, a reasonable target sits between 0.5 and 2 seconds P95. For coding assistants with large prompts (5–10 K tokens of context), between 2 and 4 s P95. Below 500 ms you enter UX territory for voice-style conversations, but that demands serious architectural compromises.

TPOT P95 (Time-Per-Output-Token). The time between consecutive tokens during decode. It dominates the “perceived fluency” of streaming. Above 80 ms/token the human reader perceives pauses; below 30 ms/token the output flows faster than it can be read. The usual industry target: 40–60 ms P95.

Sustained RPS meeting the SLO. The throughput the system must support without violating TTFT or TPOT. This is DistServe’s key metric, called goodput (see Continuous batching). “200 RPS peak” is not the same as “200 RPS with TTFT P95 ≤ 1.5 s”. Without the SLO condition, the RPS number means nothing.

These three dimensions come with a workload profile: the distribution of prompt and output lengths. Medians are not enough; you need P50, P95, P99. A badly measured profile is the main reason sizings fail.

The central formula: two budgets that cross

The calculation resolves into two independent sums that are then crossed. The smaller of the two rules.

VRAM budget

For a GPU with total VRAM $V$, the space available for KV cache is:

$$V_{\text{kv}} = V - V_{\text{model}} - V_{\text{activations}} - V_{\text{overhead}}$$

where:

$V_{\text{model}}$ is the size of the weights: for a model of $P$ parameters in a format of $b$ bytes/parameter, $V_{\text{model}} = P \cdot b$. Llama 70B BF16 = $70 \times 10^9 \times 2 = 140$ GB. Under tensor parallel TP=4, each GPU carries $140 / 4 = 35$ GB.
$V_{\text{activations}}$ are the intermediate buffers of the forward pass. For vLLM with a reasonable batch, between 2 and 6 GB per GPU depending on batch size and maximum length.
$V_{\text{overhead}}$ is the CUDA context, NCCL buffers, the PagedAttention pool, reserved paged blocks. 2–4 GB typical.

The KV cache budget per GPU is what is left over. For an H100 SXM 80 GB with Llama 70B TP=4 BF16:

$$V_{\text{kv}} = 80 - 35 - 4 - 3 = 38 \text{ GB per GPU} = 152 \text{ GB aggregated over TP=4}$$

The KV cache cost per token for a model with $L$ layers, $H_{\text{kv}}$ KV heads (GQA), per-head dimension $d_h$, in a format of $b$ bytes:

$$\text{kv per token} = 2 \cdot L \cdot H_{\text{kv}} \cdot d_h \cdot b$$

The factor of 2 is because both K and V are stored. For Llama 70B (L=80, $H_{\text{kv}}$=8 with GQA, $d_h$=128, BF16 = 2 bytes):

$$\text{kv per token} = 2 \cdot 80 \cdot 8 \cdot 128 \cdot 2 = 327,680 \text{ bytes} = 320 \text{ KB/token}$$

And the maximum concurrency at average context $C$:

$$N_{\text{max}} = \frac{V_{\text{kv}}}{C \cdot \text{kv per token}}$$

With an aggregated $V_{\text{kv}}$ of 152 GB and an average context of 1,500 tokens (800 prompt + 700 generated at the worst moment of the conversation):

$$N_{\text{max}} = \frac{152 \times 10^9}{1,500 \cdot 320 \times 10^3} \approx 316 \text{ concurrent requests}$$

This is the physical ceiling of concurrency for that replica. It is not what you are going to use; it is what you cannot exceed without OOM. The operational number sits well below it (head-room for spikes).

Time budget

Two sub-calculations come in here: decode (memory-bound) and prefill (compute-bound).

Decode TPOT. For every token generated, the model weights (the ones relevant to that request) have to be walked through and the accumulated KV cache has to be read. The bottleneck is HBM bandwidth. For a GPU with bandwidth $B$ GB/s and a model of $V_{\text{model per gpu}}$ GB of weights:

$$\text{tpot}_{\text{theoretical}} \approx \frac{V_{\text{model per gpu}}}{B}$$

For an H100 SXM with HBM3 at 3.35 TB/s and Llama 70B TP=4 BF16 (35 GB/GPU):

$$\text{tpot}_{\text{theoretical}} \approx \frac{35}{3,350} \approx 10.4 \text{ ms/token}$$

This is the best theoretical case with batch=1 and 100 % HBM efficiency. In practice vLLM on H100 with Llama 70B TP=4 reaches 12–18 ms/token at low batch and 30–45 ms/token at high batch (at concurrency 32, tokens compete for the shared HBM). The defensible operational number: 35 ms/token at concurrency 24–32.

Prefill throughput. Prefill processes N prompt tokens in a single forward pass. It is compute-bound: the bottleneck is FLOPs. For an H100 SXM with 989 sustained BF16 TFLOPs and Llama 70B (each forward pass costs roughly $2 \cdot P \cdot N$ FLOPs per sequence of length N):

$$\text{prefill tps} = \frac{4 \cdot \text{TFLOPs} \cdot \eta}{2 \cdot P} = \frac{4 \cdot 989 \times 10^{12} \cdot 0.5}{2 \cdot 70 \times 10^9} \approx 14,000 \text{ tokens/s}$$

(the factor of 4 is the GPUs in TP, $\eta$ is real efficiency between 0.4 and 0.6 on H100). An 800-token prompt takes this long in prefill:

$$\text{prefill time} = \frac{800}{14,000} \approx 57 \text{ ms}$$

Adding a typical queue of 100–300 ms at high concurrency, TTFT P95 ≈ 350–500 ms for that profile. Well below the 1.5 s target; there is margin.

The crossing

The real operational concurrency $N_{\text{op}}$ is the minimum of the VRAM ceiling, the concurrency at which TPOT starts to degrade past the SLO, and the concurrency at which TTFT starts to degrade past the SLO (prefill queue). For the example:

VRAM ceiling: 316.
TPOT degrades to 80 ms (SLO) at around concurrency ~80–100 (measured empirically with a benchmark, not a closed formula).
TTFT degrades to 1.5 s at around concurrency ~40–60 because of the prefill queue.

The replica’s operational concurrency is ~50. Applying 25 % head-room for spikes and rebalancing, target concurrency per replica ≈ 35–40.

Step-by-step worksheet: Llama 70B BF16 on 4×H100 SXM

The exercise’s input:

SLO: TTFT P95 ≤ 1.5 s; TPOT P95 ≤ 60 ms; 200 sustained RPS.
Workload: prompt P50=600, P95=1,200, P99=2,500; output P50=180, P95=500, P99=900. Average prompt 800, output 250.
Generic hardware: 4×H100 SXM 80 GB with NVLink, vLLM v1 engine, tensor parallel 4, BF16.

Step 1 — VRAM per GPU. Weights 35 GB, activations 4 GB, overhead 3 GB → KV budget 38 GB/GPU = 152 GB aggregated. KV/token for Llama 70B GQA = 320 KB. Ceiling of live tokens in cache: $152 \times 10^9 / 320 \times 10^3 \approx 475,000$ tokens. At the operational average context (800 prompt + 200 already generated = 1,000 live tokens per request), concurrency ceiling $\approx 475$.

Step 2 — average duration of a request. Prefill 800 tokens / 14,000 tps = 57 ms. Decode 250 tokens × 35 ms/token = 8,750 ms. Total $\approx 8.8$ s per request.

Step 3 — replica throughput. If the replica sustains operational concurrency 40 and each request lasts 8.8 s, the replica delivers roughly $40 / 8.8 \approx 4.5$ requests/s in steady state.

Step 4 — number of replicas. For a 200 RPS target: $200 / 4.5 \approx 45$ replicas. That is 45 × 4 = 180 GPUs. Too much: this sizing does not work because the cost per request is high.

Step 5 — review the levers. Before buying more hardware, there are three levers to explore, in this order:

Quantisation. Dropping to FP8 reduces weights to 17.5 GB/GPU (leaving more VRAM for KV cache → more concurrency), roughly doubles decode tokens/s (HBM saturated by half), and degrades MMLU quality typically by 0.5–1.5 points on models like Llama 70B. Rewriting the calculation in FP8: TPOT drops to ~18 ms, total time per request to 4.7 s, RPS per replica rises to ~8.5, replicas needed ≈ 24, equivalent to 96 GPUs.
Speculative decoding. With a small drafter and 60–70 % acceptance, effective TPOT falls 30–40 %. RPS per replica rises to ~12, replicas ≈ 17 = 68 GPUs.
Disaggregated serving. Separating prefill workers and decode workers lets each be scaled to the real mix of the workload (see Disaggregated serving). It usually shaves off another 20–40 % under asymmetric workloads.

Step 6 — recommended sizing. For the example, with FP8 + speculative decoding and 25 % head-room: 20 vLLM replicas at TP=4 over 80 H100 SXM. If the team does not want to depend on aggressive quantisation (pure BF16 for maximum fidelity), the calculation rises to 30 replicas = 120 GPUs and forces a renegotiation of the SLO or the budget.

Step 7 — write down the assumptions. This is the part no valid sizing skips. In the team’s repo, next to the calculation:

# sizing/llama70b-prod.yaml
fecha: 2026-06-01
slo:
 ttft_p95_ms: 1500
 tpot_p95_ms: 60
 rps_target: 200
workload:
 prompt_tokens_p50: 600
 prompt_tokens_p95: 1200
 output_tokens_p50: 180
 output_tokens_p95: 500
 asunto: chat productivo con RAG ligero
modelo:
 arquitectura: llama-70b-instruct
 formato_pesos: fp8
 motor: vllm-v1
hardware:
 gpu: H100-SXM-80GB
 topologia: TP=4 con NVLink intra-nodo
 red_inter_replica: 25 GbE
optimizaciones:
 - paged_attention
 - chunked_prefill
 - speculative_decoding (drafter llama-1.1b, aceptación esperada 65%)
asunciones_criticas:
 - utilizacion_hbm_eficiente: 0.55
 - head_room_pico_sobre_p95: 0.25
 - aceptacion_speculative_min: 0.55
plan_validacion:
 - benchmark vllm bench serve antes de procurement
 - canary 10% durante 7 días post-deploy

Without this YAML, the calculation is not reproducible a month later.

MoE case: Mixtral 8×22B (~141 B total, 39 B active)

MoEs change the calculation along one key dimension: the total weights are large but the active weights per token are small. For Mixtral 8×22B with top-2 routing:

Weight VRAM: $141 \times 2 = 282$ GB BF16. With TP=4 → 70 GB/GPU. It does not fit in an H100 80 GB alongside KV cache + activations. You need TP=8 (~35 GB/GPU) or FP8 with TP=4 (~35 GB/GPU).
Decode TPOT: dominated by the weights read per token, which are $\sim 39 / 8 \cdot 2 \approx 9.75$ GB/GPU with TP=4 (one top-2 expert per token, divided across 4 GPUs). On an H100 with HBM at 3.35 TB/s, theoretical TPOT ≈ 3 ms/token. In practice, 10–20 ms at reasonable concurrency.
Prefill: similar to the dense model of the active weights, ~39 B FLOPs/token.

MoE sizing usually delivers more RPS per GPU than an equivalent dense model: the low cost per token compensates for the extra VRAM. See MoE inference for the detail of routing and why a high batch is decisive so that each expert sees enough tokens.

Sensitivity table: context and quantisation

For Llama 70B on 4×H100 SXM (TP=4), operational concurrency per replica with an SLO of TTFT 1.5 s / TPOT 60 ms:

Average context	BF16	FP8	INT4 (AWQ)
500 tokens	55	110	180
1,000 tokens	40	80	130
2,000 tokens	24	50	85
4,000 tokens	12	26	48
8,000 tokens	6	13	25

Approximate numbers from public vLLM benchmarks as of June 2026, with ±20 % variation depending on engine version and the headroom adopted. To validate on your own hardware: vllm bench serve with your profile of real prompts.

The five habitual traps

Trap 1 — confusing the mean with P95. Average throughput over an hour may be 50 RPS while the 5-minute peak reaches 180 RPS. Sizing against the mean guarantees breaking the SLO at every peak. Rule: size against the hourly P95, with 20–30 % head-room over P95.

Trap 2 — not measuring the real prefill/decode split. A “RAG with short answers” workload spends 70–80 % of GPU time in prefill; a “writing assistant generating essays” spends 80 % in decode. The useful optimisations (chunked prefill vs speculative decoding) change radically. Without measuring it, you buy badly balanced hardware.

Trap 3 — sizing with no head-room for retraining or rollback. The production cluster is not just the inference engine: there is a re-embedding batch when the embeddings model changes, continuous canary evals (see Canary, blue-green and shadow), light fine-tuning, hot stand-by for rollback. Reserve 15–25 % of capacity for those non-negotiable workloads.

Trap 4 — “GPU at 100 % SM utilization” as a target. SM occupancy of 95 % with saturated HBM produces the same throughput as SM at 60 % with saturated HBM. The bottleneck in decode is the HBM. Optimising for “GPU usage 100 %” without looking at HBM utilisation and arithmetic intensity makes you spend more on GPU without gaining throughput. See GPU observability for LLM inference for which metrics to actually watch.

Trap 5 — not documenting the assumptions. A sizing with no reproducible YAML (workload, model, engine, head-room, critical assumptions) leaves the team with no way of knowing what changed when the cluster stops meeting the SLO six months later. Documenting is cheap; losing a quarter to debugging is not.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB with intra-node NVLink and 25 GbE between nodes, the recurring configurations in May 2026 are:

Model	Format	TP	Replicas that fit	Typical RPS per node (ctx 1K)
Llama 8B	BF16	1	4 (one per GPU)	240–320
Llama 8B	FP8	1	4	450–600
Llama 70B	BF16	4	1	30–45
Llama 70B	FP8	4	1	60–90
Llama 70B	INT4 AWQ	2	2	90–130
Mixtral 8×22B	FP8	4	1	90–140
Qwen 72B	BF16	4	1	28–42

These numbers are orders of magnitude to start the conversation, not commitments. The definitive sizing is validated with vllm bench serve or genai-perf (NVIDIA) using the customer’s real prompt/output profile. The prefill/decode asymmetry of each case’s workload can move these numbers 30–50 % up or down.

For clusters of 8×H100 SXM (typical of DGX servers or equivalent replicas), the options open up to TP=8 for 405B-class models or multi-replica TP=2 for 70B models with greater density. The metric that decides is always the same: tokens meeting SLO per kW and per euro of amortised hardware.

How the sizing is validated before buying

The spreadsheet sizing is the first half. The second is the validation benchmark.

Stage 1 — back-of-the-envelope sizing. This post’s formulas applied to the SLO and the expected workload. Output: an approximate number of replicas and a topology.

Stage 2 — synthetic micro-benchmark. On a borrowed GPU or one rented by the day, bring up the engine with the chosen model and run vllm bench serve with prompts of representative lengths. Validate TPOT, prefill TPS and the concurrency ceiling. Calibrate the HBM efficiency factor ($\eta$) used in the formulas.

Stage 3 — load test with realistic traffic. Generate traffic following the real distribution of the customer’s workload (not Poisson, not constant: the real trace). Measure P50/P95/P99 of TTFT, TPOT, throughput. Confirm the head-room.

Stage 4 — canary in production. With the cluster sized, route 5–10 % of real traffic for 7–14 days before closing the procurement of additional hardware. See Canary, blue-green and shadow for the mechanics.

Jumping from Stage 1 to full procurement is the most frequent cause of a cluster that is 40 % oversized and 60 % undersized at the same time, in different regions of the workload. Four weeks of validation done well save four months of refactoring.

What we have not covered (upcoming articles)

The observability metrics that close the sizing loop in production, see GPU observability for LLM inference.
Autoscaling that adjusts replicas to the real traffic curve, see Autoscaling LLM on Kubernetes.
Detailed cost accounting per tenant (showback / chargeback) over the sized hardware.
Sizing for continuous fine-tuning (PEFT and light training) that shares a cluster with inference.

References

Kwon et al. — vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (SOSP 2023).
Zhong et al. — DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024).
Agrawal et al. — Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI 2024).
NVIDIA — H100 Tensor Core GPU Architecture Whitepaper (HBM3 memory, bandwidth, sustained FLOPs).
vLLM project — vllm bench serve reference (benchmarking CLI included in the repo).
NVIDIA — genai-perf (official tool for benchmarking LLM services).

Throughput on lo0 — Tech Blog

Cost, performance and energy: the three axes of LLM inference and the identity that ties them together

What this series is

The three axes and their metrics

Cost axis in detail

Performance axis in detail

Energy axis in detail

The identity that ties the three axes together: throughput

Worked example: unit economics of an 8×H100 node in Europe

Energy is inside the cost

Sensitivity analysis: what moves the cost per token

Simplified TCO model (on-premise)

Reference: the price of a GPU-hour in European sovereign cloud

Energy and carbon reference: France, Germany, Spain

Sovereignty and regulation: the axis that does not appear in the CPM

Methodology: how to read (and not fool yourself with) these figures

The combined scorecard

Where each axis is measured: from design to production

Glossary of units and conversions

How the rest of the series fits together

See also

Sources

LLM inference benchmarking: frameworks, metrics and the state of the art (tool by tool)

What this introduction covers

The performance metrics

The latency decomposition

Goodput: the honest metric

How each metric is instrumented (and where the error creeps in)

The architecture of the tool biases the figure

Frameworks, one by one

vLLM bench and SGLang bench — engine micro-bench

AIPerf — the NVIDIA one (formerly genai-perf), multi-process

GuideLLM — from the vLLM project, SLO-oriented

LLMPerf — the Anyscale/Ray classic

MLPerf Inference — the industry standard

Comparison table

The concurrency sweep: finding the knee

Worked example: reading a sweep

MLPerf Inference: the comparability standard

Measurement bias and reproducibility

Checklist for a reproducible benchmark

Performance ≠ quality

The latency versus throughput trade-off

The connection with cost and energy

State of the art 2026

Limits and traps (data-driven)

Closing

See also

Sources

Batch sizing in vLLM: the two-hour grid search worth weeks of hardware

TL;DR

The analogy

The problem: the defaults are not for your hardware

The arithmetic that matters

The grid search: methodology

Step 1: measure the real workload

Step 2: work out the KV budget

Step 3: the grid

Step 4: read the surface

Confirmation with OTel

Reference configurations by profile

When not to touch the defaults

See also

In this same series

References

One big replica or many small ones: the decision that defines your platform

TL;DR

The analogy

The two architectures in vLLM

Architecture A: TP=4, one replica

Architecture B: TP=2 × 2 replicas

Why TP=4 has higher individual latency than TP=2

The crossover point: when TP=2×2 beats TP=4×1

The three implications nobody mentions

1. Fault tolerance

2. Autoscaling granularity

3. Quality degradation under load

Measuring the crossover point with OTel

The decision by workload profile

Kubernetes configuration for both architectures