H100 on lo0 — Tech Blog

Full TCO of an on-premise GPU cluster: from capex to the all-in €/GPU-hour and the break-even against cloud

Tue, 16 Jun 2026 06:00:00 +0200

Notation: amounts in N € or N USD (source denominated in dollars); decimal point; comma as thousands separator. The dollar sign is not used (it is the formula delimiter). Data centred on Europe/Spain. Generic example hardware: a cluster of N nodes, each with 4×H100 SXM5 80 GB.

TL;DR

A 4×H100 SXM5 node costs between 150,000 USD and 200,000 USD in total capex (GPUs + server + network + storage + prorated rack). Amortised over 3 years with European opex (energy at ~0.116 €/kWh industrial, PUE 1.54 average or 1.2 with liquid, 0.3 FTE of staff), the all-in cost ranges from 3.10 USD/GPU-hour (100 % utilisation) to 6.20 USD/GPU-hour (50 % utilisation). The break-even against AWS p5 on-demand (~6.88 USD/GPU-hour) is crossed at around 70 % utilisation; against a neocloud 3-year reserved (~1.49–2.10 USD/GPU-hour), on-prem never closes the gap in that scenario. Utilisation is the variable that decides the cost axis, not the price of the hardware.

The model: declared assumptions

Every calculation below starts from these assumptions. Changing any of them moves the conclusion; the sensitivity section quantifies by how much.

Parameter	Base value	Sensitivity range
Node	4×H100 SXM5 80 GB (HGX baseboard)	—
Cluster	N nodes (per-node model; scales linearly)	1–32 nodes
Capex amortisation	3 years (straight line)	3–5 years
GPU utilisation	70 %	30 %–100 %
Energy price	0.116 €/kWh (industrial Spain, Sept. 2025)	0.06–0.20 €/kWh
PUE	1.54 (global average, Uptime Institute 2025)	1.15–1.80
Exchange rate	1 USD = 0.93 € (reference Jun. 2026)	—

Energy source: GlobalPetrolPrices · Spain Business Electricity, Sept. 2025. PUE source: Uptime Institute Global Data Center Survey 2025 — the global average PUE has been stuck at 1.54 for the sixth consecutive year; hyperscalers 1.10–1.15; colocation/enterprise 1.58–1.80; facilities less than 5 years old, 1.48. PUE 1.2 is achievable with direct-to-chip liquid cooling.

Capex breakdown per 4×H100 SXM5 node

GPUs

Component	Unit price (USD)	Quantity	Subtotal (USD)	Source and date
H100 SXM5 80 GB (card)	30,000–40,000	4	120,000–160,000	GMI Cloud, Apr. 2026 · Introl, Apr. 2026

The range reflects market variability and volume discounts (5–15 % for orders above 50 units). H100 SXM5 cards require NVIDIA’s HGX baseboard; they are not sold loose for direct installation in standard servers.

Server / HGX baseboard

A complete 4×H100 SXM node uses NVIDIA’s HGX H100 4-GPU baseboard plus a compatible host server. Reference models: Supermicro SYS-421GU-TNXR (4U, dual Intel Xeon 4th Gen, HGX H100 4-GPU) and its Dell equivalent.

Component	Estimated cost (USD)	Note
Server chassis + CPU (2× Xeon) + RAM (512 GB DDR5) + redundant PSU	18,000–25,000	Based on a Supermicro SYS-821GE bare without GPUs at ~24,806 USD (xicomputer.com, Jun. 2026); scaled to the 4-GPU
HGX H100 4-GPU baseboard	included in the GPU price	NVIDIA HGX platform; no separate public price
Inter-GPU NVLink (inside the node)	included in the baseboard	4 GPUs connected by NVLink 4.0 on the HGX baseboard

Marketing claim (no independent verification): Supermicro announces datacenter energy cost reductions of up to 40 % with liquid cooling in its HGX H100 servers (Supermicro press release).

InfiniBand NDR network

For a multi-node cluster with tensor parallelism across nodes, the GPU-to-GPU network is critical. NDR InfiniBand (400 Gb/s per port) is the de facto standard for HGX clusters.

Component	Estimated cost per node (USD)	Source / Note
NVIDIA Quantum-2 NDR 400G switch (64 ports, prorated across N nodes)	2,000–4,000	Switch ~35,000 USD (Introl, Apr. 2026); at 16 nodes, ~2,200 USD/node
InfiniBand NDR cables/transceivers (4 ports per node × ~1,000 USD/port)	4,000	Estimate based on ~1,000 USD per optical transceiver (Introl, Apr. 2026)
InfiniBand network (prorated per 4-GPU node)	~6,000–8,000	—

For inference serving inside a single node (4 GPUs with NVLink), the inter-node network is less critical than for multi-node training. For disaggregated prefill-decode workloads spread across nodes, InfiniBand NDR is necessary.

NVMe storage

Component	Estimated cost (USD)	Note
Local NVMe (4 TB × 2 U.2/E1.S drives, working datasets and checkpoints)	2,000–4,000	~500–1,000 USD/TB enterprise NVMe, 2025
Shared object storage (NAS/MinIO, prorated per node)	2,000–5,000	Varies with the total capacity of the cluster
Total storage per node	~4,000–9,000	—

Introl models 50 TB per GPU for effective operation in training clusters (Introl, Apr. 2026); for pure inference the requirement is significantly lower (model weights + logs).

Rack, PDU and datacenter connectivity

Component	Estimated cost per node (USD/year)	Source
Rack colocation (high density, 10–15 kW per node)	5,000–12,000/year	Encoradvisors · Colocation Pricing 2026: high density 3,000–6,000 USD/month per rack; at 2 nodes per rack, ~1,500–3,000 USD/month per node = 18,000–36,000 USD/year in tier-1; lower in Spain
Rack PDU, electrical cabling (prorated)	500–1,000 per node (amortised capex)	Inside the colocation line item or in an own datacenter

Colocation in Spain/Europe is structurally cheaper than in US tier-1 markets (New York, Silicon Valley). For an own datacenter, replace it with the cost of your own space plus amortisation of the electrical and cooling infrastructure.

Capex summary per 4×H100 SXM5 node

Line item	Range (USD)	Midpoint
GPUs (4× H100 SXM5)	120,000–160,000	140,000
Server chassis + CPU/RAM/PSU	18,000–25,000	21,500
InfiniBand NDR network (prorated)	6,000–8,000	7,000
NVMe + object storage	4,000–9,000	6,500
PDU/rack/other (capex)	2,000–5,000	3,500
Total capex per node	150,000–207,000	178,500

Sources: GMI Cloud (Apr. 2026), Introl (Apr. 2026), Spheron (Apr. 2026), xicomputer.com (Jun. 2026).

Opex breakdown per 4×H100 SXM5 node (annual)

Energy

A 4×H100 SXM5 node at full load draws approximately:

$$P_{\text{node}} = 4 \times 700\,\text{W (TDP H100 SXM5)} + 800\,\text{W (server)} \approx 3.6\,\text{kW (IT)}$$

The total datacenter power includes the cooling overhead, expressed by the PUE:

$$P_{\text{total}} = P_{\text{IT}} \times \text{PUE}$$ $$\text{annual energy cost} = P_{\text{IT}} \times \text{PUE} \times 8{,}760\,\text{h} \times \text{kWh price}$$

With the base values (PUE 1.54; 0.116 €/kWh):

$$\text{energy/year} = 3.6\,\text{kW} \times 1.54 \times 8{,}760\,\text{h} \times 0.116\,\text{EUR/kWh} \approx 5{,}475\,\text{EUR}$$

With a Spanish solar PPA (Q3 2025 reference price: ~34 €/MWh = 0.034 €/kWh according to PV Tech, Oct. 2025):

$$\text{energy/year (solar PPA)} = 3.6 \times 1.54 \times 8{,}760 \times 0.034 \approx 1{,}604\,\text{EUR}$$

Energy scenario	Price (€/kWh)	Energy cost/year per 4-GPU node
Spanish solar PPA (Q3 2025)	0.034	~1,604 €
Industrial Spain (Sept. 2025)	0.116	~5,475 €
European average (industrial tariff)	0.160	~7,550 €
Worst case (no PPA, high tariff)	0.200	~9,437 €

Staff / operations

Staff cost is the most variable line item with cluster size. For a small cluster (2–8 nodes), the rule of thumb is 0.3–0.5 FTE per cluster of GPU infrastructure support (Spheron, Apr. 2026).

Cluster size	Estimated FTE	FTE cost (€/year, Western Europe)	Cost per 4-GPU node (€/year)
2–4 nodes	0.3 FTE	~120,000	36,000–18,000
8–16 nodes	0.5 FTE	~120,000	7,500
32+ nodes	1–2 FTE	~120,000	3,750–7,500

Indicative salary reference: a GPU infrastructure engineer with CUDA, InfiniBand and Kubernetes knowledge in Western Europe, ~90,000–140,000 €/year fully loaded. Introl’s figures (Apr. 2026) in USD (~275,000 USD/year for the US) reflect the North American market, which is appreciably higher.

Maintenance and support

Line item	Annual cost (% of hardware capex)	Per 4-GPU node (midpoint)
Vendor maintenance / support	5–10 % of capex	~7,000–14,000 USD → ~6,500–13,000 €
GPU failure rate (~5 % annual) × replacement cost	5 % × 4 GPUs × ~35,000 USD = ~7,000 USD expected	~6,500 € (amortised as a provision)
Minor spares (cables, modules)	~500–1,000 €	—

Introl quotes GPU failure rates of 2–3 % per year in small clusters; Google Research documented ~9 % annualised in Meta’s 16,384-GPU H100 cluster (Introl, Apr. 2026). 5 % is used here as a conservative intermediate value.

Depreciation (for accounting purposes)

Straight-line depreciation turns capex into an annual flow comparable to the cost of committed cloud:

$$\text{annual depreciation} = \frac{\text{node capex}}{\text{amortisation years}}$$

Node capex (USD)	3-year amortisation (USD/year)	5-year amortisation (USD/year)
150,000 (minimum)	50,000	30,000
178,500 (average)	59,500	35,700
207,000 (maximum)	69,000	41,400

H100 hardware depreciates fast: secondary-market analyses put the residual value at 20–40 % of the purchase price after 3 years (Introl, Apr. 2026). The arrival of Blackwell GB200/GB300 accelerates the perceived obsolescence.

Annual opex summary per 4×H100 SXM5 node (base scenario, 8-node cluster)

Line item	Base scenario (€/year)	Range
Energy (PUE 1.54; 0.116 €/kWh)	5,475	1,604–9,437
Staff (0.5 FTE × 8 nodes, prorated)	7,500	3,750–36,000
Maintenance / support / failures	9,000	5,000–15,000
Rack colocation (Spain, high density)	6,000	3,000–15,000
Total opex per node	~28,000	~13,000–75,000

The extreme range reflects the difference between a well amortised own datacenter with a solar PPA and cheap energy (minimum opex) and tier-1 colocation with market tariffs and senior staff.

Deriving the all-in €/GPU-hour

Formula

$$\text{EUR/GPU-hour all-in} = \frac{\frac{\text{node capex}}{\text{years}} + \text{node annual opex}}{4\,\text{GPUs} \times 8{,}760\,\text{h} \times u}$$

where $u$ is the average annual utilisation (0 to 1).

See the cost-per-token identity in cost per token and per request for the connection with throughput.

€/GPU-hour table by utilisation and scenario

Average capex (178,500 USD → ~166,000 €), 3-year amortisation → 55,300 €/year.

Utilisation	Opex/year (base, €)	Total cost/year (€)	Useful GPU-hours/year	€/GPU-hour
30 %	28,000	83,300	10,512	7.93
50 %	28,000	83,300	17,520	4.75
70 %	28,000	83,300	24,528	3.39
80 %	28,000	83,300	28,032	2.97
100 %	28,000	83,300	35,040	2.38

Low-opex scenario (solar PPA, own datacenter, large cluster): opex/year ~13,000 €.

Utilisation	Total cost/year (€)	€/GPU-hour
50 %	68,300	3.90
70 %	68,300	2.78
80 %	68,300	2.43
100 %	68,300	1.95

High-opex scenario (market tariff, expensive colocation, small cluster): opex/year ~75,000 €.

Utilisation	Total cost/year (€)	€/GPU-hour
50 %	130,300	7.44
70 %	130,300	5.31
80 %	130,300	4.65
100 %	130,300	3.72

From €/GPU-hour to €/1M tokens

The cost-per-token identity connects the hardware cost with the inference cost:

$$\text{EUR/1M tokens} = \frac{\text{EUR/GPU-hour} \times 10^6}{\text{throughput (tok/s)} \times 3{,}600}$$

For reference throughputs on H100 SXM5 with vLLM (see capacity planning for on-premise inference):

Model	Typical throughput (tok/s per GPU)	Source
Llama-3 70B FP8, high batch	~2,800	Series B benchmarks
Llama-3 8B FP16, medium batch	~9,000	Series B benchmarks
Mixtral 8×7B, high batch	~4,500	Series B benchmarks

€/1M tokens table in the base scenario (€/GPU-hour 3.39 at 70 % utilisation):

Model	Throughput (tok/s)	€/1M tokens
Llama-3 70B FP8	2,800	~0.336
Llama-3 8B FP16	9,000	~0.105
Mixtral 8×7B	4,500	~0.209

At 50 % utilisation (€/GPU-hour 4.75):

Model	€/1M tokens
Llama-3 70B FP8	~0.471
Llama-3 8B FP16	~0.147

Occupancy (batching) multiplies the effective throughput and lowers the €/1M tokens without changing the hardware; it is analysed in GPU utilisation as a FinOps lever.

Break-even on-prem vs cloud

The break-even formula

Break-even happens when the total annual on-prem cost equals the annual cloud cost at the same utilisation:

$$\text{annual cloud cost} = \text{cloud GPU-hour price} \times 4\,\text{GPUs} \times 8{,}760\,\text{h} \times u$$ $$\text{break-even}: \quad \frac{\text{capex/year} + \text{opex/year}}{4 \times 8{,}760 \times u} = \text{cloud GPU-hour price}$$

Solving for the break-even utilisation:

$$u^* = \frac{\text{capex/year} + \text{opex/year}}{4 \times 8{,}760 \times \text{cloud GPU-hour price}}$$

Break-even table by cloud mode and on-prem scenario

Base on-prem scenario (capex/year 55,300 €, opex/year 28,000 €, total 83,300 €/year per 4-GPU node):

Cloud reference (price/GPU-hour)	USD equiv.	Break-even utilisation $u^*$	Note
Neocloud on-demand (Lambda/Spheron ~2.90 USD)	2.90 USD (~2.70 €)	>100 %, on-prem does not compete	Neocloud on-demand is cheaper even at full utilisation
Neocloud 3-year reserved (CoreWeave ~1.49–2.10 USD)	~1.80 USD (~1.67 €)	>100 %, impossible	Neocloud reserved beats on-prem in every scenario of this model
AWS p5 on-demand (6.88 USD/GPU-hour)	6.88 USD (~6.40 €)	~47 %	Above 47 %, the average on-prem beats AWS on-demand
AWS p5 3-year reserved (~2.97 USD/GPU-hour)	2.97 USD (~2.76 €)	>100 %	—
GCP A3 on-demand (~10.98 USD/GPU-hour)	10.98 USD (~10.21 €)	~29 %	Above 29 %, on-prem beats GCP on-demand
Azure ND H100 v5 on-demand (~12.29 USD/GPU-hour)	12.29 USD (~11.43 €)	~26 %	—

Low-opex scenario (total 68,300 €/year):

Cloud reference	Break-even utilisation
AWS p5 on-demand (6.88 USD ≈ 6.40 €)	~38 %
Neocloud on-demand (2.90 USD ≈ 2.70 €)	~91 %
Neocloud 3-yr reserved (1.80 USD ≈ 1.67 €)	>100 %

High-opex scenario (total 130,300 €/year):

Cloud reference	Break-even utilisation
AWS p5 on-demand (6.88 USD ≈ 6.40 €)	~72 %
GCP A3 on-demand (~10.21 €)	~45 %
Azure on-demand (~11.43 €)	~41 %

Reading the break-even table

Against neoclouds (on-demand or reserved), on-prem TCO does not reach break-even in any scenario of the base model. Neocloud reserved beats on-prem even at 100 % utilisation, because its hourly price is below the all-in cost of your own hardware. This is consistent with the analysis in cloud GPU: on-demand, reserved and spot.
Against on-demand hyperscalers (AWS, GCP, Azure), on-prem does have an achievable break-even: around 26–72 % utilisation depending on the scenario. At medium-high utilisation (>70 %), on-prem clearly beats AWS/GCP/Azure on-demand.
The variable that moves the break-even most is opex (staff above all), not hardware capex. A well sized cluster in cheap colocation with PPA energy can lower the threshold by 20 percentage points compared with the high scenario.
For GDPR data, the break-even against US hyperscalers is skewed: the sovereignty axis rules out US hyperscalers before cost does (see sovereign on-premise vs hyperscalers).

Sensitivity analysis

TCO vs utilisation

The all-in cost per GPU-hour varies inversely with utilisation because capex is fixed:

$$\frac{d(\text{EUR/GPU-hour})}{du} = -\frac{\text{capex/year} + \text{opex/year}}{4 \times 8{,}760 \times u^2} < 0$$

Going from 50 % to 80 % utilisation cuts the €/GPU-hour by $\frac{4.75 - 2.97}{4.75} \approx 37\,\%$ in the base scenario. That 37 % reduction requires no hardware change, only more efficient scheduling (see GPU utilisation as a FinOps lever).

Utilisation	€/GPU-hour (base scenario)	Change vs 50 %
30 %	7.93	+67 %
50 %	4.75	reference
70 %	3.39	−29 %
80 %	2.97	−37 %
100 %	2.38	−50 %

TCO vs energy price

Energy price (€/kWh)	Energy opex/year	€/GPU-hour (70 % util.)	Change vs base
0.034 (solar PPA)	1,604 €	3.00	−12 %
0.116 (industrial ES, base)	5,475 €	3.39	reference
0.160 (European average)	7,550 €	3.54	+4 %
0.200 (high tariff)	9,437 €	3.67	+8 %

Energy has a moderate impact on total TCO (8–12 % variation between the extremes), because hardware capex dominates. Over a very long amortisation (5 years) with a solar PPA, however, energy drops from 6 % to 1 % of total TCO and the differential is amplified. The price of energy matters more for the carbon footprint (CSRD) than for TCO when capex is dominant.

TCO vs PUE

PUE	Cooling overhead	Energy/year (0.116 €/kWh)	€/GPU-hour (70 % util.)
1.15 (liquid cooling, new facilities)	+15 %	2,166 €	3.21
1.20 (liquid, modern datacenter)	+20 %	2,259 €	3.23
1.48 (facilities <5 years, Uptime 2025)	+48 %	3,490 €	3.33
1.54 (global average, Uptime 2025)	+54 %	3,627 €	3.39
1.80 (legacy colocation)	+80 %	4,260 €	3.47

The difference between PUE 1.15 (liquid) and 1.80 (legacy) is barely ~8 % of the €/GPU-hour at 70 % utilisation, because energy is only a fraction of TCO. PUE matters far more for the absolute energy cost and for CSRD reporting than for total TCO when hardware is the dominant component.

TCO vs amortisation years

Amortisation	Capex/year (average node, USD)	€/GPU-hour (70 % util., base opex scenario)
3 years	59,500 USD (~55,300 €)	3.39
4 years	44,625 USD (~41,500 €)	2.99
5 years	35,700 USD (~33,200 €)	2.72

Stretching the amortisation from 3 to 5 years lowers the €/GPU-hour by ~20 %, assuming the hardware remains competitive and the resale market supports the residual value. With the refresh cycle accelerated by Blackwell GB200/GB300, a 5-year amortisation carries a higher risk of technological obsolescence.

Sensitivity heat map (€/GPU-hour at 70 % utilisation, base scenario)

	PUE 1.15	PUE 1.54	PUE 1.80
3-yr amort., solar PPA (0.034 €)	2.72	2.74	2.76
3-yr amort., industrial (0.116 €)	3.21	3.39	3.47
5-yr amort., industrial (0.116 €)	2.54	2.72	2.80
3-yr amort., high tariff (0.200 €)	3.44	3.67	3.78

Decision table: cost/control/sovereignty Pareto

The table below crosses the four dimensions with no implicit hierarchy; the ordinal reading depends on each organisation’s constraints.

Option	€/GPU-hour	Initial capex	Full stack control	EU sovereignty	Elasticity	Operational risk
On-prem (util. >70 %, low opex)	2.40–3.00	high (150–207k USD/node)	total	total	none	hardware failure, idle
On-prem (util. <50 %, base opex)	4.75–7.93	high	total	total	none	capex with no return
Neocloud 3-year reserved (CoreWeave, Lambda)	1.49–2.10 USD	none	partial (API)	depends on the provider	rigid contract	minimal interruption
Neocloud on-demand (Lambda, Spheron)	2.49–3.44 USD	none	partial	depends	total	no interruption
AWS p5 on-demand	6.88 USD	none	minimal	NO (CLOUD Act)	total	no interruption
AWS p5 3-year reserved	~2.97 USD	financial commitment	minimal	NO (CLOUD Act)	rigid	no interruption
Sovereign EU cloud (Scaleway, Nebius EU)	2.15–3.85 USD	none	partial	yes (EU)	total	no interruption
Hybrid on-prem base + EU cloud peak	2.00–3.50 (weighted)	medium	high	yes (EU)	elastic peak	operational complexity

“EU sovereignty” column: US hyperscalers (AWS, GCP, Azure) are subject to the US CLOUD Act regardless of the datacenter region. Nebius has a Dutch legal entity; CoreWeave is a US company. See the full analysis in sovereign on-premise vs hyperscalers.

“Full stack control” column: on-prem lets you choose the driver version, the kernel, the NCCL configuration, MIG partitioning, and any system parameter. Cloud options offer control at container/pod level, with the hypervisor and firmware opaque.

The cost/sovereignty Pareto frontier for GDPR data excludes US hyperscalers, leaving: on-prem, sovereign EU cloud, and the hybrid. Among those three, the deciding variable is sustained utilisation and the predictability of traffic (see capacity planning for on-premise LLM inference).

Integration with the series FinOps model

The all-in €/GPU-hour of on-prem is the number that feeds the cost allocation pipeline of the series:

Cost-per-token identity (cost per token and per request): engine throughput × €/GPU-hour → €/1M tokens.
Chargeback and showback (chargeback and showback in GPU multitenancy): the all-in €/GPU-hour is the internal price charged to each tenant of the multi-tenant cluster.
Utilisation as a lever (GPU utilisation as FinOps): raising utilisation from 50 % to 80 % cuts the €/GPU-hour by 37 % without changing the hardware, the highest ROI in on-prem FinOps.
Capacity planning (capacity planning for on-premise LLM inference): the number of nodes to buy depends on the percentile of base load you want to cover in iron.
Cloud comparison (cloud GPU: on-demand, reserved and spot): the all-in €/GPU-hour goes head to head with the cloud price in table A7 to compute the break-even.

Sources

Spheron · LLM Inference On-Premise vs GPU Cloud: 2026 Cost and Break-Even Analysis (abr. 2026) — https://www.spheron.network/blog/llm-inference-on-premise-vs-cloud/
Introl · GPU Infrastructure TCO Model: 5-Year Cost Analysis for Enterprise AI (abr. 2026) — https://introl.com/blog/gpu-infrastructure-tco-5-year-cost-model
GMI Cloud · NVIDIA H100 GPU Pricing: 2026 Rent vs. Buy Cost Analysis (abr. 2026) — https://www.gmicloud.ai/en/blog/nvidia-h100-gpu-pricing-2026-rent-vs-buy-cost-analysis
Uptime Institute · Global Data Center Survey 2025 (PDF oficial) — https://datacenter.uptimeinstitute.com/rs/711-RIA-145/images/2025.Annual.Survey.Report.pdf
Uptime Institute · Global Data Center PUE Stalls at 1.54 (comunicado, oct. 2025) — https://mgrid.org/2025/10/01/uptime-institute-data-center-pue-stagnation-2025-liquid-cooling/
GlobalPetrolPrices · Spain Business Electricity Price (sept. 2025) — https://www.globalpetrolprices.com/Spain/electricity_prices/
PV Tech · European Solar PPA Prices Fall Below 35 €/MWh in Q3 2025 (oct. 2025) — https://www.pv-tech.org/european-solar-ppa-prices-fall-below-35-mwh-q3-2025/
Xi Computers · Supermicro SYS-821GE-TNHR 8U GPU Server Pricing (jun. 2026) — https://www.xicomputer.com/Solutions/Data-Center-Servers/SYS-821GE-TNHR.asp
Encoradvisors · Data Center Colocation Pricing 2026 — https://encoradvisors.com/data-center-colocation-pricing/
CloudZero · H100 GPU Cost In 2026: Buy, Rent, and Cloud Pricing Compared — https://www.cloudzero.com/blog/h100-gpu-cost/
Spheron · AI Inference Power Consumption and GPU Electricity Costs: 2026 Guide — https://www.spheron.network/blog/ai-inference-power-electricity-cost-2026/
NVIDIA · DGX SuperPOD H100 Electrical Specifications (documentación oficial) — https://docs.nvidia.com/dgx-superpod/design-guides/dgx-superpod-data-center-design-h100/latest/electrical.html

Sharing one GPU between several workloads: time-slicing, MPS and MIG

Thu, 11 Jun 2026 02:00:00 +0000

This post opens an operational series on how to squeeze a generic on-premise 4×H100 SXM LLM cluster. The sibling pieces: Serving several models on one GPU: swap and sleep (what to do when the models do not fit at the same time and have to take turns in memory), RAG on CPU: separating the data plane from generation (moving retrieval off the GPU to free it up) and Sovereign end-to-end assistant with LibreChat, LiteLLM and RAG, the final assembly, a fourth instalment in preparation. Here we start with the most basic question: you have one GPU and you want to put several workloads on it. How do you split it?

TL;DR

You have one GPU, or a few, and several workloads that want to run on it: a chat model, an embeddings service, a reranker, a queue of dev jobs. The GPU is underused if it only runs one thing, but piling several on carelessly causes contention, OOM or cascading failures. There are three mechanisms and they share out different things. Time-slicing (replicas from the NVIDIA k8s device-plugin) multiplexes in time: it advertises the GPU as “N GPUs” and the processes take turns at the compute, but they share the full physical VRAM, with no memory isolation, no fault isolation and no QoS. Its trap is an OOM that does not show up in the Kubernetes scheduler but at run time, when the sum of VRAM allocations exceeds the real memory. MPS (Multi-Process Service) multiplexes in space: it shares out the SMs between processes that execute kernels concurrently, reduces context-switch overhead and allows SM and memory limits per process. It raises throughput when there are many small kernels, but fault isolation is still weak. MIG (Multi-Instance GPU) partitions in hardware: it cuts the Hopper GPU into up to seven instances with dedicated SMs, L2, memory and bandwidth, with real memory, fault and performance isolation; datacenter only (A100/H100/H200/B200), never on an RTX 5090. The rule: real isolation / multi-tenant / compliance → MIG (if it is Hopper); many small concurrent kernels and trust between workloads → MPS; dev, bursts, consumer GPU or no need to isolate → time-slicing. This post works it through with numbers: the VRAM budget of four vLLM instances on an H100 advertised as four replicas, and what fits into a 10 GB MIG instance.

The analogy: a shared hob, one kitchen with several cooks, several kitchens

Imagine you have a single professional hob and three orders to cook at the same time. There are three ways to organise it, and they are exactly the three mechanisms.

Time-slicing is one hob taken in turns, with no larder of your own. Each cook comes in, cooks their dish, leaves, and the next one comes in. The split is temporal: nobody cooks at the same time, they take turns. The problem is not the hob, which gets shared out fine, but the common larder: the ingredients sit in a single shared cupboard and nobody has their own. If the three cooks reserve more flour than there is in total, it is not that they wait their turn: it is that there is no flour. The service goes down for everyone. And if one cook leaves a pan burning and starts a fire, it burns the whole kitchen, not just their corner.

MPS is several cooks coordinated on the same worktop. Now they do cook at the same time, sharing out the space on the worktop (the SMs). A head chef (the MPS daemon) coordinates so they do not collide and so the worktop does not sit empty while one of them waits for water to boil. You can assign each cook a percentage of the worktop and a larder limit. They work faster as a group because the worktop does not sit idle between small tasks. But they still share the kitchen: if one starts a serious fire, the others notice.

MIG is several independent kitchens in the same building. A concrete wall separates each kitchen: its own hob, its own larder, its own door and its own electrical panel. What happens in kitchen 3, a fire, an empty larder, a slow cook, does not touch kitchen 1. It is the only split with genuine isolation. The price: you have to decide in advance how many kitchens and of what size, the walls are fixed, and only expensive buildings (datacenter) come ready to put them up.

The rest of the post is, essentially, when you want cheap turns, when you want coordinated cooks and when you need concrete walls.

An H100 SXM 80 GB does not fill up with just any workload. A bge-reranker-v2-m3 reranker takes a few hundred MB and saturates a handful of SMs; a bge-m3 embeddings service is just as small; a 1B guardrail model in INT4 fits in a couple of GB. Dedicating 80 GB of HBM3 and 132 SMs to serving embeddings is using a hydraulic press to hammer in a drawing pin, the same argument as in mixed environments, but now inside the GPU instead of moving the workload to different silicon.

The aim of sharing is to raise the useful utilisation of fixed capital. But sharing badly introduces three pathologies:

Compute contention: two workloads fight over the same SMs and both run slowly, with unpredictable latency jitter.
Memory contention: the sum of requested VRAM exceeds the physical amount and something dies with a CUDA out of memory.
Cascading failure: a workload that blows up (an illegal kernel, an OOM) can drag its neighbours down with it if they share a context.

The three mechanisms attack these pathologies to different depths. None of them solves all three except MIG, and MIG costs specific hardware. Let us look at them one by one.

Time-slicing: compute turns, shared larder

Time-slicing is temporal multiplexing in software. On Kubernetes, the NVIDIA GPU Operator configures the device-plugin to advertise N replicas of each physical GPU. An H100 declared with replicas: 4 appears to the scheduler as four nvidia.com/gpu resources, and Kubernetes can place four pods on it. Internally, the GPU scheduler hands out compute turns to each process: it runs a bit of process A, switches to B, to C, to D, then back to A. It is the same time-sharing an operating system does with the CPU.

The key idea, and the one that causes most confusion, is this: a replica is NOT a fraction of the GPU. It is a compute turn. NVIDIA’s documentation is explicit: unlike MIG, there is no memory or fault isolation between replicas. The four replicas of the H100 see the full 80 GB of VRAM, unpartitioned. There is no 20 GB per replica. There are 80 GB for all four, handed out in cudaMalloc arrival order.

This has three consequences you need to internalise:

It does not isolate memory. If the sum of what the four processes reserve exceeds 80 GB, the fourth cudaMalloc fails with OOM. The Kubernetes scheduler does not see it coming: it counted four available nvidia.com/gpu resources and happily placed four pods. The OOM appears at run time, not at scheduling time. This is trap number one of time-slicing.
It does not isolate faults. A process that triggers an unrecoverable CUDA error can leave the GPU context in a state that affects its neighbours. They share the same device with no barriers.
It gives no compute QoS. Under contention, the turn-taking does not guarantee a minimum fraction to anyone. Each workload’s latency suffers jitter proportional to how many active replicas are fighting over the GPU at that instant. A latency-sensitive workload (an interactive chat) can see its TTFT dance around depending on what its neighbours are doing.

What is it good for, then? For dev, bursts and low utilisation. If you have four developers who touch the GPU sporadically, advertising four replicas lets all four have access without fighting almost ever (they rarely overlap while active). For batch workloads that tolerate jitter. And, a decisive advantage, it works on consumer GPUs: an RTX 5090 32 GB does not support MIG, but it does support time-slicing. It is the only “Kubernetes-native” way to share a 5090 between several pods.

The VRAM budget in time-slicing (the calculation that avoids the OOM)

Here is the maths you have to do before deploying, because Kubernetes will not do it for you. Suppose an H100 80 GB advertised as 4 replicas and we want to run four vLLM instances on it, one per replica.

vLLM reserves memory with the --gpu-memory-utilization parameter, which is the fraction of the total physical VRAM that each instance keeps (for model weights plus KV-cache). The detail that kills: that fraction is computed over the 80 physical GB, not over a supposed “20 GB of my replica”, because the replica does not have 20 GB. Remember, there is no memory partitioning. Each vLLM sees the 80 GB and reserves its fraction of them.

The no-OOM constraint is therefore that the sum of fractions be less than 1:

$$\sum_{i=1}^{N} g_i < 1 \quad\Longleftrightarrow\quad \sum_{i=1}^{N} g_i \cdot V_{\text{HBM}} < V_{\text{HBM}}$$

where $g_i$ is the --gpu-memory-utilization of instance $i$ and $V_{\text{HBM}} = 80$ GB. It is wise to leave headroom (runtime overhead, fragmentation, CUDA context), so in practice you aim for the sum to stay comfortably below 1, say $\le 0.9$.

A case that works. Four vLLM instances at $g_i = 0.20$:

$$\sum_{i=1}^{4} 0.20 = 0.80 \quad\Rightarrow\quad 0.80 \times 80\ \text{GB} = 64\ \text{GB} < 80\ \text{GB} \quad\checkmark$$

Each instance reserves $0.20 \times 80 = 16$ GB. Four instances add up to 64 GB, leaving a 16 GB cushion. There is no OOM. Each vLLM has 16 GB for weights plus KV-cache: enough for a 7B–8B model in FP8/INT4 with a modest KV-cache.

A case that blows up. The same four instances, but somebody raises $g_i = 0.30$ thinking “I have four replicas, I can give each one more”:

$$\sum_{i=1}^{4} 0.30 = 1.20 \quad\Rightarrow\quad 1.20 \times 80\ \text{GB} = 96\ \text{GB} > 80\ \text{GB} \quad\times$$

The first instances start up and reserve $0.30 \times 80 = 24$ GB each. Three instances are already at $72$ GB. The fourth tries to reserve another 24 GB, there is nothing left, and it dies with CUDA out of memory. And worse: Kubernetes will reschedule it on the same GPU (it still sees four replicas), where it will die again, in a CrashLoopBackOff that makes no sense if you only look at the pod manifest.

The operational rule is brutally simple: in time-slicing, you manage the VRAM budget by hand, adding up the --gpu-memory-utilization values. The replica count controls how many pods fit in compute turns, but it does not reserve a single byte of memory. Confusing the two is the recurring mistake.

MPS: coordinated cooks on the same worktop

The Multi-Process Service (MPS) attacks a different problem. By default, when several processes use the same GPU without MPS, each one has its own CUDA context, and the GPU alternates between contexts (time-slicing at driver level): they do not execute kernels at the same time, they take turns, with context-switch overhead. If your kernels are small and do not fill the GPU on their own, this leaves SMs idle: process A uses 30 % of the SMs during its turn and the other 70 % is wasted.

MPS introduces a daemon that shares a single CUDA context between processes, so that their kernels can execute concurrently occupying different SMs at the same time. It is spatial sharing of compute: instead of taking turns at the whole worktop, each cook occupies a part and they work in parallel. This reduces context-switch overhead and raises throughput when there are many small concurrent kernels that individually do not saturate the GPU.

And, unlike pure time-slicing, MPS allows per-process limits, which gives you a form of QoS:

CUDA_MPS_ACTIVE_THREAD_PERCENTAGE limits the percentage of SMs an MPS client can use. By default each client gets $100 / \text{MaxSharedClients}$. Pinning it to, say, 40 % caps that process’s compute ceiling (MPS docs).
CUDA_MPS_PINNED_DEVICE_MEM_LIMIT imposes a memory cap per client (valid from CUDA 11.5). This is what time-slicing does not have: a per-process VRAM limit that the runtime enforces.

These two limits turn MPS into a resource provisioning mechanism that mitigates the noisy neighbour: you can guarantee that a process does not eat more than X % of the SMs or more than Y GB. The combination gives reasonable QoS, not perfect but real.

The limitation MPS does not solve: fault isolation is weak. Since clients share the daemon’s CUDA context, a fatal error in one client can affect the daemon and therefore the other clients (historically, a client dying dirtily could require restarting the daemon). It is better than time-slicing in this respect, but a long way from hardware isolation. That is why MPS fits when there is trust between the workloads, processes from your own team, not third-party tenants.

The canonical use case: many small concurrent inference requests that individually leave the GPU half empty. MPS overlaps them and raises aggregate throughput. Serving several small models, or several light replicas of the same model, on a datacenter GPU where you trust all the workloads, is MPS territory.

MIG: concrete walls

The Multi-Instance GPU (MIG) is the only one of the three that gives genuine isolation, because it cuts the GPU in hardware. Available on modern datacenter GPUs, A100 (Ampere), H100/H200 (Hopper), B200 (Blackwell), and never on consumer ones: an RTX 5090 (consumer Blackwell) does not support MIG, nor do GeForce cards in general.

MIG divides the GPU into up to seven instances (GPU Instances), and each instance gets a dedicated portion of:

SMs (compute slices): compute is split into 7 slices, each one roughly 1/7 of the SMs.
L2 cache and memory: each instance has its slab of HBM and its portion of L2 cache.
Memory bandwidth: dedicated, not shared.
Data paths and engines: with fault barriers between instances.

The result is that a MIG instance behaves like a smaller, independent GPU: whatever happens in one, an OOM, a kernel blowing up, a workload saturating its compute, does not affect its neighbours. Memory, fault and performance (QoS) isolation, the three things time-slicing does not give and MPS only half gives.

The H100 80GB profiles

MIG does not allow arbitrary sizes: it has fixed profiles. On the H100 80GB, the profile catalogue (notation <compute>g.<memory>gb) is:

Profile	Compute (slices)	Memory	Max. instances
`1g.10gb`	1/7	10 GB	7
`1g.20gb`	1/7	20 GB	4
`2g.20gb`	2/7	20 GB	3
`3g.40gb`	3/7	40 GB	2
`4g.40gb`	4/7	40 GB	1
`7g.80gb`	7/7	80 GB	1 (whole GPU)

(There is also 1g.10gb+me, a variant with media engines for video encoding.) The base memory unit on the H100 80GB is 10 GB per slice (80 GB / 8, with one slice reserved), and the compute unit is 1/7 of the SMs. The profiles combine these units. Note 1g.20gb: the same compute fraction as 1g.10gb (1/7 of the SMs) but twice the memory, useful when a workload needs more VRAM than compute.

One important detail: MIG partitions do not mix freely. The GPU is divided following a valid geometry (the profiles fit together like pieces in a grid), and the profiles are fixed when you configure the GPU; changing them requires draining and repartitioning. They are concrete walls: solid, but they do not move while hot.

The calculation: 7×1g.10gb against 1×7g.80gb

Let us compare the two extremes. On the left, seven 1g.10gb instances: seven isolated 10 GB GPUs. On the right, a single 7g.80gb: the whole H100, unpartitioned.

The operational question is what fits in 10 GB. An instance’s VRAM budget is split between model weights and KV-cache:

$$V_{\text{inst}} = V_{\text{weights}} + V_{\text{KV}} + V_{\text{overhead}}$$

Take a model of 7B parameters in FP8 (1 byte/parameter):

$$V_{\text{weights}} \approx 7 \times 10^9 \times 1\ \text{byte} = 7\ \text{GB}$$

On a 1g.10gb instance (10 GB), after the 7 GB of weights and subtracting roughly 0.5–1 GB of runtime overhead, about 2 GB are left for KV-cache. That is enough for a modest context window and low concurrency, fine for a guardrail service, a classifier or an extraction model that handles short prompts one at a time. A 7B in INT4 (~3.5 GB of weights) leaves ~5.5 GB of KV-cache, much roomier. But a 13B model in FP8 (~13 GB of weights) does not fit in a 10 GB instance: not even the weights get in. For that you need 1g.20gb, 2g.20gb or larger.

Against this, the 7g.80gb (whole GPU) gives you the 80 GB for one large model: a 70B in FP8 (~70 GB of weights) fits with a tight KV-cache, or a 70B with more room spread across several whole GPUs in tensor-parallel (see TP against replicas: one big one versus N small ones).

The reading is clear: fine partitioning (7×1g.10gb) maximises the number of small isolated workloads; not partitioning (1×7g.80gb) maximises the model size that fits. The KV-cache available per instance shrinks proportionally as you partition, so fine-grained MIG serves many light isolated services, not one large model chopped up. If your workload is a single large model, MIG is not for you: use the whole GPU or several in TP.

The decision tree

The three questions, in order:

Do you need REAL isolation?
(multi-tenant, compliance, one workload failing must not touch another)
│
┌────┴────┐
YES NO
│ │
Is it Hopper/ Many SMALL concurrent kernels
Ampere/ AND you trust every workload?
Blackwell? │
│ ┌────┴────┐
┌──┴──┐ YES NO
YES NO │ │
│ │ MPS Dev / bursts / consumer GPU
MIG │ (spatial, / no need to isolate?
│ │ per-process │
│ no real QoS) YES
│ isolation: │
│ rethink TIME-SLICING
│ (move to (temporal, cheap,
│ CPU, another works on a 5090)
│ GPU, or accept
│ the risk with
│ time-slicing)

And each branch in one sentence:

MIG when isolation is a requirement (compliance, multi-tenant, hard SLA) and you have datacenter hardware that supports it. Concrete walls cost money, but if you need them there is no substitute.
MPS when you have many small concurrent workloads that leave the GPU half empty and you trust all of them (same team, not third-party tenants). You raise throughput with reasonable QoS, accepting imperfect fault isolation.
Time-slicing when it is dev, bursts, low utilisation, a consumer GPU, or you simply do not need to isolate anything. Cheap and universal, but manage the VRAM budget by hand.

One nuance the recent documentation records: they can be combined. You can do time-slicing on top of a MIG instance (hardware isolation at the instance boundary, software turns inside), or use MPS inside a MIG instance. The layers are not mutually exclusive; the tree picks the primary strategy.

Applied to the generic 4×H100 cluster

Let us get down to numbers with a generic on-premise cluster of 4×H100 SXM 80 GB with NVLink. It is common to have a heterogeneous menu of workloads: a large chat model, light services (embeddings, reranker, guardrails) and a dev/experimentation queue. Each type calls for a different mechanism. A reasoned split:

GPU 0 and GPU 1, large model in tensor-parallel (not shared). A 70B model in FP8 takes ~70 GB of weights; served comfortably with a generous KV-cache it needs more than one H100. We spread it in tensor-parallel over two whole H100s joined by NVLink (intra-node bandwidth is what makes TP viable; the detail is in TP against replicas). Here we do not share: these two GPUs belong to the large model, full stop. Total isolation by dedication.

$$V_{\text{available}} = 2 \times 80 = 160\ \text{GB};\quad V_{\text{weights}} \approx 70\ \text{GB};\quad V_{\text{KV}} \approx 80\ \text{GB of KV-cache}$$

There is memory to spare for a long request queue and high concurrency.

GPU 2, split with MIG into small instances for light services. Embeddings (bge-m3), reranker (bge-reranker-v2-m3) and a couple of guardrail models (1B–3B) are different workloads, potentially from different teams, and you want a failure or a spike in one not to touch the others. Light multi-tenancy with isolation → MIG. A reasonable partitioning of the H100:

$$\underbrace{3 \times \texttt{1g.10gb}}_{\text{30 GB, 3/7 SMs}} \;+\; \underbrace{1 \times \texttt{4g.40gb}}_{\text{40 GB, 4/7 SMs}}$$

The three 1g.10gb instances (10 GB, 1/7 of the SMs each) host embeddings, reranker and a 1B INT4 guardrail, each isolated, with no noisy neighbour. The 4g.40gb (40 GB, 4/7 of the SMs) hosts an intermediate 7B–13B model with a decent KV-cache for a support service. Each service has its larder and its wall; if the reranker blows up, the chat never finds out.

GPU 3, time-slicing for dev and bursts. The developers touch the GPU sporadically: experiments, short fine-tunes, model trials. They do not need isolation (it is the same team) and they rarely overlap while active. We advertise it as 4 replicas via the device-plugin. Four dev pods fit in turns. VRAM budget with the formula above: if each dev brings up a vLLM at --gpu-memory-utilization 0.20, they add up to $4 \times 16 = 64$ GB < 80 GB, with no OOM. If somebody needs more, lower the replica count or coordinate with the team. The cost of flexibility is manual discipline.

Summary of the split:

Resource	Mechanism	Workload	Isolation
GPU 0 + GPU 1	Dedication (TP)	70B chat in tensor-parallel	total (dedicated)
GPU 2	MIG (3×1g.10gb + 1×4g.40gb)	embeddings, reranker, guardrails, 7B–13B	real hardware
GPU 3	Time-slicing (4 replicas)	dev, bursts, experiments	none (trust)

The logic is always the same: spend isolation (MIG) where you need it, spend cheap concurrency (time-slicing) where you do not, and reserve whole GPUs for what genuinely fills them. An H100 serving embeddings on a 7g.80gb would be as absurd as an RTX 5090 trying MIG: the tool does not match the workload.

What we have not covered

What happens when they do not even fit at the same time: if you have more models than VRAM and they have to take turns in memory (loading/unloading weights, not just taking turns at compute), you enter swap and sleep territory, the sibling piece Serving several models on one GPU.
NUMA-aware scheduling: on multi-socket nodes, which GPU goes with which CPU/memory matters for latency; see Kubelet resource managers on RKE2.
Autoscaling the replicas: how many instances to bring up according to real load, with KEDA and queue metrics; see Autoscaling LLMs on Kubernetes with KEDA.
A jitter benchmark under contention: how much TTFT really wobbles in time-slicing with 4 active replicas compared with MIG, material that deserves its own measurement, not an estimate.

References

NVIDIA — Time-Slicing GPUs in Kubernetes (GPU Operator). https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html
NVIDIA — Multi-Process Service (MPS) Overview. https://docs.nvidia.com/deploy/mps/index.html
NVIDIA — MPS: Tools and Interface Reference (CUDA_MPS_ACTIVE_THREAD_PERCENTAGE, CUDA_MPS_PINNED_DEVICE_MEM_LIMIT). https://docs.nvidia.com/deploy/mps/appendix-tools-and-interface-reference.html
NVIDIA — Multi-Instance GPU (MIG) User Guide. https://docs.nvidia.com/datacenter/tesla/mig-user-guide/
NVIDIA — Supported MIG Profiles (H100 80GB catalogue). https://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-mig-profiles.html
NVIDIA — k8s-device-plugin (time-slicing replicas). https://github.com/NVIDIA/k8s-device-plugin
vLLM — Engine Arguments (--gpu-memory-utilization). https://docs.vllm.ai/en/latest/serving/engine_args.html

The shared bench: NVLink, NVSwitch and NCCL, the cable every token crosses in tensor parallel

Sat, 06 Jun 2026 07:00:00 +0200

This post goes one floor below the engine. In the seven-layer inference stack and in one big replica or many small ones the decision was how many GPUs and how to split the model; here we explain the cable that makes that split work, or that strangles it. It is the first of a mini-series “below the engine”: interconnect (this one) → kernel and NUMA → Kubernetes resource managers.

TL;DR

Tensor parallelism (TP) does not split a model into four chunks that run on their own. It spreads each layer across the GPUs, but after attention and after the MLP the GPUs have to add up their partial results with an all-reduce. On a Llama-70B with 80 layers, that is around 160 all-reduces per generated token. That all-reduce travels over the interconnect, so the interconnect sits on the critical path of every token, not in the background plumbing. On an HGX H100 baseboard, the 8 GPUs all talk to each other at 900 GB/s bidirectional via four NVSwitch chips; without NVSwitch/NVLink, that same traffic falls back to the CPU over PCIe and loses an order of magnitude. NCCL is the library that decides how each collective is done (ring, tree, or NVLS = NVLink-SHARP, which offloads the summation to the switch itself). And there is an asymmetry almost nobody keeps in mind: decode is latency-bound (tiny messages, 16 KB) and prefill is bandwidth-bound (huge batched activations), which is why “more NVLink bandwidth” speeds up prefill but barely touches token-by-token decode. This post explains the mechanism, gives the 10 real NCCL/driver knobs worth touching, and connects with vLLM’s custom all-reduce, disaggregated serving and GPU observability. With scepticism about which levers move the needle.

Where you are: the floor below the engine

The analogy: four mechanics and a single bench

Four mechanics assemble one and the same car engine. It is not that each builds their own engine in parallel, that would be four cars (four replicas of the model, a different strategy). Here they build a single one, at the same time, splitting the parts: one does the pistons, another the cylinder head, another the crankshaft. The problem is that the parts fit into each other: before going on, all four have to put together what they have and check that it matches. That “put together and check” happens dozens of times during the build.

There are two ways to organise the workshop:

A single large bench, everyone around it (NVSwitch). Each mechanic reaches out and hands their part directly to any other, all at once, without standing up. It is instant and simultaneous. This is NVLink + NVSwitch: the GPUs form an all-to-all where anyone talks to anyone at 900 GB/s at the same time.
Four separate workshops with a courier (PCIe via CPU). Every part a mechanic wants to hand to another goes into a box, goes down to reception (host memory, via CPU), and from there up to the destination workshop. Slower, and serialised by reception. This is what happens when there is no NVLink: inter-GPU traffic falls back to PCIe and bounces through the CPU, around 14× slower than NVLink.

The thesis of the post follows on its own: tensor parallelism only makes sense if the mechanics share the bench. As soon as the “put together and check” (the all-reduce) has to go through reception, splitting the work costs more than it saves. That is why, on a serious platform, TP does not cross the NVLink boundary: TP=4 or 8 inside the baseboard where there is NVSwitch, and above that you replicate or use pipeline, never stretch TP over PCIe or over the network. When each option makes sense is covered in one big replica or many small ones; here we explain why the cable dictates that decision.

The mechanism: what an all-reduce really is and why there are 160 per token

Tensor parallelism splits the weight matrices by columns/rows across the $N$ GPUs. Each GPU computes a portion of the layer output. But the next operation needs the complete output, so it has to be recombined. That recombination is a collective operation: an all-reduce, which adds the partial tensors of all the GPUs element by element and leaves the result identical on all of them.

In a standard transformer block there are two synchronisation points per layer:

After the output projection of attention (the o_proj that recombines the split heads).
After the second matrix of the MLP (the down_proj that recombines the split feed-forward).

$$ \text{all-reduces per token} = 2 \times L_{\text{layers}} $$

For a Llama-70B ($L = 80$): $2 \times 80 = 160$ all-reduces per generated token. Not per request, not per sequence: per token. Multiply by decode throughput and you understand why the interconnect is not background infrastructure but a hot path.

How the all-reduce is done: ring, tree, NVLS

NCCL does not have a single way of doing an all-reduce; it picks an algorithm according to topology and message size:

Ring. The GPUs form a ring; each one passes a chunk to its neighbour, adds, and rotates. It takes $2(N-1)$ steps. It is bandwidth-optimal for large messages: the cost of moving the data is $\frac{2(N-1)}{N} \times M$ bytes over the link, almost independent of $N$. The downside: $2(N-1)$ latency hops, bad for small messages.
Tree. Tree reduction: $\log N$ levels. Better latency for small messages and many nodes, worse bandwidth utilisation.
NVLS (NVLink SHARP). The Hopper trick: the summation is not done by the GPUs, it is done by the NVSwitch. The switch has reduction units; the GPUs send their tensors, the switch adds them in flight and returns the result. It takes work away from the GPUs (frees up SMs) and reduces hops. Available only with a 3rd-generation NVSwitch (NVLink4) plus Hopper or newer.

The mental rule: decode (tiny messages) wants latency → tree/LL or vLLM’s custom kernel; prefill (huge messages) wants bandwidth → ring/NVLS. That is why there is no global “optimal NCCL_ALGO”; it depends on which phase you are looking at.

The maths that matter: why decode and prefill stress the cable the other way round

Here is the asymmetry almost everyone skips. The size of the tensor that gets all-reduced in each layer is, approximately:

$$ M \approx B \times S \times h \times 2\ \text{bytes (BF16)} $$

where $B$ = batch, $S$ = tokens processed in this forward pass, $h$ = hidden size.

In decode, you generate 1 token per sequence per iteration. For a single sequence ($B \times S = 1$) and $h = 8192$ (Llama-70B):

$$ M_{\text{decode}} \approx 1 \times 8192 \times 2 = 16\ \text{KB per all-reduce} $$

16 KB is tiny. At 900 GB/s, moving 16 KB takes around 18 nanoseconds of pure transfer, but the real cost is dominated by the launch latency of the collective (synchronisation, kernel launch), of the order of single-digit microseconds. With 160 all-reduces per token:

$$ t_{\text{comms/token}} \approx 160 \times (5\text{–}10\,\mu s) \approx 0.8\text{–}1.6\ \text{ms} $$

That is the communication floor per token, independent of bandwidth. An uncomfortable and counter-intuitive implication: buying more NVLink bandwidth does not speed up token-by-token decode of a single sequence. What helps in decode is lowering the latency per collective (LL protocol, vLLM’s custom all-reduce, NVLS to remove hops) and batching (raising $B$ amortises the fixed latency over more tokens, the deep reason continuous batching exists, covered in continuous batching).

In prefill, you process the whole prompt at once: $S$ can be thousands of tokens, and with batching $B \times S$ reaches tens of thousands. There:

$$ M_{\text{prefill}} \approx 8000 \times 8192 \times 2 \approx 131\ \text{MB per all-reduce} $$

131 MB does stress the bandwidth. At 900 GB/s (NVSwitch) the ring all-reduce moves $\frac{2 \cdot 3}{4} \times 131 \approx 196$ MB effective in around 0.22 ms; over PCIe (around 64 GB/s aggregate, bouncing through the CPU) it would be around 3 ms and serialised. Here the cable is the bottleneck and NVLS/bandwidth rule.

Summary in one line: prefill is bandwidth-bound, decode is latency-bound. Any interconnect tuning that does not say which phase it helps is noise.

The hardware: NVLink 4 and NVSwitch on the HGX baseboard

On the generic reference cluster, 4×H100 SXM inside an HGX baseboard, the concrete figures:

Each H100 SXM5 has 18 NVLink 4 links, each one 50 GB/s bidirectional ⇒ 900 GB/s bidirectional aggregate per GPU. That is >14× the bandwidth of a PCIe Gen4 x16 (around 64 GB/s bidir).
On an 8-GPU HGX H100 baseboard, the 18 links of each GPU are spread against four 3rd-generation NVSwitch chips (grouped 5+4+4+5). The result is all-to-all: any GPU talks to any other at 900 GB/s simultaneously, without going through CPU or PCIe.
A 4-GPU baseboard is half a board: same principle, via NVSwitch. Design key: if your 4 H100s are connected by NVSwitch, you have real all-to-all; if they are on different boards connected by PCIe (some “4×PCIe” configurations), you do not have NVLink between all of them and TP=4 suffers. Verify it, do not assume it.

The 10 knobs worth touching

Almost all of them are NCCL environment variables (injected into the inference engine process) or driver settings. Ordered by impact/frequency in an on-premise deployment. The canonical detail is in the NCCL env vars docs.

Knob 1 — `NCCL_DEBUG` + topology dump: see what is happening before touching anything

Do not optimise blind: first confirm which topology and algorithms NCCL picked. This tells you whether it really is using NVLink or whether, silently, it fell back to PCIe/SHM, the number one failure and the most expensive one.

NCCL_DEBUG=INFO # prints topology, rings/trees built, transport chosen
NCCL_DEBUG_SUBSYS=GRAPH,TUNING,NET # narrows it down to what matters
# look in the log for: "via NVLink" / "via P2P" (good) vs "via SHM" / "via PCI" (bad)

If you see via SHM or via PCI between GPUs that should have NVLink, you have a topology problem (PCIe ACS enabled, IOMMU, GPUs on different boards) and no other knob fixes it. This is knob 1 for a reason: half of the “NVLink is slow” cases are “NVLink is not being used”.

Knob 2 — `NCCL_ALGO`: ring vs tree vs NVLS

Forces or excludes algorithms. By default NCCL picks according to size, and it usually gets it right; touch it only with a measurement in front of you.

NCCL_ALGO=NVLS,Tree,Ring # order of preference
NCCL_ALGO=^Ring # exclude Ring (^ prefix)

Rule: prefill/training (bandwidth) ⇒ Ring/NVLS; decode (latency) ⇒ Tree or, better, vLLM’s custom kernel (knob 10/stack). In most inference workloads, leaving it on auto and validating with knob 1 is the right thing; forcing it “just in case” usually makes things worse.

Knob 3 — `NCCL_PROTO`: LL / LL128 / Simple

The protocol controls the latency/bandwidth trade-off at low level:

NCCL_PROTO=Simple # maximum bandwidth, more latency (large messages)
NCCL_PROTO=LL # low-latency, half-bandwidth (tiny messages: decode)
NCCL_PROTO=LL128 # compromise, default on platforms that support it

LL (low-latency) uses flags instead of barriers and wins on the 16 KB messages of decode; Simple wins on the 131 MB of prefill. The default LL,LL128,Simple lets NCCL choose by size, which again is normally the best option.

Knob 4 — `NCCL_NVLS_ENABLE`: offload the summation to the NVSwitch

NVLink SHARP (NVLS) makes the switch do the reduction, freeing up GPU SMs:

NCCL_NVLS_ENABLE=1 # default: ON where there is an NVLink4+ NVSwitch (Hopper)

An important sceptical caveat: NVLS requires NVSwitch (3rd gen, NVLink4). On a node with NVLink through direct GPU-to-GPU bridges (no switch) or on 4×PCIe, NVLS is not available and this knob does nothing. Before “enabling it”, confirm with knob 1 that your topology has a switch. Where it does apply, its biggest advantage is freeing SMs for compute, which is relevant when comms and kernels compete (knob 5).

Knob 5 — `NCCL_MIN_NCHANNELS` / `NCCL_MAX_NCHANNELS`: how many SMs communication steals

Each NCCL “channel” consumes GPU SMs to move data. More channels = more collective bandwidth, but fewer SMs for the inference kernel. It is a split of a fixed resource.

NCCL_MIN_NCHANNELS=4
NCCL_MAX_NCHANNELS=16 # raising it helps prefill (bandwidth); steals SMs from decode

In decode, where the GPU is under-used on compute but tied to latency, trimming channels rarely hurts and sometimes helps; in prefill, more channels squeeze out the bandwidth. A measurement knob, not a faith knob.

Knob 6 — `NCCL_BUFFSIZE`: the buffer size per channel

NCCL_BUFFSIZE=8388608 # 8 MB (default 4 MB); larger buffers → better BW on large messages

Raising it helps bandwidth-bound prefill at the cost of memory per channel. For workloads dominated by small messages (pure decode), the default is plenty.

Knob 7 — `NCCL_P2P_LEVEL` / `NCCL_P2P_DISABLE`: guarantee P2P over NVLink

P2P is what lets one GPU read another’s memory directly over NVLink without going through the host. If it is disabled or degraded, traffic falls back to SHM/PCIe.

NCCL_P2P_LEVEL=NVL # use P2P up to the NVLink level
# NCCL_P2P_DISABLE=1 ← only as a workaround if P2P HANGS (multi-NUMA PCIe, certain Blackwell)

Watch out for the trap: NCCL_P2P_DISABLE=1 and --disable-custom-all-reduce are recommended as a patch when vLLM hangs on PCIe-only multi-NUMA topologies. It is a robustness patch that sacrifices performance: use it if it hangs, never “by default”.

Knob 8 — GPUDirect RDMA for multi-node: `NCCL_NET_GDR_LEVEL`

When TP fits in one node, this does not apply. When you have to cross nodes (huge model, pipeline parallel between baseboards), GPUDirect RDMA lets the GPU talk to the NIC without bouncing through host memory:

NCCL_NET_GDR_LEVEL=PHB # enables GDR according to GPU–NIC proximity on the PCIe bus

Without GDR, every inter-node hop adds a host copy. With InfiniBand/RoCE plus GDR, the KV or the activations travel GPU→NIC→network→NIC→GPU. It is the basis of serious multi-node work and of mixed environments.

Knob 9 — `NCCL_IB_HCA` / `NCCL_SOCKET_IFNAME`: pin the right NIC

The most common and silent multi-node mistake: NCCL picks the management NIC (1 GbE) instead of the fabric one (InfiniBand/100 GbE). Result: collectives at a snail’s pace with no visible error.

NCCL_SOCKET_IFNAME=eth0 # control interface (bootstrap)
NCCL_IB_HCA=mlx5_0,mlx5_1 # the real InfiniBand HCAs of the fabric
NCCL_IB_GID_INDEX=3 # correct GID for RoCE v2

Pin them explicitly. “Auto” gets it right on clean clusters and fails as soon as there is more than one NIC.

Knob 10 — Driver: persistence mode, clocks and NVLink error counters

Below NCCL, the driver has levers and, above all, telemetry you have to look at:

nvidia-smi -pm 1 # persistence mode: avoids driver re-init (latency/jitter)
nvidia-smi nvlink --status # are all 18 links active and at full speed?
nvidia-smi nvlink -e # error/CRC counters per link
nvidia-smi -q -d ECC # memory errors that degrade silently

An NVLink link that negotiates at half speed or accumulates CRC errors degrades the all-reduce without raising any error: the system “works”, it is just slower. These counters are the difference between diagnosing in five minutes and chasing a ghost for days. They are integrated in DCGM (knob/stack: observability).

Summary table

#	Knob	Variable / command	Phase it helps
1	Topology diagnosis	`NCCL_DEBUG=INFO` + `SUBSYS=GRAPH`	always, first
2	Collective algorithm	`NCCL_ALGO` (NVLS/Tree/Ring)	depends on phase; auto usually wins
3	Protocol	`NCCL_PROTO` (LL/LL128/Simple)	LL=decode, Simple=prefill
4	NVLink SHARP	`NCCL_NVLS_ENABLE=1`	prefill; frees SMs (requires NVSwitch)
5	Channels (SMs)	`NCCL_MIN/MAX_NCHANNELS`	+prefill bandwidth / −SM theft in decode
6	Buffer	`NCCL_BUFFSIZE`	bandwidth-bound prefill
7	NVLink P2P	`NCCL_P2P_LEVEL=NVL`	critical; disable only if it hangs
8	GPUDirect RDMA	`NCCL_NET_GDR_LEVEL`	multi-node
9	Fabric NIC	`NCCL_IB_HCA`/`SOCKET_IFNAME`	multi-node (avoids mgmt NIC)
10	Driver + telemetry	`nvidia-smi -pm 1` / `nvlink -e`	jitter + silent diagnosis

How it connects with the rest of the stack

The interconnect is not an island; it touches almost every layer above.

With vLLM, the custom all-reduce. vLLM does not always use NCCL: for the tiny messages of decode (world_size==2 or a fully-connected NVLink topology, below a certain max_size) it uses its own all-reduce kernel that beats NCCL on latency, exactly the decode bottleneck we saw in the maths. It falls back to NCCL for large messages and for topologies without NVLink (where its custom kernel “adds little over NCCL”). The flag --disable-custom-all-reduce / VLLM_DISABLE_CUSTOM_ALL_REDUCE turns it off; it is the patch for hangs on multi-NUMA PCIe. Translation: the most effective decode latency knob is sometimes not an NCCL one, it is choosing well between vLLM’s custom kernel and NCCL.

With TP vs replicas. Everything in one big replica or many small ones rests on this: high TP is only viable inside the NVLink domain. The boundary of “TP=4 or 4 replicas at TP=1?” is drawn by the cable: crossing NVLink with TP means paying for the all-reduce at PCIe prices.

With disaggregated serving. In disaggregated prefill/decode, the KV cache generated in the prefill pool has to travel to the decode pool. That transfer is another consumer of the interconnect (NVLink intra-node, GPUDirect RDMA inter-node) and competes with the all-reduces. Designing the disaggregation without counting the cost of transferring the KV is the classic trap.

With MoE. Mixture-of-Experts models add expert parallelism: an all-to-all (not an all-reduce) that routes each token to its expert, possibly on another GPU. It is a different communication pattern and heavier on bandwidth; MoE in inference lives or dies by the same cable, with an even more demanding collective.

With GPU observability. The NVLink counters (nvidia-smi nvlink -e, TX/RX bytes per link, CRC errors) and NVSwitch utilisation are exposed via DCGM and land in Prometheus/Grafana. The question “is the interconnect healthy and saturated?” is answered there, alongside the rest of GPU observability with DCGM. A slow all-reduce shows up sooner in an NVLink error counter than in API latency.

With capacity planning. The inference sizing that assumes “TP=4 scales almost linearly” only holds inside NVLink. Outside it, scaling efficiency collapses and the capacity plan lies. The cable is a parameter of the capacity model, not a detail.

Traps and things that are not what they look like

“More NVLink bandwidth = faster decode.” False for a single sequence. Decode is latency-bound; bandwidth is barely touched by 16 KB messages. What speeds up decode is batching (amortising the fixed latency) and lowering latency per collective (LL, custom kernel, NVLS). Bandwidth rules in prefill.

“I have 4 H100s, so I have NVLink between all four.” Not necessarily. There are configurations where the GPUs are on different boards joined by PCIe, or with NVLink bridges only in pairs. Confirm it with nvidia-smi nvlink --status and knob 1 before planning TP=4. A TP=4 over P2P-over-PCIe performs far worse than the brochure says.

Forcing NCCL_ALGO/NCCL_PROTO “to go faster”. NCCL picks well by size in most cases. Forcing an algorithm without measuring usually makes one of the two phases worse. The correct sequence is: knob 1 (see what it is doing) → measure → touch only if there is evidence.

Disabling P2P/custom all-reduce by default. They are robustness patches for broken topologies (multi-NUMA PCIe, certain Blackwell). Leaving them on “for stability” on a node with healthy NVLink throws performance in the bin.

Stretching TP over the network. TP=8 crossing two nodes over InfiniBand because “there is bandwidth” ignores that the per-layer all-reduce now pays network latency ×160 per token. To cross nodes, pipeline parallel (which communicates once per micro-batch, not per layer) almost always wins. The communication pattern, not just the bandwidth, decides.

Ignoring the NVLink error counters. A degraded link does not raise an exception: the system works, it is just slow. Without watching nvlink -e and ECC, you chase a performance ghost that a counter would have pointed out in five minutes.

Conclusion

Tensor parallelism sells a simple promise, split the model, multiply the VRAM, serve models that do not fit in one GPU, but the small print is that every layer forces the GPUs to come together and add up, twice, dozens of times per token. That all-reduce is the real hidden protagonist of performance, and it lives in the cable: NVLink does it on the shared bench of the NVSwitch at 900 GB/s, or PCIe drags it through the CPU’s reception desk 14× slower. Of the ten knobs, the first one, looking with NCCL_DEBUG at what is really happening, solves half the problems, because half of the “NVLink is slow” cases are “NVLink is not being used”. The rest are refinements that only mean something if you know which phase you are in: prefill wants bandwidth (NVLS, Simple, channels, buffer), decode wants latency (LL, vLLM’s custom kernel, batching). And above all, an idea that reorders the intuition: in on-premise inference, the interconnect is not plumbing you install and forget, it is a hot path, a capacity parameter and, when it degrades silently, the root cause that no API dashboard will point out to you unless you look at the counters of the cable itself.

References

NVIDIA, NVIDIA Hopper Architecture In-Depth (NVLink 4, 900 GB/s): https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/.
NVIDIA, Introducing NVIDIA HGX H100 (4× NVSwitch, all-to-all): https://developer.nvidia.com/blog/introducing-nvidia-hgx-h100-an-accelerated-server-platform-for-ai-and-high-performance-computing/.
NVIDIA, NCCL Environment Variables (all the knobs in this post): https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html.
NVIDIA, The NVLink-Network Switch (Hot Chips 2022, NVLink SHARP): https://hc34.hotchips.org/.
vLLM, Why does vLLM use a custom all-reduce method? (discussion #6159) and custom_all_reduce.py: https://github.com/vllm-project/vllm/discussions/6159.
NVIDIA, NCCL Multi-Node NVLink Tuning Guide: https://docs.nvidia.com/multi-node-nvlink-systems/multi-node-tuning-guide/nccl.html.

FP8 end to end: enable it, measure quality and decide with data

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

FP8 is the configuration change with the highest impact per unit of effort available on H100 and Ada Lovelace hardware. On an H100 it enables native FP8 tensor cores: +40-60% decode throughput and ×2 the VRAM available for the KV cache. On an RTX 4090 and an L40 the compute benefit is smaller, but the ×2 VRAM is real and translates directly into twice the concurrency. The risk is quality degradation, which on well-calibrated modern models is <0.5% on standard benchmarks but can be larger on formal reasoning. The correct workflow is not to switch it on and pray: it is to switch it on in staging, run the eval suite, correlate quality with throughput in OTel, and decide with data.

The analogy

A photographer who works with 35 mm negatives and moves to digital. Digital photos take up less space and are processed faster. But a low-resolution photo of a landscape can be indistinguishable from the high-resolution one to the human eye, while a low-resolution photo of text loses letters. Exactly the same trade-off applies to FP8: for tasks where numerical imprecision is averaged out over thousands of activations (conversation, summarisation, RAG), it is practically invisible. For tasks where a single wrong multiplication propagates into an incorrect answer (formal mathematics, critical code), it can be decisive.

The three layers of FP8 in vLLM

FP8 is not a single flag: it is three independent layers that are enabled separately and have different benefits.

Layer 1 — Model weights (--quantization fp8): The model weights are stored and computed in FP8 E4M3. The models have to be pre-quantised (available on HuggingFace with a -FP8 or -fp8 suffix) or quantised at load time with calibration. The benefit: the model takes up half the VRAM and the weight matmuls are 2× faster on an H100.

# Pre-quantised model (recommended for production)
vllm serve neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 \
 --quantization fp8

# Or on-the-fly quantisation (no extra files, somewhat slower on the first tokens)
vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
 --quantization fp8 \
 --kv-cache-dtype auto

Layer 2 — KV cache (--kv-cache-dtype fp8): The K and V tensors of the KV cache are stored in FP8 instead of BF16. It halves the size of the KV cache, doubling the number of tokens that fit in VRAM. It does not affect the model weights.

vllm serve my-model \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales # dynamic calibration, mandatory to minimise degradation

Layer 3 — Activations (automatic on an H100): On Hopper GPUs, vLLM automatically enables FP8 for the intermediate activations when both previous layers are active. No extra flag is required.

Full production configuration:

vllm serve neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 \
 --quantization fp8 \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales \
 --gpu-memory-utilization 0.92 \
 --max-model-len 16384

The measurable impact by hardware

H100 SXM (Hopper, native FP8 tensor cores)

Metric	BF16 baseline	FP8 enabled	Delta
Decode throughput (tok/s, 70B, batch 32)	~1,800	~2,700	+50%
Model VRAM (70B)	140 GB	70 GB	−50%
KV cache VRAM available (on 4×H100)	180 GB	250 GB	+39%
Maximum concurrency (ctx 8K)	~22,500 tok	~31,250 tok	+39%

This is equivalent to an extra replica for free in terms of KV cache capacity.

RTX 4090 (Ada Lovelace, FP8 in CUDA but without dedicated tensor cores)

Metric	BF16/Q4 baseline	FP8 KV cache added	Delta
Decode throughput (tok/s, 14B Q4)	~45	~47	+4%
KV cache VRAM available	15 GB	15 GB (same model)	—
Total cache tokens (ctx 8K)	~46,000	~92,000	+100%
Maximum concurrency (ctx 8K)	~5 users	~11 users	+120%

On Ada, the compute benefit is smaller (the FP8 tensor cores do not have the same width as on Hopper), but the ×2 in KV cache capacity is entirely real and translates into twice the possible concurrent users.

The correct workflow: enable, measure, decide

Enabling FP8 directly in production without validating quality is inadequate. The correct workflow has four steps.

Step 1: baseline in staging

Before enabling FP8, record the quality metrics of the current BF16 model. The most reproducible way is to run an eval suite over a fixed dataset and save the results:

# Install lm-evaluation-harness
pip install lm-eval

# BF16 baseline
lm_eval --model vllm \
 --model_args pretrained=meta-llama/Meta-Llama-3.1-70B-Instruct,dtype=bfloat16 \
 --tasks mmlu,hellaswag,gsm8k \
 --num_fewshot 5 \
 --output_path ./results/baseline_bf16.json

Step 2: enable FP8 and run the same eval suite

# FP8
lm_eval --model vllm \
 --model_args pretrained=neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8,quantization=fp8,kv_cache_dtype=fp8,calculate_kv_scales=true \
 --tasks mmlu,hellaswag,gsm8k \
 --num_fewshot 5 \
 --output_path ./results/fp8_full.json

Step 3: compute the degradation

# compare_eval.py
import json

with open("results/baseline_bf16.json") as f:
 baseline = json.load(f)
with open("results/fp8_full.json") as f:
 fp8 = json.load(f)

tasks = ["mmlu", "hellaswag", "gsm8k"]
print(f"{'Task':<15} {'BF16':>8} {'FP8':>8} {'Delta':>8} {'OK?':>6}")
print("-" * 50)
for task in tasks:
 b = baseline["results"][task]["acc,none"]
 f = fp8["results"][task]["acc,none"]
 delta = (f - b) / b * 100
 ok = "✓" if abs(delta) < 1.0 else "✗ REVIEW"
 print(f"{task:<15} {b:>8.3f} {f:>8.3f} {delta:>+7.1f}% {ok:>6}")

Decision thresholds documented in MLPerf Inference 2025:

< 0.5% degradation: enable in production without restrictions.
0.5% – 1.5%: enable with active quality monitoring via LLM-as-judge.
> 1.5%: investigate before enabling — a possible calibration problem or an incompatible model.

Step 4: domain eval with LLM-as-judge

Academic benchmarks measure what they measure. Your use case may be different. Adding 200 representative samples from your domain, evaluated by an LLM judge, closes the gap:

# domain_eval.py
from langfuse import Langfuse
from openai import OpenAI

client = Langfuse()
judge = OpenAI(base_url="http://judge-llm:8000/v1", api_key="token")

# Load the 200 curated production samples (prompt + expected answer)
samples = load_domain_samples("eval_dataset_200.json")

scores_bf16, scores_fp8 = [], []
for sample in samples:
 for model_type, endpoint in [("bf16", "http://staging-bf16:8000"), ("fp8", "http://staging-fp8:8000")]:
 response = call_model(endpoint, sample["prompt"])
 score = judge.chat.completions.create(
 model="Qwen/Qwen2.5-72B-Instruct",
 messages=[{
 "role": "user",
 "content": f"Rate this answer from 1 to 5 on accuracy and completeness.\nQuestion: {sample['prompt']}\nExpected answer: {sample['expected']}\nModel answer: {response}\n\nReply with a number from 1 to 5 only."
 }]
 ).choices[0].message.content.strip()

 if model_type == "bf16":
 scores_bf16.append(int(score))
 else:
 scores_fp8.append(int(score))

import numpy as np
print(f"Mean BF16 score: {np.mean(scores_bf16):.2f}")
print(f"Mean FP8 score: {np.mean(scores_fp8):.2f}")
print(f"Degradation: {(np.mean(scores_fp8)-np.mean(scores_bf16))/np.mean(scores_bf16)*100:.1f}%")

OTel + Langfuse correlation: the dashboard that decides

The moment of decision rests on a single dashboard with two signals on the same time axis:

Signal 1 — Throughput (Prometheus):

rate(vllm:generation_tokens_total[5m])

Signal 2 — Mean quality (Langfuse → Prometheus via exporter):

# If you have configured Langfuse with scores exported via OTel
langfuse_score_value{name="llm_judge_domain"}

The expected pattern after enabling FP8: throughput rises by 40-60% and quality stays within ±0.1 points. If quality drops by more than 0.3 points and stays low, there is a real problem.

# Alert: quality drops more than 0.2 points, sustained, after the change
ALERT FP8CalidadDegradada
 IF avg_over_time(langfuse_score_value{name="llm_judge_domain"}[30m])
 < (avg_over_time(langfuse_score_value{name="llm_judge_domain"}[1d] offset 2h) - 0.2)
 FOR 15m
 LABELS { severity = "warning" }
 ANNOTATIONS { summary = "Possible quality degradation after the FP8 configuration change" }

When NOT to enable FP8

FP8 is not always the right answer. The cases where degradation exceeds the acceptable threshold:

Formal mathematical reasoning: GSM8K and MATH are the benchmarks most sensitive to FP8. If your use case is solving mathematical problems or precise financial calculation, measure specifically on these benchmarks before enabling it.

Critical code with tests: numerical precision affects the probability of tokens at key positions in a function. The risk is not that the code “looks” bad, but that it passes superficial tests while having subtle bugs.

Very long contexts without --calculate-kv-scales: without dynamic scale calibration, the accumulated numerical error in the KV cache grows with the context. With --calculate-kv-scales active, the impact is minimal up to 32K tokens.

Small models (<7B): the FP8 conversion overhead can exceed the throughput benefit. The break-even point is around 7B parameters.

References

Capacity planning for on-premise LLM inference: how to size GPUs from an SLO

Mon, 01 Jun 2026 15:00:00 +0200

This post complements the ones on KV cache (the piece that dominates the VRAM budget), Continuous batching (what defines the effective utilisation of the compute) and Seven layers of the stack (the pieces the sizing takes for granted). Before reading this one, make sure your team has written down the SLOs it is going to chase; without that input the calculation is not defensible.

TL;DR

LLM inference capacity planning does not answer “how many tokens per second does a GPU give” — that question has no universal answer, because throughput depends on concurrency, the prefill/decode split, context length, the inference engine and quantisation. The right question has three inputs (SLO: TTFT P95, TPOT P95, sustained RPS), a hardware reference (GPU model, VRAM, HBM bandwidth, effective FLOPs) and a model (parameters, GQA/MHA/MoE architecture, weight format). The calculation resolves into two coupled budgets that cross each other. VRAM budget: from the GPU’s total you subtract model weights and activations, what remains is the KV cache budget, and from there you derive the maximum concurrency possible at the average context you expect. Time budget: the engine (vLLM, SGLang, TensorRT-LLM) has a tokens/s ceiling in decode given by HBM bandwidth and another in prefill given by useful FLOPs; from there you derive the expected TPOT and, dividing prefill_tokens by prefill throughput, the expected TTFT. Both budgets must meet the SLO simultaneously: whichever is tighter dictates the sizing. On the Llama 70B BF16 example with tensor parallel 4 on 4×H100 SXM, a single replica saturates at ~28 concurrent requests and delivers ~3,200 tokens/s of aggregate decode with a median TPOT of 35 ms; for 200 sustained RPS at a profile of 800 prompt tokens + 250 of output, you need between 4 and 5 replicas with a 25 % cushion over the observed peak. Quantisation (FP8 → INT4) divides VRAM cost and decode time by between 1.5 and 4×, but it degrades quality measurably — it is not assumed free, it is validated with evals. The five habitual traps: confusing the mean with P95, ignoring the prefill/decode split of the real workload, sizing without head-room for retraining or rollback, forgetting that a GPU at 100 % SM util means nothing if the HBM is saturated, and not documenting the assumptions behind the calculation (a sizing with no written assumptions is a throwaway calculation).

You are here: DEPLOY (with one foot in OBSERVE)

Capacity planning is a piece with dual residency. It lives in DEPLOY because without a valid sizing you do not buy hardware or configure the inference engine. But its input is real observations: the distribution of prompt and output lengths, the prefill/decode mix of the workload, the real P95s already being seen in pre-production. Without that data the calculation is a napkin sketch, defensible only until the first customer arrives who does not fit the assumed average.

The analogy: the hotel with variable-sized rooms

Imagine a hotel where the rooms have no fixed size: each guest pays for the square metres they need, and the building’s floor plan reorganises itself dynamically to accommodate whoever arrives. Management wants to maximise occupancy, but it has two real constraints and one quality metric.

Constraint 1 — physical space. The floor has 1,000 m² in total. If a family comes in needing 200 m², that family occupies that surface and it cannot be handed to the next guest. The largest room limits how many simultaneous guests fit.

Constraint 2 — service staff. There are 10 receptionists. Each can handle the check-in of one guest every two minutes. When 60 guests arrive within an hour, the last ones wait in a queue; the time from walking into reception to receiving their key depends on how many are ahead of them.

Quality metric — a promise about time. The brochure says “check-in in under 15 minutes”. If too many guests arrive at once, that promise breaks even if there is physical space free.

The physical space is the GPU’s VRAM. Each room is a request with its KV cache (larger the longer the conversation). The receptionists are the compute units (Streaming Multiprocessors + Tensor Cores). The check-in is the prefill phase; the nights the guest spends afterwards are the decode steps. The 15-minute promise is the TTFT P95 SLO.

The hotel’s capacity planning is exactly this: given the expected guest profile (how many arrive per hour, how much space they ask for on average, how many minutes of waiting they tolerate), work out how many floors and how many receptionists are needed. It is not done by estimating “rooms per hour” in the abstract; it is done by crossing the two budgets with the promise about time. The analogy holds up the calculation all the way to the end.

The three SLO inputs

Before putting a single number on the sheet, you have to write down the three dimensions of the SLO. Without this, the calculation is aesthetics, not engineering.

TTFT P95 (Time-To-First-Token). The time from the client sending the request until it receives the first token. It is dominated by the prefill phase (processing the whole prompt in one go) plus the scheduler queue. For conversational chat, a reasonable target sits between 0.5 and 2 seconds P95. For coding assistants with large prompts (5–10 K tokens of context), between 2 and 4 s P95. Below 500 ms you enter UX territory for voice-style conversations, but that demands serious architectural compromises.

TPOT P95 (Time-Per-Output-Token). The time between consecutive tokens during decode. It dominates the “perceived fluency” of streaming. Above 80 ms/token the human reader perceives pauses; below 30 ms/token the output flows faster than it can be read. The usual industry target: 40–60 ms P95.

Sustained RPS meeting the SLO. The throughput the system must support without violating TTFT or TPOT. This is DistServe’s key metric, called goodput (see Continuous batching). “200 RPS peak” is not the same as “200 RPS with TTFT P95 ≤ 1.5 s”. Without the SLO condition, the RPS number means nothing.

These three dimensions come with a workload profile: the distribution of prompt and output lengths. Medians are not enough; you need P50, P95, P99. A badly measured profile is the main reason sizings fail.

The central formula: two budgets that cross

The calculation resolves into two independent sums that are then crossed. The smaller of the two rules.

VRAM budget

For a GPU with total VRAM $V$, the space available for KV cache is:

$$V_{\text{kv}} = V - V_{\text{model}} - V_{\text{activations}} - V_{\text{overhead}}$$

where:

$V_{\text{model}}$ is the size of the weights: for a model of $P$ parameters in a format of $b$ bytes/parameter, $V_{\text{model}} = P \cdot b$. Llama 70B BF16 = $70 \times 10^9 \times 2 = 140$ GB. Under tensor parallel TP=4, each GPU carries $140 / 4 = 35$ GB.
$V_{\text{activations}}$ are the intermediate buffers of the forward pass. For vLLM with a reasonable batch, between 2 and 6 GB per GPU depending on batch size and maximum length.
$V_{\text{overhead}}$ is the CUDA context, NCCL buffers, the PagedAttention pool, reserved paged blocks. 2–4 GB typical.

The KV cache budget per GPU is what is left over. For an H100 SXM 80 GB with Llama 70B TP=4 BF16:

$$V_{\text{kv}} = 80 - 35 - 4 - 3 = 38 \text{ GB per GPU} = 152 \text{ GB aggregated over TP=4}$$

The KV cache cost per token for a model with $L$ layers, $H_{\text{kv}}$ KV heads (GQA), per-head dimension $d_h$, in a format of $b$ bytes:

$$\text{kv per token} = 2 \cdot L \cdot H_{\text{kv}} \cdot d_h \cdot b$$

The factor of 2 is because both K and V are stored. For Llama 70B (L=80, $H_{\text{kv}}$=8 with GQA, $d_h$=128, BF16 = 2 bytes):

$$\text{kv per token} = 2 \cdot 80 \cdot 8 \cdot 128 \cdot 2 = 327,680 \text{ bytes} = 320 \text{ KB/token}$$

And the maximum concurrency at average context $C$:

$$N_{\text{max}} = \frac{V_{\text{kv}}}{C \cdot \text{kv per token}}$$

With an aggregated $V_{\text{kv}}$ of 152 GB and an average context of 1,500 tokens (800 prompt + 700 generated at the worst moment of the conversation):

$$N_{\text{max}} = \frac{152 \times 10^9}{1,500 \cdot 320 \times 10^3} \approx 316 \text{ concurrent requests}$$

This is the physical ceiling of concurrency for that replica. It is not what you are going to use; it is what you cannot exceed without OOM. The operational number sits well below it (head-room for spikes).

Time budget

Two sub-calculations come in here: decode (memory-bound) and prefill (compute-bound).

Decode TPOT. For every token generated, the model weights (the ones relevant to that request) have to be walked through and the accumulated KV cache has to be read. The bottleneck is HBM bandwidth. For a GPU with bandwidth $B$ GB/s and a model of $V_{\text{model per gpu}}$ GB of weights:

$$\text{tpot}_{\text{theoretical}} \approx \frac{V_{\text{model per gpu}}}{B}$$

For an H100 SXM with HBM3 at 3.35 TB/s and Llama 70B TP=4 BF16 (35 GB/GPU):

$$\text{tpot}_{\text{theoretical}} \approx \frac{35}{3,350} \approx 10.4 \text{ ms/token}$$

This is the best theoretical case with batch=1 and 100 % HBM efficiency. In practice vLLM on H100 with Llama 70B TP=4 reaches 12–18 ms/token at low batch and 30–45 ms/token at high batch (at concurrency 32, tokens compete for the shared HBM). The defensible operational number: 35 ms/token at concurrency 24–32.

Prefill throughput. Prefill processes N prompt tokens in a single forward pass. It is compute-bound: the bottleneck is FLOPs. For an H100 SXM with 989 sustained BF16 TFLOPs and Llama 70B (each forward pass costs roughly $2 \cdot P \cdot N$ FLOPs per sequence of length N):

$$\text{prefill tps} = \frac{4 \cdot \text{TFLOPs} \cdot \eta}{2 \cdot P} = \frac{4 \cdot 989 \times 10^{12} \cdot 0.5}{2 \cdot 70 \times 10^9} \approx 14,000 \text{ tokens/s}$$

(the factor of 4 is the GPUs in TP, $\eta$ is real efficiency between 0.4 and 0.6 on H100). An 800-token prompt takes this long in prefill:

$$\text{prefill time} = \frac{800}{14,000} \approx 57 \text{ ms}$$

Adding a typical queue of 100–300 ms at high concurrency, TTFT P95 ≈ 350–500 ms for that profile. Well below the 1.5 s target; there is margin.

The crossing

The real operational concurrency $N_{\text{op}}$ is the minimum of the VRAM ceiling, the concurrency at which TPOT starts to degrade past the SLO, and the concurrency at which TTFT starts to degrade past the SLO (prefill queue). For the example:

VRAM ceiling: 316.
TPOT degrades to 80 ms (SLO) at around concurrency ~80–100 (measured empirically with a benchmark, not a closed formula).
TTFT degrades to 1.5 s at around concurrency ~40–60 because of the prefill queue.

The replica’s operational concurrency is ~50. Applying 25 % head-room for spikes and rebalancing, target concurrency per replica ≈ 35–40.

Step-by-step worksheet: Llama 70B BF16 on 4×H100 SXM

The exercise’s input:

SLO: TTFT P95 ≤ 1.5 s; TPOT P95 ≤ 60 ms; 200 sustained RPS.
Workload: prompt P50=600, P95=1,200, P99=2,500; output P50=180, P95=500, P99=900. Average prompt 800, output 250.
Generic hardware: 4×H100 SXM 80 GB with NVLink, vLLM v1 engine, tensor parallel 4, BF16.

Step 1 — VRAM per GPU. Weights 35 GB, activations 4 GB, overhead 3 GB → KV budget 38 GB/GPU = 152 GB aggregated. KV/token for Llama 70B GQA = 320 KB. Ceiling of live tokens in cache: $152 \times 10^9 / 320 \times 10^3 \approx 475,000$ tokens. At the operational average context (800 prompt + 200 already generated = 1,000 live tokens per request), concurrency ceiling $\approx 475$.

Step 2 — average duration of a request. Prefill 800 tokens / 14,000 tps = 57 ms. Decode 250 tokens × 35 ms/token = 8,750 ms. Total $\approx 8.8$ s per request.

Step 3 — replica throughput. If the replica sustains operational concurrency 40 and each request lasts 8.8 s, the replica delivers roughly $40 / 8.8 \approx 4.5$ requests/s in steady state.

Step 4 — number of replicas. For a 200 RPS target: $200 / 4.5 \approx 45$ replicas. That is 45 × 4 = 180 GPUs. Too much: this sizing does not work because the cost per request is high.

Step 5 — review the levers. Before buying more hardware, there are three levers to explore, in this order:

Quantisation. Dropping to FP8 reduces weights to 17.5 GB/GPU (leaving more VRAM for KV cache → more concurrency), roughly doubles decode tokens/s (HBM saturated by half), and degrades MMLU quality typically by 0.5–1.5 points on models like Llama 70B. Rewriting the calculation in FP8: TPOT drops to ~18 ms, total time per request to 4.7 s, RPS per replica rises to ~8.5, replicas needed ≈ 24, equivalent to 96 GPUs.
Speculative decoding. With a small drafter and 60–70 % acceptance, effective TPOT falls 30–40 %. RPS per replica rises to ~12, replicas ≈ 17 = 68 GPUs.
Disaggregated serving. Separating prefill workers and decode workers lets each be scaled to the real mix of the workload (see Disaggregated serving). It usually shaves off another 20–40 % under asymmetric workloads.

Step 6 — recommended sizing. For the example, with FP8 + speculative decoding and 25 % head-room: 20 vLLM replicas at TP=4 over 80 H100 SXM. If the team does not want to depend on aggressive quantisation (pure BF16 for maximum fidelity), the calculation rises to 30 replicas = 120 GPUs and forces a renegotiation of the SLO or the budget.

Step 7 — write down the assumptions. This is the part no valid sizing skips. In the team’s repo, next to the calculation:

# sizing/llama70b-prod.yaml
fecha: 2026-06-01
slo:
 ttft_p95_ms: 1500
 tpot_p95_ms: 60
 rps_target: 200
workload:
 prompt_tokens_p50: 600
 prompt_tokens_p95: 1200
 output_tokens_p50: 180
 output_tokens_p95: 500
 asunto: chat productivo con RAG ligero
modelo:
 arquitectura: llama-70b-instruct
 formato_pesos: fp8
 motor: vllm-v1
hardware:
 gpu: H100-SXM-80GB
 topologia: TP=4 con NVLink intra-nodo
 red_inter_replica: 25 GbE
optimizaciones:
 - paged_attention
 - chunked_prefill
 - speculative_decoding (drafter llama-1.1b, aceptación esperada 65%)
asunciones_criticas:
 - utilizacion_hbm_eficiente: 0.55
 - head_room_pico_sobre_p95: 0.25
 - aceptacion_speculative_min: 0.55
plan_validacion:
 - benchmark vllm bench serve antes de procurement
 - canary 10% durante 7 días post-deploy

Without this YAML, the calculation is not reproducible a month later.

MoE case: Mixtral 8×22B (~141 B total, 39 B active)

MoEs change the calculation along one key dimension: the total weights are large but the active weights per token are small. For Mixtral 8×22B with top-2 routing:

Weight VRAM: $141 \times 2 = 282$ GB BF16. With TP=4 → 70 GB/GPU. It does not fit in an H100 80 GB alongside KV cache + activations. You need TP=8 (~35 GB/GPU) or FP8 with TP=4 (~35 GB/GPU).
Decode TPOT: dominated by the weights read per token, which are $\sim 39 / 8 \cdot 2 \approx 9.75$ GB/GPU with TP=4 (one top-2 expert per token, divided across 4 GPUs). On an H100 with HBM at 3.35 TB/s, theoretical TPOT ≈ 3 ms/token. In practice, 10–20 ms at reasonable concurrency.
Prefill: similar to the dense model of the active weights, ~39 B FLOPs/token.

MoE sizing usually delivers more RPS per GPU than an equivalent dense model: the low cost per token compensates for the extra VRAM. See MoE inference for the detail of routing and why a high batch is decisive so that each expert sees enough tokens.

Sensitivity table: context and quantisation

For Llama 70B on 4×H100 SXM (TP=4), operational concurrency per replica with an SLO of TTFT 1.5 s / TPOT 60 ms:

Average context	BF16	FP8	INT4 (AWQ)
500 tokens	55	110	180
1,000 tokens	40	80	130
2,000 tokens	24	50	85
4,000 tokens	12	26	48
8,000 tokens	6	13	25

Approximate numbers from public vLLM benchmarks as of June 2026, with ±20 % variation depending on engine version and the headroom adopted. To validate on your own hardware: vllm bench serve with your profile of real prompts.

The five habitual traps

Trap 1 — confusing the mean with P95. Average throughput over an hour may be 50 RPS while the 5-minute peak reaches 180 RPS. Sizing against the mean guarantees breaking the SLO at every peak. Rule: size against the hourly P95, with 20–30 % head-room over P95.

Trap 2 — not measuring the real prefill/decode split. A “RAG with short answers” workload spends 70–80 % of GPU time in prefill; a “writing assistant generating essays” spends 80 % in decode. The useful optimisations (chunked prefill vs speculative decoding) change radically. Without measuring it, you buy badly balanced hardware.

Trap 3 — sizing with no head-room for retraining or rollback. The production cluster is not just the inference engine: there is a re-embedding batch when the embeddings model changes, continuous canary evals (see Canary, blue-green and shadow), light fine-tuning, hot stand-by for rollback. Reserve 15–25 % of capacity for those non-negotiable workloads.

Trap 4 — “GPU at 100 % SM utilization” as a target. SM occupancy of 95 % with saturated HBM produces the same throughput as SM at 60 % with saturated HBM. The bottleneck in decode is the HBM. Optimising for “GPU usage 100 %” without looking at HBM utilisation and arithmetic intensity makes you spend more on GPU without gaining throughput. See GPU observability for LLM inference for which metrics to actually watch.

Trap 5 — not documenting the assumptions. A sizing with no reproducible YAML (workload, model, engine, head-room, critical assumptions) leaves the team with no way of knowing what changed when the cluster stops meeting the SLO six months later. Documenting is cheap; losing a quarter to debugging is not.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB with intra-node NVLink and 25 GbE between nodes, the recurring configurations in May 2026 are:

Model	Format	TP	Replicas that fit	Typical RPS per node (ctx 1K)
Llama 8B	BF16	1	4 (one per GPU)	240–320
Llama 8B	FP8	1	4	450–600
Llama 70B	BF16	4	1	30–45
Llama 70B	FP8	4	1	60–90
Llama 70B	INT4 AWQ	2	2	90–130
Mixtral 8×22B	FP8	4	1	90–140
Qwen 72B	BF16	4	1	28–42

These numbers are orders of magnitude to start the conversation, not commitments. The definitive sizing is validated with vllm bench serve or genai-perf (NVIDIA) using the customer’s real prompt/output profile. The prefill/decode asymmetry of each case’s workload can move these numbers 30–50 % up or down.

For clusters of 8×H100 SXM (typical of DGX servers or equivalent replicas), the options open up to TP=8 for 405B-class models or multi-replica TP=2 for 70B models with greater density. The metric that decides is always the same: tokens meeting SLO per kW and per euro of amortised hardware.

How the sizing is validated before buying

The spreadsheet sizing is the first half. The second is the validation benchmark.

Stage 1 — back-of-the-envelope sizing. This post’s formulas applied to the SLO and the expected workload. Output: an approximate number of replicas and a topology.

Stage 2 — synthetic micro-benchmark. On a borrowed GPU or one rented by the day, bring up the engine with the chosen model and run vllm bench serve with prompts of representative lengths. Validate TPOT, prefill TPS and the concurrency ceiling. Calibrate the HBM efficiency factor ($\eta$) used in the formulas.

Stage 3 — load test with realistic traffic. Generate traffic following the real distribution of the customer’s workload (not Poisson, not constant: the real trace). Measure P50/P95/P99 of TTFT, TPOT, throughput. Confirm the head-room.

Stage 4 — canary in production. With the cluster sized, route 5–10 % of real traffic for 7–14 days before closing the procurement of additional hardware. See Canary, blue-green and shadow for the mechanics.

Jumping from Stage 1 to full procurement is the most frequent cause of a cluster that is 40 % oversized and 60 % undersized at the same time, in different regions of the workload. Four weeks of validation done well save four months of refactoring.

What we have not covered (upcoming articles)

The observability metrics that close the sizing loop in production, see GPU observability for LLM inference.
Autoscaling that adjusts replicas to the real traffic curve, see Autoscaling LLM on Kubernetes.
Detailed cost accounting per tenant (showback / chargeback) over the sized hardware.
Sizing for continuous fine-tuning (PEFT and light training) that shares a cluster with inference.

References

Kwon et al. — vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (SOSP 2023).
Zhong et al. — DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024).
Agrawal et al. — Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI 2024).
NVIDIA — H100 Tensor Core GPU Architecture Whitepaper (HBM3 memory, bandwidth, sustained FLOPs).
vLLM project — vllm bench serve reference (benchmarking CLI included in the repo).
NVIDIA — genai-perf (official tool for benchmarking LLM services).

FlashAttention v1/v2/v3/v4: the librarian who never clears the desk — IO-awareness, async and Blackwell's asymmetry

Fri, 29 May 2026 17:00:00 +0200

This post complements KV cache: the working memory and PagedAttention deep dive. The KV cache post explains what is stored; PagedAttention, how it is managed in memory; FlashAttention, how the computation is executed. They are three different layers of the same problem and they stack multiplicatively.

You are here: DEPLOY

TL;DR

The standard attention of a transformer has a structural problem on modern GPUs: when you look at it with a profiler, it is not compute-bound, it is memory-bound. The S = QK^T matrix of size N × N, with N the sequence length, does not fit in the fast on-chip SRAM and forces round trips to HBM that dominate the total time. FlashAttention is the family of kernels that avoids materialising that matrix by tiling over Q, K and V, computing the softmax block by block with the online version and keeping everything inside SRAM. Each version raises the utilisation ceiling: FA1 (Dao et al. 2022) abolished the N×N matrix and brought memory complexity down to O(N); FA2 (Dao 2023) parallelised along the sequence dimension and cut non-matmul FLOPs to get close to 70 % utilisation on A100; FA3 (Shah, Bikshandi, Zhang, Thakkar, Ramani, Dao 2024) exploited three Hopper-specific mechanisms, async WGMMA, TMA and FP8, to reach 740 TFLOPS BF16 (75 % of peak) and 1.2 PFLOPS FP8 on H100; FA4 (March 2026) rewrote the kernel from scratch for Blackwell, where the tensor core scales 2.25× but the SFU (where the softmax exp runs) and the SMEM bandwidth do not scale at all. The answer is a software-emulated exponential that runs on tensor cores. The result on B200: 1,605 TFLOPS BF16, 1.3× faster than cuDNN 9.13 and 2× faster than FA3 on the same GPU. This post takes apart the why (memory roofline, IO complexity), the master analogy of the librarian, the minimum mathematics and the real numbers on H100 and B200.

The analogy: the librarian who never clears the desk

A large library with two zones: a very fast but small work desk (the librarian’s desk, 200 books fit on top of it), and a giant shelving system spanning three floors where everything lives (50 million books). The librarian has to cross-reference all the books in one thematic room and produce a summary.

A naive librarian does it the direct way: fetches every relevant book from the shelves, piles them on the work desk, and since they do not fit, leaves half of them on the floor, on the chair, on top of boxes. The day is spent running between the floor and the desk, opening and closing books, never able to concentrate. The work desk is extremely fast but underused, because most of the time the librarian is moving books between the floor and the desk. That is what standard attention does: it materialises the S = QK^T matrix in HBM and keeps coming back for it piece by piece.

A FlashAttention v1 librarian changes strategy: asks for one shelf at a time, brings it to the desk, reads what is needed, jots notes in a compact notebook (“from this shelf I care about this, this and this, with these relative weights”), returns the shelf to its place and brings the next one. The notebook is the only thing that carries over between shelves. Nothing is ever piled higher than the desk can hold. The trick that makes this possible is the online softmax: instead of needing all the content at once in order to normalise, it keeps a running max and a running sum that are updated shelf by shelf.

A FlashAttention v2 librarian realises that several topics can be worked in parallel, because the desk is big and the shelves are independent along some axes. Three assistants are put to work, each with a notebook, each covering a different block of the room, and the results are combined at the end.

A FlashAttention v3 librarian manages to mechanise the flow: a conveyor belt with two stations is installed. While station A reads the current shelf and takes notes, station B already has the next shelf in transit from the shelving system. When A finishes, B hands over the new shelf with no waiting. This is the ping-pong producer/consumer pattern: the TMA offloads the data loading and the consumer warps do the work in parallel with the loads. On top of that, the notes are written in lower-precision shorthand (FP8), because the pages that matter have already had an orthogonal pre-treatment that stops them losing precision where it hurts.

A FlashAttention v4 librarian discovers something new about the remodelled library (Blackwell): two much faster conveyor belts have been installed (tensor cores 2.25×), but the shorthand typewriter has not been upgraded (the SFU is unchanged). Now the bottleneck is writing the notes, not fetching the shelves. The fix is elegant: instead of using the shorthand machine, the notes are written with polynomial formulas that the tensor core itself can evaluate (software-emulated exponential). The belt no longer sits idle waiting for the machine.

The analogy holds up with four mappings:

Work desk = SRAM per SM (228 KB on H100, 256 KB on B200).
Giant shelving system = HBM (3.35 TB/s on H100, 8 TB/s on B200).
Notebook with running max and sum = online softmax stats (m, ℓ).
Conveyor belt with two stations = TMA + WGMMA producer/consumer pipeline.

Why standard attention was the bottleneck

The naive intuition, “attention is matmuls, and GPUs are good at matmuls”, is correct but incomplete. There are two matmuls (QK^T and then softmax(S) V), and in between a non-matmul operation (softmax) that requires materialising the intermediate matrix S of size N × N.

Modern GPUs have a very concrete roofline. For H100 SXM5:

Compute peak: 989 TFLOPS BF16 (tensor core, dense, without sparsity).
Memory bandwidth: 3.35 TB/s HBM3.
Break-even point (arithmetic intensity break-even): 989 × 10¹² / (3.35 × 10¹²) ≈ 295 FLOP/byte.

Any operation that fails to perform 295 operations for every byte it moves from HBM is memory-bound: the tensor core sits waiting for data. On B200 the ratio is similar (~281 FLOP/byte) because both compute and bandwidth went up.

Standard attention materialising S reads the matrix twice (once for the softmax, once to multiply by V) and writes it once. For Llama 3 70B with head dim d = 128 and context N = 128K:

Matrix S per head per layer: N × N × 2 bytes = 128K × 128K × 2 = 34.36 GB.
80 layers × 64 Q-heads → aggregate HBM traffic (if it were materialised serially) on the order of TBs, prohibitive in transient.
Even if it is not all materialised at once, the round trips dominate the time: effective arithmetic intensity well below 295 → memory-bound operation → tensor core underused at around 25 %.

FlashAttention does not change the mathematics of attention, it changes the order of the operations so that S is never materialised in HBM. Its IO complexity is Θ(N²·d²/M) with M = SRAM size per SM, against Θ(N·d + N²) for standard attention. With d = 128 and M = 228 KB: an approximate reduction factor of M/d² ≈ 14× less HBM traffic. That is what moves the operation from memory-bound to nearly compute-bound.

The trick that made it all possible: online softmax

Without online softmax there is no FlashAttention. The idea comes from Milakov and Gimelshein, 2018 (the paper “Online normalizer calculation for softmax”, arXiv:1805.02867), and it allows softmax([x_1, ..., x_N]) to be computed in one incremental pass without needing to know the global maximum before starting.

The standard softmax is:

$$\text{softmax}(x_i) = \frac{e^{x_i - m}}{\sum_{j} e^{x_j - m}}, \quad m = \max_j x_j$$

The online trick: keep a running maximum m^{(t)} and a running sum ℓ^{(t)}. When a new block of values arrives with local maximum m_{\text{new}}:

$$m^{(t+1)} = \max(m^{(t)}, m_{\text{new}})$$ $$\ell^{(t+1)} = e^{m^{(t)} - m^{(t+1)}} \cdot \ell^{(t)} + \sum_{j \in \text{new}} e^{x_j - m^{(t+1)}}$$

And the accumulated partial outputs are rescaled by the same factor e^{m^{(t)} - m^{(t+1)}}. At the end, dividing by the final ℓ gives exactly the same result as the standard softmax. It is mathematically exact, not an approximation.

This is what allows K to be traversed block by block without materialising the whole S matrix. Each block updates the stats and the accumulated outputs, and is then discarded. The desk never fills up.

The four versions (May 2026)

	FA1 (2022)	FA2 (2023)	FA3 (2024)	FA4 (2026)
Target GPU	A100 / Ampere	A100 / H100	H100 / Hopper	B200 / Blackwell
Core idea	Tiling + online softmax	Sequence parallelism + work partitioning	Async WGMMA + TMA + FP8	Polynomial exp + 2-CTA tensor cores
Memory	O(N) (vs O(N²))	same	same	same
Typical peak util	~25 % A100	~70 % A100, ~35 % H100	75 % H100 BF16, 60 % H100 FP8	71 % B200 BF16
Effective TFLOPS	—	225 TFLOPS A100 BF16	740 H100 BF16, 1,200 H100 FP8	1,605 B200 BF16
Speedup vs previous	2-4× standard	2× FA1	1.5-2× FA2 (BF16), 2.6× (FP8)	2× FA3 on B200
Paper	arXiv:2205.14135	arXiv:2307.08691	arXiv:2407.08608	arXiv:2603.05451

FA1 — changing the order of the operations

Three ideas combined: tiling of Q/K/V into blocks that fit in SRAM, online softmax over those blocks, and recomputation in the backward pass. The N×N matrix S is not stored, only the stats (m, ℓ), and in the backward pass S is recomputed block by block from Q, K and the stats. Result: 7.6× speedup on GPT-2 versus PyTorch standard attention, memory O(N).

FA2 — parallelising along the sequence

FA1 parallelised only over batch × heads. With a small batch (1-4) and models with few heads or aggressive GQA, the GPU was left with idle SMs. FA2 also parallelises along the sequence dimension: different SMs process different stretches of Q at the same time. It also rewrites the algorithm to minimise the non-matmul operations (softmax rescaling), because those do not go through tensor cores. And it improves work partitioning between warps (split-Q instead of split-K reduces shared memory traffic). Result: around 2× over FA1 on H100 and A100, 225 TFLOPS on A100 (72 % MFU). On H100 it stays around 30-35 % of BF16 peak because it does not exploit async WGMMA.

FA3 — the Hopper moment

This is where FlashAttention stops being an algorithm and becomes a Hopper-specific artefact. Three pillars:

Async WGMMA: Hopper’s new tensor core instructions let a warpgroup fire a GEMM and have the rest of the warpgroup do something else (the softmax, for instance) while the tensor core keeps working. This is the trick that unlocks matmul/softmax overlap.
TMA (Tensor Memory Accelerator): dedicated hardware for copying tiles between HBM and SRAM. It frees the SM from computing indices and predicating out-of-bounds accesses, which used to consume SM cycles. It is the equivalent of hiring warehouse staff: the librarian no longer has to carry the books personally.
FP8 with block quantisation + incoherent processing: quantising Q and K to FP8 doubles tensor core throughput. The precision loss is mitigated with two tricks: a per-tile scale (64×d) instead of a whole-tensor scale, and a pre-multiplication by a random orthogonal Hadamard-based matrix that “spreads” the outliers before quantising. Documented result: numerical error 2.6× lower than the FP8 baseline.

These three pillars combine with producer/consumer warp specialisation (producer warps do TMA loads; consumer warps do WGMMA + softmax) and ping-pong scheduling with two warpgroups taking turns so that there are never pipeline bubbles. When WG1 does softmax, WG2 does GEMM; then they swap.

Numbers: 740 TFLOPS BF16 on H100 (75 % of the 989 peak), 1.2 PFLOPS FP8 (60 % of the 1978 FP8 dense peak). For sequences ≥ 1K it beats cuDNN. Speedup over FA2: 1.5-2× BF16, 2.6× FP8.

FA4 — Blackwell’s asymmetry

Blackwell scaled everything unevenly:

Tensor core BF16 throughput: 1 PFLOP H100 → 2.25 PFLOPS B200 (2.25×).
SFU count (where the softmax exp runs): unchanged.
Shared memory bandwidth: unchanged.

In other words, if FA3 runs as-is on a B200 with no changes, the matmul goes twice as fast but the softmax stays exactly the same, and that stalls the pipeline. It was only a matter of time before someone fixed the imbalance.

FA4 (March 2026, same team of Dao + Princeton + Together AI + Meta + NVIDIA + Colfax) is a ground-up rewrite with three ideas:

Software-emulated exponential: a polynomial approximation of exp that runs on the tensor core instead of on the SFU. It loses a little precision (carefully bounded and compensated by the rest of the kernel) but keeps the conveyor belt moving.
Conditional softmax rescaling: avoids rescaling accumulators when the running max does not change significantly. A “lazy” style optimisation: it only pays the cost when the cost is needed.
2-CTA tensor core: two CTAs (Cooperative Thread Arrays) cooperate to feed the tensor cores with larger tiles. It gets more out of Blackwell’s new capabilities.

Written in CuTeDSL (NVIDIA’s CUTLASS Python DSL, not CUDA C++ directly). Result on B200 BF16: 1,605 TFLOPS (71 % of the 2250 peak). 1.3× over cuDNN 9.13. 2.7× over Triton. 2× over FA3 run as-is on B200 (which was the previous baseline). It is the first attention kernel to pass 1 PFLOPS.

Note: there is recurring confusion about “FP4 attention”. Blackwell’s NVFP4/MXFP4 extensions apply to weights, not to attention. FA4 can be combined with NVFP4 weights, but the attention computation itself is still BF16 or FP8 depending on the configuration. FP4 quantisation of QK^T does exist in some proprietary kernels (Fireworks AI’s FireAttention V4 combines it) but it is not standard practice.

Implementations and libraries in 2026

Dao-AILab/flash-attention (the canonical repo): supports SM 8.0 (Ampere) with FA2, SM 9.0 (Hopper) with FA3, SM 10.0 (Blackwell datacenter B100/B200/B300) with FA4. The consumer Blackwell version (5090, SM 12.0) has partial support as this post goes out.
FlashInfer (flashinfer-ai/flashinfer, arXiv:2501.01005): an attention engine aimed at serving (not training). Its conceptual contribution is the Block-Sparse Row (BSR), a unified abstraction covering paged KV cache, the radix tree of prefix caching and the tree masks of speculative decoding. Internally it can call FA2/FA3, cuDNN, CUTLASS or trtllm-gen FMHA kernels depending on the case. It JIT-compiles specific variants at runtime. Integrated into vLLM, SGLang and TensorRT-LLM.
vLLM (May 2026): automatic backend selection by GPU. Default FA4 on SM 10.0+, FA3 on SM 9.0, FA2 on the rest. Fallbacks on Blackwell: TRT-LLM Ragged → FlashInfer → TokenSpeed MLA. For FP8 KV cache on B200, FlashInfer is competitive.
SGLang: uses FlashInfer as its attention backend; RadixAttention is the prefix caching layer on top (a radix tree over the KV cache).
TensorRT-LLM: its own fused kernels (trtllm-gen FMHA). XQA is NVIDIA’s own optimisation for GQA in decode.
PyTorch SDPA and FlexAttention: torch.nn.functional.scaled_dot_product_attention selects the backend automatically. FlexAttention (new) lets you define custom masks declaratively and compiles to kernels that can use FA4 as a backend.
xFormers: still alive but residual. The built-in PyTorch SDPA covers most cases.

Cases where FlashAttention does not help

Very short contexts (N < 512): the tiling and kernel launch overhead does not pay off; cuDNN can win.
Non-standard custom masks: FA ships causal, sliding window and ALiBi only. Arbitrary masks need FlexAttention or the JIT variants of FlashInfer.
Non-standard head dim: FA optimises for d = 64, 128, 256. Odd dimensions (d = 96, d = 192) fall into slow paths.
GQA/MQA with extreme ratios: natively supported, but the speedup versus pure MHA depends on the Q-heads : KV-heads ratio.
Cross-attention: supported but less optimised; self-attention is where the gain is largest.
FP8 without block quantisation or incoherent processing: it loses several points in benchmarks. If your serving framework does not implement the two tricks from the FA3 paper, FP8 attention can be a bad idea.

Implications on on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace, SM 8.9)

The 4090 is Ada Lovelace, not Hopper. It runs neither FA3 nor FA4; it runs FA2. That means around 70 % utilisation in BF16 attention (~250 effective TFLOPS against the 4090’s 330 TFLOPS BF16 peak). It is not a disaster, FA2 is already very good compared with standard attention, but the ceiling is clearly below that of an H100. For consumer deploys on a 4090 with Llama 3 8B BF16 or any 14B-32B INT4 AWQ, FA2 is what you will be using, and it is perfectly reasonable.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

This is where FA3 shines and it is what vLLM/SGLang/TRT-LLM will select by default. Two common configurations:

Llama 3 70B FP8 with FA3 FP8 attention: 1.2 PFLOPS peak on the GPU, aggregate cluster throughput on the order of 8,000-12,000 tokens/s at medium batch depending on TP and context. For FP8 attention to deliver its full performance it is crucial to use the block quantisation + incoherent processing techniques from the FA3 paper (they are enabled by default in vLLM).
DeepSeek-V3 671B FP8 + MLA with FlashInfer: DeepSeek uses Multi-head Latent Attention (MLA), a different variant of standard attention. FlashInfer has specific kernels (FlashMLA). The typical stack is vLLM/SGLang + FlashInfer + FlashMLA + an FA3 fallback for the non-MLA layers.

If the infrastructure is Blackwell (B200/B300, which some clusters start receiving in 2026), FA4 is the right option and it should be enabled by default in vLLM 0.16+ and SGLang 0.5.11+.

What we have not covered

MLA (Multi-head Latent Attention) from DeepSeek and the specific FlashMLA kernels: they optimise KV cache compression but require different kernels.
Flexible masking and the FlexAttention use cases (PyTorch 2.5+): how to declare arbitrary masks without paying the cost of a custom kernel.
Hardware assistance for sparse attention (NVIDIA 2:4 sparse tensor cores) and why sparse attention has not established itself as a higher ceiling than dense FA.
FA in the backward pass of fine-tuning: this post focuses on inference, but FA3/FA4 also go through the backward pass and they are what makes training models with long contexts viable on H100/B200.

References

Dao, T., Fu, D., Ermon, S., Rudra, A., Ré, C. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 2022. https://arxiv.org/abs/2205.14135
Dao, T. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. 2023. https://arxiv.org/abs/2307.08691
Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., Dao, T. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision. NeurIPS 2024. https://arxiv.org/abs/2407.08608
Dao, T. et al. FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling. 2026. https://arxiv.org/abs/2603.05451
Milakov, M., Gimelshein, N. Online normalizer calculation for softmax. 2018. https://arxiv.org/abs/1805.02867
Ye, Z. et al. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving. MLSys 2025. https://arxiv.org/abs/2501.01005
Tri Dao FA3 blog: https://tridao.me/blog/2024/flash3/
Tri Dao FA4 blog: https://tridao.me/blog/2026/flash4/
PyTorch FlashAttention-3 announcement: https://pytorch.org/blog/flashattention-3/
PyTorch FlexAttention + FA4: https://pytorch.org/blog/flexattention-flashattention-4-fast-and-flexible/
Together AI FA4 blog: https://www.together.ai/blog/flashattention-4
Colfax Research FA3: https://research.colfax-intl.com/flashattention-3-fast-and-accurate-attention-with-asynchrony-and-low-precision/
Colfax Research FA4: https://research.colfax-intl.com/flashattention-4-algorithm-and-kernel-pipelining-co-design-for-asymmetric-hardware-scaling/
Repo Dao-AILab/flash-attention: https://github.com/Dao-AILab/flash-attention
Repo flashinfer-ai/flashinfer: https://github.com/flashinfer-ai/flashinfer
vLLM attention backends: https://docs.vllm.ai/en/latest/design/attention_backends/
NVIDIA Hopper Architecture in Depth: https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/

The GPU cluster as a platform: turning a shared cluster into a multi-tenant service your teams can consume

Thu, 21 May 2026 07:15:00 +0200

TL;DR

Having an expensive GPU cluster and many different workloads that want to use it is not an infrastructure problem: it is an internal product problem. What separates “we have a cluster” from “we have an inference platform” are four layers the market consolidated in 2026: a gateway layer that centralises authentication, routing and policies (LiteLLM, Portkey, Kong AI Gateway); a GPU isolation model appropriate to the tenants’ profile (MIG hardware isolation for untrusted multi-tenant, MPS for processes from the same team, time-slicing only for dev); a quota and rate limiting system with budgets per tenant/team/project (LiteLLM does this in its core at team/user/api-key level with descriptive 429s); and a multi-tenant observability plane that enables real cost attribution (showback as an intermediate step, chargeback as the destination), per-tenant tracing and differentiated dashboards. Applied to a typical mid-scale GPU cluster (one node with 4-8 H100 SXM and NVLink, a common place to start in production), this translates into concrete decisions: with ~640 GB of aggregate VRAM across 8 GPUs and two typical production models (a large 70B+ model with tensor parallel and a replicated mid-sized model), the cluster serves between tens and low hundreds of simultaneous sessions depending on the mix; GPU isolation is usually resolved with MIG on smaller workloads and per-model dedication on large ones; and the platform’s success metric is effective utilisation, which in typical production sits at 30-40% and where the reasonable optimisation target is to push it to 60-70% without degrading SLAs.

This is the fifth post in the MLOps for LLMs series. It is the most operationally oriented one and cuts across several pipeline stages (Deploy + Observe plus cross-cutting concerns). The “you are here” marker flags the two active stages because the notion of a multi-tenant platform does not live in just one.

You are here: Deploy + Observe (the cluster as a product)

The question that changes the framing

When a platform team buys expensive GPU hardware and starts building inference, the first version is almost always single-purpose: one model, one client, one target latency. It works. When the second team turns up asking for the same resource, that single purpose becomes internal politics: how many replicas do we give them? What do we do if the SLAs clash? Who pays for the tokens of team B’s experiment? And when the third one arrives, what was an SRE project becomes an internal product project.

The distinction is not technical, it is one of framing. A cluster is infrastructure. A platform is a service with customers, contracts and success metrics. The change of framing implies:

Identifiable customers (tenants), not anonymous users.
Contracts (latency SLA, guaranteed throughput, available models), not “whatever we manage”.
Success metrics that are not technical but product ones: adoption, satisfaction, cost per query per tenant, time to the first “hello world”.

This post walks through how that change of framing is carried out. It lands it on a mid-scale cluster (4-8 H100 SXM with NVLink in a single node), the usual configuration when starting with serious LLM inference; but the principles generalise to any topology, from a single node with two GPUs to multi-node clusters with InfiniBand.

The four layers of a multi-tenant inference platform

The canonical architecture that settled in 2026 has four layers that any serious multi-tenant platform implements, from the outside in:

Each layer solves a concrete problem. Let us take them one by one.

Layer 1 — AI Gateway: the single front door

The AI Gateway is the component your tenants see. It is an HTTP/gRPC API compatible with OpenAI (typically /v1/chat/completions, /v1/embeddings, /v1/models) that centralises everything that happens before touching the inference backends.

Why centralise

Without a gateway, tenants connect directly to vLLM or whatever the model is. Every change (rotating an endpoint, adding a model, changing credentials, applying a policy) means notifying every tenant. Every tenant has its own retry logic, its own logging, its own auth model. It is unworkable from the third customer onwards.

With a gateway, the change is made in one place. Tenants have a stable URL and a set of credentials; the rest is the gateway’s problem.

The three dominant options in 2026

LiteLLM is the most popular OSS option, Python-first, deployed as a proxy. It supports 100+ providers (OpenAI, Anthropic, Bedrock, self-hosted vLLM, Ollama, and so on) behind a unified OpenAI-compatible API. It has a native multi-tenant hierarchy with Organizations → Teams → Users → API Keys, each level with its own budget. The Apache 2.0 version covers the basics; RBAC, SSO, audit logs and team-level enforcement require the paid Enterprise version. Deployed on K8s with the official Helm chart.

Portkey is the most mature commercial / SaaS option. A single control plane that enforces budgets, quotas, permissions and compliance. Real-time spending tracking with alerting. RBAC, audit, workspaces and SSO included. The trade-off: dependency on an external service and a per-request pricing model.

Kong AI Gateway is the option for organisations that already have Kong as their API gateway. An AI plug-in on top of the existing Kong gateway, integrated with its plugin, consumer and rate-limit model. If your platform team already runs Kong, it is the lowest friction.

When to choose each one

Situation	Gateway
Pure OSS, self-hosted, Python-first team	LiteLLM
You need RBAC, SSO, audit log out of the box, and have budget	Portkey
You already run Kong as a corporate API gateway	Kong AI Gateway
Greenfield enterprise with strict compliance	Portkey (probably)
Mid-sized OSS-first company without regulated compliance	LiteLLM (typically)

The minimum the gateway has to do

Whichever option you pick, this is what any serious deployment must enforce:

Auth and identity: every request carries an API key resolvable to a tenant + user + team.
Routing by model: the tenant asks for model: "gpt-4o"; the gateway decides whether it goes to OpenAI, to Azure OpenAI, or to your vLLM with Qwen3 32B (a cheaper fallback), according to policy.
Rate limiting: RPS per tenant, TPM (tokens per minute), concurrency limits.
Caching of identical responses: 5-30% of RAG queries are repeats; caching saves latency and cost.
OTel emission: every call produces a span with gen_ai.* semantic conventions and tenant_id as an attribute. Covered in the Evals post and MCP observability.
Failover: if vLLM goes down, the gateway redirects to the OpenAI API. If OpenAI rate-limits, the gateway falls back to Anthropic. Configurable policy.

Example of a multi-tenant LiteLLM configuration

# litellm-config.yaml — simplified example
model_list:
 - model_name: llama-3-70b
 litellm_params:
 model: openai/llama-3-70b
 api_base: http://vllm-llama3-70b.inference/v1
 api_key: os.environ/VLLM_API_KEY

 - model_name: qwen3-32b
 litellm_params:
 model: openai/qwen3-32b
 api_base: http://vllm-qwen3-32b.inference/v1
 api_key: os.environ/VLLM_API_KEY

 - model_name: gpt-4o
 litellm_params:
 model: openai/gpt-4o
 api_key: os.environ/OPENAI_API_KEY

router_settings:
 routing_strategy: usage-based-routing-v2
 fallbacks:
 - llama-3-70b: [qwen3-32b, gpt-4o]  # if vLLM goes down, fall back to the external one

general_settings:
 master_key: os.environ/LITELLM_MASTER_KEY
 database_url: os.environ/DATABASE_URL  # Postgres for budgets/keys

# Hierarchy: Organizations → Teams → Users → API Keys
# Created via the API, not in static YAML

Creating a team with a budget:

curl -X POST http://litellm/team/new \
 -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
 -d '{
 "team_alias": "soporte-chat",
 "max_budget": 500, # 500 USD/month
 "budget_duration": "30d",
 "tpm_limit": 100000, # 100K tokens/min
 "rpm_limit": 1000, # 1000 requests/min
 "models": ["llama-3-70b", "qwen3-32b"] # access to these
 }'

And the team’s API key:

curl -X POST http://litellm/key/generate \
 -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
 -d '{
 "team_id": "<team-id>",
 "duration": "30d",
 "metadata": {"environment": "production", "app": "support-bot"}
 }'

That API key is what the tenant uses. Every request made with it consumes from the team’s budget. When it runs out, LiteLLM returns HTTP 429 with a description.

Layer 2 — Policy & Quota Plane: what each tenant can do

The gateway is where enforcement happens. Policy is what gets enforced. Five axes of multi-tenant policy:

Technical quotas

TPM (tokens per minute): the hard consumption limit. For a Llama 3 70B at TP=5, ~3000 sustained output tokens/s = 180K aggregate TPM. If you have 10 tenants, assign 18K each as the ceiling.
RPS / RPM: load control, not consumption control. A 4K-token session counts as one request; so does a batch of 100 mini-completions. Useful against abuse.
Concurrency: how many simultaneous active requests per tenant. Important for latency SLAs: 100 RPS with concurrency=50 works out at 2 seconds per request.

Economic budgets

Monthly per tenant: a hard cap in USD.
Daily and hourly: soft caps to avoid a runaway in a single day.
Per project / API key: fine granularity within a single tenant.

LiteLLM has a max_budget field at every level of the hierarchy (organization, team, user, api key). Budgets are inherited and constrained downwards.

Model whitelist and blacklist

Tenants with critical workloads → only stable models (llama-3-70b, gpt-4o). Research tenants → access to experimental models as well.

Priority classes

Not all requests are equal. Three typical classes:

Guaranteed: workloads with an SLA, latency respected even under pressure.
Best-effort: normal workloads without a strict SLA.
Spot: batches that can wait, evictable if a guaranteed one arrives.

The Token Management in Multi-Tenant AI Inference Platforms paper (2026) formalises this with a model of token pools per priority class that has started to be adopted in production. It maintains guaranteed P99 latency for guaranteed workloads even under overload, with selective throttling of spot.

Admission control

Before accepting a request: is there capacity? If not, return 429 immediately instead of queueing and degrading everyone. It is the most underrated operational discipline. A cluster with admission control done properly has predictable latency; without it, catastrophic degradation when the peak arrives.

The typical pattern in 2026

# Conceptual policy for a "soporte-chat" tenant
tenant: soporte-chat
quotas:
 tpm: 50000
 rpm: 500
 max_concurrency: 30
budget:
 monthly_usd: 800
 alert_thresholds: [0.5, 0.8, 0.95] # warn when you get there
models_allowed:
 - llama-3-70b
 - qwen3-32b
priority: guaranteed
fallback_on_overload:
 - qwen3-32b  # if guaranteed fills up, fall back
 - gpt-4o-mini  # last resort, external model

Layer 3 — Isolation Plane: isolating workloads physically

This is the technically densest layer. You have a node with several H100 SXM GPUs interconnected by NVLink. How do you partition them between tenants?

MIG (Multi-Instance GPU) is the strongest isolation. It partitions the GPU into up to 7 instances with physically separate HBM memory and dedicated compute units (SMs). Tenants in different MIG instances cannot touch each other: one workload does not consume memory another needs, one does not degrade another’s throughput. Hardware isolation. Available on A100, H100, B100 and B200.

MPS (Multi-Process Service) is soft. Several processes share the GPU concurrently and NVIDIA distributes SMs according to usage. Good performance if all the processes are yours and you trust them. Worse for multi-tenant setups between customers who do not know each other, because a noisy process can degrade the others.

Time-slicing is the simplest: the GPU is assigned alternately, slot by slot, to different processes. Much worse latency (waits between slots); not recommended for production workloads with an SLA.

The choice for multi-tenant in 2026

According to the enterprise adoption survey: 80% use MIG for untrusted multi-tenant (different customers who do not know each other) and MPS for trusted environments (processes from the same team) where you want to maximise throughput. Time-slicing is only used in dev/staging so that every developer can touch a GPU without the cost of exclusivity.

An important limitation of MIG: it isolates compute and HBM memory, but the PCIe path remains shared. For PCIe-bound workloads (a lot of host↔device traffic), tenants in different MIG instances can still affect each other. For LLM inference the main path is HBM, so this is rarely a problem. But it is worth knowing.

MIG partitions on the H100

An H100 (80GB HBM3) can be partitioned into fixed profiles:

Profile	SM	Memory	Max instances per GPU
1g.10gb	14	10 GB	7
1g.20gb	14	20 GB	4
2g.20gb	28	20 GB	3
3g.40gb	42	40 GB	2
7g.80gb	98	80 GB	1 (the whole GPU)

For a mid-scale cluster with NVLink, MIG has a fundamental problem: when you partition with MIG, NVLink between GPUs is disabled. An H100 in MIG mode does not take part in multi-GPU tensor parallel. If you are going to serve a large model with tensor parallel (Llama 3 70B with TP=4 or TP=8, for instance), those GPUs have to be whole, without MIG.

That defines the architectural decision. There are two main approaches:

Approach A — A shared large model with quotas at the gateway

All the node’s GPUs serve a single large model with tensor parallel spanning the whole node. All tenants share that instance. Isolation happens at the gateway layer (quotas, rate limiting) and the policy layer (priority classes). The cluster’s kernel is one enormous vLLM instance with --max-num-seqs=128 or similar; internally, vLLM shares GPU time between the active requests with continuous batching.

Advantages: you use all the GPUs to the full, NVLink stays active, better KV cache utilisation. Disadvantages: soft isolation. A tenant that saturates does not degrade the others directly (vLLM batches), but it does compete for batch slots. You need serious priority classes.

Approach B — Dedicating GPUs per model / tenant

You split the GPUs into pools dedicated to different models. Examples on an 8-GPU node:

4 GPUs: a large 70B model with TP=4.
2 GPUs: a mid-sized 32B model replicated (2 independent instances) for tenants with a strict SLA.
2 GPUs: miscellaneous workloads (smaller models, experimentation).

Advantages: physical isolation between critical models / tenants. Disadvantages: worse aggregate utilisation; some GPUs idle while others saturate.

Approach C (advanced) — MIG on some GPUs plus dedication for the rest

If you have small workloads (4B, 7B models), you can use MIG on 1-2 GPUs to serve them and dedicate the rest to tensor parallel for the large model. It combines strong isolation for the small workloads with full use of NVLink for the large model.

The operational choice: start with A, move up to C if you need to

In most deployments, Approach A (a shared large model plus quotas) is the right starting point. Utilisation is better, operation is simpler, and the gateway’s soft isolation works for reasonable workloads.

When there is a tenant with a strict SLA that cannot tolerate competing with others, you move to Approach B for that tenant in particular (dedicating GPUs to an instance of the model just for them), keeping the rest of the cluster shared.

Approach C is for when you have 10+ tenants with very heterogeneous profiles.

Isolation at the Kubernetes level

Independently of GPU isolation, pod-level isolation is applied in K8s:

Namespaces per tenant: tenant-soporte, tenant-legal, and so on.
ResourceQuotas and LimitRanges: CPU/memory limits per namespace.
NetworkPolicies: tenant A cannot talk to tenant B’s namespaces.
K8s PriorityClasses: classes with a numeric value defining preemption order if a more critical pod arrives.
PodDisruptionBudgets: how many pods of each deployment can go down simultaneously.

Layer 4 — Observability Plane: seeing what happens per tenant

The fourth layer: observability with a tenant dimension. Without it you cannot do cost attribution, you cannot debug incidents for a single tenant, you cannot show dashboards to stakeholders.

The four mandatory properties

1. tenant_id on every span. The AI gateway resolves the API key and attributes a tenant_id. That ID is propagated via params._meta or OTel headers to every downstream component (vLLM, retrieval, MCP servers, tools). Any span in any system carries that label. It is what lets you reconstruct tenant-specific traces.

2. Metrics labelled per tenant. gen_ai.usage.input_tokens{tenant="soporte-chat"} or equivalents. Prometheus, Grafana, groupable by tenant.

3. Real cost attribution. The sum of tokens × cost/token per tenant gives the cost. For self-hosted vLLM, the cost is per GPU hour plus the proportional share of tokens (you can compute an equivalent cost per 1k tokens).

4. An immutable audit log. Every API key used, every model invoked, every quota change, every budget exceeded. For compliance.

Showback vs chargeback

An important FinOps distinction that gained clarity in 2026:

Showback: visibility without consequence. “Support team, you have consumed 623 USD this month on LLMs”. Information, not an invoice. It lets you spot abuse without penalising anyone before the team understands what is going on.

Chargeback: the cost is charged to the team’s budget. When it runs out, it runs out. It changes behaviour.

The practice that works: 6-18 months on showback while tags are calibrated, misattributions are identified and teams are trained. Then chargeback, once the numbers are credible. Launching chargeback on day 1 when the costs are still dirty creates an immediate political fight; launching showback prepares the ground so that chargeback lands in an orderly way.

Only 14% of organisations have chargeback active according to a recent survey, which indicates this is still mostly showback in real production.

Tools

Kubecost: cost allocation per namespace, deployment and pod on Kubernetes. For the cost of the shared GPU, allocate proportionally to the tokens consumed per tenant.
Finout: a FinOps platform that combines cloud bills and LLM API costs in a unified view with virtual tagging.
Langfuse: already covered. Cost tracking per trace, groupable by user or session metadata.
LiteLLM native tracking: LiteLLM’s master DB keeps running spend per team, user and API key, accessible via the API or the UI.

The minimum multi-tenant dashboard

Any platform should have:

Summary per tenant: monthly spend, current RPS, TPM consumed, % of budget spent, active sessions.
Top users within each tenant (for detecting internal abuse).
p95 latency per tenant: SLA tracking.
429 / 503 errors: how many requests are being rate-limited or rejected because of overload.
Cost trend: the monthly trajectory with a projection.
Drift per tenant (from the post-tracing series): if a tenant starts getting worse results, raise an alert.

Sizing on mid-scale GPU clusters: concrete decisions

Let us get down to hardware. As a reference we take a node with N H100 SXM (between 4 and 8) with NVLink/NVSwitch, 80 GB HBM3 each. That gives between 320 GB and 640 GB of aggregate VRAM. Inter-GPU connectivity is 900 GB/s (NVLink 4) or 600 GB/s (NVLink 3) depending on generation. HBM bandwidth per GPU is 3.35 TB/s.

Default decisions

Start with Approach A: all the node’s GPUs serving a single large 70B model in BF16 with tensor parallel = N. Expected real capacity (computed for a standard 8-GPU HGX node as an example; it scales roughly linearly with N):

Model VRAM (70B BF16): ~140 GB (≈ 17.5 GB/GPU at TP=8).
vLLM overhead + activations VRAM: ~10 GB/GPU.
VRAM free for KV cache: ~52 GB/GPU. On an 8-GPU node that is ~416 GB aggregate; on a 4-GPU one, ~210 GB.
With --kv-cache-dtype=fp8 and a 70B GQA model: ~320 KB/token.
Aggregate cache capacity (8-GPU node): ~1.3M tokens to distribute between simultaneous sessions.

That translates into throughput and concurrency (indicative figures for an 8-GPU node):

Simultaneous sessions	Average context per session	Aggregate throughput (tokens/s)
32	16K	~5000
64	8K	~8000
128	4K	~12000

Typical latencies: TTFT ~150ms at low traffic, TPOT ~15-20 ms/tok. Under high concurrency, TTFT rises to ~500ms if the queue is saturated.

An example tenant scheme

A cluster with 4 tenants and a research pool:

Tenant	TPM cap	RPM cap	Concurrency	Budget	Priority	Models
Support chat	80K	800	50	1500 USD/month	Guaranteed	llama-3-70b, qwen3-32b
Legal RAG	30K	200	15	600 USD/month	Guaranteed	llama-3-70b
Code agent	50K	300	25	1200 USD/month	Best-effort	llama-3-70b, qwen-coder
Data extr. batch	40K	1000	40	400 USD/month	Spot	llama-3-70b, qwen3-32b
Research / notebooks	10K	100	5	200 USD/month	Spot	all

TPM total: 210K. Aggregate cluster capacity: ~180K sustained TPM. It is overcommitted by about 15%, assuming not every tenant hits the ceiling at the same time. That is normal and desirable; if they all do it at once, the priority classes degrade in an orderly way.

When to add hardware

Signals that the node has become too small:

Sustained p95 TTFT > 500 ms during peak hours → the queue is building up.
vllm:num_requests_waiting constantly > 20 → admission control starting to reject.
Sustained GPU utilisation > 80% at critical hours without dropping in quiet hours → there is no headroom.
429 rate on guaranteed tenants > 1% → the platform is breaking its SLA in production.

When several of these hold, the natural next step is to add another HGX node with internal NVLink and stand up a second vLLM instance of the same model. The gateway load-balances between the two instances. Aggregate throughput doubles; latency stays the same.

Common operational traps

A gateway without auth: a backdoor into the cluster

Your vLLM sits on a ClusterIP Service and the main app talks to it. Some tenant discovers the endpoint directly and hits it without going through the gateway. Quotas and costs are silently evaded. Use a strict NetworkPolicy: only the gateway can talk to the vLLM Services; the rest of the cluster cannot.

MIG and NVLink are incompatible

You enable MIG on a GPU thinking you will get isolation plus multi-GPU; you discover that MIG disables NVLink. Any large model with TP becomes unusable. Decide MIG vs NVLink globally per cluster, not per individual GPU.

Quotas pinned to the cluster’s ceiling

You add up the TPM of all the tenants and it comes to exactly the cluster’s capacity. When two tenants peak at the same time, both wait or one gets rejected. An overcommit of 10-20% is healthy (it assumes they do not all peak at once); more than that is dangerous.

No multi-tenant observability from day 1

You launch with quotas and isolation but without tenant_id on spans. Three months later, your CFO asks “how much does the support agent cost versus the legal one?” and you cannot answer. OTel with a mandatory tenant_id from the first version, even if there are no dashboards yet; having the data is worth more than having perfect dashboards with no data.

Showback that never reaches chargeback

You have been on showback for 18 months, the teams know the numbers, nobody changes behaviour. Without the pressure of real chargeback, the incentive dilutes. Set an explicit calendar for the transition to chargeback, with an owner and a deadline.

Non-whitelisted models eating the budget

A team discovers that LiteLLM has gpt-4o configured. They use it without permission. The budget burns on an external API when the idea was to use the cheap self-hosted one. Keep an explicit per-team whitelist of accessible models.

Badly calibrated priority classes

Everyone declares themselves “guaranteed”. At the first peak, there is nothing left to degrade and everything suffers. Priority classes only for critical cases, with justification. The majority should be best-effort.

No failover from the gateway

Your vLLM goes down. The gateway has no fallback configured and returns 503 to every tenant. Configure a fallback to another model, ideally an external one (OpenAI) for guaranteed workloads, even if it costs more per hour. Availability is worth more than cost per hour.

An operational roadmap for getting started

If you start from scratch with an empty GPU node, the minimum order is as follows. Each milestone is a day’s work with slack, not a tight one:

Day 1-2 — Base K8s infrastructure. NVIDIA GPU Operator + nvidia-device-plugin + dcgm-exporter + cluster-default NetworkPolicies. Validation: a basic pod with nvidia.com/gpu: 1 gets scheduled.

Day 3 — vLLM with a large model and tensor parallel across the whole node. The vLLM Production Stack Helm chart (or bare vLLM manifests). Model weights on a shared PVC (CephFS or NFS). Validation: a curl request against the internal Service responds.

Day 4 — AI Gateway: LiteLLM. Helm chart, Postgres for budgets, master key, a first model_list pointing at vLLM. Validation: an OpenAI-compatible request through LiteLLM responds with the same content as vLLM directly.

Day 5 — Basic multi-tenancy. Create teams, API keys, budgets, model whitelists. Test with two teams. Validation: the second team, using the model it does not have whitelisted, gets a 403.

Day 6 — Minimum observability. Prometheus + Grafana scraping vLLM and LiteLLM. A dashboard with TTFT, TPOT, throughput, num_requests_waiting and budget_consumed_per_team. Validation: visible in Grafana with real data.

Day 7-8 — Pilot customer. One real tenant (ideally a controlled internal one) starts using it. Measure real latencies, discover the first operational incidents.

Day 9-10 — Tuning. Adjust --max-num-seqs, --gpu-memory-utilization, priority classes and quotas based on what the pilot taught you.

Day 11-14 — Onboarding the second tenant plus iteration. Repeat. Every new tenant onboarded reveals new cases.

After two weeks you have an operational platform with two real tenants and data to decide whether it is ready for more. The line of progress from here on is horizontal (more tenants) until you saturate; from then on, vertical (more hardware).

What we have not covered (upcoming posts)

Continuous fine-tuning in production (post 6, already decided): LoRA/QLoRA/DPO, dataset curation, eval gates, A/B versioning with real traffic between model versions.
Constitutional AI and alignment at runtime: an option still on the table.
Edge LLMs: when an H100 cluster is too expensive for a specific workload, distilled models running on NPUs or consumer GPUs.
GPU networking deep dive: NCCL, InfiniBand, GPUDirect, RDMA. For multi-node clusters with cross-host tensor parallel.

References

Multi-tenancy and GPU isolation:

Multitenant GPU Infrastructure: 4 Powerful Design Rules — survey of enterprise patterns.
Run Multiple LLMs on One GPU: MIG, Time-Slicing, and MPS Guide (Spheron).
A Practical Guide to GPU Partitioning with MIG (Medium).
GPU Partitioning for AI Workloads: NVIDIA MIG with SUSE Virtualization (KubeCon EU 2026).
Predictable LLM Serving on GPU Clusters (arxiv 2508.20274).
Token Management in Multi-Tenant AI Inference Platforms (arxiv 2603.00356) — paper on priority + admission control.

AI Gateways:

Multi-tenant FinOps:

Cross-references:

Previous posts in series 4: MLOps for LLMs overview, RAG over Kafka, The six-stage pipeline, PostgreSQL + Qdrant.
Relevant posts from the inference series: vLLM on Kubernetes — the multi-GPU HGX node scenario we develop here. LLM operators on K8s — the vLLM Production Stack and OME that the gateway can direct.
Observability: Evals, MCP observability, eBPF + drift.

H100 on lo0 — Tech Blog

Full TCO of an on-premise GPU cluster: from capex to the all-in €/GPU-hour and the break-even against cloud

TL;DR

The model: declared assumptions

Capex breakdown per 4×H100 SXM5 node

GPUs

Server / HGX baseboard

InfiniBand NDR network

NVMe storage

Rack, PDU and datacenter connectivity

Capex summary per 4×H100 SXM5 node

Opex breakdown per 4×H100 SXM5 node (annual)

Energy

Staff / operations

Maintenance and support

Depreciation (for accounting purposes)

Annual opex summary per 4×H100 SXM5 node (base scenario, 8-node cluster)

Deriving the all-in €/GPU-hour

Formula

€/GPU-hour table by utilisation and scenario

From €/GPU-hour to €/1M tokens

Break-even on-prem vs cloud

The break-even formula

Break-even table by cloud mode and on-prem scenario

Reading the break-even table

Sensitivity analysis

TCO vs utilisation

TCO vs energy price

TCO vs PUE

TCO vs amortisation years

Sensitivity heat map (€/GPU-hour at 70 % utilisation, base scenario)

Decision table: cost/control/sovereignty Pareto

Integration with the series FinOps model

Sources

Sharing one GPU between several workloads: time-slicing, MPS and MIG

TL;DR

The analogy: a shared hob, one kitchen with several cooks, several kitchens

Why share: the operational problem

Time-slicing: compute turns, shared larder

The VRAM budget in time-slicing (the calculation that avoids the OOM)

MPS: coordinated cooks on the same worktop

MIG: concrete walls

The H100 80GB profiles

The calculation: 7×1g.10gb against 1×7g.80gb

The decision tree

Applied to the generic 4×H100 cluster

What we have not covered

See also

References

The shared bench: NVLink, NVSwitch and NCCL, the cable every token crosses in tensor parallel

TL;DR

Where you are: the floor below the engine

The analogy: four mechanics and a single bench

The mechanism: what an all-reduce really is and why there are 160 per token

How the all-reduce is done: ring, tree, NVLS

The maths that matter: why decode and prefill stress the cable the other way round

The hardware: NVLink 4 and NVSwitch on the HGX baseboard

The 10 knobs worth touching

Knob 1 — NCCL_DEBUG + topology dump: see what is happening before touching anything

Knob 2 — NCCL_ALGO: ring vs tree vs NVLS

Knob 3 — NCCL_PROTO: LL / LL128 / Simple

Knob 4 — NCCL_NVLS_ENABLE: offload the summation to the NVSwitch

Knob 5 — NCCL_MIN_NCHANNELS / NCCL_MAX_NCHANNELS: how many SMs communication steals

Knob 6 — NCCL_BUFFSIZE: the buffer size per channel

Knob 7 — NCCL_P2P_LEVEL / NCCL_P2P_DISABLE: guarantee P2P over NVLink

Knob 8 — GPUDirect RDMA for multi-node: NCCL_NET_GDR_LEVEL

Knob 9 — NCCL_IB_HCA / NCCL_SOCKET_IFNAME: pin the right NIC

Knob 10 — Driver: persistence mode, clocks and NVLink error counters

Summary table

How it connects with the rest of the stack

Traps and things that are not what they look like

Conclusion

See also

References

FP8 end to end: enable it, measure quality and decide with data

TL;DR

The analogy

The three layers of FP8 in vLLM

The measurable impact by hardware

H100 SXM (Hopper, native FP8 tensor cores)

Knob 1 — `NCCL_DEBUG` + topology dump: see what is happening before touching anything

Knob 2 — `NCCL_ALGO`: ring vs tree vs NVLS

Knob 3 — `NCCL_PROTO`: LL / LL128 / Simple

Knob 4 — `NCCL_NVLS_ENABLE`: offload the summation to the NVSwitch

Knob 5 — `NCCL_MIN_NCHANNELS` / `NCCL_MAX_NCHANNELS`: how many SMs communication steals

Knob 6 — `NCCL_BUFFSIZE`: the buffer size per channel

Knob 7 — `NCCL_P2P_LEVEL` / `NCCL_P2P_DISABLE`: guarantee P2P over NVLink

Knob 8 — GPUDirect RDMA for multi-node: `NCCL_NET_GDR_LEVEL`

Knob 9 — `NCCL_IB_HCA` / `NCCL_SOCKET_IFNAME`: pin the right NIC