Slo on lo0 — Tech Blog

GuideLLM in depth: validating the SLO under load and sizing from the knee

Sun, 14 Jun 2026 04:30:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Third article of the benchmarking track (B3). B2 covered the tool catalogue; here we go into GuideLLM in depth, because it answers the operational question that sizes a platform: how far can I load this engine without breaking the SLO? It is not a catalogue benchmark (“X tokens/s”), it is an SLO-driven sweep that finds the knee, the safe capacity, and from which the real replica count and cost per token come out. We will look at its load modes, how the SLO is defined, how to read the output and how to turn the knee into sizing and into euros. With real commands; no recommendations, only the mechanics.

Why GuideLLM

GuideLLM (a vLLM project) generates realistic, configurable traffic patterns and captures full distributions of TTFT, ITL and end-to-end behaviour, for SLO-driven evaluation, with reproducible sweeps that identify the safe operating range (Red Hat). Unlike a micro-benchmark, it measures the engine, not the client (multi-process load generation), and unlike MLPerf, it sizes your specific case (your model, your SLO, your load). It is the middle tool: not for tuning a flag, not for comparing vendors, but for deciding how many GPUs you need.

The load modes (rate-types)

GuideLLM offers several modes through --rate-type, and picking the right one is half the work:

`--rate-type`	What it does	When
synchronous	one request at a time	baseline latency, no concurrency
concurrent	keeps N fixed simultaneous requests	measuring at a specific concurrency
poisson	requests per second following Poisson	simulating real traffic (random arrivals)
throughput	saturates the engine to the maximum	raw peak capacity
sweep	automatic ramp from idle to maximum	finding the knee (the most useful one)

In concurrent mode, --rate "1,2,4" launches three benchmarks at those concurrencies (Medium · GuideLLM on OpenShift). The poisson mode is the most realistic for online inference (requests do not arrive at regular intervals), and the sweep is the one that automates the search for the saturation point.

The automatic sweep: from idle to the knee

The flagship mode. When running a sweep profile, GuideLLM automatically increases the request rate from idle to maximum throughput across 10 rounds, and produces an interactive HTML report with detailed latency and throughput (Red Hat · GuideLLM on Kubernetes). During the sweep, on detecting saturation it identifies the previous iteration, the knee, and returns it as estimated capacity; if it does not detect saturation, the sweep has to be extended beyond the knee.

A worked example: reading a 10-round sweep

An illustrative output from a sweep on a 70B on 8×H100 (SLO: TTFT P99 < 500 ms), per round:

Round	Rate (req/s)	TTFT P99 (ms)	ITL P50 (ms)	Throughput (tok/s)	Goodput (tok/s)
1	2	95	19	480	480
3	8	140	20	1,900	1,900
5	14	240	22	3,100	3,060
6	17	460	24	3,400	3,330
7	20	760	31	3,700	2,520
9	26	1,500	48	3,950	900
10	30	2,400	71	4,000	380

How to read it: up to round 6 (17 req/s) the TTFT P99 meets the SLO (460 ms) and goodput ≈ throughput (3,330 ≈ 3,400). In round 7 P99 already breaks (760 ms) and goodput drops to 2,520. The knee sits between 6 and 7: the safe capacity is that of round 6, ~3,330 tok/s of goodput. Rounds 9 and 10 give more raw throughput (3,950, 4,000) but with goodput of 900 and 380; the system is “performing” by mostly serving requests that miss the target. Whoever reports “4,000 tok/s” is describing round 10, where the system is broken. The defensible number is 3,330 tok/s under TTFT P99 < 500 ms, and that is the one that goes into the sizing. Note as well that the sweep had to reach round 10 (P99 of 2,400 ms) to see where it broke: that is why it has to be extended beyond the knee.

Defining the SLO: the number that decides the knee

The “knee” is not absolute: it depends on the SLO you define. A typical online inference SLO:

Metric	Example threshold	What it protects
TTFT P99	< 500 ms	the wait for the first token (interactivity)
TPOT / ITL P95	< 50 ms/token	the perceived “typing speed”
Error rate	< 0.1 %	reliability

Goodput, the throughput that meets that SLO, is the metric that defines the knee: the safe capacity is the maximum load where goodput ≈ throughput. Changing the SLO moves the knee: a stricter TTFT SLO (200 ms) gives an earlier knee (less safe capacity) than a loose one (1 s). That is why the SLO is fixed before the sweep, and reported alongside the result: a capacity “of 3,330 tok/s” without saying under which SLO means nothing.

Execution: the command

An SLO-driven sweep against a vLLM endpoint:

guidellm benchmark \
 --target "http://vllm:8000" \
 --rate-type sweep \
 --max-seconds 120 \
 --data "prompt_tokens=1024,output_tokens=256" \
 --output-path resultados.json

Key parameters: --rate-type sweep (the automatic ramp), --data (the length distribution of the load, which must resemble your real traffic), --max-seconds (duration per round), --target (the endpoint). For the SLO, GuideLLM allows the latency constraints that define goodput to be set. The output goes to --output-path in JSON, YAML or CSV, in addition to the interactive HTML report.

Defining the load: `--data` decides the result

The --data parameter (the distribution of prompt and output lengths) changes the knee as much as the SLO does. Three forms, from least to most faithful:

Load	`--data`	Fidelity
Fixed length	`prompt_tokens=1024,output_tokens=256`	low: real traffic is not fixed
Synthetic distribution	length ranges	medium
Real traces	a dataset of your traffic	high: the knee that applies

The trap: a sweep with short fixed-length prompts gives an optimistic knee that does not resemble production, where long prompts dominate the prefill cost and bring the knee forward. To size properly, feed GuideLLM the real distribution of your traffic (prompt and output lengths measured in production). The knee of a sweep is only as realistic as the load you put into it; with an unrealistic --data, you are sizing for traffic that does not exist.

The arrival profile matters too: --rate-type poisson simulates random arrivals (like real online traffic), while concurrent keeps N fixed (like a batch). An online service measured with concurrent can give a knee different from the real one; for interactive inference, poisson is the honest profile.

The output: what to read

GuideLLM produces standardised, exportable reports for dashboards, analysis and regression tracking, in JSON, YAML and CSV (Red Hat). What matters to read, per sweep round:

TTFT (P50, P95, P99) — the latency to the first token.
ITL/TPOT (P50, P95, P99) — between tokens.
Throughput (req/s and tok/s) — the raw figure.
Goodput — the part that meets the SLO (the honest number).

The interactive HTML report lets you see the full distributions (not just means), which is where the latency tail that a mean hides becomes visible. For sizing and comparison, the JSON is what gets versioned and fed into the reproducible harness.

From the knee to sizing and cost

This is where GuideLLM connects with the rest of the series. The replica count and the cost per token come out of the knee:

The calculation: if the knee gives a goodput of 3,330 tok/s per replica and your peak target load is 20,000 tok/s, you need 6 replicas (20,000 ÷ 3,330 ≈ 6.0). And the cost per token comes from the replica cost (from OpenCost, ~11 €/h) divided by its goodput: ~0.92 €/1M tokens. It is the bridge to capacity planning and to cost per token: the correct denominator is the goodput at the knee, not catalogue throughput. Sizing with raw throughput leaves you short of useful capacity and breaks the SLO in production.

Deployment as a Kubernetes Job

GuideLLM runs as a Kubernetes Job inside the cluster to benchmark models served by the orchestration platform (Red Hat). This matters for fidelity: the load generator runs inside the cluster, on the same network as the endpoint, so it measures the engine without the network latency of an external client. The Job is parameterised with the target, the rate-type and the load, and dumps the JSON to a volume or a store for versioning. Running it as a Job also makes automation easier (a CronJob for periodic benchmarking, or triggered from CI).

SLO validation in CI: catching regressions

The most valuable use in the medium term: a Job that runs a short sweep on every change (a vLLM release, a config change) and compares the goodput against the baseline. If goodput drops by more than a threshold or the knee moves earlier, the pipeline fails. Since GuideLLM exports standardised JSON for regression tracking, comparing two runs is trivial. That way a performance regression, which is a cost and capacity regression, is caught at the commit, not in production. The full sweep is not needed on every commit: a short sweep covering the knee is enough to catch the regression; the exhaustive one is for releases.

Comparing configurations with sweeps

GuideLLM shines for deciding between configurations by running the same sweep against each one. The fair protocol: same load (--data), same SLO, same hardware, varying only the config. Examples of decisions that a sweep settles with data:

Decision	What to compare	What the knee reveals
FP16 vs FP8	two deployments of the same model	FP8 usually gives a higher knee (more goodput)
`max-num-seqs`	different values of the flag	the one that maximises goodput under the SLO
vLLM vs SGLang	two engines, same model	which engine gives more goodput at your SLO
KV size/precision	KV cache configs	the effect on capacity

Each sweep gives a knee; the highest knee under the same SLO wins. It is the controlled experiment that fills the scorecard row (article B8): not “X is faster” in the abstract, but “X gives Y tok/s more goodput under this specific SLO, which translates into Z fewer replicas and W € less per million tokens”.

Watching the three axes during the sweep

A trick that connects with the cost and energy track: while the sweep runs, capture the GPU metrics with DCGM. That way each round yields the goodput (GuideLLM), the average power (DCGM → J/token) and the cost (node price → €/token) at once. In a single sweep campaign you measure the three axes of the scorecard for every operating point, and they come out coherent by construction (same instant, same load). Instead of an isolated performance benchmark, you get the complete row, cost, performance, energy, for the knee, which is exactly what the proposal needs. Align the time windows (the DCGM power and the GuideLLM metrics must cover the same interval) or the J/token does not correspond to the measured goodput.

Interpreting the distributions, not just the knee

The knee is the headline, but the GuideLLM HTML report gives full distributions, and there is information there that the knee summarises away. Three additional readings:

The shape of the tail. Two configs with the same P99 can have different tails: one with P99 at 500 ms and P99.9 at 600 ms is stable; another with P99 at 500 ms and P99.9 at 3,000 ms has a long tail that will hit some users severely. The mean, and even P99, hide it; the distribution shows it.
The TTFT/ITL split. Seeing the two distributions separately tells you whether the bottleneck is prefill (high TTFT) or decode (high ITL), which steers the optimisation (more concurrency vs chunked prefill, and so on).
The spread across rounds. If goodput varies a lot between rounds at the same rate, the system is unstable under load, something a single capacity number does not reveal.

The knee sizes; the distributions diagnose. For a proposal, the knee is the datum; for operating and optimising, the distributions are where you see what to fix.

GuideLLM in the reproducible harness

GuideLLM fits as the load engine of the reproducible harness (article S4). The pattern:

A script/Job that deploys the engine with the pinned config, warms up, runs the sweep and dumps the JSON with all the metadata (model, version, hardware, --data, SLO).
Named by date and config to version the runs.
A store (a git repo of JSONs or a bucket) to reproduce and compare.
DCGM capturing in parallel to add the energy to each point.

That way “reproduce the knee of this config” is one command, not an afternoon, and comparing two releases is a JSON diff. It is what turns benchmarking from a one-off activity into a continuous capability, and what lets the capacity figure in the proposal come with the bench to reproduce it.

The cost of a sweep (in euros)

A 10-round sweep at ~2 minutes per round occupies the node for ~20–30 minutes. At an amortised cost of ~11 €/h, that is ~4–5.5 € per full sweep of a model. Not much per run, but a continuous benchmarking programme (every release, every config, several models) adds up; hence the value of automating the sweep as a reproducible Job and running short sweeps in CI (only around the knee), reserving the exhaustive one for releases. The cost of measuring is part of the cost of the platform, and small next to the cost of sizing badly.

The SLO by use case: the knee moves with it

Since the SLO defines the knee, different use cases give different capacities on the same hardware. Worth keeping in mind when sizing:

Use case	Typical SLO	Effect on the knee
Interactive chat	TTFT P99 < 500 ms, ITL < 50 ms	early knee (latency rules)
Code copilot	TTFT P99 < 300 ms	even earlier knee
Batch / summarisation	no TTFT SLO, maximise throughput	late knee (almost pure throughput)
Agent (multi-step)	bounded per-step latency	depends on the number of steps

The same GPU serves more batch load than interactive chat load, because the batch SLO is loose. That is why sizing requires one sweep per load profile: there is no “the node’s capacity”, there is “the node’s capacity for this SLO”. Mixing loads with different SLOs in the same pool without separating them is a classic source of broken SLOs, with batch saturating and interactive chat paying for it. The fix connects with scheduling: separate pools or priorities by SLO.

GuideLLM vs the others: when to use which

To place it against the tools from B2:

Question	Tool	Why
“How far can I load without breaking the SLO?”	GuideLLM	SLO-driven sweep, the knee
“What is the endpoint’s maximum capacity?”	AIPerf	automatic saturation detection
“Does my vLLM flag change help?”	vllm bench serve	fast, native to the engine
“Which hardware/engine is better in the abstract?”	read MLPerf	cross-vendor comparability

GuideLLM is the tool for sizing under an SLO: when the question is operational (how many replicas, what cost per token at my SLO), it is the most direct answer. AIPerf and GuideLLM overlap (both multi-process); the practical difference is that GuideLLM is more oriented to the SLO and the reproducible report, and AIPerf to maximum capacity with automatic detection. Many teams use both; what must not be done is to compare a result from one with a result from the other as if they were the same measurement.

Checklist for a defensible sweep

For the knee of a sweep to support a sizing decision:

Step	Check
Realistic load	`--data` with your traffic’s distribution (or poisson)
Declared SLO	TTFT/TPOT with a fixed percentile and threshold
Extended sweep	reaches the point where P99 breaks (past the knee)
Goodput, not throughput	capacity is the goodput under the SLO
Job in the cluster	no network latency from an external client
Versioned output	JSON with all the metadata
DCGM in parallel	energy per point, optional but recommended

If you can tick the seven boxes, the knee is an auditable datum; if any is missing, it is an anecdote. The sizing in the proposal hangs on these seven being green.

Limits and traps (data-driven)

Not extending the sweep beyond the knee. If saturation is not detected, the estimated capacity is the last round tested, not the real knee. Extend it until P99 breaks.
Unrealistic load. A fixed-length --data does not resemble your traffic; use the real distribution (or poisson) for a knee that applies.
Reporting throughput, not goodput. The knee is defined by goodput under the SLO; raw throughput overestimates capacity.
Undeclared SLO. A capacity without the SLO it was measured under is neither comparable nor defensible.
Client outside the cluster. Running GuideLLM from outside puts network latency into the measurement; run it as a Job inside the cluster.

With GuideLLM mastered, you have the tool that turns performance into an actionable datum: the knee, the sizing and the cost per token, all from a reproducible sweep. The next article (B4) goes into NVIDIA’s AIPerf; this one closes SLO validation, which is what sizes the proposal.

Closing

GuideLLM answers the only performance question that sizes a platform: how far can I load without breaking the SLO? And it answers it with the right discipline: a sweep from idle to saturation that finds the knee, defined by goodput under a declared SLO, over a load that resembles the real one. Everything the proposal needs comes out of that knee: the replica count, the cost per token (with goodput as the denominator, not catalogue throughput) and, if you capture DCGM during the sweep, the energy per token as well. The mistake that invalidates the exercise is the usual one: reporting peak throughput instead of goodput, measuring with an unrealistic load, or not extending the sweep until you see where it breaks. Done properly, as a Job inside the cluster, with the real distribution, the SLO fixed and the output in versioned JSON, GuideLLM turns performance from a marketing figure into a reproducible datum that survives an audit and sizes a sovereign architecture with numbers. The knee is not the fastest point; it is the point where your platform still delivers what it promised.

Sources

Red Hat · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
Red Hat · desplegar y benchmarkear vLLM con GuideLLM en Kubernetes — https://developers.redhat.com/articles/2025/12/24/how-deploy-and-benchmark-vllm-guidellm-kubernetes
GuideLLM · PyPI — https://pypi.org/project/guidellm/
GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
Medium · GuideLLM en OpenShift (rate-types, ejemplo) — https://medium.com/@jajodia.nirjhar/exploring-guidellm-benchmarking-a-live-llm-on-openshift-ccc2d0841794

GPU observability for LLM inference: the twelve DCGM and vLLM metrics that dictate the health of your production

Mon, 01 Jun 2026 15:30:00 +0200

This post complements those on LLM tracing with OpenTelemetry GenAI (the tracing layer above the metrics), Capacity planning (what was sized and what must be watched) and Continuous batching (the mechanism that explains several of the engine metrics).

TL;DR

Observability for an LLM inference cluster is built on two complementary sources: the GPU hardware metrics exposed by the DCGM (Data Center GPU Manager) Exporter, part of the NVIDIA GPU Operator, and the metrics from the inference engine (vLLM, SGLang, TensorRT-LLM) exposed on a Prometheus-compatible /metrics. Neither is enough on its own. The classic nvidia-smi metric called GPU utilization is misleading for LLMs: it reads high whenever any kernel is running, without distinguishing tensor cores burning from SMs waiting on HBM. The full cockpit has twelve DCGM metrics in four families (compute: DCGM_FI_PROF_SM_OCCUPANCY, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, DCGM_FI_PROF_DRAM_ACTIVE; memory: DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_FB_FREE, DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL; thermal-power: DCGM_FI_DEV_GPU_TEMP, DCGM_FI_DEV_POWER_USAGE, DCGM_FI_DEV_CLOCK_THROTTLE_REASONS; health: DCGM_FI_DEV_XID_ERRORS, DCGM_FI_DEV_ECC_DBE_VOL_TOTAL, DCGM_FI_DEV_RETIRED_DBE) and five vLLM engine metrics (vllm:num_requests_running, vllm:num_requests_waiting, vllm:gpu_cache_usage_perc, vllm:time_to_first_token_seconds, vllm:time_per_output_token_seconds). Each one has a defensible green/amber/red threshold, a PromQL query for alerting, and at least one common false reading that confuses the junior operator. The six critical alerts that any production cluster must fire are: HBM > 92 %, thermal or power throttle, XID error, ECC double-bit, KV cache pool > 95 %, and TTFT P95 outside SLO for 5 minutes. The point of having this panel: that the operator on shift diagnoses the origin of a degradation in under five minutes, without opening an SSH console to the GPUs. When that holds, the cluster has moved to professional operation; until then, it is operated on intuition.

You are here: OBSERVE (the other half of tracing)

Tracing, already covered in LLM tracing with OpenTelemetry GenAI, answers what happened in this specific request. Metrics answer what is happening across the cluster in aggregate. They are complementary: an alert on the metrics side tells you “the cluster is degrading”, tracing tells you “and here is the specific trace that proves it”. A cluster without tracing but with metrics operates; a cluster without metrics but with tracing does not operate, it debugs.

The analogy: the cockpit of a modern aircraft

In a modern commercial aircraft, the pilot’s instrument panel has more than 70 active indicators. If there were only one, the altimeter, say, the aircraft would fly into the ground at the first moment of low visibility. You need the altimeter and the attitude indicator, and the airspeed indicator, and the turn indicator, and the fuel gauge, and the oil pressure gauges for each engine, and the turbine exit temperatures. Each answers a different question. And together they cover the operational question: is the aircraft healthy, is it where it should be, and is it going where we want?

Observability for an LLM inference cluster works the same way. A single metric, “GPU utilization 99 %”, answers nothing useful. It is like looking only at the car’s odometer to diagnose why the engine is making a noise. The full cockpit is twelve instruments on the hardware side plus five on the inference engine side, organised into families that answer different questions:

Compute and efficiency: are the tensor cores doing the work we expect, or are they waiting?
Memory: is there VRAM left for new requests, or are we on the edge of OOM?
Thermal and power: is the hardware healthy, or is it silently limiting throughput?
Health and errors: is there hardware degradation under way (ECC, XID, NVLink)?
Inference engine: is the queue growing, is the KV pool saturated, is the SLO being met?

The first four answer “is the GPU fine?”. The fifth answers “is it delivering the service we promised?”. The two questions are different and both must have an answer at a glance.

Why `nvidia-smi` `GPU-Util` misleads with LLMs

The classic metric that appears in nvidia-smi as GPU-Util corresponds to DCGM_FI_DEV_GPU_UTIL. Its official definition: “percentage of time during which one or more kernels were executing on the GPU”. The problem with LLMs: the decode phase is memory-bound, not compute-bound. When the inference engine decodes token by token, the GPU spends 90 % of the time waiting for the HBM to finish delivering the model weights and the KV cache. There is a kernel running (the HBM read), so GPU-Util reports values close to 100 %. But the tensor cores are idle: the bottleneck is memory, not compute.

Practical result: the operator sees “GPU-Util 99 %” in Grafana and assumes “GPU saturated, no more load can go in”. But the reality may be “compute at 25 %, HBM saturated at 95 %”, which changes the operational decisions (quantisation, batch size, parallelism). The classic metric lies by oversimplifying.

The right move is to look at the three DCGM profiling metrics of the _FI_PROF_* subsystem:

DCGM_FI_PROF_SM_OCCUPANCY: ratio of active warps over the maximum per SM. Is there parallel work?
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE: % of cycles with tensor cores effectively active. Is compute working?
DCGM_FI_PROF_DRAM_ACTIVE: % of cycles with HBM transferring. Is memory saturated?

A typical decode-bound GPU running Llama 70B on an H100 shows: SM occupancy 35–55 %, tensor active 15–30 %, DRAM active 80–95 %. That is the real “saturated GPU” for LLMs. The three together tell the regimes apart; none of them alone says anything actionable.

How they are wired up in production

The platform side is covered in Five maturity levels (level 4, the GPU plane) and Seven deployment phases (phase F5). For the observer, the key pieces are:

NVIDIA GPU Operator. Helm manifests that deploy on each GPU node: drivers, container toolkit, MIG manager and DCGM Exporter. The last one exposes /metrics in Prometheus format with all the DCGM_FI_* listed above. It is scraped from the cluster’s internal Prometheus.

Inference engine. vLLM exposes /metrics on port 8000 (default) with vllm:* metrics. SGLang exposes it too, with the sglang: prefix. TensorRT-LLM exposes it through Triton Inference Server with the nv_inference: prefix. The basic naming convention is similar across the three engines; the thresholds and queries in this post assume vLLM, but they translate.

ServiceMonitor / PodMonitor. The Prometheus operator resource that says what to scrape. Minimal example:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
 name: vllm-inference
spec:
 selector:
 matchLabels: { app: vllm }
 podMetricsEndpoints:
 - port: metrics
 interval: 15s

Dashboards. The NVIDIA operator publishes reference Grafana dashboards for DCGM in nvidia/dcgm-exporter (the official repo). vLLM publishes one in vllm-project/vllm (the examples/ folder). Both work as a base; each team adds the panels specific to its own SLO.

The twelve DCGM metrics organised by family

Family 1 — Compute

DCGM_FI_PROF_SM_OCCUPANCY: ratio of active warps per SM over the maximum possible. Value between 0 and 1.

Green: 0.30–0.70 (typical LLM decode regime).
Amber: < 0.20 sustained (batch too small, GPU under-used in parallelism).
Red: 0.95 sustained with low DRAM_ACTIVE (pathological kernel saturating the SMs).

DCGM_FI_PROF_PIPE_TENSOR_ACTIVE: % of cycles with tensor cores executing. The key metric for “is compute producing?”.

Green in prefill: 50–80 %.
Green in decode: 15–30 % (decode is memory-bound, this is not a symptom of a problem).
Red: < 5 % sustained in prefill, or the engine is not using the tensor cores at all (bad config, incompatible format).

DCGM_FI_PROF_DRAM_ACTIVE: % of cycles with HBM transferring data. The key metric for detecting memory saturation.

Green in decode: 60–85 %.
Amber: > 90 % sustained (HBM is a firm bottleneck, which explains high TPOT).
Red: > 95 % sustained with KV cache pool < 70 % (something other than the engine is asking for HBM; investigate leaks).

Family 2 — Memory

DCGM_FI_DEV_FB_USED: Frame Buffer (HBM) used, in MiB.

Green: 70–85 % of total.
Amber: 86–92 %.
Red: > 92 % (risk of OOM on the next paged-attention allocation).

PromQL for the cluster-wide percentage: 100 * sum(DCGM_FI_DEV_FB_USED) / sum(DCGM_FI_DEV_FB_TOTAL).

DCGM_FI_DEV_FB_FREE: free Frame Buffer. Complementary to the previous one; useful for absolute alerts (< 4096 MiB free).

DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL: aggregate NVLink bandwidth in MB/s. For TP (tensor parallel) topologies that cross GPUs over NVLink, this metric reveals whether the parallelism split is saturating the bus.

Green: varies with topology. On 4×H100 SXM with NVLink 4.0, theoretical capacity is 450 GB/s per GPU. A typical TP=4 regime: 50–150 GB/s sustained.
Red: > 90 % of capacity sustained (check whether the model would fit with lower TP or with pipeline parallel).

Family 3 — Thermal and power

DCGM_FI_DEV_GPU_TEMP: die temperature in °C.

Green: < 75 °C.
Amber: 75–82 °C.
Red: > 83 °C (close to the H100’s automatic thermal throttle; check ventilation, air flow, rack inlet temperature).

DCGM_FI_DEV_POWER_USAGE: draw in watts. For the H100 SXM, nominal TDP is 700 W. Useful for three things: spotting an unusually low workload (suspect idle or a stall), billing real energy cost, and firing alerts if the draw approaches the PDU limit.

DCGM_FI_DEV_CLOCK_THROTTLE_REASONS: encoded bitmap with the active throttle reasons. This is the metric that silently explains TPOT degradations.

Relevant bits:

0x0000000000000001: Idle (not a problem).
0x0000000000000002: App clocks setting.
0x0000000000000004: SW Power Cap (software limit, e.g. from nvidia-smi -pl).
0x0000000000000008: HW Slowdown.
0x0000000000000010: Sync Boost (NVIDIA Sync).
0x0000000000000020: SW Thermal Slowdown (software thermal limit).
0x0000000000000040: HW Thermal Slowdown (hardware thermal limit, an emergency).
0x0000000000000080: HW Power Brake Slowdown (PSU voltage drop).
0x0000000000000100: Display Clock Setting.

Any throttle other than Idle with a sustained value > 0 is an alert. TPOT degradation with DRAM_ACTIVE already high and thermal throttle active is the classic “the rack is badly ventilated, it is not the engine’s fault”.

Family 4 — Health

DCGM_FI_DEV_XID_ERRORS: cumulative counter of driver XID errors. XIDs are critical event codes that NVIDIA documents exhaustively (XID 13: graphics engine exception; XID 31: GPU memory page fault; XID 43: reset channel verif error; XID 79: GPU has fallen off the bus; XID 95: uncontained ECC error; and so on). Any increment is an immediate alert: many XIDs require a node reset or an RMA of the GPU.

DCGM_FI_DEV_ECC_DBE_VOL_TOTAL: volatile double-bit ECC errors (uncorrectable). Unlike single-bit ones (which ECC corrects silently and which are counted in DCGM_FI_DEV_ECC_SBE_*), double-bit errors corrupt data. Any value > 0 is a critical alert: the GPU must be drained and inspected.

DCGM_FI_DEV_RETIRED_DBE: physical HBM pages retired because of accumulated double-bit errors. NVIDIA retires faulty pages automatically to prevent future corruption. More than 4–8 retired pages on one GPU suggests silicon degradation: document it and plan a replacement in the next maintenance window.

The five inference engine metrics (vLLM)

The DCGM metrics answer “is the GPU healthy?”. The engine metrics answer “is the service meeting the SLO?”. Without them, you know the hardware works but you do not know whether the clients are happy.

vllm:num_requests_running: requests currently in the batch. If it reaches the configured --max-num-seqs and does not come down, the engine is saturated on concurrency (check VRAM and rebalance through the autoscaler, see Autoscaling LLM on Kubernetes).

vllm:num_requests_waiting: requests queued, not yet in the batch. Any value > 0 sustained over minutes means the cluster is not scaling with the load. This is the primary metric for HPA.

vllm:gpu_cache_usage_perc: % of the KV cache pool in use.

Green: 50–80 %.
Amber: 80–92 %.
Red: > 92 % (risk of preempt-on-OOM: vLLM will drop requests to free memory, which raises TTFT visibly).

vllm:time_to_first_token_seconds: histogram of TTFT per request. Consumed as histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket[5m]))). Compared against the TTFT P95 SLO, it fires the primary service alert.

vllm:time_per_output_token_seconds: histogram of TPOT. Equivalent to the previous one but for streaming smoothness. Compared against the TPOT P95 SLO, it fires the secondary alert.

The six alerts that must page in production

Any serious production cluster fires these six alerts to a channel with an on-call rotation. Without them, the SLO is met by luck, not by process.

groups:
 - name: gpu-llm-critical
 rules:
 - alert: GpuHbmNearOom
 expr: 100 * (DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL) > 92
 for: 2m
 labels: { severity: critical }
 annotations:
 summary: "HBM on {{ $labels.gpu }} at {{ $value }}% — OOM risk"

 - alert: GpuThermalOrPowerThrottle
 expr: (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS != 0) and ignoring(reason) (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS != 1)
 for: 1m
 labels: { severity: warning }
 annotations:
 summary: "GPU {{ $labels.gpu }} throttling (reasons={{ $value }})"

 - alert: GpuXidErrorDetected
 expr: increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0
 labels: { severity: critical }
 annotations:
 summary: "XID error on GPU {{ $labels.gpu }} — investigate immediately"

 - alert: GpuEccDoubleBit
 expr: DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0
 labels: { severity: critical }
 annotations:
 summary: "ECC double-bit on GPU {{ $labels.gpu }} — drain node"

 - alert: VllmKvCachePoolNearFull
 expr: vllm:gpu_cache_usage_perc > 0.95
 for: 3m
 labels: { severity: warning }
 annotations:
 summary: "KV cache pool > 95% on {{ $labels.instance }}"

 - alert: VllmTtftP95OutOfSlo
 expr: histogram_quantile(0.95, sum by(le, instance)(rate(vllm:time_to_first_token_seconds_bucket[5m]))) > 1.5
 for: 5m
 labels: { severity: warning }
 annotations:
 summary: "TTFT P95 over SLO ({{ $value }}s > 1.5s)"

These six cover 80 % of the incidents that affect the SLO. The remaining 20 % demand investigation with tracing (see LLM tracing with OpenTelemetry GenAI).

Master table: thresholds and queries

Metric	Green	Amber	Red	Base query (PromQL)
SM occupancy	0.30–0.70	0.15–0.30	< 0.10 sustained	`DCGM_FI_PROF_SM_OCCUPANCY`
Tensor active (decode)	15–30 %	< 10 %	< 3 %	`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`
DRAM active	60–85 %	85–95 %	> 95 % with low KV	`DCGM_FI_PROF_DRAM_ACTIVE`
FB used	70–85 %	86–92 %	> 92 %	`100 * DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL`
NVLink BW	< 70 % cap	70–90 % cap	> 90 % cap	`DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL`
GPU temp	< 75 °C	75–82 °C	> 83 °C	`DCGM_FI_DEV_GPU_TEMP`
Power usage	< 90 % TDP	90–98 % TDP	> 98 % TDP	`DCGM_FI_DEV_POWER_USAGE`
Throttle reasons	0 or Idle	App/SW	HW Therm/Power	`DCGM_FI_DEV_CLOCK_THROTTLE_REASONS`
XID errors	no change	—	any delta	`increase(DCGM_FI_DEV_XID_ERRORS[5m])`
ECC DBE	0	—	> 0	`DCGM_FI_DEV_ECC_DBE_VOL_TOTAL`
Retired pages	< 4	4–8	> 8	`DCGM_FI_DEV_RETIRED_DBE`
KV cache used	50–80 %	80–92 %	> 92 %	`vllm:gpu_cache_usage_perc`
Requests waiting	0	1–5 sustained	> 10 sustained	`vllm:num_requests_waiting`
TTFT P95	< SLO	80–100 % SLO	> SLO	see alert query above
TPOT P95	< SLO	80–100 % SLO	> SLO	`histogram_quantile(0.95, sum by(le)(rate(vllm:time_per_output_token_seconds_bucket[5m])))`

Three pitfalls that confuse the junior operator

Pitfall 1 — “GPU-Util at 99 % = saturated”. As explained at the start, DCGM_FI_DEV_GPU_UTIL lights up with any kernel. The right move is to look at the three _PROF_* metrics (SM occupancy, tensor active, DRAM active) together. GPU util 99 % + tensor active 8 % + DRAM active 92 % means “saturated by memory, not compute”; GPU util 99 % + tensor active 75 % + DRAM active 50 % means “saturated by compute, prefill heavy”. The two situations call for different levers.

Pitfall 2 — confusing single-bit ECC (SBE) with double-bit (DBE). Single-bit errors are corrected silently and are unavoidable in any HBM under load (cosmic radiation, voltage fluctuations). An SBE counter growing slowly is not an alert, it is physics. DBE is: it corrupts data. Telling the two metrics apart avoids false alarms and false negatives in equal measure.

Pitfall 3 — alerting on num_requests_waiting > 0 without context. An instantaneous value of 1 or 2 during a spike is normal. What matters is a sustained queue: use for: 5m with a threshold of 3–5. Without that window, the system floods the alert channel with noise.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB with intra-node NVLink:

DCGM Exporter deployed through the NVIDIA GPU Operator, one DaemonSet per GPU node.
Internal Prometheus with 30-day retention for high-frequency metrics, 1 year for downsampled ones (Thanos/Mimir if the volume justifies it).
Grafana with three standard dashboards: GPU hardware (DCGM), engine (vLLM), SLO (TTFT/TPOT/RPS against written targets).
Alertmanager with an on-call rotation and rate limiting through silences grouped by node.
Cardinality kept under control: gpu (local id), node, pod, model. Do not add request_id or other high-cardinality labels to metrics; that is tracing’s job.

Estimated volume for a 16-GPU cluster scraped every 15 s: around 2 million samples/min, around 25 GB/day of raw Prometheus. Manageable with one Prometheus per cluster plus retention; if the team scales beyond 64 GPUs, consider a Thanos sidecar or VictoriaMetrics. See OSS LLMOps tooling catalogue for equivalent alternatives.

What we have not covered (upcoming articles)

Tracing of LLM workloads: already covered in LLM tracing with OpenTelemetry GenAI.
Autoscaling based on these metrics: see Autoscaling LLM on Kubernetes.
Incident response runbooks: how each of these alerts translates into concrete action (drain, restart, RMA, scaling, rollback).
Cost accounting: using DCGM_FI_DEV_POWER_USAGE and vllm:request_success_total for per-tenant cost showback.
Multi-tenant fairness monitoring: when several tenants share a cluster, which metrics detect that one of them is hogging the KV cache.

References

NVIDIA — DCGM Exporter (repo nvidia/dcgm-exporter, documented metrics and units).
NVIDIA — DCGM Field Identifiers reference (complete list of DCGM_FI_*).
NVIDIA — XID Errors documentation (catalogue of XID codes and remediation procedures).
NVIDIA — NVIDIA GPU Operator (official Helm chart).
vLLM project — examples/production_monitoring/ (reference PromQL and Grafana dashboards).
Prometheus — Histogram and summary best practices (for building defensible percentile queries).
NVIDIA — H100 Tensor Core GPU datasheet (TDP, HBM bandwidth, NVLink capacities).

Capacity planning for on-premise LLM inference: how to size GPUs from an SLO

Mon, 01 Jun 2026 15:00:00 +0200

This post complements the ones on KV cache (the piece that dominates the VRAM budget), Continuous batching (what defines the effective utilisation of the compute) and Seven layers of the stack (the pieces the sizing takes for granted). Before reading this one, make sure your team has written down the SLOs it is going to chase; without that input the calculation is not defensible.

TL;DR

LLM inference capacity planning does not answer “how many tokens per second does a GPU give” — that question has no universal answer, because throughput depends on concurrency, the prefill/decode split, context length, the inference engine and quantisation. The right question has three inputs (SLO: TTFT P95, TPOT P95, sustained RPS), a hardware reference (GPU model, VRAM, HBM bandwidth, effective FLOPs) and a model (parameters, GQA/MHA/MoE architecture, weight format). The calculation resolves into two coupled budgets that cross each other. VRAM budget: from the GPU’s total you subtract model weights and activations, what remains is the KV cache budget, and from there you derive the maximum concurrency possible at the average context you expect. Time budget: the engine (vLLM, SGLang, TensorRT-LLM) has a tokens/s ceiling in decode given by HBM bandwidth and another in prefill given by useful FLOPs; from there you derive the expected TPOT and, dividing prefill_tokens by prefill throughput, the expected TTFT. Both budgets must meet the SLO simultaneously: whichever is tighter dictates the sizing. On the Llama 70B BF16 example with tensor parallel 4 on 4×H100 SXM, a single replica saturates at ~28 concurrent requests and delivers ~3,200 tokens/s of aggregate decode with a median TPOT of 35 ms; for 200 sustained RPS at a profile of 800 prompt tokens + 250 of output, you need between 4 and 5 replicas with a 25 % cushion over the observed peak. Quantisation (FP8 → INT4) divides VRAM cost and decode time by between 1.5 and 4×, but it degrades quality measurably — it is not assumed free, it is validated with evals. The five habitual traps: confusing the mean with P95, ignoring the prefill/decode split of the real workload, sizing without head-room for retraining or rollback, forgetting that a GPU at 100 % SM util means nothing if the HBM is saturated, and not documenting the assumptions behind the calculation (a sizing with no written assumptions is a throwaway calculation).

You are here: DEPLOY (with one foot in OBSERVE)

Capacity planning is a piece with dual residency. It lives in DEPLOY because without a valid sizing you do not buy hardware or configure the inference engine. But its input is real observations: the distribution of prompt and output lengths, the prefill/decode mix of the workload, the real P95s already being seen in pre-production. Without that data the calculation is a napkin sketch, defensible only until the first customer arrives who does not fit the assumed average.

The analogy: the hotel with variable-sized rooms

Imagine a hotel where the rooms have no fixed size: each guest pays for the square metres they need, and the building’s floor plan reorganises itself dynamically to accommodate whoever arrives. Management wants to maximise occupancy, but it has two real constraints and one quality metric.

Constraint 1 — physical space. The floor has 1,000 m² in total. If a family comes in needing 200 m², that family occupies that surface and it cannot be handed to the next guest. The largest room limits how many simultaneous guests fit.

Constraint 2 — service staff. There are 10 receptionists. Each can handle the check-in of one guest every two minutes. When 60 guests arrive within an hour, the last ones wait in a queue; the time from walking into reception to receiving their key depends on how many are ahead of them.

Quality metric — a promise about time. The brochure says “check-in in under 15 minutes”. If too many guests arrive at once, that promise breaks even if there is physical space free.

The physical space is the GPU’s VRAM. Each room is a request with its KV cache (larger the longer the conversation). The receptionists are the compute units (Streaming Multiprocessors + Tensor Cores). The check-in is the prefill phase; the nights the guest spends afterwards are the decode steps. The 15-minute promise is the TTFT P95 SLO.

The hotel’s capacity planning is exactly this: given the expected guest profile (how many arrive per hour, how much space they ask for on average, how many minutes of waiting they tolerate), work out how many floors and how many receptionists are needed. It is not done by estimating “rooms per hour” in the abstract; it is done by crossing the two budgets with the promise about time. The analogy holds up the calculation all the way to the end.

The three SLO inputs

Before putting a single number on the sheet, you have to write down the three dimensions of the SLO. Without this, the calculation is aesthetics, not engineering.

TTFT P95 (Time-To-First-Token). The time from the client sending the request until it receives the first token. It is dominated by the prefill phase (processing the whole prompt in one go) plus the scheduler queue. For conversational chat, a reasonable target sits between 0.5 and 2 seconds P95. For coding assistants with large prompts (5–10 K tokens of context), between 2 and 4 s P95. Below 500 ms you enter UX territory for voice-style conversations, but that demands serious architectural compromises.

TPOT P95 (Time-Per-Output-Token). The time between consecutive tokens during decode. It dominates the “perceived fluency” of streaming. Above 80 ms/token the human reader perceives pauses; below 30 ms/token the output flows faster than it can be read. The usual industry target: 40–60 ms P95.

Sustained RPS meeting the SLO. The throughput the system must support without violating TTFT or TPOT. This is DistServe’s key metric, called goodput (see Continuous batching). “200 RPS peak” is not the same as “200 RPS with TTFT P95 ≤ 1.5 s”. Without the SLO condition, the RPS number means nothing.

These three dimensions come with a workload profile: the distribution of prompt and output lengths. Medians are not enough; you need P50, P95, P99. A badly measured profile is the main reason sizings fail.

The central formula: two budgets that cross

The calculation resolves into two independent sums that are then crossed. The smaller of the two rules.

VRAM budget

For a GPU with total VRAM $V$, the space available for KV cache is:

$$V_{\text{kv}} = V - V_{\text{model}} - V_{\text{activations}} - V_{\text{overhead}}$$

where:

$V_{\text{model}}$ is the size of the weights: for a model of $P$ parameters in a format of $b$ bytes/parameter, $V_{\text{model}} = P \cdot b$. Llama 70B BF16 = $70 \times 10^9 \times 2 = 140$ GB. Under tensor parallel TP=4, each GPU carries $140 / 4 = 35$ GB.
$V_{\text{activations}}$ are the intermediate buffers of the forward pass. For vLLM with a reasonable batch, between 2 and 6 GB per GPU depending on batch size and maximum length.
$V_{\text{overhead}}$ is the CUDA context, NCCL buffers, the PagedAttention pool, reserved paged blocks. 2–4 GB typical.

The KV cache budget per GPU is what is left over. For an H100 SXM 80 GB with Llama 70B TP=4 BF16:

$$V_{\text{kv}} = 80 - 35 - 4 - 3 = 38 \text{ GB per GPU} = 152 \text{ GB aggregated over TP=4}$$

The KV cache cost per token for a model with $L$ layers, $H_{\text{kv}}$ KV heads (GQA), per-head dimension $d_h$, in a format of $b$ bytes:

$$\text{kv per token} = 2 \cdot L \cdot H_{\text{kv}} \cdot d_h \cdot b$$

The factor of 2 is because both K and V are stored. For Llama 70B (L=80, $H_{\text{kv}}$=8 with GQA, $d_h$=128, BF16 = 2 bytes):

$$\text{kv per token} = 2 \cdot 80 \cdot 8 \cdot 128 \cdot 2 = 327,680 \text{ bytes} = 320 \text{ KB/token}$$

And the maximum concurrency at average context $C$:

$$N_{\text{max}} = \frac{V_{\text{kv}}}{C \cdot \text{kv per token}}$$

With an aggregated $V_{\text{kv}}$ of 152 GB and an average context of 1,500 tokens (800 prompt + 700 generated at the worst moment of the conversation):

$$N_{\text{max}} = \frac{152 \times 10^9}{1,500 \cdot 320 \times 10^3} \approx 316 \text{ concurrent requests}$$

This is the physical ceiling of concurrency for that replica. It is not what you are going to use; it is what you cannot exceed without OOM. The operational number sits well below it (head-room for spikes).

Time budget

Two sub-calculations come in here: decode (memory-bound) and prefill (compute-bound).

Decode TPOT. For every token generated, the model weights (the ones relevant to that request) have to be walked through and the accumulated KV cache has to be read. The bottleneck is HBM bandwidth. For a GPU with bandwidth $B$ GB/s and a model of $V_{\text{model per gpu}}$ GB of weights:

$$\text{tpot}_{\text{theoretical}} \approx \frac{V_{\text{model per gpu}}}{B}$$

For an H100 SXM with HBM3 at 3.35 TB/s and Llama 70B TP=4 BF16 (35 GB/GPU):

$$\text{tpot}_{\text{theoretical}} \approx \frac{35}{3,350} \approx 10.4 \text{ ms/token}$$

This is the best theoretical case with batch=1 and 100 % HBM efficiency. In practice vLLM on H100 with Llama 70B TP=4 reaches 12–18 ms/token at low batch and 30–45 ms/token at high batch (at concurrency 32, tokens compete for the shared HBM). The defensible operational number: 35 ms/token at concurrency 24–32.

Prefill throughput. Prefill processes N prompt tokens in a single forward pass. It is compute-bound: the bottleneck is FLOPs. For an H100 SXM with 989 sustained BF16 TFLOPs and Llama 70B (each forward pass costs roughly $2 \cdot P \cdot N$ FLOPs per sequence of length N):

$$\text{prefill tps} = \frac{4 \cdot \text{TFLOPs} \cdot \eta}{2 \cdot P} = \frac{4 \cdot 989 \times 10^{12} \cdot 0.5}{2 \cdot 70 \times 10^9} \approx 14,000 \text{ tokens/s}$$

(the factor of 4 is the GPUs in TP, $\eta$ is real efficiency between 0.4 and 0.6 on H100). An 800-token prompt takes this long in prefill:

$$\text{prefill time} = \frac{800}{14,000} \approx 57 \text{ ms}$$

Adding a typical queue of 100–300 ms at high concurrency, TTFT P95 ≈ 350–500 ms for that profile. Well below the 1.5 s target; there is margin.

The crossing

The real operational concurrency $N_{\text{op}}$ is the minimum of the VRAM ceiling, the concurrency at which TPOT starts to degrade past the SLO, and the concurrency at which TTFT starts to degrade past the SLO (prefill queue). For the example:

VRAM ceiling: 316.
TPOT degrades to 80 ms (SLO) at around concurrency ~80–100 (measured empirically with a benchmark, not a closed formula).
TTFT degrades to 1.5 s at around concurrency ~40–60 because of the prefill queue.

The replica’s operational concurrency is ~50. Applying 25 % head-room for spikes and rebalancing, target concurrency per replica ≈ 35–40.

Step-by-step worksheet: Llama 70B BF16 on 4×H100 SXM

The exercise’s input:

SLO: TTFT P95 ≤ 1.5 s; TPOT P95 ≤ 60 ms; 200 sustained RPS.
Workload: prompt P50=600, P95=1,200, P99=2,500; output P50=180, P95=500, P99=900. Average prompt 800, output 250.
Generic hardware: 4×H100 SXM 80 GB with NVLink, vLLM v1 engine, tensor parallel 4, BF16.

Step 1 — VRAM per GPU. Weights 35 GB, activations 4 GB, overhead 3 GB → KV budget 38 GB/GPU = 152 GB aggregated. KV/token for Llama 70B GQA = 320 KB. Ceiling of live tokens in cache: $152 \times 10^9 / 320 \times 10^3 \approx 475,000$ tokens. At the operational average context (800 prompt + 200 already generated = 1,000 live tokens per request), concurrency ceiling $\approx 475$.

Step 2 — average duration of a request. Prefill 800 tokens / 14,000 tps = 57 ms. Decode 250 tokens × 35 ms/token = 8,750 ms. Total $\approx 8.8$ s per request.

Step 3 — replica throughput. If the replica sustains operational concurrency 40 and each request lasts 8.8 s, the replica delivers roughly $40 / 8.8 \approx 4.5$ requests/s in steady state.

Step 4 — number of replicas. For a 200 RPS target: $200 / 4.5 \approx 45$ replicas. That is 45 × 4 = 180 GPUs. Too much: this sizing does not work because the cost per request is high.

Step 5 — review the levers. Before buying more hardware, there are three levers to explore, in this order:

Quantisation. Dropping to FP8 reduces weights to 17.5 GB/GPU (leaving more VRAM for KV cache → more concurrency), roughly doubles decode tokens/s (HBM saturated by half), and degrades MMLU quality typically by 0.5–1.5 points on models like Llama 70B. Rewriting the calculation in FP8: TPOT drops to ~18 ms, total time per request to 4.7 s, RPS per replica rises to ~8.5, replicas needed ≈ 24, equivalent to 96 GPUs.
Speculative decoding. With a small drafter and 60–70 % acceptance, effective TPOT falls 30–40 %. RPS per replica rises to ~12, replicas ≈ 17 = 68 GPUs.
Disaggregated serving. Separating prefill workers and decode workers lets each be scaled to the real mix of the workload (see Disaggregated serving). It usually shaves off another 20–40 % under asymmetric workloads.

Step 6 — recommended sizing. For the example, with FP8 + speculative decoding and 25 % head-room: 20 vLLM replicas at TP=4 over 80 H100 SXM. If the team does not want to depend on aggressive quantisation (pure BF16 for maximum fidelity), the calculation rises to 30 replicas = 120 GPUs and forces a renegotiation of the SLO or the budget.

Step 7 — write down the assumptions. This is the part no valid sizing skips. In the team’s repo, next to the calculation:

# sizing/llama70b-prod.yaml
fecha: 2026-06-01
slo:
 ttft_p95_ms: 1500
 tpot_p95_ms: 60
 rps_target: 200
workload:
 prompt_tokens_p50: 600
 prompt_tokens_p95: 1200
 output_tokens_p50: 180
 output_tokens_p95: 500
 asunto: chat productivo con RAG ligero
modelo:
 arquitectura: llama-70b-instruct
 formato_pesos: fp8
 motor: vllm-v1
hardware:
 gpu: H100-SXM-80GB
 topologia: TP=4 con NVLink intra-nodo
 red_inter_replica: 25 GbE
optimizaciones:
 - paged_attention
 - chunked_prefill
 - speculative_decoding (drafter llama-1.1b, aceptación esperada 65%)
asunciones_criticas:
 - utilizacion_hbm_eficiente: 0.55
 - head_room_pico_sobre_p95: 0.25
 - aceptacion_speculative_min: 0.55
plan_validacion:
 - benchmark vllm bench serve antes de procurement
 - canary 10% durante 7 días post-deploy

Without this YAML, the calculation is not reproducible a month later.

MoE case: Mixtral 8×22B (~141 B total, 39 B active)

MoEs change the calculation along one key dimension: the total weights are large but the active weights per token are small. For Mixtral 8×22B with top-2 routing:

Weight VRAM: $141 \times 2 = 282$ GB BF16. With TP=4 → 70 GB/GPU. It does not fit in an H100 80 GB alongside KV cache + activations. You need TP=8 (~35 GB/GPU) or FP8 with TP=4 (~35 GB/GPU).
Decode TPOT: dominated by the weights read per token, which are $\sim 39 / 8 \cdot 2 \approx 9.75$ GB/GPU with TP=4 (one top-2 expert per token, divided across 4 GPUs). On an H100 with HBM at 3.35 TB/s, theoretical TPOT ≈ 3 ms/token. In practice, 10–20 ms at reasonable concurrency.
Prefill: similar to the dense model of the active weights, ~39 B FLOPs/token.

MoE sizing usually delivers more RPS per GPU than an equivalent dense model: the low cost per token compensates for the extra VRAM. See MoE inference for the detail of routing and why a high batch is decisive so that each expert sees enough tokens.

Sensitivity table: context and quantisation

For Llama 70B on 4×H100 SXM (TP=4), operational concurrency per replica with an SLO of TTFT 1.5 s / TPOT 60 ms:

Average context	BF16	FP8	INT4 (AWQ)
500 tokens	55	110	180
1,000 tokens	40	80	130
2,000 tokens	24	50	85
4,000 tokens	12	26	48
8,000 tokens	6	13	25

Approximate numbers from public vLLM benchmarks as of June 2026, with ±20 % variation depending on engine version and the headroom adopted. To validate on your own hardware: vllm bench serve with your profile of real prompts.

The five habitual traps

Trap 1 — confusing the mean with P95. Average throughput over an hour may be 50 RPS while the 5-minute peak reaches 180 RPS. Sizing against the mean guarantees breaking the SLO at every peak. Rule: size against the hourly P95, with 20–30 % head-room over P95.

Trap 2 — not measuring the real prefill/decode split. A “RAG with short answers” workload spends 70–80 % of GPU time in prefill; a “writing assistant generating essays” spends 80 % in decode. The useful optimisations (chunked prefill vs speculative decoding) change radically. Without measuring it, you buy badly balanced hardware.

Trap 3 — sizing with no head-room for retraining or rollback. The production cluster is not just the inference engine: there is a re-embedding batch when the embeddings model changes, continuous canary evals (see Canary, blue-green and shadow), light fine-tuning, hot stand-by for rollback. Reserve 15–25 % of capacity for those non-negotiable workloads.

Trap 4 — “GPU at 100 % SM utilization” as a target. SM occupancy of 95 % with saturated HBM produces the same throughput as SM at 60 % with saturated HBM. The bottleneck in decode is the HBM. Optimising for “GPU usage 100 %” without looking at HBM utilisation and arithmetic intensity makes you spend more on GPU without gaining throughput. See GPU observability for LLM inference for which metrics to actually watch.

Trap 5 — not documenting the assumptions. A sizing with no reproducible YAML (workload, model, engine, head-room, critical assumptions) leaves the team with no way of knowing what changed when the cluster stops meeting the SLO six months later. Documenting is cheap; losing a quarter to debugging is not.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB with intra-node NVLink and 25 GbE between nodes, the recurring configurations in May 2026 are:

Model	Format	TP	Replicas that fit	Typical RPS per node (ctx 1K)
Llama 8B	BF16	1	4 (one per GPU)	240–320
Llama 8B	FP8	1	4	450–600
Llama 70B	BF16	4	1	30–45
Llama 70B	FP8	4	1	60–90
Llama 70B	INT4 AWQ	2	2	90–130
Mixtral 8×22B	FP8	4	1	90–140
Qwen 72B	BF16	4	1	28–42

These numbers are orders of magnitude to start the conversation, not commitments. The definitive sizing is validated with vllm bench serve or genai-perf (NVIDIA) using the customer’s real prompt/output profile. The prefill/decode asymmetry of each case’s workload can move these numbers 30–50 % up or down.

For clusters of 8×H100 SXM (typical of DGX servers or equivalent replicas), the options open up to TP=8 for 405B-class models or multi-replica TP=2 for 70B models with greater density. The metric that decides is always the same: tokens meeting SLO per kW and per euro of amortised hardware.

How the sizing is validated before buying

The spreadsheet sizing is the first half. The second is the validation benchmark.

Stage 1 — back-of-the-envelope sizing. This post’s formulas applied to the SLO and the expected workload. Output: an approximate number of replicas and a topology.

Stage 2 — synthetic micro-benchmark. On a borrowed GPU or one rented by the day, bring up the engine with the chosen model and run vllm bench serve with prompts of representative lengths. Validate TPOT, prefill TPS and the concurrency ceiling. Calibrate the HBM efficiency factor ($\eta$) used in the formulas.

Stage 3 — load test with realistic traffic. Generate traffic following the real distribution of the customer’s workload (not Poisson, not constant: the real trace). Measure P50/P95/P99 of TTFT, TPOT, throughput. Confirm the head-room.

Stage 4 — canary in production. With the cluster sized, route 5–10 % of real traffic for 7–14 days before closing the procurement of additional hardware. See Canary, blue-green and shadow for the mechanics.

Jumping from Stage 1 to full procurement is the most frequent cause of a cluster that is 40 % oversized and 60 % undersized at the same time, in different regions of the workload. Four weeks of validation done well save four months of refactoring.

What we have not covered (upcoming articles)

The observability metrics that close the sizing loop in production, see GPU observability for LLM inference.
Autoscaling that adjusts replicas to the real traffic curve, see Autoscaling LLM on Kubernetes.
Detailed cost accounting per tenant (showback / chargeback) over the sized hardware.
Sizing for continuous fine-tuning (PEFT and light training) that shares a cluster with inference.

References

Kwon et al. — vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (SOSP 2023).
Zhong et al. — DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024).
Agrawal et al. — Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI 2024).
NVIDIA — H100 Tensor Core GPU Architecture Whitepaper (HBM3 memory, bandwidth, sustained FLOPs).
vLLM project — vllm bench serve reference (benchmarking CLI included in the repo).
NVIDIA — genai-perf (official tool for benchmarking LLM services).

Slo on lo0 — Tech Blog

GuideLLM in depth: validating the SLO under load and sizing from the knee

What this article covers

Why GuideLLM

The load modes (rate-types)

The automatic sweep: from idle to the knee

A worked example: reading a 10-round sweep

Defining the SLO: the number that decides the knee

Execution: the command

Defining the load: --data decides the result

The output: what to read

From the knee to sizing and cost

Deployment as a Kubernetes Job

SLO validation in CI: catching regressions

Comparing configurations with sweeps

Watching the three axes during the sweep

Interpreting the distributions, not just the knee

GuideLLM in the reproducible harness

The cost of a sweep (in euros)

The SLO by use case: the knee moves with it

GuideLLM vs the others: when to use which

Checklist for a defensible sweep

Limits and traps (data-driven)

Closing

See also

Sources

GPU observability for LLM inference: the twelve DCGM and vLLM metrics that dictate the health of your production

TL;DR

You are here: OBSERVE (the other half of tracing)

The analogy: the cockpit of a modern aircraft

Why nvidia-smi GPU-Util misleads with LLMs

How they are wired up in production

The twelve DCGM metrics organised by family

Family 1 — Compute

Family 2 — Memory

Family 3 — Thermal and power

Family 4 — Health

The five inference engine metrics (vLLM)

The six alerts that must page in production

Master table: thresholds and queries

Three pitfalls that confuse the junior operator

Applied to typical on-premise hardware

What we have not covered (upcoming articles)

See also

References

Capacity planning for on-premise LLM inference: how to size GPUs from an SLO

TL;DR

You are here: DEPLOY (with one foot in OBSERVE)

The analogy: the hotel with variable-sized rooms

The three SLO inputs

The central formula: two budgets that cross

VRAM budget

Time budget

The crossing

Step-by-step worksheet: Llama 70B BF16 on 4×H100 SXM

MoE case: Mixtral 8×22B (~141 B total, 39 B active)

Sensitivity table: context and quantisation

The five habitual traps

Applied to typical on-premise hardware

How the sizing is validated before buying

What we have not covered (upcoming articles)

See also

References

Defining the load: `--data` decides the result

Why `nvidia-smi` `GPU-Util` misleads with LLMs