Tpot on lo0 — Tech Blog

MLPerf Inference: how to read it, what LoadGen enforces and what comparability it offers

Sat, 29 Aug 2026 09:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

MLPerf Inference is maintained by MLCommons on a six-monthly cadence. It imposes three things no home-made benchmark has: a mandatory load generator (LoadGen, compiled from a tagged revision and unmodified), model-equivalence rules that bound what can be optimised, and an audit process that reviews up to two submissions per round. The four scenarios (SingleStream, MultiStream, Server, Offline) have a minimum duration of 600 s and different metrics: Server reports the sustained Poisson rate and is the only one with a latency constraint. For the LLM benchmarks that constraint is not a request latency but two simultaneous SLOs at the 99th percentile: TTFT and TPOT, with per-model figures ranging from 2000 ms / 200 ms on Llama 2 70B to 450 ms / 40 ms on its Interactive variant. Quality is required relative to the reference model (99 % or 99.9 %) and the generation length is audited as well (Llama 2 70B: more than 90 % of 294.45 tokens per sample). Three reading rules sum up the real comparability: results are only compared within the same version, division, category and scenario; per-accelerator is not an official MLCommons metric, it is derived; and neither cost nor energy enters the primary metric, to the point that round v5.1 had 2 power submissions against 27 participating organisations.

What it is and how it is organised

MLPerf Inference is maintained by the MLCommons Association through the MLPerf Inference Working Group, whose participation is limited to MLCommons members and affiliates (MLPerf Inference Working Group). The declared cadence for publishing results is roughly six months.

Round	Results published
v4.0	27 March 2024
v4.1	28 August 2024
v5.0	2 April 2025
v5.1	9 September 2025
v6.0	1 April 2026
v6.1	in preparation as of this data sheet

The suite splits into two system categories: Datacenter and Edge, the latter defined in the rules as everything that is not datacenter (inference_rules.adoc). A Datacenter system has two requirements that do not apply in Edge:

Mandatory ECC on DRAM and HBM, active during all performance and accuracy runs. There is no requirement on SRAM.
Mandatory networking since round v3.0, with a minimum bandwidth computed from the achieved throughput. For Llama 3.1 405B the minimum ingress is throughput × 20000 × dtype_size bytes/s; for Llama 2 70B the minimum egress is throughput × 1024 × dtype_size.

Edge additionally allows inferred results: a MultiStream derived from SingleStream counts as 8 times the p99 latency; an Offline derived from MultiStream, 8000 divided by the mean latency in milliseconds.

The four scenarios

This is the normative table, transcribed from the rules:

Scenario	Query generation	Duration	Samples/query	Latency constraint	Tail latency	Reported metric
SingleStream	The next query is issued when the SUT completes the previous one	600 s	1	None	90 %	p90 latency with early stopping
Server / Interactive	LoadGen issues queries following a Poisson distribution	600 s	1	Benchmark-specific	99 %	Maximum Poisson parameter supported
Offline	All samples are delivered at the start in a single query	1 query, 600 s	At least 24,576	None	N/A	Measured throughput
MultiStream	The next query is issued when the SUT completes the previous one	600 s	8	None	99 %	p99 query latency with early stopping

Four operational clarifications that change how each number is interpreted:

In Server, LoadGen does not measure a latency: it runs a binary search over the target rate. For a given value it generates queries at that QPS with Poisson arrivals; if the run fails the latency gate, it lowers the value and repeats. The final published metric is, per the official FAQ, scheduled samples per second, not completed samples nor the target entered by the user.
In MultiStream, the latency of a query is the maximum of the latencies of its samples, and the 8 samples of each query are contiguous in the order in which they were loaded. The value multi_stream_samples_per_query = 8 is fixed in the official configuration.
In Offline there is no latency constraint at all. The rules themselves forbid techniques that only work in fixed-length experiments except in the Offline scenario, which is a normative acknowledgement that an Offline result says nothing about behaviour under an SLO.
The percentiles in the table are the lower theoretical bound for runs with very many queries. With early stopping, the percentile actually computed is somewhat above the nominal one, as a penalty for processing few queries.

The mandatory scenarios depend on the benchmark. In Datacenter, the summarisation, question answering, text generation and VLM tasks require (Server | Interactive), Offline, that is, the submitter chooses between Server and Interactive. Reasoning and recommendation require Server, Offline. Medical segmentation, node classification, speech-to-text and RAG require only Offline.

LoadGen: what it measures and what it does not

LoadGen is the module that generates the load and computes the metrics. Its use is mandatory for all submissions, and it must be compiled from a tagged and approved revision of the repository, unaltered; the README is explicit that local modifications of the C++ library are not accepted.

Its four declared responsibilities are generating the queries according to the scenario, tracking the latency of each query, validating the accuracy of the results and computing the final metrics. Latency is defined as the time from when LoadGen was scheduled to pass a query to the system under test until it receives the response.

Equally important is what it declares out of scope: LoadGen does not know the model, does not know the data formats, cannot score accuracy and does not know the scenario constraints of the MLPerf rules. The consequence is written in its own README: being model-agnostic, it cannot enforce the MLPerf requirements, for example the target percentiles and latencies. Those limits are injected from mlperf.conf and user.conf. The justification for making it mandatory appears in the founding ISCA 2020 paper: it establishes a clear boundary between the components that belong to the submitter and those that belong to MLPerf, and it measures the performance of the complete system instead of that of a single piece.

Duration, query count and early stopping

The minimum duration is 600,000 ms in all four scenarios. The minimum number of queries comes from a confidence interval table that the rules publish:

Tail percentile	Confidence	Margin of error	Inferences	Rounded
90 %	99 %	0.50 %	23,886	24,576
95 %	99 %	0.25 %	50,425	57,344
97 %	99 %	0.15 %	85,811	90,112
99 %	99 %	0.05 %	262,742	270,336

The submission checker applies Server: 270336, SingleStream: 1024 and MultiStream: 662. For Offline, the minimum is set per benchmark: 24,576 on Llama 2 70B, 15,000 on Mixtral 8x7B, 13,368 on Llama 3.1 8B, 8,313 on Llama 3.1 405B, 4,388 on DeepSeek-R1, 1,633 on Whisper and 43 on 3D UNet.

The early stopping mechanism allows runs to be shortened while keeping the statistical guarantee. With tolerance d = 0 and confidence c = 0.99, the algorithm resolves by binary search the smallest number of queries below the latency threshold that satisfies the criterion, given the number of queries observed above it.

LoadGen’s modes are SubmissionRun (accuracy followed by performance), AccuracyOnly, PerformanceOnly and FindPeakPerformance. The last, applicable only to Server, takes target_qps as a lower bound if it passes, estimates the upper bound at double and doubles it until failure, and then does a binary search. In performance mode LoadGen selects queries uniformly at random with replacement from a set of size QSL; in accuracy mode it uses a copy of the validation dataset, each sample exactly once. One accuracy run must be executed for every performance result submitted, and the same code in both modes.

The seeds are announced four weeks before the deadline and the mandatory generator is Mersenne Twister 19937. They change every round: in v5.1, qsl_rng_seed = 1780908523862526354; in v5.0 it was 6023615788873153749.

The compliance tests

They are activated by placing an audit.config file in the working directory, whose parameters override those of mlperf.conf and user.conf. Their logs are mandatory in the submission package. The declared purpose is to detect anomalies, not to diagnose their cause.

For the LLM benchmarks the required test is TEST06, specifically against the end-of-sequence token exploit. It runs 100 samples and demands three conditions: that the first token reported separately matches the first token of the model’s output, that the output ends with zero or a single EOS token, and that the number of tokens reported matches the real length of the generated sequence.

Divisions, categories and what can be touched

The three divisions

Closed: requires preprocessing, postprocessing and model equivalent to the reference implementation. It allows calibration for quantisation and does not allow retraining. It is the only one that can use the MLPerf name unqualified.
Open: allows arbitrary preprocessing, postprocessing and model, including retraining. The accuracy, latency and scenario constraints do not apply: instead, the accuracy obtained and the latency constraints under which the performance was obtained must be reported. The model may have any origin, be quantised in any way and be sparsified in any way. It must use the same validation dataset as the corresponding Closed and use all of it.
Network: inherits all the Closed requirements, applies only to Datacenter, and separates the LoadGen node from the system under test through a fabric. The submitter’s dispatch library cannot preprocess, postprocess, batch, pad or cache. Bus interconnects are forbidden by name (PCIe, CXL, CCIX, HyperTransport, NVLink, QPI, UPI, ICI) and only Ethernet, IEEE 802.11, InfiniBand and 3GPP are admitted, with a requirement to work chassis to chassis at more than ten metres.

The availability categories

Category	Hardware	Software
Available in cloud	Available for cloud rental	Available
Available on premise	Available for purchase	Available
Preview	Must be available for the next submission, or the one after 140 days, whichever is longer	Available except what is needed for substantially new hardware
RDI	Does not meet the above	Does not meet the above

Available demands four cumulative conditions: price available, having been rented or shipped to at least one third party, public evidence of availability and reasonable availability to additional third parties on the submission date. A Preview result obliges a resubmission as Available with equal or better performance, tolerating up to 2 % degradation from noise; if there is no resubmission, the Preview result is marked invalid. RDI components cannot be submitted as Available until the cycle after the next one, or 221 days, whichever is longer.

What Closed allows and forbids

MLPerf delivers the weights in fp16 or fp32. The central rule is that the submitter may perform purely mathematical and reproducible quantisation, using only the calibration data and the tensors of the delivered model, to any numerical format that reaches the required quality, and that the method must be described publicly at a level that allows it to be reproduced. The test against back doors is elegant: the description of the quantisation method must be much smaller than the non-zero weights it produces.

Allowed, among other things: any framework or runtime, arbitrary data layout, varying the matrix multiplication algorithm, mathematically equivalent transformations, polynomial approximations of transcendentals, processing queries out of order within what the scenario admits, replacing dense operations with mathematically equivalent sparse operations, hand-picking different precisions per operation, fusing and unfusing, dynamic batching, and mixtures of experts combining weights with different quantisation.

Forbidden: replacing or supplementing weights wholesale, discarding non-zero weights, including pruning, caching queries or responses, coalescing identical queries, modifying weights during the timed portion, quantisation algorithms of size comparable to the weights they produce, hardcoding the total number of queries, using knowledge of the LoadGen implementation to predict peaks or troughs in the Server scenario, changing the number of search beams, and incorporating statistics from the performance or accuracy sets.

Two clarifications that matter to anyone serving LLMs in production: the KV cache is allowed just as in the reference model provided it is not applied across queries; PagedAttention is admitted if the block is reused only within the batch; continuous or dynamic batching is allowed; and KV cache entries are treated as activations for quantisation purposes, and cannot be pruned. Speculative decoding is only admitted in the benchmark and scenario combinations explicitly listed, with the reference head at the same precision, and implementations that artificially manipulate the acceptance rate are forbidden.

The SLOs almost nobody cites: TTFT and TPOT

This is the point where an MLPerf table is most often misread. In the LLM benchmarks of the Server scenario, the constraint is not a request latency: the rules set target_latency = 0 and activate use_token_latencies = 1, so that the gate is formed by two simultaneous metrics, both evaluated at the 99th percentile:

TTFT (time to first token): latency of the first token.
TPOT (time per output token): mean interval between all generated tokens.

The figures are in mlperf.conf and change per round and per benchmark:

Benchmark (Server scenario)	TTFT	TPOT	Round
`llama2-70b`	2000 ms	200 ms	v5.0, v5.1, v6.x
`llama2-70b-interactive`	450 ms	40 ms	v5.1, v6.x
`mixtral-8x7b`	2000 ms	200 ms	v5.0, v5.1, v6.x
`llama3_1-405b`	6000 ms	175 ms	v5.0, v5.1, v6.x
`llama3_1-405b-interactive`	4500 ms	80 ms	v6.x
`llama3_1-8b`	2000 ms	100 ms	v5.1, v6.x
`llama3_1-8b-interactive`	500 ms	30 ms	v6.x
`deepseek-r1`	2000 ms	80 ms	v5.1, v6.x
`deepseek-r1-interactive`	1500 ms	15 ms	v6.x
`gpt-oss-120b`	3000 ms	80 ms	v6.x

A TPOT of 40 ms is equivalent to 25 tokens per second per user, which is the order of magnitude of a conversational interface; one of 200 ms is equivalent to 5 tokens per second, which is a batch use case dressed up as a service. Two results for the same model under the Server and Interactive labels are not the same measurement. Two documented discrepancies between primary sources remain: the llama2-70b-interactive variant does not appear in the mlperf.conf tagged as v5.0 despite having been announced in that round, and in the v6.x cycle the TPOT of gpt-oss-120b-interactive appears as 15 ms in mlperf.conf and as 20 ms in the text of the rules.

For contrast, the constraints of the non-LLM benchmarks in the Server scenario are conventional request latencies: ResNet-50 15 ms, RetinaNet 100 ms, BERT 130 ms, DLRMv2 60 ms, DLRMv3 80 ms, RNN-T 1000 ms, GPT-J and Stable Diffusion XL 20,000 ms.

Quality: the 99 % and the token count

The quality targets are expressed relative to the reference model, never in absolute terms, and each benchmark demands one variant, the other or both. In the current Datacenter suite, 3D UNet, Llama 3.1 8B and Whisper ask for 99 % and 99.9 % of FP32, which generates two distinct results; Llama 2 70B asks only for 99.9 %; Llama 3.1 405B, Mixtral, DeepSeek-R1, GPT-OSS-120B, Qwen3-VL, RGAT and WAN-2.2 ask only for 99 %; and E2E-RAG is the exception at 97 %. In the checker this materialises as separate benchmarks (bert-99 and bert-99.9), each with its computed numerical threshold.

On top of that metric there is a second LLM-specific constraint that is cited very rarely: the generation length is audited as well.

Benchmark	Length constraint
Llama 2 70B	tokens per sample > 90 % of 294.45
Llama 3.1 405B	tokens per sample between 90 % and 110 % of 684.68
Mixtral 8x7B	tokens per sample between 90 % and 110 % of 144.84
Llama 3.1 8B	total generated length > 90 % of 8,167,644

The FAQ settles the obvious shortcut: reducing the maximum output length below the reference is not allowed, and truncating tokens to improve performance or reach the accuracy target is not permitted. The inference parameters are fixed in Closed (max_new_tokens = 1024 on Llama 2 70B, 20000 on Llama 3.1 405B and DeepSeek-R1), as is the decoding algorithm: greedy search in the Llama family, Mixtral and DeepSeek-R1; sampling with temperature = 1.0 and top_p = 1.0 in GPT-OSS-120B and Qwen3-VL. Accuracy is reported to five significant figures with round-half-to-even.

The current Datacenter suite

Area	Task	Model	Dataset	QSL	Required quality
Vision	Medical segmentation	3D UNet	KiTS 2019	42	99 % and 99.9 % of FP32 (DICE 0.86330)
Language	Summarisation	Llama 3.1 8B	CNN DailyMail v3.0.0	13,368	99 % and 99.9 % of FP32
Language	Question answering	Llama 2 70B	OpenOrca	24,576	99.9 % of FP32
Language	Text generation	Llama 3.1 405B	LongBench, Ruler, GovReport	8,313	99 % of FP16
Language	QA, maths and code	Mixtral 8x7B	OpenOrca, GSM8K, MBXP	15,000	99 % of FP16
Language	Reasoning	DeepSeek-R1	`mlperf_deepseek_r1`	4,388	99 % of FP16 (exact match 81.9132 %)
Language	QA, maths and code	GPT-OSS-120B	AIME25, GPQA Diamond, LiveCodeBench v6	6,396	99 % of 83.13 %
Vision	Vision-language model	Qwen3-VL-235B-A22B	Shopify product catalogue	48,289	99 % of BF16 (hierarchical F1 0.7824)
Commerce	Recommendation	DLRMv3	Synthetic Streaming 100B	34,996	99.9 % of FP32
Generative	Text to video	WAN-2.2-T2V-A14B	VBench prompts	248	99 % of BF16 (VBench 69.7752)
Graphs	Node classification	RGAT	IGBH	788,379	99 % of FP32 (72.86 %)
Audio	Speech to text	Whisper	LibriSpeech	1,633	99 % and 99.9 % of FP32 (WER 2.0671 %)
Language	End-to-end RAG	E2E-RAG	FRAMES	824	97 % of FP32

The E2E-RAG benchmark deserves attention from anyone designing a RAG platform, because it fixes the whole pipeline: a FAISS HNSW index with mandatory parameters M = 32, efConstruction = 200 and efSearch = 100; e5-base-v2 embeddings; ColBERTv2 reranking; generation and query rewriting with GPT-OSS-120B; evaluation with Llama 3.1 8B as judge; and a maximum of 5 retrieval iterations, mandatory for all submitters. Model loading is not timed; building the vector database is.

The definition of sample changes per model and is a frequent source of confusion: in the LLMs it is a sequence, in DLRMv3 it is a request with a user history and 2048 candidates, in WAN-2.2 it is a pair of positive and negative prompts, and in PointPainting it is five images and a lidar point cloud.

How to read a results row

The columns of the official table in the Closed division are Submitter, Software, System, Benchmark Results, Processor/Count, Details, Accelerator/Count and Code. The Open division adds Model Used and Notes. Rows with power measurement add System Power in Server and Offline, or Energy Per Stream in the stream scenarios. Any citation must use the identifier in the format major-version.minor-version.entry.benchmark, of the style 5.1-0053, and carry a footnote with suite, version, division, benchmark, scenario, date and source.

Per-accelerator is not an official metric

There is no per-accelerator column. It is a derived metric, and the MLCommons messaging guidelines are blunt: any comparison based on a different or derived metric, such as power, cost, model size or accuracy, must make the basis of comparison clear in the text and in a footnote, and secondary and derived metrics cannot be presented as official or verified MLPerf metrics. NVIDIA itself states this with the standard formula in its round blogs: per-GPU performance is not a primary MLPerf Inference metric and is computed by dividing the reported throughput by the number of reported accelerators. The same applies to combining results from several benchmarks: MLCommons allows it but does not endorse it, and the composite cannot be presented as an official result.

The comparison rules

MLPerf results can only be compared with compatible MLPerf results: same benchmark, same scenario and compatible versions per the normative compatibility table.
MLPerf results cannot be compared with non-MLPerf results.
When comparing, any difference in version, division, category, verification status, scenario or number of chips must be clearly identified. When comparing Open with Closed, it must be identified in what sense the Open result would not qualify as Closed.
Submitters cannot publish results of a version before its official date; non-submitters must wait two weeks from that date.
The sanctions regime goes as far as barring an infringer from submitting results in the future and marking their results as non-compliant permanently in the database, with a deadline of three working days to withdraw the infringing content.

How much audit sits behind a row

In each round up to two submissions are audited: one chosen at random from all of them and zero or one chosen by the review committee. And there is a constraint that conditions the reading of the whole table: only Available submissions in the Closed division are auditable. Preview, RDI and Open fall outside the process. There is an exemption from the random audit if the system is equivalent to another already audited and neither the aggregate performance nor the per-accelerator performance differ by more than 10 % from those of the previous audit. The deadlines are 28 days to select an auditor, 30 for the report after signing the confidentiality agreements and around 90 days for the complete process, with two days of hardware access for the auditor. The rules close with three sentences that define the standard of proof: results that cannot be replicated are not valid results, benchmark detection is not allowed and input-based optimisation is not allowed either.

The recent rounds, in figures

Round	Organisations	Performance results	Power results	New benchmarks
v4.0	23	more than 8,500	900	2
v4.1	22	964	31	1
v5.0	23	17,457	n/a	4
v5.1	27	n/a	2	3
v6.0	24	n/a	n/a	5 of 11 Datacenter tests

These counts do not form a time series: the counting criteria differ between press releases and none documents the criterion, to the point that v4.0 and v4.1 differ by a factor close to nine.

The published performance jumps are comparable within each announcement. Between v4.0 and v5.0, twelve months apart, the number of Llama 2 70B submissions multiplied by 2.5, the median score doubled and the best score was 3.3 times faster. Between v5.0 and v5.1, in six months, the best systems improved by up to 50 % in some scenarios. Round v6.0 changed axis and its press release talks about scale rather than speed: 30 % more multi-node submissions, 10 % of the systems with more than ten nodes against 2 % in the previous round, and a system larger than 72 nodes and 288 accelerators, quadrupling the previous maximum node count.

Hardware that debuted in each round: MI300X, TPU v6e, Xeon Granite Rapids and B200 in v4.1; MI325X, Xeon 6980P, GB200 and Jetson AGX Thor in v5.0; MI355X, Intel Arc Pro B60, GB300 and RTX Pro 6000 Blackwell Server Edition in v5.1.

What MLPerf Inference does not measure

Cost. There is no cost or TCO metric, and cost is explicitly classified as a derived metric that cannot be presented as official.
Energy, except as an optional extension. MLPerf Power is regulated in a separate document, requires measuring at the wall with the power group’s workflow and PTDaemon, and forbids any other method. Its real adoption is marginal: 2 power submissions in v5.1 against 27 participating organisations. In addition, MLCommons bounds what that figure means: the measured power is only valid for the benchmark it accompanies, and any other reference to power, such as a TDP setting or a power supply rating, is neither measured nor validated by MLCommons. The messaging rule forbids submitters from publishing per-watt normalised comparisons using any proxy other than the measured power.
Configurations not optimised by the vendor. The bias is acknowledged in the audit rule itself, which contemplates submissions whose performance is not consistent with the known characteristics of the hardware, or where the committee lacks visibility into how it was achieved, or where hardware and software are not reasonably available to the public.
Barrier to entry. Only MLCommons members and test partners can submit results for review. Anyone measuring with the same code without submitting must label every figure as unverified with the corresponding legend.

Usage flow for an on-premise platform

MLPerf Inference is useful in a purchasing decision if it is used for what it is, a test bench with written rules, and not as a ranking.

Filter by division and category before looking at any number: keep Closed and Available. A Preview or RDI result describes hardware that cannot yet be bought, and neither of the two is auditable.
Choose the scenario that matches the load pattern: Server or Interactive if there are users waiting, Offline if the work is batch. An Offline result does not authorise promising a latency.
Read the SLO before the throughput: in any LLM benchmark, the number of queries per second only means something alongside its TTFT/TPOT pair. The same machine publishes very different figures under Server and under Interactive.
Check the quality variant: 99 % and 99.9 % are different results for the same model, and the second usually costs performance.
Translate to your own configuration carefully: dividing by the number of accelerators is a derived metric, not an official one, and it has to be declared as such.
Close the gap MLPerf leaves open with your own measurement: cost, energy and behaviour under real traffic. That is exactly the function of the reproducible harness and of the track B bench tools, and the reason no platform decision stands on a public table alone.

Cross-links from the benchmarking track

B1/B2 — LLM inference benchmarking: frameworks, metrics and state of the art: the TTFT, TPOT and goodput metrics outside the MLPerf corset, and which tool measures each one.
B2 — LLM benchmark tool catalogue: the tools with which you reproduce at home what MLPerf standardises.
B3 — GuideLLM and SLO validation under load: the practical equivalent of LoadGen’s binary search in the Server scenario.
B6 — Measurement bias and reproducibility: why LoadGen exists, told from the side of what happens when there is none.
B8 — Serving engines compared on the Pareto frontier: the comparison MLPerf does not make, with engines instead of systems.
C4 — MLPerf Power: the energy extension of this same machinery, with its certified measurement at the wall.

Sources

MLCommons, MLPerf Inference Working Group — https://mlcommons.org/working-groups/benchmarks/inference/
MLCommons, MLPerf Inference Rules (inference_rules.adoc) — https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc
MLCommons, MLPerf Inference Power Measurement (power_measurement.adoc) — https://github.com/mlcommons/inference_policies/blob/master/power_measurement.adoc
MLCommons, MLPerf Submission Rules (submission_rules.adoc) — https://github.com/mlcommons/policies/blob/master/submission_rules.adoc
MLCommons, MLPerf Results Messaging Guidelines — https://github.com/mlcommons/policies/blob/master/MLPerf_Results_Messaging_Guidelines.adoc
MLCommons, LoadGen README — https://github.com/mlcommons/inference/blob/master/loadgen/README.md
MLCommons, loadgen/test_settings.h — https://github.com/mlcommons/inference/blob/master/loadgen/test_settings.h
MLCommons, loadgen/mlperf.conf (master) — https://github.com/mlcommons/inference/blob/master/loadgen/mlperf.conf
MLCommons, loadgen/mlperf.conf (tag v5.1) — https://github.com/mlcommons/inference/blob/v5.1/loadgen/mlperf.conf
MLCommons, loadgen/mlperf.conf (tag v5.0) — https://github.com/mlcommons/inference/blob/v5.0/loadgen/mlperf.conf
MLCommons, tools/submission/submission_checker.py (tag v5.0) — https://github.com/mlcommons/inference/blob/v5.0/tools/submission/submission_checker.py
MLCommons, compliance/nvidia TEST06 README (tag v5.1) — https://github.com/mlcommons/inference/blob/v5.1/compliance/nvidia/TEST06/README.md
MLCommons, Benchmark results: Inference Datacenter — https://mlcommons.org/benchmarks/inference-datacenter/
Reddi et al., MLPerf Inference Benchmark, ISCA 2020 — https://arxiv.org/abs/1911.02549
MLCommons, MLPerf Inference v5.0 Benchmark Results (abril 2025) — https://www.businesswire.com/news/home/20250402313932/en/MLCommons-Releases-New-MLPerf-Inference-v5.0-Benchmark-Results
MLCommons, MLPerf Inference v5.1 Benchmark Results (septiembre 2025) — https://www.globenewswire.com/news-release/2025/09/09/3147136/0/en/MLCommons-Releases-New-MLPerf-Inference-v5-1-Benchmark-Results.html
MLCommons, MLPerf Inference v6.0 Benchmark Results (abril 2026) — https://www.globenewswire.com/news-release/2026/04/01/3266801/0/en/MLCommons-Releases-New-MLPerf-Inference-v6-0-Benchmark-Results.html
MLCommons, MLPerf Inference v4.1 Benchmark Results (agosto 2024) — https://www.businesswire.com/news/home/20240828886616/en/New-MLPerf-Inference-v4.1-Benchmark-Results-Highlight-Rapid-Hardware-and-Software-Innovations-in-Generative-AI-Systems
NVIDIA Developer Blog, NVIDIA Blackwell delivers massive performance leaps in MLPerf Inference v5.0 — https://developer.nvidia.com/blog/nvidia-blackwell-delivers-massive-performance-leaps-in-mlperf-inference-v5-0/
NVIDIA Developer Blog, NVIDIA Blackwell Ultra sets new inference records in MLPerf debut — https://developer.nvidia.com/blog/nvidia-blackwell-ultra-sets-new-inference-records-in-mlperf-debut/
MLCommons, MLPerf Automotive (octubre 2025) — https://arxiv.org/html/2510.27065v1
Tschand et al., MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts — https://arxiv.org/abs/2410.12032

Capacity planning for on-premise LLM inference: how to size GPUs from an SLO

Mon, 01 Jun 2026 15:00:00 +0200

This post complements the ones on KV cache (the piece that dominates the VRAM budget), Continuous batching (what defines the effective utilisation of the compute) and Seven layers of the stack (the pieces the sizing takes for granted). Before reading this one, make sure your team has written down the SLOs it is going to chase; without that input the calculation is not defensible.

TL;DR

LLM inference capacity planning does not answer “how many tokens per second does a GPU give” — that question has no universal answer, because throughput depends on concurrency, the prefill/decode split, context length, the inference engine and quantisation. The right question has three inputs (SLO: TTFT P95, TPOT P95, sustained RPS), a hardware reference (GPU model, VRAM, HBM bandwidth, effective FLOPs) and a model (parameters, GQA/MHA/MoE architecture, weight format). The calculation resolves into two coupled budgets that cross each other. VRAM budget: from the GPU’s total you subtract model weights and activations, what remains is the KV cache budget, and from there you derive the maximum concurrency possible at the average context you expect. Time budget: the engine (vLLM, SGLang, TensorRT-LLM) has a tokens/s ceiling in decode given by HBM bandwidth and another in prefill given by useful FLOPs; from there you derive the expected TPOT and, dividing prefill_tokens by prefill throughput, the expected TTFT. Both budgets must meet the SLO simultaneously: whichever is tighter dictates the sizing. On the Llama 70B BF16 example with tensor parallel 4 on 4×H100 SXM, a single replica saturates at ~28 concurrent requests and delivers ~3,200 tokens/s of aggregate decode with a median TPOT of 35 ms; for 200 sustained RPS at a profile of 800 prompt tokens + 250 of output, you need between 4 and 5 replicas with a 25 % cushion over the observed peak. Quantisation (FP8 → INT4) divides VRAM cost and decode time by between 1.5 and 4×, but it degrades quality measurably — it is not assumed free, it is validated with evals. The five habitual traps: confusing the mean with P95, ignoring the prefill/decode split of the real workload, sizing without head-room for retraining or rollback, forgetting that a GPU at 100 % SM util means nothing if the HBM is saturated, and not documenting the assumptions behind the calculation (a sizing with no written assumptions is a throwaway calculation).

You are here: DEPLOY (with one foot in OBSERVE)

Capacity planning is a piece with dual residency. It lives in DEPLOY because without a valid sizing you do not buy hardware or configure the inference engine. But its input is real observations: the distribution of prompt and output lengths, the prefill/decode mix of the workload, the real P95s already being seen in pre-production. Without that data the calculation is a napkin sketch, defensible only until the first customer arrives who does not fit the assumed average.

The analogy: the hotel with variable-sized rooms

Imagine a hotel where the rooms have no fixed size: each guest pays for the square metres they need, and the building’s floor plan reorganises itself dynamically to accommodate whoever arrives. Management wants to maximise occupancy, but it has two real constraints and one quality metric.

Constraint 1 — physical space. The floor has 1,000 m² in total. If a family comes in needing 200 m², that family occupies that surface and it cannot be handed to the next guest. The largest room limits how many simultaneous guests fit.

Constraint 2 — service staff. There are 10 receptionists. Each can handle the check-in of one guest every two minutes. When 60 guests arrive within an hour, the last ones wait in a queue; the time from walking into reception to receiving their key depends on how many are ahead of them.

Quality metric — a promise about time. The brochure says “check-in in under 15 minutes”. If too many guests arrive at once, that promise breaks even if there is physical space free.

The physical space is the GPU’s VRAM. Each room is a request with its KV cache (larger the longer the conversation). The receptionists are the compute units (Streaming Multiprocessors + Tensor Cores). The check-in is the prefill phase; the nights the guest spends afterwards are the decode steps. The 15-minute promise is the TTFT P95 SLO.

The hotel’s capacity planning is exactly this: given the expected guest profile (how many arrive per hour, how much space they ask for on average, how many minutes of waiting they tolerate), work out how many floors and how many receptionists are needed. It is not done by estimating “rooms per hour” in the abstract; it is done by crossing the two budgets with the promise about time. The analogy holds up the calculation all the way to the end.

The three SLO inputs

Before putting a single number on the sheet, you have to write down the three dimensions of the SLO. Without this, the calculation is aesthetics, not engineering.

TTFT P95 (Time-To-First-Token). The time from the client sending the request until it receives the first token. It is dominated by the prefill phase (processing the whole prompt in one go) plus the scheduler queue. For conversational chat, a reasonable target sits between 0.5 and 2 seconds P95. For coding assistants with large prompts (5–10 K tokens of context), between 2 and 4 s P95. Below 500 ms you enter UX territory for voice-style conversations, but that demands serious architectural compromises.

TPOT P95 (Time-Per-Output-Token). The time between consecutive tokens during decode. It dominates the “perceived fluency” of streaming. Above 80 ms/token the human reader perceives pauses; below 30 ms/token the output flows faster than it can be read. The usual industry target: 40–60 ms P95.

Sustained RPS meeting the SLO. The throughput the system must support without violating TTFT or TPOT. This is DistServe’s key metric, called goodput (see Continuous batching). “200 RPS peak” is not the same as “200 RPS with TTFT P95 ≤ 1.5 s”. Without the SLO condition, the RPS number means nothing.

These three dimensions come with a workload profile: the distribution of prompt and output lengths. Medians are not enough; you need P50, P95, P99. A badly measured profile is the main reason sizings fail.

The central formula: two budgets that cross

The calculation resolves into two independent sums that are then crossed. The smaller of the two rules.

VRAM budget

For a GPU with total VRAM $V$, the space available for KV cache is:

$$V_{\text{kv}} = V - V_{\text{model}} - V_{\text{activations}} - V_{\text{overhead}}$$

where:

$V_{\text{model}}$ is the size of the weights: for a model of $P$ parameters in a format of $b$ bytes/parameter, $V_{\text{model}} = P \cdot b$. Llama 70B BF16 = $70 \times 10^9 \times 2 = 140$ GB. Under tensor parallel TP=4, each GPU carries $140 / 4 = 35$ GB.
$V_{\text{activations}}$ are the intermediate buffers of the forward pass. For vLLM with a reasonable batch, between 2 and 6 GB per GPU depending on batch size and maximum length.
$V_{\text{overhead}}$ is the CUDA context, NCCL buffers, the PagedAttention pool, reserved paged blocks. 2–4 GB typical.

The KV cache budget per GPU is what is left over. For an H100 SXM 80 GB with Llama 70B TP=4 BF16:

$$V_{\text{kv}} = 80 - 35 - 4 - 3 = 38 \text{ GB per GPU} = 152 \text{ GB aggregated over TP=4}$$

The KV cache cost per token for a model with $L$ layers, $H_{\text{kv}}$ KV heads (GQA), per-head dimension $d_h$, in a format of $b$ bytes:

$$\text{kv per token} = 2 \cdot L \cdot H_{\text{kv}} \cdot d_h \cdot b$$

The factor of 2 is because both K and V are stored. For Llama 70B (L=80, $H_{\text{kv}}$=8 with GQA, $d_h$=128, BF16 = 2 bytes):

$$\text{kv per token} = 2 \cdot 80 \cdot 8 \cdot 128 \cdot 2 = 327,680 \text{ bytes} = 320 \text{ KB/token}$$

And the maximum concurrency at average context $C$:

$$N_{\text{max}} = \frac{V_{\text{kv}}}{C \cdot \text{kv per token}}$$

With an aggregated $V_{\text{kv}}$ of 152 GB and an average context of 1,500 tokens (800 prompt + 700 generated at the worst moment of the conversation):

$$N_{\text{max}} = \frac{152 \times 10^9}{1,500 \cdot 320 \times 10^3} \approx 316 \text{ concurrent requests}$$

This is the physical ceiling of concurrency for that replica. It is not what you are going to use; it is what you cannot exceed without OOM. The operational number sits well below it (head-room for spikes).

Time budget

Two sub-calculations come in here: decode (memory-bound) and prefill (compute-bound).

Decode TPOT. For every token generated, the model weights (the ones relevant to that request) have to be walked through and the accumulated KV cache has to be read. The bottleneck is HBM bandwidth. For a GPU with bandwidth $B$ GB/s and a model of $V_{\text{model per gpu}}$ GB of weights:

$$\text{tpot}_{\text{theoretical}} \approx \frac{V_{\text{model per gpu}}}{B}$$

For an H100 SXM with HBM3 at 3.35 TB/s and Llama 70B TP=4 BF16 (35 GB/GPU):

$$\text{tpot}_{\text{theoretical}} \approx \frac{35}{3,350} \approx 10.4 \text{ ms/token}$$

This is the best theoretical case with batch=1 and 100 % HBM efficiency. In practice vLLM on H100 with Llama 70B TP=4 reaches 12–18 ms/token at low batch and 30–45 ms/token at high batch (at concurrency 32, tokens compete for the shared HBM). The defensible operational number: 35 ms/token at concurrency 24–32.

Prefill throughput. Prefill processes N prompt tokens in a single forward pass. It is compute-bound: the bottleneck is FLOPs. For an H100 SXM with 989 sustained BF16 TFLOPs and Llama 70B (each forward pass costs roughly $2 \cdot P \cdot N$ FLOPs per sequence of length N):

$$\text{prefill tps} = \frac{4 \cdot \text{TFLOPs} \cdot \eta}{2 \cdot P} = \frac{4 \cdot 989 \times 10^{12} \cdot 0.5}{2 \cdot 70 \times 10^9} \approx 14,000 \text{ tokens/s}$$

(the factor of 4 is the GPUs in TP, $\eta$ is real efficiency between 0.4 and 0.6 on H100). An 800-token prompt takes this long in prefill:

$$\text{prefill time} = \frac{800}{14,000} \approx 57 \text{ ms}$$

Adding a typical queue of 100–300 ms at high concurrency, TTFT P95 ≈ 350–500 ms for that profile. Well below the 1.5 s target; there is margin.

The crossing

The real operational concurrency $N_{\text{op}}$ is the minimum of the VRAM ceiling, the concurrency at which TPOT starts to degrade past the SLO, and the concurrency at which TTFT starts to degrade past the SLO (prefill queue). For the example:

VRAM ceiling: 316.
TPOT degrades to 80 ms (SLO) at around concurrency ~80–100 (measured empirically with a benchmark, not a closed formula).
TTFT degrades to 1.5 s at around concurrency ~40–60 because of the prefill queue.

The replica’s operational concurrency is ~50. Applying 25 % head-room for spikes and rebalancing, target concurrency per replica ≈ 35–40.

Step-by-step worksheet: Llama 70B BF16 on 4×H100 SXM

The exercise’s input:

SLO: TTFT P95 ≤ 1.5 s; TPOT P95 ≤ 60 ms; 200 sustained RPS.
Workload: prompt P50=600, P95=1,200, P99=2,500; output P50=180, P95=500, P99=900. Average prompt 800, output 250.
Generic hardware: 4×H100 SXM 80 GB with NVLink, vLLM v1 engine, tensor parallel 4, BF16.

Step 1 — VRAM per GPU. Weights 35 GB, activations 4 GB, overhead 3 GB → KV budget 38 GB/GPU = 152 GB aggregated. KV/token for Llama 70B GQA = 320 KB. Ceiling of live tokens in cache: $152 \times 10^9 / 320 \times 10^3 \approx 475,000$ tokens. At the operational average context (800 prompt + 200 already generated = 1,000 live tokens per request), concurrency ceiling $\approx 475$.

Step 2 — average duration of a request. Prefill 800 tokens / 14,000 tps = 57 ms. Decode 250 tokens × 35 ms/token = 8,750 ms. Total $\approx 8.8$ s per request.

Step 3 — replica throughput. If the replica sustains operational concurrency 40 and each request lasts 8.8 s, the replica delivers roughly $40 / 8.8 \approx 4.5$ requests/s in steady state.

Step 4 — number of replicas. For a 200 RPS target: $200 / 4.5 \approx 45$ replicas. That is 45 × 4 = 180 GPUs. Too much: this sizing does not work because the cost per request is high.

Step 5 — review the levers. Before buying more hardware, there are three levers to explore, in this order:

Quantisation. Dropping to FP8 reduces weights to 17.5 GB/GPU (leaving more VRAM for KV cache → more concurrency), roughly doubles decode tokens/s (HBM saturated by half), and degrades MMLU quality typically by 0.5–1.5 points on models like Llama 70B. Rewriting the calculation in FP8: TPOT drops to ~18 ms, total time per request to 4.7 s, RPS per replica rises to ~8.5, replicas needed ≈ 24, equivalent to 96 GPUs.
Speculative decoding. With a small drafter and 60–70 % acceptance, effective TPOT falls 30–40 %. RPS per replica rises to ~12, replicas ≈ 17 = 68 GPUs.
Disaggregated serving. Separating prefill workers and decode workers lets each be scaled to the real mix of the workload (see Disaggregated serving). It usually shaves off another 20–40 % under asymmetric workloads.

Step 6 — recommended sizing. For the example, with FP8 + speculative decoding and 25 % head-room: 20 vLLM replicas at TP=4 over 80 H100 SXM. If the team does not want to depend on aggressive quantisation (pure BF16 for maximum fidelity), the calculation rises to 30 replicas = 120 GPUs and forces a renegotiation of the SLO or the budget.

Step 7 — write down the assumptions. This is the part no valid sizing skips. In the team’s repo, next to the calculation:

# sizing/llama70b-prod.yaml
fecha: 2026-06-01
slo:
 ttft_p95_ms: 1500
 tpot_p95_ms: 60
 rps_target: 200
workload:
 prompt_tokens_p50: 600
 prompt_tokens_p95: 1200
 output_tokens_p50: 180
 output_tokens_p95: 500
 asunto: chat productivo con RAG ligero
modelo:
 arquitectura: llama-70b-instruct
 formato_pesos: fp8
 motor: vllm-v1
hardware:
 gpu: H100-SXM-80GB
 topologia: TP=4 con NVLink intra-nodo
 red_inter_replica: 25 GbE
optimizaciones:
 - paged_attention
 - chunked_prefill
 - speculative_decoding (drafter llama-1.1b, aceptación esperada 65%)
asunciones_criticas:
 - utilizacion_hbm_eficiente: 0.55
 - head_room_pico_sobre_p95: 0.25
 - aceptacion_speculative_min: 0.55
plan_validacion:
 - benchmark vllm bench serve antes de procurement
 - canary 10% durante 7 días post-deploy

Without this YAML, the calculation is not reproducible a month later.

MoE case: Mixtral 8×22B (~141 B total, 39 B active)

MoEs change the calculation along one key dimension: the total weights are large but the active weights per token are small. For Mixtral 8×22B with top-2 routing:

Weight VRAM: $141 \times 2 = 282$ GB BF16. With TP=4 → 70 GB/GPU. It does not fit in an H100 80 GB alongside KV cache + activations. You need TP=8 (~35 GB/GPU) or FP8 with TP=4 (~35 GB/GPU).
Decode TPOT: dominated by the weights read per token, which are $\sim 39 / 8 \cdot 2 \approx 9.75$ GB/GPU with TP=4 (one top-2 expert per token, divided across 4 GPUs). On an H100 with HBM at 3.35 TB/s, theoretical TPOT ≈ 3 ms/token. In practice, 10–20 ms at reasonable concurrency.
Prefill: similar to the dense model of the active weights, ~39 B FLOPs/token.

MoE sizing usually delivers more RPS per GPU than an equivalent dense model: the low cost per token compensates for the extra VRAM. See MoE inference for the detail of routing and why a high batch is decisive so that each expert sees enough tokens.

Sensitivity table: context and quantisation

For Llama 70B on 4×H100 SXM (TP=4), operational concurrency per replica with an SLO of TTFT 1.5 s / TPOT 60 ms:

Average context	BF16	FP8	INT4 (AWQ)
500 tokens	55	110	180
1,000 tokens	40	80	130
2,000 tokens	24	50	85
4,000 tokens	12	26	48
8,000 tokens	6	13	25

Approximate numbers from public vLLM benchmarks as of June 2026, with ±20 % variation depending on engine version and the headroom adopted. To validate on your own hardware: vllm bench serve with your profile of real prompts.

The five habitual traps

Trap 1 — confusing the mean with P95. Average throughput over an hour may be 50 RPS while the 5-minute peak reaches 180 RPS. Sizing against the mean guarantees breaking the SLO at every peak. Rule: size against the hourly P95, with 20–30 % head-room over P95.

Trap 2 — not measuring the real prefill/decode split. A “RAG with short answers” workload spends 70–80 % of GPU time in prefill; a “writing assistant generating essays” spends 80 % in decode. The useful optimisations (chunked prefill vs speculative decoding) change radically. Without measuring it, you buy badly balanced hardware.

Trap 3 — sizing with no head-room for retraining or rollback. The production cluster is not just the inference engine: there is a re-embedding batch when the embeddings model changes, continuous canary evals (see Canary, blue-green and shadow), light fine-tuning, hot stand-by for rollback. Reserve 15–25 % of capacity for those non-negotiable workloads.

Trap 4 — “GPU at 100 % SM utilization” as a target. SM occupancy of 95 % with saturated HBM produces the same throughput as SM at 60 % with saturated HBM. The bottleneck in decode is the HBM. Optimising for “GPU usage 100 %” without looking at HBM utilisation and arithmetic intensity makes you spend more on GPU without gaining throughput. See GPU observability for LLM inference for which metrics to actually watch.

Trap 5 — not documenting the assumptions. A sizing with no reproducible YAML (workload, model, engine, head-room, critical assumptions) leaves the team with no way of knowing what changed when the cluster stops meeting the SLO six months later. Documenting is cheap; losing a quarter to debugging is not.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB with intra-node NVLink and 25 GbE between nodes, the recurring configurations in May 2026 are:

Model	Format	TP	Replicas that fit	Typical RPS per node (ctx 1K)
Llama 8B	BF16	1	4 (one per GPU)	240–320
Llama 8B	FP8	1	4	450–600
Llama 70B	BF16	4	1	30–45
Llama 70B	FP8	4	1	60–90
Llama 70B	INT4 AWQ	2	2	90–130
Mixtral 8×22B	FP8	4	1	90–140
Qwen 72B	BF16	4	1	28–42

These numbers are orders of magnitude to start the conversation, not commitments. The definitive sizing is validated with vllm bench serve or genai-perf (NVIDIA) using the customer’s real prompt/output profile. The prefill/decode asymmetry of each case’s workload can move these numbers 30–50 % up or down.

For clusters of 8×H100 SXM (typical of DGX servers or equivalent replicas), the options open up to TP=8 for 405B-class models or multi-replica TP=2 for 70B models with greater density. The metric that decides is always the same: tokens meeting SLO per kW and per euro of amortised hardware.

How the sizing is validated before buying

The spreadsheet sizing is the first half. The second is the validation benchmark.

Stage 1 — back-of-the-envelope sizing. This post’s formulas applied to the SLO and the expected workload. Output: an approximate number of replicas and a topology.

Stage 2 — synthetic micro-benchmark. On a borrowed GPU or one rented by the day, bring up the engine with the chosen model and run vllm bench serve with prompts of representative lengths. Validate TPOT, prefill TPS and the concurrency ceiling. Calibrate the HBM efficiency factor ($\eta$) used in the formulas.

Stage 3 — load test with realistic traffic. Generate traffic following the real distribution of the customer’s workload (not Poisson, not constant: the real trace). Measure P50/P95/P99 of TTFT, TPOT, throughput. Confirm the head-room.

Stage 4 — canary in production. With the cluster sized, route 5–10 % of real traffic for 7–14 days before closing the procurement of additional hardware. See Canary, blue-green and shadow for the mechanics.

Jumping from Stage 1 to full procurement is the most frequent cause of a cluster that is 40 % oversized and 60 % undersized at the same time, in different regions of the workload. Four weeks of validation done well save four months of refactoring.

What we have not covered (upcoming articles)

The observability metrics that close the sizing loop in production, see GPU observability for LLM inference.
Autoscaling that adjusts replicas to the real traffic curve, see Autoscaling LLM on Kubernetes.
Detailed cost accounting per tenant (showback / chargeback) over the sized hardware.
Sizing for continuous fine-tuning (PEFT and light training) that shares a cluster with inference.

References

Kwon et al. — vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (SOSP 2023).
Zhong et al. — DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024).
Agrawal et al. — Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI 2024).
NVIDIA — H100 Tensor Core GPU Architecture Whitepaper (HBM3 memory, bandwidth, sustained FLOPs).
vLLM project — vllm bench serve reference (benchmarking CLI included in the repo).
NVIDIA — genai-perf (official tool for benchmarking LLM services).

Tpot on lo0 — Tech Blog

MLPerf Inference: how to read it, what LoadGen enforces and what comparability it offers

TL;DR

What it is and how it is organised

The four scenarios

LoadGen: what it measures and what it does not

Duration, query count and early stopping

The compliance tests

Divisions, categories and what can be touched

The three divisions

The availability categories

What Closed allows and forbids

The SLOs almost nobody cites: TTFT and TPOT

Quality: the 99 % and the token count

The current Datacenter suite

How to read a results row

Per-accelerator is not an official metric

The comparison rules

How much audit sits behind a row

The recent rounds, in figures

What MLPerf Inference does not measure

Usage flow for an on-premise platform

Cross-links from the benchmarking track

See also

Sources

Capacity planning for on-premise LLM inference: how to size GPUs from an SLO

TL;DR

You are here: DEPLOY (with one foot in OBSERVE)

The analogy: the hotel with variable-sized rooms

The three SLO inputs

The central formula: two budgets that cross

VRAM budget

Time budget

The crossing

Step-by-step worksheet: Llama 70B BF16 on 4×H100 SXM

MoE case: Mixtral 8×22B (~141 B total, 39 B active)

Sensitivity table: context and quantisation

The five habitual traps

Applied to typical on-premise hardware

How the sizing is validated before buying

What we have not covered (upcoming articles)

See also

References