Mlperf on lo0 — Tech Blog

MLPerf Inference: how to read it, what LoadGen enforces and what comparability it offers

Sat, 29 Aug 2026 09:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

MLPerf Inference is maintained by MLCommons on a six-monthly cadence. It imposes three things no home-made benchmark has: a mandatory load generator (LoadGen, compiled from a tagged revision and unmodified), model-equivalence rules that bound what can be optimised, and an audit process that reviews up to two submissions per round. The four scenarios (SingleStream, MultiStream, Server, Offline) have a minimum duration of 600 s and different metrics: Server reports the sustained Poisson rate and is the only one with a latency constraint. For the LLM benchmarks that constraint is not a request latency but two simultaneous SLOs at the 99th percentile: TTFT and TPOT, with per-model figures ranging from 2000 ms / 200 ms on Llama 2 70B to 450 ms / 40 ms on its Interactive variant. Quality is required relative to the reference model (99 % or 99.9 %) and the generation length is audited as well (Llama 2 70B: more than 90 % of 294.45 tokens per sample). Three reading rules sum up the real comparability: results are only compared within the same version, division, category and scenario; per-accelerator is not an official MLCommons metric, it is derived; and neither cost nor energy enters the primary metric, to the point that round v5.1 had 2 power submissions against 27 participating organisations.

What it is and how it is organised

MLPerf Inference is maintained by the MLCommons Association through the MLPerf Inference Working Group, whose participation is limited to MLCommons members and affiliates (MLPerf Inference Working Group). The declared cadence for publishing results is roughly six months.

Round	Results published
v4.0	27 March 2024
v4.1	28 August 2024
v5.0	2 April 2025
v5.1	9 September 2025
v6.0	1 April 2026
v6.1	in preparation as of this data sheet

The suite splits into two system categories: Datacenter and Edge, the latter defined in the rules as everything that is not datacenter (inference_rules.adoc). A Datacenter system has two requirements that do not apply in Edge:

Mandatory ECC on DRAM and HBM, active during all performance and accuracy runs. There is no requirement on SRAM.
Mandatory networking since round v3.0, with a minimum bandwidth computed from the achieved throughput. For Llama 3.1 405B the minimum ingress is throughput × 20000 × dtype_size bytes/s; for Llama 2 70B the minimum egress is throughput × 1024 × dtype_size.

Edge additionally allows inferred results: a MultiStream derived from SingleStream counts as 8 times the p99 latency; an Offline derived from MultiStream, 8000 divided by the mean latency in milliseconds.

The four scenarios

This is the normative table, transcribed from the rules:

Scenario	Query generation	Duration	Samples/query	Latency constraint	Tail latency	Reported metric
SingleStream	The next query is issued when the SUT completes the previous one	600 s	1	None	90 %	p90 latency with early stopping
Server / Interactive	LoadGen issues queries following a Poisson distribution	600 s	1	Benchmark-specific	99 %	Maximum Poisson parameter supported
Offline	All samples are delivered at the start in a single query	1 query, 600 s	At least 24,576	None	N/A	Measured throughput
MultiStream	The next query is issued when the SUT completes the previous one	600 s	8	None	99 %	p99 query latency with early stopping

Four operational clarifications that change how each number is interpreted:

In Server, LoadGen does not measure a latency: it runs a binary search over the target rate. For a given value it generates queries at that QPS with Poisson arrivals; if the run fails the latency gate, it lowers the value and repeats. The final published metric is, per the official FAQ, scheduled samples per second, not completed samples nor the target entered by the user.
In MultiStream, the latency of a query is the maximum of the latencies of its samples, and the 8 samples of each query are contiguous in the order in which they were loaded. The value multi_stream_samples_per_query = 8 is fixed in the official configuration.
In Offline there is no latency constraint at all. The rules themselves forbid techniques that only work in fixed-length experiments except in the Offline scenario, which is a normative acknowledgement that an Offline result says nothing about behaviour under an SLO.
The percentiles in the table are the lower theoretical bound for runs with very many queries. With early stopping, the percentile actually computed is somewhat above the nominal one, as a penalty for processing few queries.

The mandatory scenarios depend on the benchmark. In Datacenter, the summarisation, question answering, text generation and VLM tasks require (Server | Interactive), Offline, that is, the submitter chooses between Server and Interactive. Reasoning and recommendation require Server, Offline. Medical segmentation, node classification, speech-to-text and RAG require only Offline.

LoadGen: what it measures and what it does not

LoadGen is the module that generates the load and computes the metrics. Its use is mandatory for all submissions, and it must be compiled from a tagged and approved revision of the repository, unaltered; the README is explicit that local modifications of the C++ library are not accepted.

Its four declared responsibilities are generating the queries according to the scenario, tracking the latency of each query, validating the accuracy of the results and computing the final metrics. Latency is defined as the time from when LoadGen was scheduled to pass a query to the system under test until it receives the response.

Equally important is what it declares out of scope: LoadGen does not know the model, does not know the data formats, cannot score accuracy and does not know the scenario constraints of the MLPerf rules. The consequence is written in its own README: being model-agnostic, it cannot enforce the MLPerf requirements, for example the target percentiles and latencies. Those limits are injected from mlperf.conf and user.conf. The justification for making it mandatory appears in the founding ISCA 2020 paper: it establishes a clear boundary between the components that belong to the submitter and those that belong to MLPerf, and it measures the performance of the complete system instead of that of a single piece.

Duration, query count and early stopping

The minimum duration is 600,000 ms in all four scenarios. The minimum number of queries comes from a confidence interval table that the rules publish:

Tail percentile	Confidence	Margin of error	Inferences	Rounded
90 %	99 %	0.50 %	23,886	24,576
95 %	99 %	0.25 %	50,425	57,344
97 %	99 %	0.15 %	85,811	90,112
99 %	99 %	0.05 %	262,742	270,336

The submission checker applies Server: 270336, SingleStream: 1024 and MultiStream: 662. For Offline, the minimum is set per benchmark: 24,576 on Llama 2 70B, 15,000 on Mixtral 8x7B, 13,368 on Llama 3.1 8B, 8,313 on Llama 3.1 405B, 4,388 on DeepSeek-R1, 1,633 on Whisper and 43 on 3D UNet.

The early stopping mechanism allows runs to be shortened while keeping the statistical guarantee. With tolerance d = 0 and confidence c = 0.99, the algorithm resolves by binary search the smallest number of queries below the latency threshold that satisfies the criterion, given the number of queries observed above it.

LoadGen’s modes are SubmissionRun (accuracy followed by performance), AccuracyOnly, PerformanceOnly and FindPeakPerformance. The last, applicable only to Server, takes target_qps as a lower bound if it passes, estimates the upper bound at double and doubles it until failure, and then does a binary search. In performance mode LoadGen selects queries uniformly at random with replacement from a set of size QSL; in accuracy mode it uses a copy of the validation dataset, each sample exactly once. One accuracy run must be executed for every performance result submitted, and the same code in both modes.

The seeds are announced four weeks before the deadline and the mandatory generator is Mersenne Twister 19937. They change every round: in v5.1, qsl_rng_seed = 1780908523862526354; in v5.0 it was 6023615788873153749.

The compliance tests

They are activated by placing an audit.config file in the working directory, whose parameters override those of mlperf.conf and user.conf. Their logs are mandatory in the submission package. The declared purpose is to detect anomalies, not to diagnose their cause.

For the LLM benchmarks the required test is TEST06, specifically against the end-of-sequence token exploit. It runs 100 samples and demands three conditions: that the first token reported separately matches the first token of the model’s output, that the output ends with zero or a single EOS token, and that the number of tokens reported matches the real length of the generated sequence.

Divisions, categories and what can be touched

The three divisions

Closed: requires preprocessing, postprocessing and model equivalent to the reference implementation. It allows calibration for quantisation and does not allow retraining. It is the only one that can use the MLPerf name unqualified.
Open: allows arbitrary preprocessing, postprocessing and model, including retraining. The accuracy, latency and scenario constraints do not apply: instead, the accuracy obtained and the latency constraints under which the performance was obtained must be reported. The model may have any origin, be quantised in any way and be sparsified in any way. It must use the same validation dataset as the corresponding Closed and use all of it.
Network: inherits all the Closed requirements, applies only to Datacenter, and separates the LoadGen node from the system under test through a fabric. The submitter’s dispatch library cannot preprocess, postprocess, batch, pad or cache. Bus interconnects are forbidden by name (PCIe, CXL, CCIX, HyperTransport, NVLink, QPI, UPI, ICI) and only Ethernet, IEEE 802.11, InfiniBand and 3GPP are admitted, with a requirement to work chassis to chassis at more than ten metres.

The availability categories

Category	Hardware	Software
Available in cloud	Available for cloud rental	Available
Available on premise	Available for purchase	Available
Preview	Must be available for the next submission, or the one after 140 days, whichever is longer	Available except what is needed for substantially new hardware
RDI	Does not meet the above	Does not meet the above

Available demands four cumulative conditions: price available, having been rented or shipped to at least one third party, public evidence of availability and reasonable availability to additional third parties on the submission date. A Preview result obliges a resubmission as Available with equal or better performance, tolerating up to 2 % degradation from noise; if there is no resubmission, the Preview result is marked invalid. RDI components cannot be submitted as Available until the cycle after the next one, or 221 days, whichever is longer.

What Closed allows and forbids

MLPerf delivers the weights in fp16 or fp32. The central rule is that the submitter may perform purely mathematical and reproducible quantisation, using only the calibration data and the tensors of the delivered model, to any numerical format that reaches the required quality, and that the method must be described publicly at a level that allows it to be reproduced. The test against back doors is elegant: the description of the quantisation method must be much smaller than the non-zero weights it produces.

Allowed, among other things: any framework or runtime, arbitrary data layout, varying the matrix multiplication algorithm, mathematically equivalent transformations, polynomial approximations of transcendentals, processing queries out of order within what the scenario admits, replacing dense operations with mathematically equivalent sparse operations, hand-picking different precisions per operation, fusing and unfusing, dynamic batching, and mixtures of experts combining weights with different quantisation.

Forbidden: replacing or supplementing weights wholesale, discarding non-zero weights, including pruning, caching queries or responses, coalescing identical queries, modifying weights during the timed portion, quantisation algorithms of size comparable to the weights they produce, hardcoding the total number of queries, using knowledge of the LoadGen implementation to predict peaks or troughs in the Server scenario, changing the number of search beams, and incorporating statistics from the performance or accuracy sets.

Two clarifications that matter to anyone serving LLMs in production: the KV cache is allowed just as in the reference model provided it is not applied across queries; PagedAttention is admitted if the block is reused only within the batch; continuous or dynamic batching is allowed; and KV cache entries are treated as activations for quantisation purposes, and cannot be pruned. Speculative decoding is only admitted in the benchmark and scenario combinations explicitly listed, with the reference head at the same precision, and implementations that artificially manipulate the acceptance rate are forbidden.

The SLOs almost nobody cites: TTFT and TPOT

This is the point where an MLPerf table is most often misread. In the LLM benchmarks of the Server scenario, the constraint is not a request latency: the rules set target_latency = 0 and activate use_token_latencies = 1, so that the gate is formed by two simultaneous metrics, both evaluated at the 99th percentile:

TTFT (time to first token): latency of the first token.
TPOT (time per output token): mean interval between all generated tokens.

The figures are in mlperf.conf and change per round and per benchmark:

Benchmark (Server scenario)	TTFT	TPOT	Round
`llama2-70b`	2000 ms	200 ms	v5.0, v5.1, v6.x
`llama2-70b-interactive`	450 ms	40 ms	v5.1, v6.x
`mixtral-8x7b`	2000 ms	200 ms	v5.0, v5.1, v6.x
`llama3_1-405b`	6000 ms	175 ms	v5.0, v5.1, v6.x
`llama3_1-405b-interactive`	4500 ms	80 ms	v6.x
`llama3_1-8b`	2000 ms	100 ms	v5.1, v6.x
`llama3_1-8b-interactive`	500 ms	30 ms	v6.x
`deepseek-r1`	2000 ms	80 ms	v5.1, v6.x
`deepseek-r1-interactive`	1500 ms	15 ms	v6.x
`gpt-oss-120b`	3000 ms	80 ms	v6.x

A TPOT of 40 ms is equivalent to 25 tokens per second per user, which is the order of magnitude of a conversational interface; one of 200 ms is equivalent to 5 tokens per second, which is a batch use case dressed up as a service. Two results for the same model under the Server and Interactive labels are not the same measurement. Two documented discrepancies between primary sources remain: the llama2-70b-interactive variant does not appear in the mlperf.conf tagged as v5.0 despite having been announced in that round, and in the v6.x cycle the TPOT of gpt-oss-120b-interactive appears as 15 ms in mlperf.conf and as 20 ms in the text of the rules.

For contrast, the constraints of the non-LLM benchmarks in the Server scenario are conventional request latencies: ResNet-50 15 ms, RetinaNet 100 ms, BERT 130 ms, DLRMv2 60 ms, DLRMv3 80 ms, RNN-T 1000 ms, GPT-J and Stable Diffusion XL 20,000 ms.

Quality: the 99 % and the token count

The quality targets are expressed relative to the reference model, never in absolute terms, and each benchmark demands one variant, the other or both. In the current Datacenter suite, 3D UNet, Llama 3.1 8B and Whisper ask for 99 % and 99.9 % of FP32, which generates two distinct results; Llama 2 70B asks only for 99.9 %; Llama 3.1 405B, Mixtral, DeepSeek-R1, GPT-OSS-120B, Qwen3-VL, RGAT and WAN-2.2 ask only for 99 %; and E2E-RAG is the exception at 97 %. In the checker this materialises as separate benchmarks (bert-99 and bert-99.9), each with its computed numerical threshold.

On top of that metric there is a second LLM-specific constraint that is cited very rarely: the generation length is audited as well.

Benchmark	Length constraint
Llama 2 70B	tokens per sample > 90 % of 294.45
Llama 3.1 405B	tokens per sample between 90 % and 110 % of 684.68
Mixtral 8x7B	tokens per sample between 90 % and 110 % of 144.84
Llama 3.1 8B	total generated length > 90 % of 8,167,644

The FAQ settles the obvious shortcut: reducing the maximum output length below the reference is not allowed, and truncating tokens to improve performance or reach the accuracy target is not permitted. The inference parameters are fixed in Closed (max_new_tokens = 1024 on Llama 2 70B, 20000 on Llama 3.1 405B and DeepSeek-R1), as is the decoding algorithm: greedy search in the Llama family, Mixtral and DeepSeek-R1; sampling with temperature = 1.0 and top_p = 1.0 in GPT-OSS-120B and Qwen3-VL. Accuracy is reported to five significant figures with round-half-to-even.

The current Datacenter suite

Area	Task	Model	Dataset	QSL	Required quality
Vision	Medical segmentation	3D UNet	KiTS 2019	42	99 % and 99.9 % of FP32 (DICE 0.86330)
Language	Summarisation	Llama 3.1 8B	CNN DailyMail v3.0.0	13,368	99 % and 99.9 % of FP32
Language	Question answering	Llama 2 70B	OpenOrca	24,576	99.9 % of FP32
Language	Text generation	Llama 3.1 405B	LongBench, Ruler, GovReport	8,313	99 % of FP16
Language	QA, maths and code	Mixtral 8x7B	OpenOrca, GSM8K, MBXP	15,000	99 % of FP16
Language	Reasoning	DeepSeek-R1	`mlperf_deepseek_r1`	4,388	99 % of FP16 (exact match 81.9132 %)
Language	QA, maths and code	GPT-OSS-120B	AIME25, GPQA Diamond, LiveCodeBench v6	6,396	99 % of 83.13 %
Vision	Vision-language model	Qwen3-VL-235B-A22B	Shopify product catalogue	48,289	99 % of BF16 (hierarchical F1 0.7824)
Commerce	Recommendation	DLRMv3	Synthetic Streaming 100B	34,996	99.9 % of FP32
Generative	Text to video	WAN-2.2-T2V-A14B	VBench prompts	248	99 % of BF16 (VBench 69.7752)
Graphs	Node classification	RGAT	IGBH	788,379	99 % of FP32 (72.86 %)
Audio	Speech to text	Whisper	LibriSpeech	1,633	99 % and 99.9 % of FP32 (WER 2.0671 %)
Language	End-to-end RAG	E2E-RAG	FRAMES	824	97 % of FP32

The E2E-RAG benchmark deserves attention from anyone designing a RAG platform, because it fixes the whole pipeline: a FAISS HNSW index with mandatory parameters M = 32, efConstruction = 200 and efSearch = 100; e5-base-v2 embeddings; ColBERTv2 reranking; generation and query rewriting with GPT-OSS-120B; evaluation with Llama 3.1 8B as judge; and a maximum of 5 retrieval iterations, mandatory for all submitters. Model loading is not timed; building the vector database is.

The definition of sample changes per model and is a frequent source of confusion: in the LLMs it is a sequence, in DLRMv3 it is a request with a user history and 2048 candidates, in WAN-2.2 it is a pair of positive and negative prompts, and in PointPainting it is five images and a lidar point cloud.

How to read a results row

The columns of the official table in the Closed division are Submitter, Software, System, Benchmark Results, Processor/Count, Details, Accelerator/Count and Code. The Open division adds Model Used and Notes. Rows with power measurement add System Power in Server and Offline, or Energy Per Stream in the stream scenarios. Any citation must use the identifier in the format major-version.minor-version.entry.benchmark, of the style 5.1-0053, and carry a footnote with suite, version, division, benchmark, scenario, date and source.

Per-accelerator is not an official metric

There is no per-accelerator column. It is a derived metric, and the MLCommons messaging guidelines are blunt: any comparison based on a different or derived metric, such as power, cost, model size or accuracy, must make the basis of comparison clear in the text and in a footnote, and secondary and derived metrics cannot be presented as official or verified MLPerf metrics. NVIDIA itself states this with the standard formula in its round blogs: per-GPU performance is not a primary MLPerf Inference metric and is computed by dividing the reported throughput by the number of reported accelerators. The same applies to combining results from several benchmarks: MLCommons allows it but does not endorse it, and the composite cannot be presented as an official result.

The comparison rules

MLPerf results can only be compared with compatible MLPerf results: same benchmark, same scenario and compatible versions per the normative compatibility table.
MLPerf results cannot be compared with non-MLPerf results.
When comparing, any difference in version, division, category, verification status, scenario or number of chips must be clearly identified. When comparing Open with Closed, it must be identified in what sense the Open result would not qualify as Closed.
Submitters cannot publish results of a version before its official date; non-submitters must wait two weeks from that date.
The sanctions regime goes as far as barring an infringer from submitting results in the future and marking their results as non-compliant permanently in the database, with a deadline of three working days to withdraw the infringing content.

How much audit sits behind a row

In each round up to two submissions are audited: one chosen at random from all of them and zero or one chosen by the review committee. And there is a constraint that conditions the reading of the whole table: only Available submissions in the Closed division are auditable. Preview, RDI and Open fall outside the process. There is an exemption from the random audit if the system is equivalent to another already audited and neither the aggregate performance nor the per-accelerator performance differ by more than 10 % from those of the previous audit. The deadlines are 28 days to select an auditor, 30 for the report after signing the confidentiality agreements and around 90 days for the complete process, with two days of hardware access for the auditor. The rules close with three sentences that define the standard of proof: results that cannot be replicated are not valid results, benchmark detection is not allowed and input-based optimisation is not allowed either.

The recent rounds, in figures

Round	Organisations	Performance results	Power results	New benchmarks
v4.0	23	more than 8,500	900	2
v4.1	22	964	31	1
v5.0	23	17,457	n/a	4
v5.1	27	n/a	2	3
v6.0	24	n/a	n/a	5 of 11 Datacenter tests

These counts do not form a time series: the counting criteria differ between press releases and none documents the criterion, to the point that v4.0 and v4.1 differ by a factor close to nine.

The published performance jumps are comparable within each announcement. Between v4.0 and v5.0, twelve months apart, the number of Llama 2 70B submissions multiplied by 2.5, the median score doubled and the best score was 3.3 times faster. Between v5.0 and v5.1, in six months, the best systems improved by up to 50 % in some scenarios. Round v6.0 changed axis and its press release talks about scale rather than speed: 30 % more multi-node submissions, 10 % of the systems with more than ten nodes against 2 % in the previous round, and a system larger than 72 nodes and 288 accelerators, quadrupling the previous maximum node count.

Hardware that debuted in each round: MI300X, TPU v6e, Xeon Granite Rapids and B200 in v4.1; MI325X, Xeon 6980P, GB200 and Jetson AGX Thor in v5.0; MI355X, Intel Arc Pro B60, GB300 and RTX Pro 6000 Blackwell Server Edition in v5.1.

What MLPerf Inference does not measure

Cost. There is no cost or TCO metric, and cost is explicitly classified as a derived metric that cannot be presented as official.
Energy, except as an optional extension. MLPerf Power is regulated in a separate document, requires measuring at the wall with the power group’s workflow and PTDaemon, and forbids any other method. Its real adoption is marginal: 2 power submissions in v5.1 against 27 participating organisations. In addition, MLCommons bounds what that figure means: the measured power is only valid for the benchmark it accompanies, and any other reference to power, such as a TDP setting or a power supply rating, is neither measured nor validated by MLCommons. The messaging rule forbids submitters from publishing per-watt normalised comparisons using any proxy other than the measured power.
Configurations not optimised by the vendor. The bias is acknowledged in the audit rule itself, which contemplates submissions whose performance is not consistent with the known characteristics of the hardware, or where the committee lacks visibility into how it was achieved, or where hardware and software are not reasonably available to the public.
Barrier to entry. Only MLCommons members and test partners can submit results for review. Anyone measuring with the same code without submitting must label every figure as unverified with the corresponding legend.

Usage flow for an on-premise platform

MLPerf Inference is useful in a purchasing decision if it is used for what it is, a test bench with written rules, and not as a ranking.

Filter by division and category before looking at any number: keep Closed and Available. A Preview or RDI result describes hardware that cannot yet be bought, and neither of the two is auditable.
Choose the scenario that matches the load pattern: Server or Interactive if there are users waiting, Offline if the work is batch. An Offline result does not authorise promising a latency.
Read the SLO before the throughput: in any LLM benchmark, the number of queries per second only means something alongside its TTFT/TPOT pair. The same machine publishes very different figures under Server and under Interactive.
Check the quality variant: 99 % and 99.9 % are different results for the same model, and the second usually costs performance.
Translate to your own configuration carefully: dividing by the number of accelerators is a derived metric, not an official one, and it has to be declared as such.
Close the gap MLPerf leaves open with your own measurement: cost, energy and behaviour under real traffic. That is exactly the function of the reproducible harness and of the track B bench tools, and the reason no platform decision stands on a public table alone.

Cross-links from the benchmarking track

B1/B2 — LLM inference benchmarking: frameworks, metrics and state of the art: the TTFT, TPOT and goodput metrics outside the MLPerf corset, and which tool measures each one.
B2 — LLM benchmark tool catalogue: the tools with which you reproduce at home what MLPerf standardises.
B3 — GuideLLM and SLO validation under load: the practical equivalent of LoadGen’s binary search in the Server scenario.
B6 — Measurement bias and reproducibility: why LoadGen exists, told from the side of what happens when there is none.
B8 — Serving engines compared on the Pareto frontier: the comparison MLPerf does not make, with engines instead of systems.
C4 — MLPerf Power: the energy extension of this same machinery, with its certified measurement at the wall.

Sources

MLCommons, MLPerf Inference Working Group — https://mlcommons.org/working-groups/benchmarks/inference/
MLCommons, MLPerf Inference Rules (inference_rules.adoc) — https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc
MLCommons, MLPerf Inference Power Measurement (power_measurement.adoc) — https://github.com/mlcommons/inference_policies/blob/master/power_measurement.adoc
MLCommons, MLPerf Submission Rules (submission_rules.adoc) — https://github.com/mlcommons/policies/blob/master/submission_rules.adoc
MLCommons, MLPerf Results Messaging Guidelines — https://github.com/mlcommons/policies/blob/master/MLPerf_Results_Messaging_Guidelines.adoc
MLCommons, LoadGen README — https://github.com/mlcommons/inference/blob/master/loadgen/README.md
MLCommons, loadgen/test_settings.h — https://github.com/mlcommons/inference/blob/master/loadgen/test_settings.h
MLCommons, loadgen/mlperf.conf (master) — https://github.com/mlcommons/inference/blob/master/loadgen/mlperf.conf
MLCommons, loadgen/mlperf.conf (tag v5.1) — https://github.com/mlcommons/inference/blob/v5.1/loadgen/mlperf.conf
MLCommons, loadgen/mlperf.conf (tag v5.0) — https://github.com/mlcommons/inference/blob/v5.0/loadgen/mlperf.conf
MLCommons, tools/submission/submission_checker.py (tag v5.0) — https://github.com/mlcommons/inference/blob/v5.0/tools/submission/submission_checker.py
MLCommons, compliance/nvidia TEST06 README (tag v5.1) — https://github.com/mlcommons/inference/blob/v5.1/compliance/nvidia/TEST06/README.md
MLCommons, Benchmark results: Inference Datacenter — https://mlcommons.org/benchmarks/inference-datacenter/
Reddi et al., MLPerf Inference Benchmark, ISCA 2020 — https://arxiv.org/abs/1911.02549
MLCommons, MLPerf Inference v5.0 Benchmark Results (abril 2025) — https://www.businesswire.com/news/home/20250402313932/en/MLCommons-Releases-New-MLPerf-Inference-v5.0-Benchmark-Results
MLCommons, MLPerf Inference v5.1 Benchmark Results (septiembre 2025) — https://www.globenewswire.com/news-release/2025/09/09/3147136/0/en/MLCommons-Releases-New-MLPerf-Inference-v5-1-Benchmark-Results.html
MLCommons, MLPerf Inference v6.0 Benchmark Results (abril 2026) — https://www.globenewswire.com/news-release/2026/04/01/3266801/0/en/MLCommons-Releases-New-MLPerf-Inference-v6-0-Benchmark-Results.html
MLCommons, MLPerf Inference v4.1 Benchmark Results (agosto 2024) — https://www.businesswire.com/news/home/20240828886616/en/New-MLPerf-Inference-v4.1-Benchmark-Results-Highlight-Rapid-Hardware-and-Software-Innovations-in-Generative-AI-Systems
NVIDIA Developer Blog, NVIDIA Blackwell delivers massive performance leaps in MLPerf Inference v5.0 — https://developer.nvidia.com/blog/nvidia-blackwell-delivers-massive-performance-leaps-in-mlperf-inference-v5-0/
NVIDIA Developer Blog, NVIDIA Blackwell Ultra sets new inference records in MLPerf debut — https://developer.nvidia.com/blog/nvidia-blackwell-ultra-sets-new-inference-records-in-mlperf-debut/
MLCommons, MLPerf Automotive (octubre 2025) — https://arxiv.org/html/2510.27065v1
Tschand et al., MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts — https://arxiv.org/abs/2410.12032

Storage in the AI era (2/4): performance

Mon, 22 Jun 2026 11:00:00 +0200

In the first article we drew the map: the memory-storage hierarchy, the media technologies and the software stack that dominates GPU clusters. This second article goes into the property that justifies all that complexity: performance. The question an architect has to answer is not “how many GB/s does my array deliver?”, but “can I get my GPUs working at 95 % instead of 40 %?”. That difference is, literally, half the compute bill.

The metric that really matters: accelerator utilisation

It is tempting to measure AI storage with the classic metrics, sequential throughput, random IOPS, latency, and we do need them. But none of them captures what the business pays for. The integrating metric is accelerator utilisation: what fraction of the time the GPU spends computing rather than waiting for data.

Most real ML pipelines run at between 30 % and 60 % GPU utilisation. Industry surveys are damning: only around 7 % of AI/ML teams manage to exceed 85 % at peak, and poor forecasting or weak autoscaling can leave GPUs idle 70-85 % of the time. A job running at 30 % utilisation is paying for 70 % of its compute capacity and getting nothing for it.

We can formalise this. If $ t_{\text{cmp}} $ is the compute time per batch and $ t_{\text{io}} $ the time the GPU spends blocked waiting for I/O that fails to overlap with compute, effective utilisation is:

$$U = \frac{t_{\text{cmp}}}{t_{\text{cmp}} + t_{\text{io}}}$$

The goal of all AI storage engineering is to drive $ t_{\text{io}} $ towards zero, whether by increasing bandwidth, reducing latency or overlapping I/O with compute through prefetching and pipelining. And there is an aggravating factor: the faster the GPU, the harder it is to feed. Moving from V100 to H100 cut the per-batch compute time of the 3D U-Net model by 76 %, turning a bandwidth-sensitive workload into a latency-sensitive one. Every accelerator generation raises the bar for storage.

This change of regime, from bandwidth-limited to latency-limited, has a design consequence worth internalising. When a workload is bandwidth-limited, the fix is to add transfer capacity: more NVMe, more lanes, more nodes. When it becomes latency-limited, that recipe stops working; what matters then is response time per operation, which is attacked with cache, with intelligent prefetching and with data paths that remove hops (GPUDirect, for instance). Diagnosing correctly which of the two regimes each workload sits in is the first task of any optimisation: throwing bandwidth at a latency-limited workload is spending money without moving utilisation.

MLPerf Storage: the benchmark that does measure the right thing

For years there was no neutral way to compare AI storage platforms. MLPerf Storage, from MLCommons, has provided one. Its design insight is not to measure GB/s in the abstract but to simulate the accelerators’ think time so as to generate a realistic I/O pattern, and to require those simulated accelerators to sustain a minimum utilisation level. In other words, it measures exactly what matters: how many accelerators a platform can keep fed without their utilisation dropping below the threshold.

v1.0 (September 2024) used the 3D U-Net, ResNet-50 and CosmoFlow models, simulating A100 and H100, with utilisation thresholds of 90 % for the first two and 70 % for CosmoFlow. v2.0 (August 2025) marked a leap: more than 200 results from 26 organisations, including Alluxio, DDN, Hammerspace, HPE, IBM, KIOXIA, Micron, Oracle, Samsung and WDC, systems able to sustain roughly twice as many accelerators as in v1.0, and, above all, a new checkpointing workload (which we will come back to).

Some v2.0 results help calibrate the state of the art:

System	Throughput	Accelerators	GPU utilisation
Volumez	1.079 TB/s	—	92.21 %
Hammerspace (5 nodes)	420.8 GB/s	140	>94.7 %
Hammerspace (3 nodes)	253.1 GB/s	84	>94.7 %
Hammerspace (1 node)	85.6 GB/s	28	>94.7 %
Alluxio	24.14 GiB/s	128	99.57 %

The Hammerspace figure illustrates a property an architect should demand: linear scaling. From 1 to 5 nodes throughput grows from 85.6 to 420.8 GB/s while keeping utilisation above 94.7 % with a minute coefficient of variation (0.08-0.14 %). Linear scaling is what separates an AI platform from a fast NAS.

The numbers from commercial platforms

Outside the benchmark, platforms certified for DGX SuperPOD publish aggregate figures worth keeping as an order-of-magnitude reference:

WEKA WEKApod Nitro: 720 GB/s read and 186 GB/s write per configuration, 18 million IOPS, with ConnectX-8 NICs at 800 Gb/s. Per node, 70 GB/s read and 40 GB/s write; the minimum 8-node configuration delivers 560 GB/s read. A single entry-level system covers the I/O demand of a 1,152-GPU GB200 scalable unit.
DDN AI400X2: more than 90 GB/s and 3 million IOPS per appliance; the Turbo variant saturates close to 100 GB/s (800 Gbps) per DGX B200. DDN’s new platform claims more than 1 TB/s read per appliance with per-rack scaling, and has fed NVIDIA’s Eos supercomputer at 4 TB/s.
VAST Data: multiple TB/s from a single mount point via NFS multipathing, RDMA over NFS and GPUDirect, managing exabytes in a single multiprotocol namespace.
Pure Storage: certified for DGX SuperPOD with GB200 and GB300.

These figures only mean something in relation to the cluster they feed. A GB200 NVL72 moves 130 TB/s of internal GPU-to-GPU communication; storage does not compete with that, but it does have to sustain data loading and checkpointing without strangling 72 GPUs that behave as one.

The I/O pattern of training and GPUDirect Storage

Understanding the I/O pattern is the key to neither over- nor under-sizing. Training combines several patterns: large sequential reads of the dataset, small random reads during shuffling between epochs, and massive periodic writes during checkpointing. Each stresses storage differently.

The traditional data path goes through the CPU: from storage to a bounce buffer in host memory and from there to GPU memory. Every byte of every checkpoint and every weight load pays the latency and CPU overhead of that double hop. GPUDirect Storage (GDS) cuts it out: it enables direct DMA between GPU memory and NVMe through the cuFile user-space library and the nvidia-fs kernel module, which intercepts POSIX calls and redirects them through the DMA engine. The result is zero-copy, with direct transfers exceeding 40 GB/s and, above all, without stealing cycles from the CPU.

There is a nuance that naive deployments miss: GDS is optimised for large, aligned, sequential transfers. For workloads dominated by small files or heavily random access, the traditional path with CPU caching can be more efficient. GDS is not a magic “more performance” switch; it is a tool for a specific pattern. Designing the data pipeline, meaning dataset format, shard size and number of DataLoader workers, matters as much as picking the array.

The data pipeline deserves a pause, because it is where performance leaks most quietly. Training does not read the dataset just any way: it walks it in batches, shuffles it between epochs to avoid ordering bias, and often applies transformations (image decoding, tokenisation, augmentation) on the CPU before handing the tensor to the GPU. Each of those steps can become the bottleneck. Too few DataLoader workers leaves the GPU waiting; a dataset format that forces millions of small files open sinks metadata performance; a badly designed shuffle turns efficient sequential reads into costly random accesses. Common practice in serious workloads is to pack the dataset into large shards (formats such as WebDataset, TFRecord or Parquet) precisely so that the read pattern is large and sequential, the terrain where GDS and parallel file systems perform best. The mental rule: the fastest storage in the world will not save a badly designed data pipeline.

The real cost of checkpointing (when there is training)

Checkpointing is the most demanding write workload in the AI world, but it needs to be placed properly: it is a training problem. A pure inference factory barely suffers from it, since its heavy writes are different ones, KV-cache, vector indexes, telemetry. We cover it here because many installations combine some fine-tuning or occasional training with serving, and because it illustrates better than any other case how asynchrony beats brute force. If your platform is for inference, you can read this section as context and jump to the next one, which is the one that really concerns you.

That said, most people’s intuition about checkpointing is wrong on two counts: size and frequency.

Size. A checkpoint is not just the weights. The rule of thumb is around 16 bytes per parameter, because the bulk is optimiser state. The figures from the MLPerf Storage v2.0 checkpointing workload, based on Meta’s Llama 3 Herd, make it plain:

Model	Processes	Checkpoint size	Of which, optimiser state
8B	8	105 GB	90 GB
70B	64	912 GB	—
405B	512	5.29 TB	—
1T	1024	15 TB	13.2 TB

Frequency. At scale, hardware failures are constant (we develop this in the article on availability). Meta’s model for a 16,000-accelerator cluster predicts a failure every few hours. To lose less than 5 % of progress between failures, you need on the order of 20 checkpoints in that interval. That translates into aggressive cadences:

16,000 accelerators: around 155 checkpoints a day, one every 9.3 minutes.
100,000 accelerators: around 967 checkpoints a day, one every 1.5 minutes.

And here is where the bandwidth problem appears. A synchronous checkpoint halts training while it writes. If $ S_{\text{ckpt}} $ is the checkpoint size and $ t_{\text{win}} $ the acceptable time window for writing it while keeping overhead below 5 %, the required bandwidth is:

$$B_{\text{req}} = \frac{S_{\text{ckpt}}}{t_{\text{win}}}$$

For a 1T model on 100,000 accelerators, the window falls to around 4.4 seconds. Writing 15 TB in that time demands on the order of 3.6 TB/s across the cluster, or some 200 GB/s sustained over the whole job. On checkpoints alone, that scenario generates more than 14 PB of writes a day. This is not a marginal workload: it is one of the most demanding write workloads in existence.

It is worth seeing the calculation laid out, because it reveals where the lever is. Take the 405B model, whose checkpoint weighs 5.29 TB. If policy requires writing it while keeping overhead below 5 % at a cadence where the acceptable window is, say, 30 seconds, the synchronous bandwidth needed is around 176 GB/s sustained, within reach of a high-end array, but only one dedicated to that task. If instead of 30 seconds the window contracts to 5 (a larger cluster, more frequent failures), the requirement jumps above 1 TB/s. Sensitivity to the window is brutal: the difference between a viable design and an impossible one lies in how time is managed, not in how many drives are bought. That is where asynchrony changes the rules.

The solution is not just more bandwidth; it is asynchrony. Asynchronous checkpointing decouples the snapshot (a fast GPU→CPU copy) from persistence (a background write, overlapped with compute). PyTorch’s Distributed Async Checkpointing cuts effective checkpoint time by 10 to 20 times; in one IBM example a 7B model brought its down time down from 148.8 to 6.3 seconds, a 23.6-fold improvement. Techniques such as CheckFreq pipeline the snapshot and persistence with training, allowing a checkpoint every 14-19 iterations. This is why vendors like VAST argue the relevant metric is not GB/s but checkpoint overlap: what fraction of the checkpoint overlaps with compute. Around 10 % overlap is usually enough.

The design lesson is clear: sizing training storage for the peak write rate of synchronous checkpointing is expensive and often unnecessary if the software stack gets asynchrony right. But trusting asynchrony without measuring overlap is playing roulette with weeks of compute.

Performance in an inference factory

Here is the heart of the matter for anyone running an inference factory. In an inference factory storage is not measured in epochs or GPU-hours but in the experience the client perceives, and that experience condenses into two metrics: TTFT (time to first token, the time until the first token of the answer) and inter-token latency (the rate at which the following tokens arrive). At fleet scale, the metric that pays the bills is throughput in tokens per second per GPU, because it determines how many requests each accelerator serves. Storage touches all three, in ways a training-centric design does not anticipate.

Prefill, decode and their disaggregation

Inference has two phases with opposite profiles. Prefill processes the entire input prompt at once, generates the key-value pairs of the KV-cache and is a compute-intensive phase; it dominates TTFT. Decode generates tokens one by one, reusing that cache, is memory-bandwidth-intensive and dominates inter-token latency. Because their bottlenecks differ, the 2025-2026 trend is to disaggregate them: run prefill and decode on separate GPU pools and transfer the KV-cache between them. That transfer, which may travel over NVLink, over the network or through a shared storage layer, makes KV-cache movement a performance-critical operation. A factory that disaggregates prefill and decode needs a very high bandwidth, low latency KV-cache path between both pools; storage (or the DPU that governs it) stops being a passive actor and enters the critical path of every request.

The KV-cache as a performance layer

The central problem is the size of the KV-cache with long contexts: it overflows HBM, which is scarce and expensive. The options are to recompute it on every request, ruinously expensive in compute and lethal for TTFT, or to offload it to a slower layer and reuse it. Two approaches compete here. Offload to CXL memory offers gains of more than 5 times against SSD or RDMA-based caching, according to CXL supply chain vendors, and allows batches up to 30 % larger while holding latency targets, which raises throughput and GPU utilisation. Offload to NVMe, standardised by NVIDIA in 2026 with its ICMS platform on BlueField-4, is slower but much cheaper and more capacious, ideal for pod-scale KV-cache.

The big performance lever here is KV-cache reuse (prefix caching). Many requests share prefixes: the same system prompt, the same context document, the same prior conversation. If the KV-cache of those prefixes is kept in a fast storage layer and reused instead of recomputed, prefill shortens drastically and TTFT collapses. This turns a low-latency storage layer into a direct multiplier of the factory’s throughput: the more useful cache fits and the faster it is served, the less compute is wasted recalculating what has already been calculated. Sizing this layer well, in capacity, bandwidth, latency and eviction policy, is one of the most profitable performance decisions in the whole factory.

Model load time and cold start

There is a storage cost specific to inference that training ignores: starting a replica. When autoscaling decides to bring up a new instance of a model, because demand is rising, because a replica has failed or because a version is being rotated, the weights have to be read from the registry and loaded into HBM. For a model of tens or hundreds of GB, that cold start can run from seconds to minutes depending on registry bandwidth and the load path. And during that time the new GPU serves no requests while the system is, perhaps, saturated. A slow model registry translates directly into missed SLAs at peak. The mitigations, keeping weights on hot local flash, using fast-loading formats, streaming weights as they are needed, or GPUDirect Storage to load straight into HBM, are storage decisions that determine the factory’s real elasticity.

RAG performance and vector search

If the factory serves RAG, there is one more link in the critical path of every request: retrieval. Before the model generates anything, the system searches a vector database for the relevant fragments, and that similarity search, over indexes that can run to TB, adds to TTFT. Vector search performance depends on keeping the indexes on low-latency flash and on an efficient random read pattern; an index that does not fit in memory and is served from slow storage adds latency to every request. Beyond that, precomputed KV-caches of the retrieved documents speed up prefill, because the model reuses keys and values already calculated and only encodes the user’s query. An inference factory with RAG therefore has two latency-sensitive read workloads coexisting, the vector index and the KV-cache, competing for the same fast flash and needing to be sized together.

For an inference architect, all this introduces a storage plane with a profile of its own: latency-sensitive random reads (indexes, reusable KV-cache prefixes), high-turnover ephemeral writes (new KV-cache), large, sporadic but urgent reads (model loading) and an append-only telemetry flow. It is not sized in the least like the storage of a training cluster.

How to measure your own: beyond the spec sheet

Vendor and MLPerf figures are useful as a reference, but real performance depends on the specific workload, and an architect must measure rather than assume. The methodology that works starts from three principles.

The first is to measure accelerator utilisation, not storage in the abstract. An array can deliver 500 GB/s in a synthetic test and still leave the GPUs at 50 % because the real pattern is small random files that the sequential test does not reproduce. The right question is always: what fraction of the time is my GPU computing, with this workload, this dataset format and this pipeline?

The second is to reproduce the real I/O pattern, not a convenient one. That means testing with the same dataset format, the same shard size, the same number of workers and the same checkpointing cadence that will be used in production. MLPerf Storage’s insight was precisely to simulate accelerator think time in order to generate a realistic pattern; replicating that rigour at your own scale is what separates a useful measurement from a marketing number.

The third is to look for scaling, not peak. A platform that delivers a very high peak in a small configuration but flattens out as nodes are added is no use for growth. Tests should be run at several scales to verify that throughput grows roughly linearly and that accelerator utilisation holds, as the Hammerspace example in MLPerf v2.0 showed. A low coefficient of variation across configurations matters as much as the peak.

The cost of getting it wrong

It is worth putting a number on the problem. The price of an H100 in the cloud in mid-2026 moves across a wide range by provider, with a market median around 2.3-3.1 USD per GPU-hour. Multiply that by thousands of GPUs and by the weeks a training run lasts, and 40 % utilisation instead of 90 % translates into millions of euros of wasted compute. That is why more than 50 % of organisations report data or storage bottlenecks limiting their AI in 2026, and why storage bandwidth has emerged as a hard ceiling on scaling, comparable to power and cooling: when there is data starvation, adding compute gives diminishing returns.

The operational recommendation that keeps coming up is to size the I/O pipeline for 300 Gbps or more in serious workloads, and to measure, not assume, accelerator utilisation as a health indicator. Optimised storage can deliver up to 5 times the throughput of standard S3 access over HTTP: more than 100 GB/s aggregate against some 20 GB/s.

The physical layer: networks and PCIe Gen6

All of the above rests on a physical layer that has also advanced. NVIDIA’s ConnectX-8 NIC combines PCIe Gen6 with 800 GbE or 800 Gb InfiniBand XDR, designed for Blackwell systems. InfiniBand XDR offers 800 Gbps per direction (double NDR, which remains the usual storage fabric), and the Quantum-X800 switch adds in-network compute that accelerates collective operations. The transition to XDR is accelerating with the Blackwell deployments of 2025-2026. The mental rule is simple: storage must match the network, and the network must match the GPU. One slow link defines the performance of the whole chain.

It is worth translating this into concrete numbers. An 800 Gb/s NIC delivers, at best, around 100 GB/s to a node. If the array can serve more than the NIC absorbs, the network is the bottleneck; if the array serves less, storage is. Balance is found by sizing each layer so none strangles the next, and by verifying it with end-to-end measurements rather than the isolated specifications of each component. It is common to find installations with an excellent array and an undersized fabric, or the reverse, performing far below the sum of their parts. The physical layer also imposes its geography: network topology (how many hops there are between the compute node and the array, whether they share a switch or cross the cluster spine) affects latency as much as the array itself, and an AI storage design that ignores network topology is incomplete.

Three questions before sizing

All this complexity can be organised into three questions an architect should answer before buying anything. What is the dominant I/O pattern of my workload, large sequential, small random, bursty writes, and therefore which metric limits me? Is my workload bandwidth-limited or latency-limited, and consequently where do I invest? And what is my checkpointing budget, in size, cadence and, above all, what fraction can I overlap with compute? Whoever answers these three with measured data, rather than intuitions, sizes well; whoever skips them ends up with idle GPUs or with an oversized, very expensive array that is never put to use. AI storage performance is, at bottom, an exercise in fit: putting the right capacity where the workload needs it, no more and no less.

Takeaways

AI storage performance is not measured in GB/s but in accelerator utilisation, and that utilisation depends on three things: enough bandwidth, low latency and, above all, overlapping I/O with compute. MLPerf Storage v2.0 has finally given a common language for comparing platforms through that lens. Checkpointing is the most demanding write workload and is mastered with asynchrony, not hardware alone. Long-context inference adds a new storage layer around the KV-cache. And the cost of getting it wrong is measured in idle GPUs, which is the most expensive line in the budget.

Performance, however, is worth nothing if the data is neither protected nor available. The third article takes on the security of this new storage layer.

Sources

MLCommons, MLPerf Storage v2.0 Results — https://mlcommons.org/2025/08/mlperf-storage-v2-0-results/
MLCommons, Announcing the MLPerf Storage v2.0 Checkpointing Workload — https://mlcommons.org/2025/08/storage-2-checkpointing/
MLCommons, MLPerf Storage v1.0 Benchmark Results — https://mlcommons.org/2024/09/mlperf-storage-v1-0-benchmark-results/
Hammerspace, MLPerf Storage v2.0 Benchmark Results — https://hammerspace.com/hammerspace-mlperf-storage-v2-0-benchmark-results/
Alluxio, Strong Performance in MLPerf Storage v2.0 — https://www.alluxio.io/blog/alluxio-demonstrates-strong-performance-in-mlperf-storage-v2-0-benchmarks
NVIDIA, GPUDirect Storage Overview Guide — https://docs.nvidia.com/gpudirect-storage/overview-guide/index.html
Spheron, GPU Direct Storage Guide 2026 — https://www.spheron.network/blog/gpu-direct-storage-nvme-ai-training-inference-guide/
PyTorch, Reducing Checkpointing Times with Async Checkpointing — https://pytorch.org/blog/reducing-checkpointing-times/
VAST Data, Optimizing Checkpoint Bandwidth for LLM Training — https://www.vastdata.com/blog/optimizing-checkpoint-bandwidth-for-llm-training
Astera Labs, How CXL Transforms RAG and KV Cache Performance — https://www.asteralabs.com/breaking-through-the-memory-wall-how-cxl-transforms-rag-and-kv-cache-performance/
MinIO, AI Storage Architecture: Overcoming the Bottleneck in 2026 — https://www.min.io/blog/ai-storage-architecture-bottleneck-2026
WEKA, WEKApod Nitro reference architecture — https://www.weka.io/resources/reference-architecture/nvidia-gb200-nvl72-systems-reference-architecture-for-cloud-partners/
Blocks & Files, Storage vendors rally behind Nvidia at GTC 2025 — https://blocksandfiles.com/2025/03/18/nvidia-storage-announcements/
ServeTheHome, NVIDIA ConnectX-8 SuperNIC PCIe Gen6 800G — https://www.servethehome.com/nvidia-connectx-8-supernic-pcie-gen6-800g-nic-detailed/
SpendArk, GPU Cloud Pricing 2026 — https://spendark.com/blog/machine-learning-cloud-cost/

LLM benchmark tool catalogue: the practical deep dive

Sun, 14 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

What this article covers

Second article in the benchmarking track (B2). The B1 introduction pinned the metrics and the methodology; this article is the practical manual: for each tool, what it measures, how it is actually invoked (with the command), what output format it produces and when to choose it. The goal is that, after reading it, you know what to run to measure your engine and you can reproduce the number. No universal recommendations; only the mechanics of each tool and the criterion for choosing.

How to choose: the tool map

Recalling the split from B1, the tools are ordered on two axes: micro-bench (single-process, for tuning an engine) versus load generator (multi-process, for measuring real capacity), and engine-native versus endpoint-agnostic.

Datasets: the synthetic load matters as much as the tool

Before the profiles, one point that changes results without anyone noticing: what load you feed it. The tools accept different kinds of dataset, and each one measures something different:

Dataset	What it simulates	Bias it introduces
random (fixed lengths)	controlled uniform load	unrealistic: traffic is not fixed length
sharegpt (real conversations)	realistic prompt distribution	the de facto standard for comparison
your own traces	your real traffic	the most faithful, but specific to your case

The trap: a benchmark with short, fixed-length prompts gives a very high throughput that looks nothing like production, where lengths vary and long prompts dominate the prefill cost. For a defensible figure, use sharegpt (comparability) or, better, traces from your own traffic (fidelity). And always declare the length distribution (prompt/output) alongside the number, because two benchmarks with different datasets are not comparable even if they use the same tool.

vLLM bench serve

What it measures: TTFT, TPOT, throughput and latencies of the vLLM server under a synthetic load. Class: micro-bench (single-process). It is the tool for tuning vLLM and seeing the effect of its optimisations (decode, prefill).

Typical invocation (vLLM · benchmark CLI):

vllm bench serve \
 --backend vllm \
 --model meta-llama/Llama-3.1-8B-Instruct \
 --endpoint /v1/completions \
 --dataset-name sharegpt \
 --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
 --num-prompts 1000

Key parameters: --num-prompts (load), --dataset-name (sharegpt, random, etc.), --request-rate (requests/s). The output is a console summary with TTFT, TPOT, throughput and percentiles, and it can be dumped to JSON. To sweep concurrencies there is vllm bench sweep serve, which automates the sweep (vLLM · sweep).

Limit: single-process, it saturates on the client at high concurrency; less flexible than GuideLLM in datasets and load patterns. Use it to iterate quickly on the vLLM config, not to measure maximum capacity at scale.

SGLang bench

What it measures: the equivalent for the SGLang engine (TTFT, TPOT, throughput). Class: micro-bench. Use: tuning SGLang and comparing it against itself across configurations. The mechanics are analogous to vllm bench serve: a synthetic load, console/JSON output with the same metrics. If you are evaluating SGLang against vLLM, do not compare them with their respective micro-benches (same class but different implementations): use an endpoint-agnostic load generator (GuideLLM/AIPerf) against both endpoints, so that the tool is not the variable.

AIPerf (NVIDIA, formerly genai-perf)

What it measures: TTFT, ITL, throughput and latency against any compatible endpoint (vLLM, NIM, TGI, SGLang). Class: multi-process load generator. It is the successor to genai-perf (retired on 15 Apr 2026). Its distinguishing feature: during the sweep it detects GPU saturation and returns the previous iteration as estimatedCapacity, which is why the sweep has to extend beyond the knee (AIPerf).

Output: structured metrics (JSON) with TTFT/ITL distributions and the estimatedCapacity. Use it when you want the real capacity of an endpoint, whatever the engine, with automatic detection of the saturation point.

GuideLLM (vLLM project)

What it measures: full distributions of TTFT, ITL and end-to-end behaviour, for SLO-driven evaluation. Class: multi-process load generator. It is the recommended tool for benchmarking vLLM servers in production: more flexible than vllm bench serve in dataset loading, request format and traffic patterns, with live progress and automatic report generation (Red Hat).

Typical invocation:

guidellm benchmark \
 --target "http://localhost:8000" \
 --rate-type throughput \
 --max-requests 1000 \
 --data "samples=1000,prompt_tokens=1024,output_tokens=256"

Key parameters: --rate-type (synchronous, concurrent, throughput, or rate-based), --data (the load specification: number of samples and prompt/output lengths), --target (the endpoint). It generates reproducible sweeps to find the safe operating range under SLO, with full distributions (not just means). Output: a report with percentiles and, often, exportable to JSON/HTML. It is the default option for answering “how hard can I load this engine without breaking the SLO?”.

LLMPerf (Anyscale/Ray)

What it measures: throughput and latency at the inference level. Class: load generator. Use: endpoint validation, historically very widespread in the Ray/Anyscale ecosystem. It is a solid and well-known option, though less focused on distributions and sweeps than GuideLLM/AIPerf. It fits if you already operate on Ray or want a simple tool to validate an endpoint.

inference-benchmarker (Hugging Face)

What it measures: latency and throughput of inference endpoints, aimed at producing comparable reports. Class: load generator. Use: an OSS alternative inside the Hugging Face ecosystem, useful if you already work with TGI or the HF stack. Like the others in its class, its value lies in measuring real capacity with distributed load; the choice between this one, GuideLLM and AIPerf usually comes down to the ecosystem you already operate in rather than to deep differences in what they measure.

GuideLLM versus AIPerf: which of the two load generators

They are the two serious multi-process load options, and they overlap a great deal. The practical differences that tip the choice:

Criterion	GuideLLM	AIPerf
Origin	vLLM project (Red Hat)	NVIDIA (successor to genai-perf)
Focus	SLO-driven evaluation, reproducible sweeps	real capacity, automatic estimatedCapacity
Load patterns	synchronous, concurrent, rate-based	sweep with saturation detection
Report	live progress + automatic report	structured metrics (JSON)
Ecosystem	vLLM / OpenShift AI	NVIDIA NIM / Triton / vLLM

In practice: if your question is “how hard can I load it without breaking the SLO?”, GuideLLM answers it more directly with its SLO-driven sweeps; if your question is “what is the maximum capacity of this endpoint?”, AIPerf gives it with its automatic knee detection. Many teams use both: GuideLLM for the operational SLO, AIPerf for the reference capacity. What you must not do is compare a GuideLLM result with an AIPerf one as if they were the same measurement: although both are multi-process, their sweep methodology differs; pick one for a given comparison and stick to it.

MLPerf Inference (MLCommons)

What it measures: performance under standardised scenarios (Offline, Server, Interactive) with strict rules. Unlike the previous ones, MLPerf is not “run” for your day-to-day case: it is a vendor comparison suite whose results are read (published by NVIDIA, AMD, Intel and others). Use it to compare hardware and engines against each other under identical rules, not to size your specific workload. For that, GuideLLM/AIPerf against your endpoint.

Practical comparison table

Tool	Class	Base command	Output	When
vllm bench serve	micro	`vllm bench serve`	console/JSON	tune vLLM
vllm bench sweep	micro (sweep)	`vllm bench sweep serve`	console/JSON	sweep concurrency in vLLM
SGLang bench	micro	bench from the SGLang repo	console/JSON	tune SGLang
AIPerf	load	`aiperf profile …`	JSON + estimatedCapacity	real capacity, multi-endpoint
GuideLLM	load	`guidellm benchmark …`	report + JSON/HTML	SLO, reproducible sweep
LLMPerf	load	Ray/LLMPerf script	JSON	validate endpoint (Ray)
MLPerf	suite	(results are read)	official results	compare vendors

Summary: what you run for each question

So as not to get lost in the catalogue, the direct mapping from question to tool:

Your question	Tool	Why
“Does my vLLM config change help?”	`vllm bench serve` (+ sweep)	fast, engine-native
“How hard can I load without breaking the SLO?”	GuideLLM	SLO-driven sweeps
“What is the maximum capacity of the endpoint?”	AIPerf	automatic knee detection
“vLLM or SGLang for my workload?”	GuideLLM/AIPerf against both	same tool, engine as the variable
“Which hardware/engine is better in the abstract?”	read MLPerf	cross-vendor comparability
“Is there a regression in this release?”	short sweep in CI against a baseline	continuous detection

The rule underlying the whole table: micro-bench to iterate on an engine, load generator to measure capacity and decide, MLPerf to compare vendors. And, for any comparison between systems, the same tool for every candidate. If these three categories are clear to you, the specific choice is secondary.

Step-by-step benchmark methodology

A reliable run is not “launching the command”: it is a procedure. The steps, with the load tool (GuideLLM/AIPerf) against your endpoint:

Deploy the engine with the exact config to be measured (model, precision, flags), pinned.
Warm up: send a few requests and discard them, so that the prefix cache and the autotuning do not inflate the first numbers.
Sweep: sweep rising concurrencies (1, 8, 16, 24, 32…) beyond the knee, to see where latency takes off.
Collect the output in JSON with percentiles (TTFT/ITL/throughput/goodput) and the metadata.
Compare: read the goodput under the SLO, not the maximum throughput.

Skipping step 2 (warm-up) or not going past the knee in step 3 are the two mistakes that bias the result the most.

Comparing two engines: the fair protocol

If the goal is to choose between vLLM and SGLang (or TRT-LLM), the protocol that avoids false conclusions:

Same load tool (GuideLLM or AIPerf) against both endpoints, never each engine’s own micro-bench.
Same dataset and length distribution.
Same hardware and precision.
Same SLO to compute the goodput of both.
Vary only the engine; everything else fixed.

Only then is the difference you measure down to the engine and not to the tool, the dataset or the hardware. It is the controlled experiment that supports the scorecard row (article B8).

Output format and comparability

What you record alongside the number is what makes it comparable. A useful output includes, in JSON so it can be versioned:

{
 "tool": "guidellm", "version": "x.y.z",
 "model": "Llama-3.1-70B", "precision": "FP16",
 "hardware": "8xH100 SXM NVLink",
 "load": {"prompt_tokens": 1024, "output_tokens": 256, "concurrency": 16},
 "results": {"ttft_p50_ms": 180, "ttft_p99_ms": 460,
 "itl_p50_ms": 22, "throughput_tok_s": 3400, "goodput_tok_s": 3330}
}

Saving this for every run turns a benchmark into an auditable datum: anyone can reproduce the figure with the same tool, version, model, hardware and load. It is the material of the reproducible harness (article S4), and the difference between a defensible number and a console screenshot.

Worked example: reading the output of a sweep

An illustrative output from a GuideLLM sweep on a 70B on 8×H100 (SLO: P99 TTFT < 500 ms), just as you would read it:

Concurrency	TTFT P99 (ms)	ITL P50 (ms)	Throughput (tok/s)	Goodput (tok/s)
8	240	20	2,100	2,100
16	460	22	3,400	3,330
24	980	31	3,900	2,420
32	1,800	54	4,000	800

How to read it: the knee sits between 16 and 24. At concurrency 16, P99 (460 ms) meets the SLO and goodput (3,330 tok/s) ≈ throughput. At 24, throughput rises little (3,400 → 3,900) but P99 already violates the SLO and goodput falls to 2,420. At 32, throughput is at its maximum (4,000) but goodput collapses to 800: the system “performs well” while serving requests that do not comply. The defensible capacity is the one at concurrency 16 (3,330 useful tok/s), and that is the number that goes into capacity planning and into cost per token. Anyone reporting “4,000 tok/s” is describing the point where the system no longer meets its SLO.

Automating the harness

A manual run does not scale to a benchmarking programme. The minimum automation:

An idempotent script that deploys the engine, warms it up, runs the sweep and saves the JSON with all the metadata (model, version, hardware, dataset, SLO).
Naming by date and config, to version the runs and compare over time.
A results store (a git repo of JSONs, or a bucket) so that anyone can reproduce and compare.

The goal is that reproducing a number is one command, not an afternoon. It is the basis of the harness in article S4, and what turns benchmarking from a one-off activity into a continuous platform capability.

CI integration: continuous benchmarking

The next level is measuring on every change: a CI job that, when the engine or the config is updated, launches a short sweep against a test environment and compares against the baseline. If goodput falls by more than a threshold, the pipeline fails. That way a performance regression is caught at the commit, not in production. Watch out for two things: benchmarking in CI consumes GPU-hours (budget for it) and it needs a stable environment (same hardware) for the comparison to be valid. You do not need the full sweep on every commit: a short sweep covering the knee is enough to detect regressions; the exhaustive sweep is for releases.

The cost of measuring (in euros)

Benchmarking consumes GPU-hours, and that has a cost worth budgeting for. A serious sweep on an 8×H100 node can occupy the cards for a couple of hours; at an amortised cost of ~11 €/h, that is ~22 € per full sweep, more if you sweep several models and precisions. It is not much per run, but a continuous benchmarking programme (every release, every config change) adds up. The practical rule: automate the harness so that each run is cheap and reproducible, and measure what you are going to use to decide, not for the sake of exhaustiveness. The cost of measuring is part of the cost of the platform, small next to the cost of serving but real.

Observing during the benchmark: one run, three axes

A trick that saves work and connects the series: while the sweep runs, capture the GPU metrics too. With DCGM exporting to Prometheus during the run, you record at the same time:

Axis	Source during the sweep	Metric
Performance	the tool (GuideLLM/AIPerf)	TTFT, ITL, throughput, goodput
Energy	DCGM	power (W) → J/token
Cost	node price (OpenCost)	€/hour → CPM per point

That way, from a single run you get the three numbers for the same operating point: at concurrency 16, the goodput (3,330 tok/s), the average power (from DCGM, which divided by the throughput gives J/token) and the cost per token (with the node price). Instead of three separate campaigns, you measure the three axes at once, and they end up coherent by construction because they correspond to the same instant and the same load. It is what the harness in article S4 does, and the reason to export DCGM during the benchmark even if you are only after performance: energy and cost come almost free if you capture them in the same window.

The methodological warning: align the time windows. The DCGM power figures and the tool’s metrics have to cover exactly the same interval (no warm-up, no shutdown), or the J/token does not correspond to the throughput measured. Same window, same three numbers.

The benchmark is also FinOps and energy

An idea that connects this article with the rest of the series: measuring performance is, in fact, measuring cost and energy. By the identity of the opening article, goodput is the denominator of cost per token and of energy per token. When a sweep reveals that config A gives 3,330 tok/s of goodput and config B gives 4,000, you are not only measuring speed: you are measuring that B costs fewer euros and fewer watts per token. That is why the output JSON of a benchmark should be accompanied by the iron cost (from OpenCost) to compute the real CPM of every point in the sweep: throughput × node price = cost per token. Benchmarking is not an isolated axis; it is the tool that, indirectly, moves the cost and the energy of the platform the most, and the one that fills the performance column of the scorecard with numbers that translate directly into euros.

The operational consequence: do not benchmark performance in a vacuum. Every run you save with its throughput and its goodput should be crossable with the node cost (€/hour) and the energy (J/token) to give the three faces of the same operating point. That is how the benchmarking track feeds the FinOps and energy tracks instead of living apart.

State of the art 2026 and limits

Migration to multi-process: AIPerf (formerly genai-perf) and GuideLLM consolidate the measurement of real capacity; micro-benches are left for tuning engines.
GuideLLM as the OSS standard for SLO-driven evaluation with an automatic report.
Careful with comparing across classes: a micro-bench and a load generator are not comparable; pin the class and the tool.
Version and dataset matter: the same command with a different dataset or version gives a different number; pin them.
MLPerf does not size your case: it compares vendors, it does not replace a sweep against your workload.

With the practical catalogue covered, the next article in the track (B3) goes into GuideLLM and SLO validation under load in depth. The tool is the means; the reproducible datum is the end.

Mistakes that invalidate a benchmark

To finish, the list of what turns a run into rubbish, by frequency:

Mistake	Effect	Fix
Comparing different classes	up to 7× of spurious difference	same tool for everyone
Not discarding the warm-up	artificially low TTFT	warm up and discard
Not going past the knee	you do not know the safe capacity	extend the sweep
Unrealistic dataset (fixed length)	throughput that does not apply	sharegpt or your own traces
Reporting the mean, not P99	hides the tail	percentiles always
Tokenizer foreign to the model	biased tok/s and cost per token	count with the model’s own
Not pinning version/config	irreproducible	save everything in the JSON

Any one of these is enough for the number not to be defensible. A benchmark is only as good as its methodology: the tool matters less than running it properly and recording everything.

Closing

The catalogue of benchmark tools boils down to a simple decision (micro-bench to tune an engine, load generator to measure capacity, MLPerf to compare vendors) and to a discipline that weighs more than the choice: the method. The same guidellm benchmark gives a golden datum or a useless one depending on the dataset, the warm-up, how far the sweep goes and what you record in the output. For a sovereign architecture proposal, performance only counts if it comes with its command, its version, its load and its goodput under SLO, and, crossed with cost in euros and energy per token, it becomes the scorecard column that decides which engine and which configuration hold up the platform. You choose the tool in five minutes; the methodology is what makes the number survive an audit.

Sources

vLLM · CLI de benchmark (bench serve) — https://docs.vllm.ai/en/latest/cli/bench/serve/
vLLM · bench sweep serve — https://docs.vllm.ai/en/latest/cli/bench/sweep/serve/
Red Hat · desplegar y benchmarkear vLLM con GuideLLM en Kubernetes — https://developers.redhat.com/articles/2025/12/24/how-deploy-and-benchmark-vllm-guidellm-kubernetes
GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
NVIDIA AIPerf · guía de benchmarking — https://lucaberton.com/blog/nvidia-aiperf-llm-inference-benchmarking-guide/
Medium · Benchmarking LLM Serving Performance (guía) — https://medium.com/@kimdoil1211/benchmarking-llm-serving-performance-a-comprehensive-guide-db94b1bfe8cf

LLM inference benchmarking: frameworks, metrics and the state of the art (tool by tool)

Sat, 13 Jun 2026 02:30:00 +0200

Notation: amounts in euros (N €), decimals with a point. Performance is not very sensitive to the country, but its associated cost (cost per token) is expressed in € and links back to the opening article.

What this introduction covers

Third article in the data series, a deep dive into the performance axis. Measuring the performance of an inference engine looks trivial (“how many tokens per second?”) and it is exactly where people fool themselves most: two tools can report results that differ by a factor of 7 for the same system. This article inventories the metrics that matter and how they are defined, why the architecture of the tool biases the figure, how the saturation point is found with a concurrency sweep, and the profile of each framework. No recommendations: the choice of engine is settled in the Pareto article (B8); here there are only the facts and the methodology, because a benchmark without a published methodology is not comparable.

The performance metrics

There is no single performance metric, there are five, and mixing them is the first source of error:

Metric	Definition	Unit	Dominant phase
TTFT (Time To First Token)	time from sending the prompt to the first token	ms	prefill
TPOT / ITL (Time Per Output Token / Inter-Token Latency)	average time between output tokens once generation has started	ms/token	decode
Request throughput	complete request-response cycles per second at the concurrency tested	req/s	both
Token throughput	total tokens (input + output) per second across all concurrent requests	tok/s	both
Goodput	share of requests that meet the defined SLO	useful tok/s	both
P50 / P95 / P99	latency percentiles (not the mean)	ms	—

Precise definitions, because every tool computes them its own way (Anyscale · latency and throughput metrics): TTFT is what a user waits before seeing the first character, dominated by prefill compute; ITL is the average time between successive output tokens and sets the perceived “typing speed” of the answer; request throughput is complete cycles per second; token throughput is total tokens (input plus output) per second across all concurrent requests.

The latency decomposition

The total latency of a request with $N$ output tokens decomposes as:

$$\text{latency} \approx \text{TTFT} + (N-1)\times \text{TPOT}$$

That is why TTFT and TPOT are reported separately: a single mean latency hides very different profiles. A system with high TTFT and low TPOT (expensive prefill, fast decode) and another one the other way round can have the same mean latency for one particular length, yet behave in opposite ways when the response size changes. For an interactive chat experience TTFT and TPOT rule; for a batch of long summaries, token throughput does. Measuring the mean hides both realities.

Goodput: the honest metric

Raw throughput (TPS, RPS) says how much work the system does; goodput says how much of that work meets your quality-of-service standards (SLO) (Anyscale). An engine can boast 10,000 aggregate tok/s, but if half the requests violate the P99 TTFT SLO, its goodput is 5,000. The figure you defend in a proposal is the goodput, not the catalogue throughput: it is the only one that translates into satisfied users and an honest cost per token.

How each metric is instrumented (and where the error creeps in)

Before comparing numbers it helps to know where each clock starts and stops, because two tools can call different things “TTFT”:

Client-side versus server-side TTFT. The TTFT measured by the client includes network latency and gateway queueing; the one measured by the server does not. For an engine comparison the server-side figure is the one that matters; for user experience, the client-side one. Mixing the two invalidates the comparison.
Streaming is required. TTFT and ITL can only be measured if the response arrives as a stream (token by token). If the tool measures complete responses, there is no real TTFT: there is total latency in disguise.
Token counting. Throughput in tok/s depends on which tokenizer counts the tokens. If the tool uses a tokenizer different from the model’s, the token count (and therefore the tok/s and the cost per token) is biased. You have to count with the tokenizer of the model being served.
Warm-up and prefix cache. The first requests of a benchmark benefit from a warm prefix cache and give artificially low TTFT; the warm-up has to be discarded or the result inflates reality.

These four instrumentation decisions explain a good share of the discrepancies between tools. A performance figure that does not say where the clock is measured and with which tokenizer is not comparable, however precise it may look.

The architecture of the tool biases the figure

Here is the trap that invalidates half the published benchmarks. Tools split into two classes by the architecture of the client that generates the load, and that architecture determines whether the measurement is reliable at high concurrency:

Single-process micro-bench (vLLM bench, SGLang bench, genai-perf): a Python client with asyncio in a single process. Useful for quick experiments on one engine, but the single-process architecture introduces a bottleneck on the client side that biases the data at high concurrency (genAI-perf and vLLM): the client cannot generate enough load and you measure the limit of the client, not the limit of the engine.
Multi-process load (GuideLLM, AIPerf): they spread load generation across several processes, avoiding that limit. This is the class that has emerged for measuring at real scale.

The size of the bias is enormous: at 1,000 QPS, a single-process benchmark processed 75,574 tokens against the 545,733 tokens of a distributed architecture, a discrepancy of 7.2× in measurement capacity for the same system ([search]). Anyone comparing two engines with tools from different classes is not comparing the engines: they are comparing the benchmark clients.

Frameworks, one by one

vLLM bench and SGLang bench — engine micro-bench

What they measure: TTFT, TPOT and throughput of the engine itself (vLLM or SGLang). Class: single-process micro-bench. Use: quick experiments to tune one engine and see the effect of its optimisations (see decode and prefill). Limit: they saturate on the client at high concurrency; they are no good for measuring real capacity at scale.

AIPerf — the NVIDIA one (formerly genai-perf), multi-process

What it measures: TTFT, ITL, throughput and latency against vLLM, NIM, TGI and any compatible endpoint. Class: multi-process load. State-of-the-art datum: NVIDIA retired genai-perf and replaced it with AIPerf on 15 April 2026. During the sweep, AIPerf detects GPU saturation and identifies the previous iteration, returning it as estimatedCapacity; if it detects no saturation, estimatedCapacity is the last iteration tested, which is why the sweep has to extend beyond the knee (AIPerf).

GuideLLM — from the vLLM project, SLO-oriented

What it measures: full distributions of TTFT, ITL and end-to-end behaviour, for SLO-driven evaluation. Class: multi-process load. Differentiator: it generates realistic, configurable traffic patterns in synchronous, concurrent and rate-based modes, including reproducible sweeps to identify safe operating ranges (Red Hat · GuideLLM, GuideLLM · GitHub). It is the tool for answering “how hard can I load this engine without breaking the SLO?”.

LLMPerf — the Anyscale/Ray classic

What it measures: throughput and latency at the inference level. Use: endpoint validation, historically very widespread. Class: load generator. Limit: less focused on distributions and sweeps than GuideLLM/AIPerf.

MLPerf Inference — the industry standard

What it measures: performance under standardised scenarios with strict rules, for comparability across vendors. Maintainer: MLCommons. It is the gold standard of cross-vendor comparability; it is expanded on below.

Comparison table

Tool	Class	What it measures	Maintainer	When to use it
vLLM bench	single-process micro	TTFT, TPOT, throughput of vLLM	vLLM (OSS)	tune vLLM, quick experiments
SGLang bench	single-process micro	metrics of the SGLang engine	SGLang (OSS)	tune SGLang
AIPerf (formerly genai-perf)	multi-process load	TTFT, ITL, throughput; estimatedCapacity	NVIDIA (OSS)	real capacity, multi-endpoint
GuideLLM	multi-process load	distributions, SLO, sweeps	vLLM (OSS)	validate SLO, find the knee
LLMPerf	load	throughput and latency	Anyscale/Ray (OSS)	endpoint validation
MLPerf Inference	standard suite	server/offline/interactive scenarios	MLCommons	cross-vendor comparability

The concurrency sweep: finding the knee

The most useful measurement for sizing is not a number, it is a curve: how latency and throughput change as concurrency rises. Raising concurrency keeps the GPU busier and raises RPS, but past a certain point it sends TTFT, ITL and end-to-end latency through the roof ([search]). The goal of the sweep is to find the knee: the maximum concurrency where throughput still rises without latency breaking the SLO.

That is why AIPerf extends the sweep beyond the knee: only by seeing where latency takes off can it return the safe capacity (estimatedCapacity). This curve is the raw material of capacity planning: the knee yields the replica count and the cost per token at the target load.

Worked example: reading a sweep

An illustrative sweep on an example node (a 70B on 8×H100, SLO of P99 TTFT < 500 ms), to see how the knee is read:

Concurrency	RPS	TTFT P50 (ms)	TTFT P99 (ms)	Token tput (tok/s)	Goodput
1	2	80	110	350	100 %
8	14	110	240	2,100	100 %
16	22	180	460	3,400	98 %
24	26	320	980	3,900	62 %
32	27	540	1,800	4,000	20 %

Reading: up to concurrency ~16 throughput grows and P99 stays under the SLO (goodput ~100 %). Between 16 and 24 sits the knee: throughput barely rises any more (3,400 → 3,900 tok/s) but P99 takes off (460 → 980 ms) and goodput collapses (98 % → 62 %). At concurrency 32 raw throughput is at its maximum (4,000 tok/s) but goodput is 20 %: the system “performs well” while serving mostly requests that violate the SLO. The defensible safe capacity is the one at concurrency ~16, not the one at maximum throughput. This is the number that goes into capacity planning and into cost per token: at 3,400 useful tok/s, not 4,000 raw tok/s.

MLPerf Inference: the comparability standard

To compare across vendors and engines with identical rules there is MLPerf Inference (MLCommons). The datacenter category focuses on two scenarios plus an optional one (MLCommons · datacenter):

Scenario	What it simulates	Metric
Offline	raw throughput processing the whole dataset in batch	maximum throughput
Server	interactive environment: requests one at a time following Poisson at a mean RPS	RPS under TTFT and TPOT limits
Interactive (optional)	like server but with stricter latency limits	RPS under a hard SLO

The Server scenario is the realistic one for online inference: the load generator sends requests following a Poisson distribution and demands that specific TTFT and TPOT bounds be met. MLPerf Inference v5.0 (April 2025) introduced a large-scale 405B benchmark and a low-latency interactive 70B one, offering language benchmarks at every scale (7B to 405B), architectural diversity (including MoE) and scenarios (MLCommons · v5.0); v5.1 (September 2025) widened the results with record participation (MLCommons · v5.1).

The value of MLPerf is comparability: everyone measures the same thing under the same rules. Its limit is that those rules may not match your workload (your length distribution, your specific SLO), so it serves to compare hardware and engines against each other, not necessarily to size your case. For that, your own sweep.

Measurement bias and reproducibility

That two benchmarks give very different results for the same system is no accident: systematic measurement bias in production benchmarks is characterised in the literature (arXiv 2605.24217), and there is work dedicated to the meta-metrics and good practices of system-level performance benchmarking (arXiv 2508.10251). The most common sources of bias:

Source of bias	Effect	Mitigation
Single-process client	underestimates throughput at high concurrency	use multi-process load
Unrealistic length distribution	results that do not apply to your traffic	use realistic traces
Measuring the mean instead of percentiles	hides the latency tail	report P95/P99
Uncontrolled warm-up	the prefix cache inflates the first results	discard the warm-up
Not pinning engine/model version	irreproducible	pin everything and publish it

The methodological conclusion of this article: a benchmark without a published methodology is not comparable. For a performance figure to support a proposal it has to come with the tool, its version, the model and precision, the load distribution and the SLO. Synthesis article S4 builds a reproducible harness that pins all of that.

Checklist for a reproducible benchmark

For a performance figure to be defensible before a committee, it has to come with everything that allows it to be reproduced. The minimum published alongside the result:

What to pin	Why
Tool + version	each one measures differently; the version changes behaviour
Model + precision (FP16/FP8/INT4)	precision changes throughput and quality
Hardware (GPU, count, interconnect)	an 8×H100 NVLink is not an 8×H100 PCIe
Engine + version + flags	vLLM/SGLang/TRT-LLM and their configuration
Length distribution (in/out)	real traffic is not fixed length
Concurrency levels of the sweep	you have to go past the knee
SLO (which percentile, which threshold)	it defines the goodput
Warm-up handling	discard it or it biases the result
Tokenizer used for counting	it affects tok/s and cost per token

The practical rule: if you cannot hand over this table alongside the figure, the figure is not a datum, it is an anecdote. The reproducible harness of article S4 automates the recording of all these parameters so that anyone, including whoever challenges the proposal, can reproduce the exact number.

Performance ≠ quality

One warning that avoids the most expensive mistake: these tools measure speed and throughput, not accuracy. An engine can be blisteringly fast while serving a model that answers badly. Quality is measured with another family of tools (lm-evaluation-harness, HELM, task leaderboards) which is another axis of the scorecard (article B7). Confusing “fast” with “good” is how you build a platform that serves bad answers very quickly. On the final Pareto frontier, performance and quality are two distinct axes that must be looked at together, never one instead of the other.

The other family of tools, for reference (expanded in B7):

Tool	What it measures	Common trap
lm-evaluation-harness	accuracy on hundreds of standardised tasks	contamination of the test dataset
HELM	holistic evaluation (accuracy, robustness, bias, efficiency)	heavy to run
LiveBench / dynamic leaderboards	tasks that rotate to avoid contamination	comparability over time

The data point: contamination, the model having seen the test during training, inflates quality metrics just as warm-up inflates performance ones. That is why dynamic leaderboards rotate their questions. Quality and performance share that lesson: the measurement method biases the result as much as the system being measured.

The latency versus throughput trade-off

A property the sweep reveals and which is worth keeping in mind: latency and throughput pull in opposite directions. Batching groups requests to amortise the cost of moving the weights out of VRAM, which raises throughput, but each request waits for the batch to form, which raises individual latency. There are two operating regimes, and the benchmark serves to place you in the right one:

Regime	Optimises	Configuration	Case
Latency	low TTFT/TPOT	small batch, low concurrency	interactive chat, copilots
Throughput	maximum tok/s	large batch, high concurrency	overnight batch, ingestion

There is no single “best” point: there is the best point for your SLO. A benchmark that reports only maximum throughput is describing the throughput regime and ignoring whether that point meets the latency your case needs. That is why goodput, throughput under the SLO, is the metric that reconciles the two regimes: it measures how much throughput you get without stepping outside acceptable latency. The sweep traverses the curve between both regimes; your SLO marks where on that curve your system sits.

The connection with cost and energy

Performance is not an isolated axis: by the identity of the opening article, throughput is the denominator of cost per token and of energy per token. A sweep that finds a knee at 4,000 tok/s rather than 2,800 is not just “faster”: it lowers the CPM from ~1.09 to ~0.76 €/1M tok and the energy per token in the same proportion, on the same iron. That is why performance benchmarking is the tool that, indirectly, moves cost the most: every goodput improvement translates into euros and into watts per token. The number that connects the three axes is goodput, the throughput that meets the SLO, not catalogue throughput.

State of the art 2026

genai-perf → AIPerf migration (15 Apr 2026): NVIDIA consolidates its benchmarking into a multi-process tool with saturation detection.
GuideLLM as the OSS standard for SLO-driven evaluation with reproducible sweeps.
MLPerf Inference v5.0/v5.1 extends to 405B, interactive 70B and MoE, with record participation: cross-vendor comparability is maturing.
Measurement bias characterised: the community accepts that the measurement method matters as much as the system measured; the emphasis on reproducibility and meta-metrics is growing.

Limits and traps (data-driven)

Comparing tools from different classes. A single-process micro-bench and a multi-process load generator are not comparable; the difference can be 7×. Pin the class.
Catalogue throughput instead of goodput. The honest number is the one that meets the SLO.
Means instead of percentiles. The mean hides the tail; report P95/P99.
Not extending the sweep beyond the knee. Without seeing where latency takes off you do not know the safe capacity.
Confusing performance with quality. They are distinct axes; fast is not good.
Not pinning versions. Engine, model, precision and load left unpinned = irreproducible = not defensible.

The next article in the track (B2) goes into the tool catalogue in depth; this one pins the metrics and the methodology. With performance measured reproducibly, the scorecard can cross it with cost (in €) and energy for the final decision.

Closing

Performance benchmarking looks like the most “objective” of the three axes (they are tokens per second, after all) and it is exactly where the most manipulation happens, almost always without bad intent: a single-process tool here, a mean instead of a P99 there, a catalogue throughput instead of the goodput. The difference between a marketing number and a defensible datum lies not in the engine measured but in the methodology: the class of tool, where the clock is measured, with which tokenizer, how far the sweep goes and which SLO defines the goodput. For a sovereign architecture proposal, performance only counts if it is delivered with that reproducibility record, and, crossed with cost in euros and energy per token, it becomes the column of the Pareto frontier that decides which engine and which configuration hold up the platform. The number you defend is not the highest one: it is the reproducible goodput.

Sources

Anyscale · métricas de latencia y throughput de LLM — https://docs.anyscale.com/llm/serving/benchmarking/metrics
Red Hat · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
NVIDIA AIPerf · guía de benchmarking (ex genai-perf) — https://lucaberton.com/blog/nvidia-aiperf-llm-inference-benchmarking-guide/
MLCommons · MLPerf Inference datacenter (escenarios) — https://mlcommons.org/benchmarks/inference-datacenter/
MLCommons · MLPerf Inference v5.0 (405B + 70B interactivo) — https://mlcommons.org/2025/04/llm-inference-v5/
MLCommons · MLPerf Inference v5.1 — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
arXiv 2605.24217 · sesgo sistemático de medición en benchmarks de inferencia LLM — https://arxiv.org/html/2605.24217
arXiv 2508.10251 · meta-métricas y buenas prácticas de benchmarking de rendimiento — https://arxiv.org/pdf/2508.10251
Medium · LLM Inference Benchmarking (genAI-perf y vLLM) — https://kchandan.medium.com/llm-inference-benchmarking-genai-perf-and-vllm-5dd06b57428e

Mlperf on lo0 — Tech Blog

MLPerf Inference: how to read it, what LoadGen enforces and what comparability it offers

TL;DR

What it is and how it is organised

The four scenarios

LoadGen: what it measures and what it does not

Duration, query count and early stopping

The compliance tests

Divisions, categories and what can be touched

The three divisions

The availability categories

What Closed allows and forbids

The SLOs almost nobody cites: TTFT and TPOT

Quality: the 99 % and the token count

The current Datacenter suite

How to read a results row

Per-accelerator is not an official metric

The comparison rules

How much audit sits behind a row

The recent rounds, in figures

What MLPerf Inference does not measure

Usage flow for an on-premise platform

Cross-links from the benchmarking track

See also

Sources

Storage in the AI era (2/4): performance

The metric that really matters: accelerator utilisation

MLPerf Storage: the benchmark that does measure the right thing

The numbers from commercial platforms

The I/O pattern of training and GPUDirect Storage

The real cost of checkpointing (when there is training)

Performance in an inference factory

Prefill, decode and their disaggregation

The KV-cache as a performance layer

Model load time and cold start

RAG performance and vector search

How to measure your own: beyond the spec sheet

The cost of getting it wrong

The physical layer: networks and PCIe Gen6

Three questions before sizing

Takeaways

See also

Sources

LLM benchmark tool catalogue: the practical deep dive

What this article covers

How to choose: the tool map

Datasets: the synthetic load matters as much as the tool

vLLM bench serve

SGLang bench

AIPerf (NVIDIA, formerly genai-perf)

GuideLLM (vLLM project)

LLMPerf (Anyscale/Ray)

inference-benchmarker (Hugging Face)

GuideLLM versus AIPerf: which of the two load generators

MLPerf Inference (MLCommons)

Practical comparison table

Summary: what you run for each question

Step-by-step benchmark methodology

Comparing two engines: the fair protocol

Output format and comparability

Worked example: reading the output of a sweep

Automating the harness

CI integration: continuous benchmarking

The cost of measuring (in euros)

Observing during the benchmark: one run, three axes

The benchmark is also FinOps and energy

State of the art 2026 and limits

Mistakes that invalidate a benchmark

Closing

See also

Sources

LLM inference benchmarking: frameworks, metrics and the state of the art (tool by tool)

What this introduction covers

The performance metrics

The latency decomposition

Goodput: the honest metric

How each metric is instrumented (and where the error creeps in)

The architecture of the tool biases the figure

Frameworks, one by one

vLLM bench and SGLang bench — engine micro-bench