Benchmarking on lo0 — Tech Blog

MLPerf Inference: how to read it, what LoadGen enforces and what comparability it offers

Sat, 29 Aug 2026 09:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

MLPerf Inference is maintained by MLCommons on a six-monthly cadence. It imposes three things no home-made benchmark has: a mandatory load generator (LoadGen, compiled from a tagged revision and unmodified), model-equivalence rules that bound what can be optimised, and an audit process that reviews up to two submissions per round. The four scenarios (SingleStream, MultiStream, Server, Offline) have a minimum duration of 600 s and different metrics: Server reports the sustained Poisson rate and is the only one with a latency constraint. For the LLM benchmarks that constraint is not a request latency but two simultaneous SLOs at the 99th percentile: TTFT and TPOT, with per-model figures ranging from 2000 ms / 200 ms on Llama 2 70B to 450 ms / 40 ms on its Interactive variant. Quality is required relative to the reference model (99 % or 99.9 %) and the generation length is audited as well (Llama 2 70B: more than 90 % of 294.45 tokens per sample). Three reading rules sum up the real comparability: results are only compared within the same version, division, category and scenario; per-accelerator is not an official MLCommons metric, it is derived; and neither cost nor energy enters the primary metric, to the point that round v5.1 had 2 power submissions against 27 participating organisations.

What it is and how it is organised

MLPerf Inference is maintained by the MLCommons Association through the MLPerf Inference Working Group, whose participation is limited to MLCommons members and affiliates (MLPerf Inference Working Group). The declared cadence for publishing results is roughly six months.

Round	Results published
v4.0	27 March 2024
v4.1	28 August 2024
v5.0	2 April 2025
v5.1	9 September 2025
v6.0	1 April 2026
v6.1	in preparation as of this data sheet

The suite splits into two system categories: Datacenter and Edge, the latter defined in the rules as everything that is not datacenter (inference_rules.adoc). A Datacenter system has two requirements that do not apply in Edge:

Mandatory ECC on DRAM and HBM, active during all performance and accuracy runs. There is no requirement on SRAM.
Mandatory networking since round v3.0, with a minimum bandwidth computed from the achieved throughput. For Llama 3.1 405B the minimum ingress is throughput × 20000 × dtype_size bytes/s; for Llama 2 70B the minimum egress is throughput × 1024 × dtype_size.

Edge additionally allows inferred results: a MultiStream derived from SingleStream counts as 8 times the p99 latency; an Offline derived from MultiStream, 8000 divided by the mean latency in milliseconds.

The four scenarios

This is the normative table, transcribed from the rules:

Scenario	Query generation	Duration	Samples/query	Latency constraint	Tail latency	Reported metric
SingleStream	The next query is issued when the SUT completes the previous one	600 s	1	None	90 %	p90 latency with early stopping
Server / Interactive	LoadGen issues queries following a Poisson distribution	600 s	1	Benchmark-specific	99 %	Maximum Poisson parameter supported
Offline	All samples are delivered at the start in a single query	1 query, 600 s	At least 24,576	None	N/A	Measured throughput
MultiStream	The next query is issued when the SUT completes the previous one	600 s	8	None	99 %	p99 query latency with early stopping

Four operational clarifications that change how each number is interpreted:

In Server, LoadGen does not measure a latency: it runs a binary search over the target rate. For a given value it generates queries at that QPS with Poisson arrivals; if the run fails the latency gate, it lowers the value and repeats. The final published metric is, per the official FAQ, scheduled samples per second, not completed samples nor the target entered by the user.
In MultiStream, the latency of a query is the maximum of the latencies of its samples, and the 8 samples of each query are contiguous in the order in which they were loaded. The value multi_stream_samples_per_query = 8 is fixed in the official configuration.
In Offline there is no latency constraint at all. The rules themselves forbid techniques that only work in fixed-length experiments except in the Offline scenario, which is a normative acknowledgement that an Offline result says nothing about behaviour under an SLO.
The percentiles in the table are the lower theoretical bound for runs with very many queries. With early stopping, the percentile actually computed is somewhat above the nominal one, as a penalty for processing few queries.

The mandatory scenarios depend on the benchmark. In Datacenter, the summarisation, question answering, text generation and VLM tasks require (Server | Interactive), Offline, that is, the submitter chooses between Server and Interactive. Reasoning and recommendation require Server, Offline. Medical segmentation, node classification, speech-to-text and RAG require only Offline.

LoadGen: what it measures and what it does not

LoadGen is the module that generates the load and computes the metrics. Its use is mandatory for all submissions, and it must be compiled from a tagged and approved revision of the repository, unaltered; the README is explicit that local modifications of the C++ library are not accepted.

Its four declared responsibilities are generating the queries according to the scenario, tracking the latency of each query, validating the accuracy of the results and computing the final metrics. Latency is defined as the time from when LoadGen was scheduled to pass a query to the system under test until it receives the response.

Equally important is what it declares out of scope: LoadGen does not know the model, does not know the data formats, cannot score accuracy and does not know the scenario constraints of the MLPerf rules. The consequence is written in its own README: being model-agnostic, it cannot enforce the MLPerf requirements, for example the target percentiles and latencies. Those limits are injected from mlperf.conf and user.conf. The justification for making it mandatory appears in the founding ISCA 2020 paper: it establishes a clear boundary between the components that belong to the submitter and those that belong to MLPerf, and it measures the performance of the complete system instead of that of a single piece.

Duration, query count and early stopping

The minimum duration is 600,000 ms in all four scenarios. The minimum number of queries comes from a confidence interval table that the rules publish:

Tail percentile	Confidence	Margin of error	Inferences	Rounded
90 %	99 %	0.50 %	23,886	24,576
95 %	99 %	0.25 %	50,425	57,344
97 %	99 %	0.15 %	85,811	90,112
99 %	99 %	0.05 %	262,742	270,336

The submission checker applies Server: 270336, SingleStream: 1024 and MultiStream: 662. For Offline, the minimum is set per benchmark: 24,576 on Llama 2 70B, 15,000 on Mixtral 8x7B, 13,368 on Llama 3.1 8B, 8,313 on Llama 3.1 405B, 4,388 on DeepSeek-R1, 1,633 on Whisper and 43 on 3D UNet.

The early stopping mechanism allows runs to be shortened while keeping the statistical guarantee. With tolerance d = 0 and confidence c = 0.99, the algorithm resolves by binary search the smallest number of queries below the latency threshold that satisfies the criterion, given the number of queries observed above it.

LoadGen’s modes are SubmissionRun (accuracy followed by performance), AccuracyOnly, PerformanceOnly and FindPeakPerformance. The last, applicable only to Server, takes target_qps as a lower bound if it passes, estimates the upper bound at double and doubles it until failure, and then does a binary search. In performance mode LoadGen selects queries uniformly at random with replacement from a set of size QSL; in accuracy mode it uses a copy of the validation dataset, each sample exactly once. One accuracy run must be executed for every performance result submitted, and the same code in both modes.

The seeds are announced four weeks before the deadline and the mandatory generator is Mersenne Twister 19937. They change every round: in v5.1, qsl_rng_seed = 1780908523862526354; in v5.0 it was 6023615788873153749.

The compliance tests

They are activated by placing an audit.config file in the working directory, whose parameters override those of mlperf.conf and user.conf. Their logs are mandatory in the submission package. The declared purpose is to detect anomalies, not to diagnose their cause.

For the LLM benchmarks the required test is TEST06, specifically against the end-of-sequence token exploit. It runs 100 samples and demands three conditions: that the first token reported separately matches the first token of the model’s output, that the output ends with zero or a single EOS token, and that the number of tokens reported matches the real length of the generated sequence.

Divisions, categories and what can be touched

The three divisions

Closed: requires preprocessing, postprocessing and model equivalent to the reference implementation. It allows calibration for quantisation and does not allow retraining. It is the only one that can use the MLPerf name unqualified.
Open: allows arbitrary preprocessing, postprocessing and model, including retraining. The accuracy, latency and scenario constraints do not apply: instead, the accuracy obtained and the latency constraints under which the performance was obtained must be reported. The model may have any origin, be quantised in any way and be sparsified in any way. It must use the same validation dataset as the corresponding Closed and use all of it.
Network: inherits all the Closed requirements, applies only to Datacenter, and separates the LoadGen node from the system under test through a fabric. The submitter’s dispatch library cannot preprocess, postprocess, batch, pad or cache. Bus interconnects are forbidden by name (PCIe, CXL, CCIX, HyperTransport, NVLink, QPI, UPI, ICI) and only Ethernet, IEEE 802.11, InfiniBand and 3GPP are admitted, with a requirement to work chassis to chassis at more than ten metres.

The availability categories

Category	Hardware	Software
Available in cloud	Available for cloud rental	Available
Available on premise	Available for purchase	Available
Preview	Must be available for the next submission, or the one after 140 days, whichever is longer	Available except what is needed for substantially new hardware
RDI	Does not meet the above	Does not meet the above

Available demands four cumulative conditions: price available, having been rented or shipped to at least one third party, public evidence of availability and reasonable availability to additional third parties on the submission date. A Preview result obliges a resubmission as Available with equal or better performance, tolerating up to 2 % degradation from noise; if there is no resubmission, the Preview result is marked invalid. RDI components cannot be submitted as Available until the cycle after the next one, or 221 days, whichever is longer.

What Closed allows and forbids

MLPerf delivers the weights in fp16 or fp32. The central rule is that the submitter may perform purely mathematical and reproducible quantisation, using only the calibration data and the tensors of the delivered model, to any numerical format that reaches the required quality, and that the method must be described publicly at a level that allows it to be reproduced. The test against back doors is elegant: the description of the quantisation method must be much smaller than the non-zero weights it produces.

Allowed, among other things: any framework or runtime, arbitrary data layout, varying the matrix multiplication algorithm, mathematically equivalent transformations, polynomial approximations of transcendentals, processing queries out of order within what the scenario admits, replacing dense operations with mathematically equivalent sparse operations, hand-picking different precisions per operation, fusing and unfusing, dynamic batching, and mixtures of experts combining weights with different quantisation.

Forbidden: replacing or supplementing weights wholesale, discarding non-zero weights, including pruning, caching queries or responses, coalescing identical queries, modifying weights during the timed portion, quantisation algorithms of size comparable to the weights they produce, hardcoding the total number of queries, using knowledge of the LoadGen implementation to predict peaks or troughs in the Server scenario, changing the number of search beams, and incorporating statistics from the performance or accuracy sets.

Two clarifications that matter to anyone serving LLMs in production: the KV cache is allowed just as in the reference model provided it is not applied across queries; PagedAttention is admitted if the block is reused only within the batch; continuous or dynamic batching is allowed; and KV cache entries are treated as activations for quantisation purposes, and cannot be pruned. Speculative decoding is only admitted in the benchmark and scenario combinations explicitly listed, with the reference head at the same precision, and implementations that artificially manipulate the acceptance rate are forbidden.

The SLOs almost nobody cites: TTFT and TPOT

This is the point where an MLPerf table is most often misread. In the LLM benchmarks of the Server scenario, the constraint is not a request latency: the rules set target_latency = 0 and activate use_token_latencies = 1, so that the gate is formed by two simultaneous metrics, both evaluated at the 99th percentile:

TTFT (time to first token): latency of the first token.
TPOT (time per output token): mean interval between all generated tokens.

The figures are in mlperf.conf and change per round and per benchmark:

Benchmark (Server scenario)	TTFT	TPOT	Round
`llama2-70b`	2000 ms	200 ms	v5.0, v5.1, v6.x
`llama2-70b-interactive`	450 ms	40 ms	v5.1, v6.x
`mixtral-8x7b`	2000 ms	200 ms	v5.0, v5.1, v6.x
`llama3_1-405b`	6000 ms	175 ms	v5.0, v5.1, v6.x
`llama3_1-405b-interactive`	4500 ms	80 ms	v6.x
`llama3_1-8b`	2000 ms	100 ms	v5.1, v6.x
`llama3_1-8b-interactive`	500 ms	30 ms	v6.x
`deepseek-r1`	2000 ms	80 ms	v5.1, v6.x
`deepseek-r1-interactive`	1500 ms	15 ms	v6.x
`gpt-oss-120b`	3000 ms	80 ms	v6.x

A TPOT of 40 ms is equivalent to 25 tokens per second per user, which is the order of magnitude of a conversational interface; one of 200 ms is equivalent to 5 tokens per second, which is a batch use case dressed up as a service. Two results for the same model under the Server and Interactive labels are not the same measurement. Two documented discrepancies between primary sources remain: the llama2-70b-interactive variant does not appear in the mlperf.conf tagged as v5.0 despite having been announced in that round, and in the v6.x cycle the TPOT of gpt-oss-120b-interactive appears as 15 ms in mlperf.conf and as 20 ms in the text of the rules.

For contrast, the constraints of the non-LLM benchmarks in the Server scenario are conventional request latencies: ResNet-50 15 ms, RetinaNet 100 ms, BERT 130 ms, DLRMv2 60 ms, DLRMv3 80 ms, RNN-T 1000 ms, GPT-J and Stable Diffusion XL 20,000 ms.

Quality: the 99 % and the token count

The quality targets are expressed relative to the reference model, never in absolute terms, and each benchmark demands one variant, the other or both. In the current Datacenter suite, 3D UNet, Llama 3.1 8B and Whisper ask for 99 % and 99.9 % of FP32, which generates two distinct results; Llama 2 70B asks only for 99.9 %; Llama 3.1 405B, Mixtral, DeepSeek-R1, GPT-OSS-120B, Qwen3-VL, RGAT and WAN-2.2 ask only for 99 %; and E2E-RAG is the exception at 97 %. In the checker this materialises as separate benchmarks (bert-99 and bert-99.9), each with its computed numerical threshold.

On top of that metric there is a second LLM-specific constraint that is cited very rarely: the generation length is audited as well.

Benchmark	Length constraint
Llama 2 70B	tokens per sample > 90 % of 294.45
Llama 3.1 405B	tokens per sample between 90 % and 110 % of 684.68
Mixtral 8x7B	tokens per sample between 90 % and 110 % of 144.84
Llama 3.1 8B	total generated length > 90 % of 8,167,644

The FAQ settles the obvious shortcut: reducing the maximum output length below the reference is not allowed, and truncating tokens to improve performance or reach the accuracy target is not permitted. The inference parameters are fixed in Closed (max_new_tokens = 1024 on Llama 2 70B, 20000 on Llama 3.1 405B and DeepSeek-R1), as is the decoding algorithm: greedy search in the Llama family, Mixtral and DeepSeek-R1; sampling with temperature = 1.0 and top_p = 1.0 in GPT-OSS-120B and Qwen3-VL. Accuracy is reported to five significant figures with round-half-to-even.

The current Datacenter suite

Area	Task	Model	Dataset	QSL	Required quality
Vision	Medical segmentation	3D UNet	KiTS 2019	42	99 % and 99.9 % of FP32 (DICE 0.86330)
Language	Summarisation	Llama 3.1 8B	CNN DailyMail v3.0.0	13,368	99 % and 99.9 % of FP32
Language	Question answering	Llama 2 70B	OpenOrca	24,576	99.9 % of FP32
Language	Text generation	Llama 3.1 405B	LongBench, Ruler, GovReport	8,313	99 % of FP16
Language	QA, maths and code	Mixtral 8x7B	OpenOrca, GSM8K, MBXP	15,000	99 % of FP16
Language	Reasoning	DeepSeek-R1	`mlperf_deepseek_r1`	4,388	99 % of FP16 (exact match 81.9132 %)
Language	QA, maths and code	GPT-OSS-120B	AIME25, GPQA Diamond, LiveCodeBench v6	6,396	99 % of 83.13 %
Vision	Vision-language model	Qwen3-VL-235B-A22B	Shopify product catalogue	48,289	99 % of BF16 (hierarchical F1 0.7824)
Commerce	Recommendation	DLRMv3	Synthetic Streaming 100B	34,996	99.9 % of FP32
Generative	Text to video	WAN-2.2-T2V-A14B	VBench prompts	248	99 % of BF16 (VBench 69.7752)
Graphs	Node classification	RGAT	IGBH	788,379	99 % of FP32 (72.86 %)
Audio	Speech to text	Whisper	LibriSpeech	1,633	99 % and 99.9 % of FP32 (WER 2.0671 %)
Language	End-to-end RAG	E2E-RAG	FRAMES	824	97 % of FP32

The E2E-RAG benchmark deserves attention from anyone designing a RAG platform, because it fixes the whole pipeline: a FAISS HNSW index with mandatory parameters M = 32, efConstruction = 200 and efSearch = 100; e5-base-v2 embeddings; ColBERTv2 reranking; generation and query rewriting with GPT-OSS-120B; evaluation with Llama 3.1 8B as judge; and a maximum of 5 retrieval iterations, mandatory for all submitters. Model loading is not timed; building the vector database is.

The definition of sample changes per model and is a frequent source of confusion: in the LLMs it is a sequence, in DLRMv3 it is a request with a user history and 2048 candidates, in WAN-2.2 it is a pair of positive and negative prompts, and in PointPainting it is five images and a lidar point cloud.

How to read a results row

The columns of the official table in the Closed division are Submitter, Software, System, Benchmark Results, Processor/Count, Details, Accelerator/Count and Code. The Open division adds Model Used and Notes. Rows with power measurement add System Power in Server and Offline, or Energy Per Stream in the stream scenarios. Any citation must use the identifier in the format major-version.minor-version.entry.benchmark, of the style 5.1-0053, and carry a footnote with suite, version, division, benchmark, scenario, date and source.

Per-accelerator is not an official metric

There is no per-accelerator column. It is a derived metric, and the MLCommons messaging guidelines are blunt: any comparison based on a different or derived metric, such as power, cost, model size or accuracy, must make the basis of comparison clear in the text and in a footnote, and secondary and derived metrics cannot be presented as official or verified MLPerf metrics. NVIDIA itself states this with the standard formula in its round blogs: per-GPU performance is not a primary MLPerf Inference metric and is computed by dividing the reported throughput by the number of reported accelerators. The same applies to combining results from several benchmarks: MLCommons allows it but does not endorse it, and the composite cannot be presented as an official result.

The comparison rules

MLPerf results can only be compared with compatible MLPerf results: same benchmark, same scenario and compatible versions per the normative compatibility table.
MLPerf results cannot be compared with non-MLPerf results.
When comparing, any difference in version, division, category, verification status, scenario or number of chips must be clearly identified. When comparing Open with Closed, it must be identified in what sense the Open result would not qualify as Closed.
Submitters cannot publish results of a version before its official date; non-submitters must wait two weeks from that date.
The sanctions regime goes as far as barring an infringer from submitting results in the future and marking their results as non-compliant permanently in the database, with a deadline of three working days to withdraw the infringing content.

How much audit sits behind a row

In each round up to two submissions are audited: one chosen at random from all of them and zero or one chosen by the review committee. And there is a constraint that conditions the reading of the whole table: only Available submissions in the Closed division are auditable. Preview, RDI and Open fall outside the process. There is an exemption from the random audit if the system is equivalent to another already audited and neither the aggregate performance nor the per-accelerator performance differ by more than 10 % from those of the previous audit. The deadlines are 28 days to select an auditor, 30 for the report after signing the confidentiality agreements and around 90 days for the complete process, with two days of hardware access for the auditor. The rules close with three sentences that define the standard of proof: results that cannot be replicated are not valid results, benchmark detection is not allowed and input-based optimisation is not allowed either.

The recent rounds, in figures

Round	Organisations	Performance results	Power results	New benchmarks
v4.0	23	more than 8,500	900	2
v4.1	22	964	31	1
v5.0	23	17,457	n/a	4
v5.1	27	n/a	2	3
v6.0	24	n/a	n/a	5 of 11 Datacenter tests

These counts do not form a time series: the counting criteria differ between press releases and none documents the criterion, to the point that v4.0 and v4.1 differ by a factor close to nine.

The published performance jumps are comparable within each announcement. Between v4.0 and v5.0, twelve months apart, the number of Llama 2 70B submissions multiplied by 2.5, the median score doubled and the best score was 3.3 times faster. Between v5.0 and v5.1, in six months, the best systems improved by up to 50 % in some scenarios. Round v6.0 changed axis and its press release talks about scale rather than speed: 30 % more multi-node submissions, 10 % of the systems with more than ten nodes against 2 % in the previous round, and a system larger than 72 nodes and 288 accelerators, quadrupling the previous maximum node count.

Hardware that debuted in each round: MI300X, TPU v6e, Xeon Granite Rapids and B200 in v4.1; MI325X, Xeon 6980P, GB200 and Jetson AGX Thor in v5.0; MI355X, Intel Arc Pro B60, GB300 and RTX Pro 6000 Blackwell Server Edition in v5.1.

What MLPerf Inference does not measure

Cost. There is no cost or TCO metric, and cost is explicitly classified as a derived metric that cannot be presented as official.
Energy, except as an optional extension. MLPerf Power is regulated in a separate document, requires measuring at the wall with the power group’s workflow and PTDaemon, and forbids any other method. Its real adoption is marginal: 2 power submissions in v5.1 against 27 participating organisations. In addition, MLCommons bounds what that figure means: the measured power is only valid for the benchmark it accompanies, and any other reference to power, such as a TDP setting or a power supply rating, is neither measured nor validated by MLCommons. The messaging rule forbids submitters from publishing per-watt normalised comparisons using any proxy other than the measured power.
Configurations not optimised by the vendor. The bias is acknowledged in the audit rule itself, which contemplates submissions whose performance is not consistent with the known characteristics of the hardware, or where the committee lacks visibility into how it was achieved, or where hardware and software are not reasonably available to the public.
Barrier to entry. Only MLCommons members and test partners can submit results for review. Anyone measuring with the same code without submitting must label every figure as unverified with the corresponding legend.

Usage flow for an on-premise platform

MLPerf Inference is useful in a purchasing decision if it is used for what it is, a test bench with written rules, and not as a ranking.

Filter by division and category before looking at any number: keep Closed and Available. A Preview or RDI result describes hardware that cannot yet be bought, and neither of the two is auditable.
Choose the scenario that matches the load pattern: Server or Interactive if there are users waiting, Offline if the work is batch. An Offline result does not authorise promising a latency.
Read the SLO before the throughput: in any LLM benchmark, the number of queries per second only means something alongside its TTFT/TPOT pair. The same machine publishes very different figures under Server and under Interactive.
Check the quality variant: 99 % and 99.9 % are different results for the same model, and the second usually costs performance.
Translate to your own configuration carefully: dividing by the number of accelerators is a derived metric, not an official one, and it has to be declared as such.
Close the gap MLPerf leaves open with your own measurement: cost, energy and behaviour under real traffic. That is exactly the function of the reproducible harness and of the track B bench tools, and the reason no platform decision stands on a public table alone.

Cross-links from the benchmarking track

B1/B2 — LLM inference benchmarking: frameworks, metrics and state of the art: the TTFT, TPOT and goodput metrics outside the MLPerf corset, and which tool measures each one.
B2 — LLM benchmark tool catalogue: the tools with which you reproduce at home what MLPerf standardises.
B3 — GuideLLM and SLO validation under load: the practical equivalent of LoadGen’s binary search in the Server scenario.
B6 — Measurement bias and reproducibility: why LoadGen exists, told from the side of what happens when there is none.
B8 — Serving engines compared on the Pareto frontier: the comparison MLPerf does not make, with engines instead of systems.
C4 — MLPerf Power: the energy extension of this same machinery, with its certified measurement at the wall.

Sources

MLCommons, MLPerf Inference Working Group — https://mlcommons.org/working-groups/benchmarks/inference/
MLCommons, MLPerf Inference Rules (inference_rules.adoc) — https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc
MLCommons, MLPerf Inference Power Measurement (power_measurement.adoc) — https://github.com/mlcommons/inference_policies/blob/master/power_measurement.adoc
MLCommons, MLPerf Submission Rules (submission_rules.adoc) — https://github.com/mlcommons/policies/blob/master/submission_rules.adoc
MLCommons, MLPerf Results Messaging Guidelines — https://github.com/mlcommons/policies/blob/master/MLPerf_Results_Messaging_Guidelines.adoc
MLCommons, LoadGen README — https://github.com/mlcommons/inference/blob/master/loadgen/README.md
MLCommons, loadgen/test_settings.h — https://github.com/mlcommons/inference/blob/master/loadgen/test_settings.h
MLCommons, loadgen/mlperf.conf (master) — https://github.com/mlcommons/inference/blob/master/loadgen/mlperf.conf
MLCommons, loadgen/mlperf.conf (tag v5.1) — https://github.com/mlcommons/inference/blob/v5.1/loadgen/mlperf.conf
MLCommons, loadgen/mlperf.conf (tag v5.0) — https://github.com/mlcommons/inference/blob/v5.0/loadgen/mlperf.conf
MLCommons, tools/submission/submission_checker.py (tag v5.0) — https://github.com/mlcommons/inference/blob/v5.0/tools/submission/submission_checker.py
MLCommons, compliance/nvidia TEST06 README (tag v5.1) — https://github.com/mlcommons/inference/blob/v5.1/compliance/nvidia/TEST06/README.md
MLCommons, Benchmark results: Inference Datacenter — https://mlcommons.org/benchmarks/inference-datacenter/
Reddi et al., MLPerf Inference Benchmark, ISCA 2020 — https://arxiv.org/abs/1911.02549
MLCommons, MLPerf Inference v5.0 Benchmark Results (abril 2025) — https://www.businesswire.com/news/home/20250402313932/en/MLCommons-Releases-New-MLPerf-Inference-v5.0-Benchmark-Results
MLCommons, MLPerf Inference v5.1 Benchmark Results (septiembre 2025) — https://www.globenewswire.com/news-release/2025/09/09/3147136/0/en/MLCommons-Releases-New-MLPerf-Inference-v5-1-Benchmark-Results.html
MLCommons, MLPerf Inference v6.0 Benchmark Results (abril 2026) — https://www.globenewswire.com/news-release/2026/04/01/3266801/0/en/MLCommons-Releases-New-MLPerf-Inference-v6-0-Benchmark-Results.html
MLCommons, MLPerf Inference v4.1 Benchmark Results (agosto 2024) — https://www.businesswire.com/news/home/20240828886616/en/New-MLPerf-Inference-v4.1-Benchmark-Results-Highlight-Rapid-Hardware-and-Software-Innovations-in-Generative-AI-Systems
NVIDIA Developer Blog, NVIDIA Blackwell delivers massive performance leaps in MLPerf Inference v5.0 — https://developer.nvidia.com/blog/nvidia-blackwell-delivers-massive-performance-leaps-in-mlperf-inference-v5-0/
NVIDIA Developer Blog, NVIDIA Blackwell Ultra sets new inference records in MLPerf debut — https://developer.nvidia.com/blog/nvidia-blackwell-ultra-sets-new-inference-records-in-mlperf-debut/
MLCommons, MLPerf Automotive (octubre 2025) — https://arxiv.org/html/2510.27065v1
Tschand et al., MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts — https://arxiv.org/abs/2410.12032

The reproducible harness: measuring cost, performance and energy in a single auditable experiment

Tue, 16 Jun 2026 12:00:00 +0200

Notation: amounts in euros (N €), decimals with a point, thousands with a thin space. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

The “data” series has produced three independent measurement axes: cost per million tokens (OpenCost + LiteLLM), performance under an SLO (GuideLLM + AIPerf) and energy per token (DCGM + Kepler). The problem is that the three have been measured in different articles, with different loads and at different times: they are not comparable with each other. This closing article describes the integrated harness that runs the three axes in the same experiment, on the same node (4×H100 SXM, a generic reference), with all the metadata fixed, the output in versioned JSON/CSV and an idempotent Kubernetes Job. The result is the 3-axis scorecard (€/1M tok, Wh/token, TTFT/ITL P99) that allows configurations to be compared on a multi-objective Pareto frontier and any figure to be audited with the bench to reproduce it.

Why the three axes must be measured together

The series’ opening post (The three axes) established the identity:

$$\text{CPM} = \frac{\text{cost/h}}{\text{throughput (tok/s)} \times 3.6 \times 10^{-3}}$$ $$\text{energy/token (Wh)} = \frac{\text{mean power (W)}}{\text{throughput (tok/s)} \times 3\,600}$$

Throughput is the common denominator. If it is measured in different experiments, different time, different load, different GPU temperature, then CPM and energy/token do not share a denominator: they are three anecdotes, not a scorecard. The harness captures them in the same time window, over the same load, with Prometheus windows aligned to the second. Only then is the scorecard row coherent by construction.

The second reason is reproducibility. The post Bias and reproducibility in benchmarking listed twelve biases that invalidate comparisons: an unpinned engine, an undeclared tokeniser, unfixed input/output lengths, missing warmup, a client outside the cluster. The harness removes all of them because the metadata are part of the Job, not of the documentation.

Architecture of the integrated bench

The harness has four layers. Each one is OSS, exports to Prometheus and lives in the same Kubernetes namespace:

Layer	Tool(s)	Primary metric	Export protocol
Performance	GuideLLM (SLO sweep) + AIPerf	TTFT P99, ITL P99, goodput (tok/s)	native JSON/CSV + `/metrics` OpenMetrics
Cost	OpenCost + LiteLLM proxy	CPM (€/1M tok), cost/request	REST API + Prometheus scrape
Energy (GPU)	DCGM Exporter	`DCGM_FI_DEV_POWER_USAGE` (W), `DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION` (mJ)	Prometheus DaemonSet
Energy (pod)	Kepler	`kepler_container_joules_total`, energy per pod	Prometheus DaemonSet

MLPerf Power is used as an external comparability reference: its published results, with hardware documented in detail, allow the harness figures to be calibrated for plausibility. The in-house bench does not aim to be an MLPerf submission, but to be reproducible on your own cluster.

The Kubernetes Job: the full YAML

The experiment runs as a versioned Kubernetes Job. All the relevant metadata are environment variables declared in the manifest: not in ad-hoc scripts, not in external documentation. The Job is idempotent (same name = same run) and leaves traces in the pod log and in the output volume.

apiVersion: batch/v1
kind: Job
metadata:
 name: bench-llama3-70b-fp8-h100x4-20260616
 namespace: benchmark
 labels:
 bench/model: llama3-70b
 bench/precision: fp8
 bench/engine: vllm-0.9.1
 bench/hardware: h100x4-sxm
 bench/isl: "1024"
 bench/osl: "256"
 bench/concurrency: "32"
 bench/tokenizer: meta-llama-3-tokenizer-v3
 bench/run-id: "20260616T1200"
spec:
 backoffLimit: 0
 template:
 spec:
 restartPolicy: Never
 serviceAccountName: bench-runner
 volumes:
 - name: results
 persistentVolumeClaim:
 claimName: bench-results-pvc
 initContainers:
 # Warmup: 60 s of traffic before the experiment
 - name: warmup
 image: ghcr.io/vllm-project/guidellm:0.4.2
 command:
 - guidellm
 - benchmark
 - --target
 - http://vllm-svc.inference.svc.cluster.local:8000
 - --rate-type
 - concurrent
 - --rate
 - "4"
 - --max-seconds
 - "60"
 - --data
 - prompt_tokens=1024,output_tokens=256
 env:
 - name: GUIDELLM_ENV
 value: production
 containers:
 - name: guidellm-sweep
 image: ghcr.io/vllm-project/guidellm:0.4.2
 command:
 - guidellm
 - benchmark
 - --target
 - http://vllm-svc.inference.svc.cluster.local:8000
 - --rate-type
 - sweep
 - --max-seconds
 - "120"
 - --data
 - prompt_tokens=1024,output_tokens=256
 - --output-path
 - /results/guidellm-$(BENCH_RUN_ID).json
 env:
 - name: BENCH_RUN_ID
 valueFrom:
 fieldRef:
 fieldPath: metadata.labels['bench/run-id']
 volumeMounts:
 - name: results
 mountPath: /results

 - name: aiperf-concurrent
 image: nvcr.io/nvidia/aiperf:0.2.0
 command:
 - aiperf
 - profile
 - --url
 - http://vllm-svc.inference.svc.cluster.local:8000/v1
 - --model
 - meta-llama/Meta-Llama-3-70B-Instruct
 - --concurrency
 - "4,8,16,32"
 - --input-tokens
 - "1024"
 - --output-tokens
 - "256"
 - --num-requests
 - "200"
 - --output
 - /results/aiperf-$(BENCH_RUN_ID).json
 volumeMounts:
 - name: results
 mountPath: /results

 - name: scorecard-exporter
 image: python:3.12-slim
 command:
 - python
 - /scripts/export_scorecard.py
 - --run-id
 - "$(BENCH_RUN_ID)"
 - --prometheus
 - http://prometheus.monitoring.svc.cluster.local:9090
 - --output
 - /results/scorecard-$(BENCH_RUN_ID).json
 env:
 - name: BENCH_RUN_ID
 valueFrom:
 fieldRef:
 fieldPath: metadata.labels['bench/run-id']
 volumeMounts:
 - name: results
 mountPath: /results

The Job name (bench-llama3-70b-fp8-h100x4-20260616) is the unique identifier of the run. Changing it is enough to register a variant. The labels are the metadata that the scorecard-exporter reads to enrich the output JSON.

Performance layer: GuideLLM and AIPerf

GuideLLM — the SLO-driven sweep

GuideLLM (a vLLM project) generates realistic traffic patterns, synchronous, concurrent, poisson, throughput, sweep, and captures full TTFT and ITL distributions (Red Hat Developer). The sweep mode ramps from idle to saturation over 10 rounds and identifies the knee: the maximum load where goodput ≈ throughput under the declared SLO. The output is JSON/CSV with all the percentiles per round, ready to be versioned.

For the harness, the bench’s reference SLO is:

Metric	Threshold	Percentile
TTFT	500 ms	P99
ITL (TPOT)	50 ms/tok	P95
Error rate	0.5 %	—

The Job command already appears in the YAML above. The JSON output includes, per round: rate (req/s), TTFT (P50/P95/P99), ITL (P50/P95/P99), throughput (tok/s) and goodput (tok/s). The goodput under the SLO at the knee is the value that enters the scorecard.

For the cross-verification layer of the data and comparability with published results, see the post GuideLLM in depth.

AIPerf — NVIDIA’s concurrency profiler

AIPerf (the successor to GenAI-Perf, repository ai-dynamo/aiperf) measures TTFT, ITL, throughput and latency distributions at fixed concurrencies (GitHub ai-dynamo/aiperf). Where GuideLLM gives the automatic sweep up to the knee, AIPerf gives the detailed profile at specific concurrencies (4, 8, 16, 32 in the example): it allows the throughput-latency curve to be characterised point by point.

The two are complementary: GuideLLM finds the knee automatically; AIPerf confirms it and characterises the behaviour in the neighbourhood of that knee. Both outputs go to the results volume with the same BENCH_RUN_ID. The deep analysis of AIPerf/GenAI-Perf is in GenAI-Perf in depth.

Metrics the harness extracts from AIPerf for the scorecard:

# TTFT P99 at concurrency 16 (the closest to the sweep's knee):
aiperf profile ... --concurrency 16 \
 | jq '.results[] | select(.concurrency==16) | .ttft_ms.p99'

# Goodput (tok/s) at concurrency 16:
aiperf profile ... --concurrency 16 \
 | jq '.results[] | select(.concurrency==16) | .output_token_throughput'

Cost layer: OpenCost and LiteLLM

OpenCost — the denominator in euros per GPU-hour

OpenCost (CNCF incubating, opencost.io) allocates Kubernetes cost to namespace, label, pod and container in real time (GitHub opencost/opencost). For the harness, OpenCost answers the question: how much does the vllm-svc pod cost in euros per hour during the experiment’s window?

The harness calls the OpenCost REST API when the experiment finishes:

# Cost of the inference namespace over the experiment window (1 hour)
curl -s "http://opencost.monitoring.svc.cluster.local:9003/allocation" \
 --data-urlencode 'window=2026-06-16T12:00:00Z,2026-06-16T13:00:00Z' \
 --data-urlencode 'aggregate=namespace' \
 --data-urlencode 'namespace=inference' \
 | jq '.data[0].inference.totalCost'

The returned cost (in EUR, configured with the node’s real prices) is divided by the throughput measured in that same window to obtain the CPM:

$$\text{CPM} = \frac{\text{cost}_\text{namespace/h} \times 10^6}{\text{goodput (tok/s)} \times 3\,600}$$

The post OpenCost: cost allocation in Kubernetes details the price configuration and the allocation by GPU label.

LiteLLM — the per-request token counter

LiteLLM (litellm.ai, GitHub BerriAI/litellm) acts as an OpenAI-compatible proxy with token accounting per request, model and team. In the harness, GuideLLM and AIPerf point at the LiteLLM endpoint (which in turn forwards to vLLM): every request is recorded with prompt_tokens, completion_tokens and cost (using the custom pricing configured for the on-prem node).

# litellm-config.yaml (fragment of custom on-prem pricing)
model_list:
 - model_name: llama3-70b-fp8
 litellm_params:
 model: openai/meta-llama/Meta-Llama-3-70B-Instruct
 api_base: http://vllm-svc.inference.svc.cluster.local:8000/v1
 api_key: sk-dummy
 input_cost_per_token: 0.00000109 # 1.09 EUR/1M tok on-prem
 output_cost_per_token: 0.00000109

The LiteLLM records are exported to Prometheus as litellm_request_total_tokens and litellm_spend_metric_total, which the scorecard exporter consumes to compute the real CPM per token type (input vs output).

Energy layer: DCGM and Kepler

DCGM Exporter — GPU power and energy

DCGM Exporter (GitHub NVIDIA/dcgm-exporter, docs) exposes GPU metrics on /metrics for Prometheus as a DaemonSet on the GPU nodes. The two energy metrics the harness uses:

DCGM metric	Type	Unit	Use in the harness
`DCGM_FI_DEV_POWER_USAGE`	gauge	W	instantaneous power per GPU
`DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION`	counter	mJ	energy accumulated since boot

To compute the energy consumed during the experiment’s window (without the idle baseline), the harness takes the difference of the counter before and after the sweep:

# Deployment as a DaemonSet (fragment of the official chart)
helm repo add gpu-helm-charts \
 https://nvidia.github.io/dcgm-exporter/helm-charts
helm install dcgm-exporter gpu-helm-charts/dcgm-exporter \
 --namespace monitoring \
 --set serviceMonitor.enabled=true \
 --set serviceMonitor.interval=15s

The DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION field (in mJ) allows the experiment’s energy to be computed with hardware-counter accuracy, not with an estimate:

$$\text{experiment energy (J)} = \left(\text{counter}_\text{end} - \text{counter}_\text{start}\right) \times 10^{-3}$$ $$\text{Wh/token} = \frac{\text{experiment energy (J)}}{3\,600 \times \text{generated tokens}}$$

Kepler — pod-level energy

Kepler (CNCF sandbox, GitHub sustainable-computing-io/kepler) uses eBPF to estimate energy consumption at container and pod level, exporting kepler_container_joules_total to Prometheus (Red Hat Emerging Technologies). It combines RAPL (CPU/DRAM), NVML (GPU) and regression models when no sensors are available.

In the harness, Kepler complements DCGM: DCGM gives the hardware measurement of the GPU (more precise for GPU-intensive loads such as LLM inference), while Kepler attributes the energy to the specific vLLM pod (including the node’s CPU contribution). The main metric the harness consumes:

# Energy of the vllm-svc pod during the experiment (J)
increase(
 kepler_container_joules_total{
 container_namespace="inference",
 container_name="vllm"
 }[${BENCH_DURATION}]
)

The comparative analysis of DCGM vs Kepler vs Zeus is in Energy tools: deployment, precision and overhead.

The MLPerf Power reference

MLPerf Power (MLCommons, IEEE HPCA 2025 paper) establishes the protocol for measuring the energy efficiency of ML systems with high-precision external measurement. The in-house bench is not an MLPerf submission (that requires external power meters and committee review), but its published results are the calibration reference: if the harness gives a result of the same order as the MLPerf submission for the same hardware with the same model, the measurement is plausible. If it differs by more than a factor of 2, there is a methodological problem. See the post MLPerf Power: energy efficiency for the current reference data.

Unifying the metrics: the PromQL queries

The scorecard-exporter is the Job container that, once GuideLLM and AIPerf have finished, collects all the metrics from Prometheus and builds the JSON of the scorecard row. The key queries:

# Mean GPU power during the sweep (W) — 4×H100 node
avg_over_time(
 sum(DCGM_FI_DEV_POWER_USAGE{Hostname=~"gpu-node-.*"})[${BENCH_DURATION}:15s]
)

# Total GPU energy during the sweep (mJ → convert to J)
(
 sum(DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION{Hostname=~"gpu-node-.*"}) offset 0
 -
 sum(DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION{Hostname=~"gpu-node-.*"}) offset ${BENCH_DURATION}
) * 0.001

# Tokens generated during the sweep (from LiteLLM)
increase(
 litellm_request_total_tokens{model="llama3-70b-fp8", token_type="completion"}[${BENCH_DURATION}]
)

# Cost of the inference namespace over the window (from the OpenCost API)
# → REST call at the start and at the end of the Job, difference of accrual

The ${BENCH_DURATION} variable is the real duration of the sweep (in Prometheus format, e.g. 22m), which the exporter computes as end_ts - start_ts and substitutes into every query.

The output JSON of each run has this structure:

{
 "run_id": "20260616T1200",
 "metadata": {
 "model": "meta-llama/Meta-Llama-3-70B-Instruct",
 "engine": "vllm-0.9.1",
 "precision": "fp8",
 "hardware": "4xH100-SXM-80GB",
 "isl_tokens": 1024,
 "osl_tokens": 256,
 "tokenizer": "meta-llama-3-tokenizer-v3",
 "slo_ttft_p99_ms": 500,
 "slo_itl_p95_ms": 50,
 "bench_tool_guidellm": "0.4.2",
 "bench_tool_aiperf": "0.2.0",
 "dcgm_exporter": "3.3.9",
 "kepler": "0.10.2"
 },
 "performance": {
 "goodput_tok_s": 3120,
 "ttft_p99_ms": 487,
 "itl_p95_ms": 42,
 "throughput_peak_tok_s": 3890,
 "elbow_concurrency": 28
 },
 "cost": {
 "cpm_eur_1m": 0.97,
 "gpu_cost_eur_h": 10.8,
 "cost_window_eur": 3.24
 },
 "energy": {
 "wh_per_token": 0.00044,
 "j_per_token": 1.58,
 "power_mean_w": 4924,
 "energy_total_kwh": 0.287,
 "pue": 1.4,
 "wh_per_token_pue_adjusted": 0.000616
 },
 "carbon": {
 "grid_intensity_gco2_kwh": 40,
 "co2_per_1m_tokens_g": 24.6
 }
}

All the fields are computed from the same time windows. The JSON is versioned in git alongside the harness code. Reproducing any scorecard row is: git checkout <run-id> && kubectl apply -f job.yaml.

The 3-axis scorecard: table and interpretation

The following table illustrates how five configurations of the same model (Llama 3 70B) on the same reference node (4×H100 SXM) compare with the harness. The figures are illustrative of the order of magnitude (the bench fills them in with real measurements):

Config	Engine	Precision	CPM (€/1M)	Goodput (tok/s)	Wh/tok	TTFT P99 (ms)	ITL P95 (ms)	CO2 /1M (g, FR)
A	vLLM 0.9	FP16	1.64	1,890	0.00074	498	49	29.6
B	vLLM 0.9	FP8	0.97	3,120	0.00044	487	42	17.6
C	SGLang 0.4	FP8	0.88	3,410	0.00041	421	38	16.4
D	vLLM 0.9	FP8, ISL 512	1.31	2,240	0.00056	294	31	22.4
E	vLLM 0.9	FP8, max-batch 128	0.84	3,590	0.00040	721	55	16.0

How to read the table:

B vs A: FP8 over FP16 cuts CPM by 41 % and energy per token by 41 % thanks to the same higher throughput. Both meet the SLO (TTFT P99 < 500 ms, ITL P95 < 50 ms). B dominates A on all three axes: it is Pareto-superior.
C vs B: SGLang gives 9 % more goodput and 9 % lower CPM with FP8. Both meet the SLO. C is Pareto-superior to B (if the engine is indifferent for the stack).
D vs B: a shorter ISL (512 tok) lowers latency (TTFT P99 294 ms vs 487 ms) but lowers goodput and raises CPM. If the use case demands TTFT < 300 ms, D is the candidate; if TTFT < 500 ms is enough, B is better on cost and energy.
E vs C: max-batch 128 raises goodput (+5 %) and lowers CPM, but the TTFT P99 breaks the SLO (721 ms > 500 ms) and so does the ITL P95 (55 ms > 50 ms). E has the best raw throughput but is outside the SLO: it is not a valid candidate for the interactive chat use case.

The scorecard’s Pareto frontier, under the declared SLO, includes only A, B, C and D (E falls out for breaking the SLO). Of those four, C dominates B which dominates A. D only enters the frontier if the use case demands TTFT P99 < 300 ms. The decision is not a number; it is the whole row plus the SLO.

The multi-objective Pareto frontier

With three minimisation metrics, CPM (€/1M tok), Wh/tok and TTFT P99 (ms), the Pareto frontier is defined as the set of configurations where none dominates another on all three axes simultaneously. Formally, config $ i $ dominates config $ j $ if:

$$\text{CPM}_i \leq \text{CPM}_j \;\land\; \text{Wh/tok}_i \leq \text{Wh/tok}_j \;\land\; \text{TTFT}_{i} \leq \text{TTFT}_{j}$$

with at least one strict inequality. The scorecard allows it to be computed over the set of runs with a simple vector operation. In the table’s example, the frontier under the SLO (with E excluded) is $\{C, D\}$: C dominates on cost/energy/goodput; D dominates on latency. Between C and D the choice depends on the TTFT SLO of the specific use case.

Reproducible design: the twelve mandatory metadata

The checklist that closes the bias dossier (Bias and reproducibility). For the harness JSON to be auditable, the twelve fields must be present in every run:

#	Field	Example	Why it is mandatory
1	Serving engine version	`vllm-0.9.1`	the same model can perform differently across versions
2	Model version (commit/tag)	`meta-llama/Meta-Llama-3-70B-Instruct@sha256:abc`	avoids ambiguity between variants of the same name
3	Precision	`fp8`	FP16 vs FP8 changes throughput and latency
4	Tokeniser and version	`meta-llama-3-tokenizer-v3`	the ISL/OSL in tokens depends on the tokeniser
5	ISL (input sequence length)	`1024` tok	changes the prefill time and the knee
6	OSL (output sequence length)	`256` tok	changes the decode time
7	Hardware (model and count)	`4×H100-SXM-80GB`	the basis of any comparison
8	Maximum concurrency tested	`32`	defines the sweep’s range
9	Bench tool version	`guidellm-0.4.2`, `aiperf-0.2.0`	different versions can give different metrics
10	Declared SLO	`TTFT P99 < 500 ms, ITL P95 < 50 ms`	the knee depends on the SLO; without it there is no goodput
11	Warmup duration	`60 s`	without warmup the KV cache is cold and the first rounds are biased
12	Harness version / run-id	`20260616T1200`	identifies the run so it can be reproduced with `git checkout`

A scorecard missing any of these twelve fields is an anecdote. With all twelve, it is an auditable datum. The run-id in the Job name (bench-llama3-70b-fp8-h100x4-20260616) implicitly incorporates fields 1-3 and 7, forcing the Job name to change with every variant, which makes it impossible to overlap runs in the same namespace.

Idempotence and versioning of results

The Kubernetes Job is idempotent: if the same manifest is applied again (same name), the Job already exists and Kubernetes does not relaunch it (backoffLimit: 0 policy). This avoids accidental duplicate runs. To relaunch, the Job has to be deleted (kubectl delete job <name>) or the run-id changed.

Results are versioned with the following directory convention in the harness’s git repository:

results/
20260616T1200/
guidellm-20260616T1200.json
aiperf-20260616T1200.json
scorecard-20260616T1200.json
20260617T0900/
guidellm-20260617T0900.json
aiperf-20260617T0900.json
scorecard-20260617T0900.json
scorecard-aggregate.csv # all rows, for analysis and charts

The scorecard-aggregate.csv is the scorecard table in flat format: each row is a run, each column a field of the JSON. It is generated with:

# Generates the aggregate CSV from all the JSON files in results/
python scripts/aggregate_scorecards.py results/ > scorecard-aggregate.csv

The CSV is what feeds the Pareto charts, the reports and the architecture decisions. It is versioned in git; its diff is the configuration change.

How it connects with the other posts in the series

The harness is not a standalone article: it is the synthesis of the twenty-eight articles. Each tool has its deep dive:

Performance: GuideLLM in depth — sweep, SLO, goodput, knee; GenAI-Perf / AIPerf in depth — concurrency, rate sweep, distribution metrics.
Cost: OpenCost: cost allocation in Kubernetes — allocation by namespace/label, custom on-prem prices.
Energy: Energy tools: deployment, precision and overhead — DCGM vs Kepler vs Zeus, instrumentation overhead, relative precision; MLPerf Power: energy efficiency — the external calibration reference.
Theoretical framework: The three axes: cost, performance and energy — the CPM/energy/throughput identity that justifies measuring them together.
Bias: Bias and reproducibility in benchmarking — the twelve biases the harness removes.

The harness closes the series’ dossier because it makes the opening article’s promise possible: “when someone disputes a number in the proposal, the answer is not ‘a blog says so’, but ’this is the bench, this is the methodology, reproduce it’”. With the run-id and the git repo, reproducing is one command.

Full flow of a benchmarking session

The harness’s standard operating procedure, from start to finish:

# 1. Make sure the inference endpoint is up
kubectl get svc vllm-svc -n inference

# 2. Verify that DCGM and Kepler are scraping
curl -s http://prometheus.monitoring.svc.cluster.local:9090/api/v1/query \
 --data-urlencode 'query=DCGM_FI_DEV_POWER_USAGE' | jq '.data.result | length'

# 3. Record the energy counter BEFORE the experiment
PRE_ENERGY=$(kubectl exec -n monitoring dcgm-exporter-xxx -- \
 curl -s localhost:9400/metrics \
 | grep DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION | awk '{sum+=$2} END{print sum}')

# 4. Launch the Job (unique name per run)
kubectl apply -f jobs/bench-llama3-70b-fp8-h100x4-20260616.yaml

# 5. Wait for it to finish
kubectl wait --for=condition=complete job/bench-llama3-70b-fp8-h100x4-20260616 \
 -n benchmark --timeout=3600s

# 6. Record the energy counter AFTER
POST_ENERGY=$(kubectl exec -n monitoring dcgm-exporter-xxx -- \
 curl -s localhost:9400/metrics \
 | grep DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION | awk '{sum+=$2} END{print sum}')

# 7. Collect the results from the volume
kubectl cp benchmark/bench-run-pod:/results/ ./results/20260616T1200/

# 8. Compute the experiment's energy (mJ → kWh)
python scripts/calc_energy.py \
 --pre $PRE_ENERGY --post $POST_ENERGY \
 --run-id 20260616T1200

# 9. Add a row to the aggregate CSV
python scripts/aggregate_scorecards.py results/ > scorecard-aggregate.csv

# 10. Version it
git add results/20260616T1200/ scorecard-aggregate.csv
git commit -m "bench: llama3-70b fp8 4xH100 ISL1024 OSL256 (20260616T1200)"

Steps 6-9 are candidates for automation as a second post-processing Job launched when the main one completes (with ownerReferences or a clean-up CronJob). In the harness’s basic state, the manual steps are precisely the ones that force a review of the result before versioning it.

Cost of the experiment

A GuideLLM sweep of 10 rounds at 120 seconds per round occupies the node for ~22-26 minutes. The AIPerf profile at 4 concurrencies (200 requests each) adds ~8-12 minutes. Total per run: ~35-40 minutes of a 4×H100 node.

At the reference amortised cost (~10.8 €/h for 4×H100 on-prem):

$$\text{cost per run} \approx 10.8 \times \frac{38}{60} \approx 6.84 \text{ EUR}$$

At the reference European cloud price (4 × 2.73 = 10.92 €/h):

$$\text{cost per run (cloud)} \approx 10.92 \times \frac{38}{60} \approx 6.92 \text{ EUR}$$

A continuous benchmarking programme (10 configurations × 3 models × 2 sweeps/week) adds up to ~415 EUR/week. That is why the harness includes short sweeps (5 rounds, ~15 minutes) for CI and full sweeps for releases.

What the harness does not measure (and why)

Missing dimension	Reason	Where it is covered
Output quality (MMLU, HumanEval)	requires a quality evaluation bench, not a serving one	LLM quality benchmarks
Training throughput	the harness is exclusively for inference	outside the scope of the “data” series
Network and storage cost	OpenCost can attribute it, but it requires additional configuration	OpenCost: cost allocation
Hourly carbon intensity	use the ElectricityMaps API + the measured energy	From the watt to carbon: PUE and grid mix
Client-server network latency	the Job runs inside the cluster; WAN latency requires an external client	document as additional metadata

Sources

GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
GuideLLM · PyPI — https://pypi.org/project/guidellm/
Red Hat Developer · desplegar y benchmarkear vLLM con GuideLLM en Kubernetes — https://developers.redhat.com/articles/2025/12/24/how-deploy-and-benchmark-vllm-guidellm-kubernetes
Red Hat Developer · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
AIPerf · GitHub (ai-dynamo/aiperf) — https://github.com/ai-dynamo/aiperf
GenAI-Perf · documentación NVIDIA Triton — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/README.html
NVIDIA · LLM performance benchmarking con GenAI-Perf — https://developer.nvidia.com/blog/llm-performance-benchmarking-measuring-nvidia-nim-performance-with-genai-perf
NVIDIA NIM Benchmarking · métricas (TTFT, ITL, throughput) — https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html
OpenCost · web oficial — https://opencost.io/
OpenCost · GitHub (opencost/opencost) — https://github.com/opencost/opencost
OpenCost · CNCF blog (sandbox → incubating) — https://www.cncf.io/blog/2022/12/06/opencost-a-new-cncf-sandbox-project-for-real-time-kubernetes-cost-monitoring/
LiteLLM · web oficial — https://www.litellm.ai/
LiteLLM · GitHub (BerriAI/litellm) — https://github.com/BerriAI/litellm
LiteLLM · Spend Tracking docs — https://docs.litellm.ai/docs/proxy/cost_tracking
NVIDIA DCGM Exporter · documentación oficial — https://docs.nvidia.com/datacenter/dcgm/latest/gpu-telemetry/dcgm-exporter.html
NVIDIA DCGM Exporter · GitHub — https://github.com/NVIDIA/dcgm-exporter
NVIDIA DCGM Exporter · métricas CSV — https://github.com/NVIDIA/dcgm-exporter/blob/main/etc/dcp-metrics-included.csv
Kepler (Kubernetes-based Efficient Power Level Exporter) · GitHub — https://github.com/sustainable-computing-io/kepler
Red Hat Emerging Technologies · Introducing Kepler — https://next.redhat.com/2023/08/22/introducing-kepler-efficient-power-monitoring-for-kubernetes/
MLCommons · MLPerf Inference Datacenter benchmark — https://mlcommons.org/benchmarks/inference-datacenter/
MLCommons · MLPerf Power presentado en IEEE HPCA 2025 — https://mlcommons.org/2025/03/ml-commons-power-hpca/
IEEE Xplore · MLPerf Power paper (μWatts to MWatts) — https://ieeexplore.ieee.org/document/10946778/
MLCommons · MLPerf Inference v5.1 results (septiembre 2025) — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
Red Hat · Efficient and reproducible LLM inference: MLPerf Inference v5.1 — https://www.redhat.com/en/blog/efficient-and-reproducible-llm-inference-red-hat-mlperf-inference-v51-results
BentoML · Beyond tokens-per-second: cost, speed and quality in LLM inference — https://www.bentoml.com/blog/beyond-tokens-per-second-how-to-balance-speed-cost-and-quality-in-llm-inference

Measurement bias and reproducibility: why two benchmarks of the same model give figures that differ by up to 7×

Tue, 16 Jun 2026 07:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The mathematical symbol is reserved for formulas: amounts are expressed in € or USD. Thousands with a thin space ($1\,000$).

TL;DR

Running the same Llama-3-70B-Instruct model on the same 4×H100 SXM 80 GB node with two different tools can produce throughput results that differ by 7.2× without the engine changing a single line of code. The cause is not the engine: it is the measurement method. At 1,000 QPS, a single-process asyncio client (vLLM bench, SGLang bench, genai-perf pre-AIPerf) processed 75,574 tokens against the 545,733 tokens of a multi-process client, both measuring the same endpoint (arXiv 2605.24217). The bias is not random: it is systematic, directional and reproducible. This article catalogues the sources of bias, quantifies their magnitude and describes the minimum harness that turns a number into an auditable datum.

The sources of bias: a catalogue with magnitudes

Each source of bias moves the published number in a given direction and with a characteristic magnitude. The table below orders the sources from largest to smallest impact observed in the literature:

Source of bias	Direction of the bias	Documented magnitude	Reference
Single-process client saturation (Python GIL, asyncio)	understates throughput; overstates latency	up to 7.2× fewer tokens processed at 1,000 QPS	arXiv 2605.24217
Tokenizer different from the model’s (LLMPerf uses a universal LlamaTokenizer)	tok/s not comparable across different vocabularies	variable; up to 33 % of client overhead in the TPS denominator	NVIDIA blog (TPS = client overhead ÷ total duration)
ignore_eos missing or inconsistent OSL	underestimates throughput; artificially shortens the real OSL	up to premature termination; an unrealistic fixed-length benchmark	vLLM bench docs; NVIDIA fundamental concepts
Unrealistic ISL/OSL distribution (fixed length instead of a realistic distribution)	an optimistic knee that does not resemble production	the knee shifts; prefill understated with short prompts	AIPerf sequence-length-distributions docs
Missing warmup / warm prefix cache	overstates TTFT (artificially low on the first requests)	TTFT underestimated by an already populated KV cache	vLLM prefix caching docs; arXiv 2605.24217
`concurrency` vs `request-rate` as the load mode	concurrency saturates symmetrically; request-rate can build a queue	ITL overestimated if the queue grows without a ceiling with request-rate	NVIDIA fundamentals; AIPerf docs
Measurement window too short	does not capture the steady state; includes ramp-up	throughput overestimated; latency underestimated	meta-metrics arXiv 2508.10251
Thermal state and GPU clocks	thermal throttling reduces throughput without prior stabilisation	temperature > 75 °C triggers throttling; a difference of ~3 W of stable power	arXiv 2604.09048
Warm prefix cache between runs	overstates TTFT; unrealistic TTFT if the cache is pre-populated from the previous run	a severe effect in multi-prompt evaluations with shared prefixes	arXiv 2605.24217
Mean instead of percentiles	hides the latency tail	P99 can be 4–6× the median at high concurrency	NVIDIA fundamentals; Anyscale docs

The dominant bias: single-process client saturation

The largest bias is not in the engine but in the client that generates the load. The single-process tools (vLLM bench, SGLang bench, genai-perf before its replacement by AIPerf on 15 April 2026) use a single Python process with asyncio to manage the concurrent requests. The Python Global Interpreter Lock (GIL) prevents a single process from using more than one CPU core at a time, which introduces a bottleneck on the client side that becomes critical at high concurrency.

The mathematical effect: as the request rate rises, the client fails to dispatch requests at the configured pace, accumulates queueing time on the client side, and that time gets recorded as engine latency. The result is that TTFT and TPOT appear inflated and throughput appears depressed, without the engine having changed anything (arXiv 2605.24217).

The documented discrepancy at 1,000 QPS:

Client architecture	Tokens processed at 1,000 QPS	Ratio
Single-process (asyncio, Python GIL)	75,574	1× (baseline)
Multi-process (distributed load)	545,733	7.2×

Note: both clients point at the same endpoint; the difference is entirely attributable to the benchmark client, not to the engine.

Tokenizer bias: tok/s are not comparable across vocabularies

Tokens are not universal. Each model has its own tokenizer with its own vocabulary. Llama-3 uses a vocabulary of 128,256 tokens; Gemma uses 256,128; earlier models used 32,000–50,000. The same English text produces a different number of tokens depending on the model’s tokenizer.

The direct consequence for benchmarking: if the tool measures tok/s with a tokenizer different from that of the model being served, the token count, and therefore the throughput in tok/s and the cost per token, are biased. LLMPerf (archived in December 2025) used LlamaTokenizer universally for all models, which guaranteed internal consistency in the leaderboard but made the tok/s not comparable with measurements of other models with different vocabularies.

The TPS formula in LLMPerf additionally included the benchmark’s full denominator, the time spent generating prompts, preparing requests and storing responses, which NVIDIA estimated at up to 33 % of the total duration at concurrency 1. This makes LLMPerf’s TPS systematically lower than that of GenAI-Perf/AIPerf for the same system, without the engine being worse:

$$\text{TPS}_{\text{LLMPerf}} = \frac{\text{output tokens}}{T_{\text{end}} - T_{\text{start}}}$$ $$\text{TPS}_{\text{GenAI-Perf}} = \frac{\text{output tokens}}{T_y - T_x}$$

Where $T_{\text{start}}$ and $T_{\text{end}}$ include the client overheads, while $T_x$ and $T_y$ are the instant of the first request and that of the last token received, respectively (NVIDIA · Fundamental Concepts).

The operational rule: always count tokens with the tokenizer of the model being served, not with a proxy tokenizer.

The bias of ignore_eos and an inconsistent OSL

Most LLM models generate a special end-of-sequence token (EOS) when they consider the response complete. If the benchmark does not set ignore_eos=True, the real output length (OSL) varies from request to request according to the model’s natural length distribution, which produces:

Inconsistent OSL: two runs with the same seed can produce different OSL distributions if the model varies its natural output.
Spurious comparison: a “faster” model may be so simply because it generates shorter responses (it hits EOS sooner), not because it has more real throughput.
Underestimated throughput: if the benchmark expects OSL=256 but the model stops at OSL=80 on average, the throughput in tok/s appears higher but measures less work.

The ignore_eos parameter (or --ignore-eos in vLLM bench) instructs the engine to ignore the EOS token and continue until reaching max_tokens. It is mandatory for the OSL to be the configured one rather than the model’s natural one, and for two runs to be comparable (vLLM benchmark CLI docs).

The bias of the ISL/OSL distribution

The distribution of input (ISL) and output (OSL) lengths determines what proportion of the compute time goes to prefill (costly in TTFT) and how much to decode (costly in ITL). A benchmark with a fixed length, say ISL=128, OSL=128, produces results that do not resemble any real traffic.

Real use cases have very different distributions:

Use case	Typical ISL (tokens)	Typical OSL (tokens)	Dominated by
Translation	500–2,000	500–2,000	balanced
Generation (code, email)	~100	~1,000	decode (long OSL)
Summarisation / RAG	~1,000	~100	prefill (long ISL)
Reasoning (CoT)	~100	1,000–10,000	very long decode

A short-ISL benchmark with a model optimised for prefill will give an artificially low TTFT and an artificially high throughput. The knee of the sweep shifts: with ISL=64 the system admits more concurrency without breaking the TTFT SLO; with ISL=1,024 prefill saturates earlier and the knee appears earlier. Using the wrong distribution means sizing for traffic that does not exist.

AIPerf introduces sequence distributions with configurable per-component variance to reproduce realistic traffic mixes (AIPerf · Sequence Length Distributions):

--sequence-distribution "64|10,32|8:70;256|40,128|20:20;1024|100,512|50:10"

This creates 70 % of requests with $\text{ISL} \sim \mathcal{N}(64, 10)$ and $\text{OSL} \sim \mathcal{N}(32, 8)$, 20 % with medium ISL/OSL, and 10 % with long ISL/OSL, far more faithful to real chatbot traffic than a fixed length.

The bias of warmup and the prefix cache

Two related but distinct sources of bias:

Missing warmup. The first requests of a benchmark hit a “cold” engine: the GPU is in a low-clock state, the KV cache is empty, and the operating system may be paging memory. The TTFT of the first requests is structurally higher than that of the steady state. If the benchmark does not discard a warmup period, the mean TTFT includes these outliers and overestimates the real production latency. Some frameworks (GenAI-Perf/AIPerf) use a sliding window that excludes the ramp-up and ramp-down requests; others do not.

Warm prefix cache between runs. The prefix KV cache (prefix cache or prompt cache) stores the computed results of repeated prompt prefixes. If the benchmark runs multiple consecutive runs with the same prompts, the second and subsequent runs find the KV cache already populated and report an artificially low TTFT, that of the decode, not of the prefill. For a baseline benchmark, the prefix cache must be cold; for one that simulates production with repeated prompts, warm. The distinction must be made explicit (arXiv 2605.24217; vLLM · Automatic Prefix Caching).

The bias of `concurrency` vs `request-rate`

The two load control modes produce different latency distributions for the same system:

Mode	Semantics	When to use	Risk
concurrency N	keeps exactly N requests in flight; as soon as one finishes, it launches another	measuring the system under a fixed concurrency load	over-represents sustained load; does not simulate real arrivals
request-rate r (constant or Poisson)	launches one request every $1/r$ seconds (constant) or with interarrival $\sim \text{Exp}(1/r)$ (Poisson)	simulating real traffic (random arrivals)	if the engine cannot absorb r req/s, the queue grows without a ceiling

NVIDIA recommends the concurrency mode for most capacity benchmarks (NVIDIA · LLM Benchmarking Fundamental Concepts). The request-rate mode is more faithful for online traffic (a Poisson distribution of arrivals), but if the rate exceeds the engine’s capacity the queue grows indefinitely and the latency metrics include queueing time that can dominate the TTFT, mixing queue behaviour with engine behaviour.

A system measured at concurrency=16 and another at request-rate=16 req/s are not under the same load: fixed concurrency guarantees 16 simultaneous requests; the rate sets the arrival pace but not the instantaneous concurrency. Comparing their results without adjustment is incorrect.

The bias of the measurement window and the thermal state

Window too short. A 30-second benchmark may measure the system’s ramp-up, not the steady state. The recommendation of arXiv 2508.10251 is that the measurement window cover at least 3-5× the system’s ramp time under load, and that the metrics be computed only over the steady window, excluding warmup and cooldown.

Thermal state of the GPU. Datacenter GPUs (H100 SXM, A100) operate with thermal throttling above ~75 °C. If the GPU has not reached its steady-state temperature before the benchmark, the first measurements correspond to a higher-clock state than the sustainable one. Controlled experiments document that power must stabilise within a range of 3 W for at least 30 seconds before the measurements are representative (arXiv 2604.09048 · Watt Counts). The effect is especially severe in prefill benchmarks (TTFT), where the high clocks of the cold state produce an artificially low TTFT.

For a reproducible benchmark on 4×H100 SXM 80 GB: before measuring, run sustained load for at least 2–3 minutes until nvidia-smi reports a stable temperature and stable power (variation < 3 W over 30 s). Only then start the measurement window.

How the tools compute ITL differently

ITL (Inter-Token Latency) appears in every tool but its formula varies, and the differences are not small:

GenAI-Perf / AIPerf:

$$\text{ITL} = \frac{\text{e2e latency} - \text{TTFT}}{\text{output tokens} - 1}$$

TTFT is excluded from the numerator and the denominator discounts the first token. ITL is a pure decode metric, with no prefill contamination.

LLMPerf (archived Dec. 2025):

$$\text{ITL}_{\text{LLMPerf}} = \frac{\text{e2e latency}}{\text{output tokens}}$$

TTFT is included in the numerator. For short sequences (OSL < 50 tokens), TTFT can represent 50–80 % of the e2e_latency, so that LLMPerf’s ITL measures mainly prefill, not decode. Two systems with the same decode but different prefill will show different ITLs in LLMPerf even if they are identical in decode speed.

vLLM bench (benchmark_serving.py):

It computes ITL as the mean of the intervals between consecutive tokens of the output stream, including intra-request variance. It can reveal decode jitter that the others average away.

The practical consequence: never cross a GenAI-Perf ITL with an LLMPerf one as if they were the same metric. They are different formulas over the same signal.

Comparability across tools: why their numbers do not cross

The following table summarises the methodological differences that make one tool’s numbers structurally incompatible with another’s without an explicit adjustment:

Dimension	vLLM bench	LLMPerf (archived)	GenAI-Perf / AIPerf	GuideLLM
Client architecture	single-process asyncio	single-process	multi-process	multi-process
ITL includes TTFT	no	yes	no	no
TPS denominator	$T_y - T_x$	$T_{\text{end}} - T_{\text{start}}$ (overhead included)	$T_y - T_x$	per round
Tokenizer	that of the model served	universal LlamaTokenizer	that of the model served	that of the model served
Warmup / sliding window	not automatic	no	yes (sliding window)	per round
ignore_eos by default	no	no	explicitly recommended	configurable
ISL/OSL distribution	manual parameter	manual parameter	distributions with variance	configurable `--data`
Primary load mode	concurrency	concurrency (drained batches)	concurrency (recommended)	sweep + poisson
Structured output	basic JSON	JSON	JSON + CSV	JSON + HTML + CSV

An LLMPerf result and a GenAI-Perf result for the same endpoint can differ in TTFT, ITL and TPS simultaneously and in all of them for methodological reasons, not because of engine differences. The only way to cross them is to run both tools on the same system under the same conditions and compute the empirical conversion factor, which in practice amounts to re-measuring with a single tool.

The honest harness: a reproducibility checklist

A benchmark whose number can be defended in an audit has to come accompanied by all the metadata that allow it to be reproduced. The minimum to demand, organised by category:

Hardware

Metadata	Example 4×H100	Effect if not fixed
GPU (model, variant, count)	4× NVIDIA H100 SXM 80 GB	H100 PCIe vs SXM differ in memory BW and NVLink
GPU-GPU interconnect	NVLink 4.0	tensor parallelism depends on the interconnect BW
CPU, RAM, CPU-GPU bandwidth	2× Intel Xeon 8480+, 512 GB DDR5, PCIe 5.0	the tokenisation bottleneck can be on the CPU
Thermal state at the start	GPU temperature < 65 °C, stable power ± 3 W	throttling alters TPS and TTFT by up to ~15 %
GPU clocks	without `nvidia-smi -pm 1` they can vary	a difference of up to ~10 % in throughput

Software and model

Metadata	Example	Effect if not fixed
Engine + version + flags	vLLM 0.8.4, `--tensor-parallel-size 4 --gpu-memory-utilization 0.90`	each version changes the scheduler and the KV cache management
Model + precision	Llama-3-70B-Instruct, FP8	FP16 vs FP8 differ by ~1.4–1.8× in throughput
Tokenizer used for counting	tokenizer of the model served (HF tokenizer)	a universal LlamaTokenizer biases tok/s across vocabularies
Generation seed	`--seed 42`	without a fixed seed, the OSL varies run-to-run
`ignore_eos`	`True`	without it, the OSL varies with the prompt content
Sampling parameters	`temperature=1.0, top_p=0.95`	greedy vs sampling affect logit speed

Load

Metadata	Example	Effect if not fixed
Bench tool + version	AIPerf 0.2.1	each version changes formulas and warmup
ISL/OSL distribution	$\mathcal{N}(512, 64)$ ISL, $\mathcal{N}(128, 20)$ OSL	changing the distribution moves the knee
Load mode	concurrency, sweep 1–64	concurrency vs request-rate: different distributions
Concurrency levels	1, 2, 4, 8, 16, 32, 64 (full sweep, beyond the knee)	without passing the knee, the safe capacity is unknown
Duration per point	120 s minimum per level	short windows capture ramp-up, not the steady state
Warmup	30 s excluded from the metric computation	without warmup, the metrics include the cold state
Prefix cache treatment	cold (flush between runs) or warm (declared explicitly)	warm cache: unrealistic TTFT for a baseline

SLO and reported metrics

Metadata	Example	Effect if not declared
Declared SLO	TTFT P99 < 500 ms, ITL P95 < 50 ms	goodput depends on the SLO; without an SLO there is no goodput
Reported percentiles	P50, P95, P99 (not just the mean)	the mean hides the tail; irrelevant for an SLO
Throughput vs goodput	goodput under the SLO, not raw throughput	catalogue throughput can be 5–10× the goodput
Total number of requests	≥ 1,000 per concurrency level	small samples: high variance in percentiles

Methodological defence of the datum before an audit

A throughput number presented without the data sheet above is not a datum: it is an anecdote. The validation pattern for a technical audit:

1. Traceability of the tool. The tool and its version must be pinnable to a Git commit or a container image hash. AIPerf and GuideLLM export JSON with version metadata; vLLM bench omits them by default and they have to be captured manually.

2. Reproducibility of the environment. The engine deployment script (or the Kubernetes manifest) and the exact benchmark command must be enough for a third party to reproduce the number on the same hardware. GuideLLM exports the benchmark file as an authoritative record of the session: configuration, metadata, per-benchmark statistics and request entries with individual timings (GitHub vllm-project/guidellm).

3. Verification of the steady state. The metrics must come from the benchmark’s steady window (warmup excluded). The throughput vs concurrency curve must show the knee, the point where throughput saturates and latency blows up, and extend beyond it. Without a visible knee, the reported capacity may be the client’s limit, not the engine’s.

4. Goodput, not raw throughput. The auditable throughput is the goodput under the declared SLO, not the peak throughput. A system that reports 4,000 tok/s but whose goodput under TTFT P99 < 500 ms is 1,800 tok/s has a real capacity of 1,800 tok/s for interactive use cases.

5. Internal comparability. If two engines or two configurations are compared, the tool, the load distribution, the SLO, the hardware and the warmup treatment must be identical. Any difference in these dimensions contaminates the comparison.

The goodput formula as an auditable metric:

$$\text{goodput} = \text{throughput} \times \Pr[\text{TTFT} \leq \text{SLO}_{\text{TTFT}}] \times \Pr[\text{ITL} \leq \text{SLO}_{\text{ITL}}]$$

Where the probabilities are estimated over the empirical distribution of the run. An engine with a throughput of 4,000 tok/s and $\Pr[\text{TTFT} \leq 500\,\text{ms}] = 0.45$ has a goodput of $4{,}000 \times 0.45 = 1{,}800$ tok/s.

A numerical example: the same node, four different measurements

Reference hardware: 4×H100 SXM 80 GB, model Llama-3-70B-Instruct FP8, tensor parallel 4, SLO TTFT P99 < 500 ms.

Benchmark configuration	Reported throughput	Real goodput	Factor vs goodput
vLLM bench, concurrency=32, no ignore_eos, no warmup, fixed OSL=64	5,800 tok/s	~1,200 tok/s (short OSL inflates TPS)	4.8×
LLMPerf, concurrency=32, LlamaTokenizer, overhead included in the denominator	3,100 tok/s	~1,600 tok/s (ITL includes TTFT)	1.9×
AIPerf, concurrency=32, ignore_eos, sliding window, model’s tokenizer	3,400 tok/s	3,350 tok/s	1.01×
GuideLLM, sweep 1–64, poisson, realistic ISL/OSL distribution, declared SLO	3,330 tok/s goodput (knee at round 6)	3,330 tok/s	1.0× (honest baseline)

The range: from 1,200 tok/s to 5,800 tok/s for the same engine, the same hardware and the same model. The maximum factor is 4.8×. The cause is not the engine; it is the instrumentation decisions.

Decomposing the total bias

The total observable bias between two tools is the multiplicative composition of the individual biases. For a comparison of single-process vLLM bench vs multi-process GuideLLM on the same system:

$$\text{total factor} \approx \underbrace{f_{\text{client}}}_{\leq 7.2\times} \times \underbrace{f_{\text{tokenizer}}}_{1.0{-}1.3\times} \times \underbrace{f_{\text{ignore-eos}}}_{1.0{-}2.0\times} \times \underbrace{f_{\text{ISL/OSL}}}_{1.0{-}1.5\times} \times \underbrace{f_{\text{warmup}}}_{1.0{-}1.2\times}$$

The client saturation factor ($\leq 7.2\times$) dominates, but the other factors multiply. Under adverse conditions (single-process client + different tokenizer + no ignore_eos + unrealistic OSL + no warmup), the compound bias can exceed 15–20× for the same system.

State of the art 2026: what has changed

genai-perf → AIPerf migration (15 Apr 2026): NVIDIA retired genai-perf and replaced it with AIPerf, which is multi-process with automatic saturation detection, configurable ISL/OSL distributions and a built-in sliding window. The migration removes the single-process client bias from NVIDIA’s official tooling.
LLMPerf archived (Dec. 2025): the ray-project/llmperf repository is archived and in read-only mode. The historical results of the LLMPerf leaderboard are comparable only among themselves; they must not be crossed with modern measurements without adjustment.
GuideLLM 0.5.x (2025–2026): a complete architectural refactor, multimodal support, and authoritative JSON export with all the session metadata. It is the OSS standard for SLO-driven evaluation.
arXiv 2605.24217 (May 2026): the first formal characterisation of systematic measurement bias in production LLM benchmarks, with a mathematical demonstration of the GIL effect and a proposal for a multi-process harness.
arXiv 2508.10251: meta-metrics and good practice for system-level performance benchmarking; it establishes the minimum dimensions of the parameter space that must be declared for a result to be reproducible.
MLPerf Inference v5.1 (Sep. 2025): 27 participants, a participation record. The MLPerf rules require declaring the exact code, the dataset and the hardware, and admit public review; it is the only industry benchmark with a formal review process for reproducibility.

Cross-links from the benchmarking track

The biases described here apply to every tool in the catalogue. For the complete data sheet of each tool:

Full catalogue: LLM benchmarking tools, data sheet by data sheet
GuideLLM and SLO validation: GuideLLM in depth: validating the SLO under load
AIPerf (ex genai-perf) and saturation detection: NVIDIA GenAI-Perf in depth
State of the art of frameworks: LLM benchmarking: frameworks and state of the art
Engine decision on the Pareto axis: Serving engines compared: the Pareto frontier

Sources

arXiv 2605.24217 · Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks — https://arxiv.org/abs/2605.24217
arXiv 2508.10251 · Meta-Metrics and Best Practices for System-Level Inference Performance Benchmarking — https://arxiv.org/pdf/2508.10251
arXiv 2604.09048 · Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures — https://arxiv.org/html/2604.09048v1
NVIDIA Technical Blog · LLM Inference Benchmarking: Fundamental Concepts (ITL, TPS, ISL/OSL, ignore_eos, concurrency vs request-rate) — https://developer.nvidia.com/blog/llm-benchmarking-fundamental-concepts/
NVIDIA AIPerf Docs · Sequence Length Distributions for Advanced Benchmarking — https://docs.nvidia.com/aiperf/tutorials/datasets-inputs/sequence-length-distributions-for-advanced-benchmarking
NVIDIA AIPerf · Request Rate with Max Concurrency — https://docs.nvidia.com/aiperf/tutorials/load-patterns-scheduling/request-rate-with-max-concurrency
NVIDIA AIPerf · Comprehensive LLM Benchmarking Guide — https://lucaberton.com/blog/nvidia-aiperf-llm-inference-benchmarking-guide/
vLLM Documentation · Benchmark CLI (ignore_eos, métricas disponibles) — https://docs.vllm.ai/en/latest/benchmarking/cli/
vLLM Documentation · Automatic Prefix Caching — https://docs.vllm.ai/en/stable/design/prefix_caching/
GitHub · ray-project/llmperf (archivado dic. 2025) — https://github.com/ray-project/llmperf
GitHub · vllm-project/guidellm (exportación JSON autoritativa, metadatos de sesión) — https://github.com/vllm-project/guidellm
Red Hat Developer · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
MLCommons · MLPerf Inference v5.1 results (récord 27 participantes, reproducibilidad formal) — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
arXiv 2502.16721 · Speed and Conversational LLMs: Not All Is About Tokens per Second (tokenizer incompatibilidad y tok/s entre vocabularios) — https://arxiv.org/pdf/2502.16721
Medium · LLM Inference Benchmarking (genAI-perf y vLLM, discrepancia 7,2×) — https://kchandan.medium.com/llm-inference-benchmarking-genai-perf-and-vllm-5dd06b57428e

LLM quality benchmarks: the contamination trap and the OSS tools for not fooling yourself

Tue, 16 Jun 2026 03:00:00 +0200

Notation: amounts in euros (N €), decimals with a full stop. The dollar sign is not used (on this site it is a formula delimiter).

Post B7 of the benchmarking track. The B1 introduction established the distinction between the three axes (performance, quality, energy). The B2 tool catalogue covered the performance axis (TTFT, ITL, goodput). This article covers the quality axis: what each benchmark measures, how it is run, and why the leaderboard numbers do not always say what they appear to say. The LLMOps eval pipeline and LLM-as-judge cover the system’s internal evaluation layer; this article covers the evaluation of the base model with standard academic benchmarks.

TL;DR

An inference engine with an 80 ms TTFT serving a low-quality model is worth nothing: the platform’s Pareto front has two independent axes. In 2025–2026, the most widely used static benchmarks (MMLU, HumanEval) have crossed the saturation threshold (HumanEval > 93% in frontier models, MMLU-Pro > 75% in the top-5), which invalidates their ability to discriminate at the high end. The main cause is not that the models are better: it is data contamination, with 18% n-gram overlap detected in Llama-2-70B against evaluation datasets. The OSS tools active in 2026 are lm-evaluation-harness (EleutherAI, MIT, 60+ tasks, the ecosystem’s de facto engine), HELM (Stanford, Apache 2.0, in maintenance since June 2026), LiveBench (MIT, 1,000 questions refreshed monthly to avoid contamination), lighteval (Hugging Face, MIT, 1,000+ tasks, vLLM support), and LMArena/Chatbot Arena (Bradley-Terry over millions of human comparisons). OpenLLM Leaderboard v2 was archived in 2025 and replaced by the OpenEvals ecosystem (200+ specialised leaderboards). Reproducing an eval number requires pinning: harness version, model version, prompt template version and number of shots.

Quality ≠ performance: the independent Pareto axis

The performance axis (TTFT, ITL, throughput) measures how long the engine takes to produce tokens. The quality axis measures how correct those tokens are. They are orthogonal: a small model served with vLLM at 3,000 tok/s can score 42% on MMLU-Pro; a frontier model may need 4× more GPU and score 72%. The product throughput × quality is the only number that matters for a platform in production.

The operational consequence: select model and engine first by quality (does it clear the minimum threshold for the task?) and then by performance (does it meet the SLO within the GPU budget?). Inverting the order, picking the fastest engine and fitting the model into it, is the most frequent mistake in initial sizing.

Reference benchmarks: what each one measures

The table below collects the standard academic benchmarks in 2026, the capability each one measures, its test set size and the current saturation level in frontier models.

Benchmark	Capability measured	Format	Test set	Frontier saturation 2026
MMLU	Multidisciplinary knowledge (57 subjects)	4-choice MCQ	14,079	High: top-5 > 88%; insufficient separation
MMLU-Pro	MMLU with 10 options, multistep reasoning	10-choice MCQ	12,032	Medium: top-5 between 72–77%; still separates
GSM8K	Primary-school maths reasoning (word problems)	answer generation	1,319	Very high: top models > 97%
MATH	Competition maths (algebra, geometry, calculus, etc.)	LaTeX generation	5,000	Medium-high: top models 85–92%
GPQA Diamond	PhD-level scientific reasoning (biology, physics, chemistry)	4-choice MCQ	198	Emerging: top > 90% in 2026; still separates in 60–90%
BBH (BIG-Bench Hard)	23 tasks where earlier models fell below human level	generation	6,511	Medium: useful for complex reasoning
IFEval	Verifiable instruction following (format, length, language)	generation + rules	541	Low: notable separation between models
HumanEval	Python code generation (164 problems with signature)	generation + execution	164	Very high: top > 93%; saturated
MBPP	Code generation (500 problems without a function signature)	generation + execution	500	High: top models 85–92%
MuSR	Multistep reasoning over long texts	generation	~1,000	Low: useful for extended reasoning
TruthfulQA	Truthfulness: resistance to asserting popular myths	generation + MCQ	817	Medium: visible separation between families

Note on saturation: a saturated benchmark does not distinguish models at the high end; it is only valuable as a floor check (confirming the model has not regressed to the level of three years ago). For the frontier zone, the useful benchmarks in 2026 are MMLU-Pro, GPQA Diamond, IFEval and MuSR.

OSS evaluation tools: data sheets

lm-evaluation-harness (EleutherAI)

Field	Data
Repository	github.com/EleutherAI/lm-evaluation-harness
Licence	MIT
Maintainer	EleutherAI
Active version (Jun 2026)	0.4.x (`main` branch)
Available tasks	60+ benchmarks, hundreds of subtasks and variants
Supported backends	Hugging Face Transformers, vLLM, litellm, OpenAI-compatible API, GGUF
Task format	YAML + Python; each task defines dataset, template, metric and number of shots
What it measures	MCQ accuracy, exact match, pass@k on code, perplexity, BLEU/ROUGE
How it is invoked	`lm_eval --model hf --model_args pretrained=<model> --tasks mmlu,gsm8k --num_fewshot 5 --output_path results/`
Output	JSON with per-task metrics + configuration metadata
Use in the ecosystem	Official engine of the OpenLLM Leaderboard (v1 and v2); de facto reference in research

The harness is the industry standard for reproducible evaluation. Its strength is the breadth of tasks and the normalisation of the prompt format. Its limitation: the tasks are static, the datasets do not change between harness versions, which exposes it to the contamination problem if the model was trained on the same data.

Full invocation with vLLM as the backend:

lm_eval \
 --model vllm \
 --model_args pretrained=meta-llama/Llama-3.1-70B-Instruct,dtype=bfloat16 \
 --tasks mmlu_pro,gpqa_diamond,ifeval,humaneval \
 --num_fewshot 5 \
 --batch_size auto \
 --output_path ./eval_results/ \
 --log_samples

HELM (Stanford CRFM)

Field	Data
Repository	github.com/stanford-crfm/helm
Official site	crfm.stanford.edu/helm
Licence	Apache 2.0
Maintainer	Stanford CRFM
Status (Jun 2026)	Maintenance mode since 1 June 2026
What it measures	Accuracy, calibration, robustness, fairness, toxicity, efficiency (latency/cost)
Task format	Python + YAML; scenarios with multiple simultaneous metrics
Supported backends	API (OpenAI, Anthropic, Cohere, etc.) + local models via Hugging Face
Active variants	HELM Classic, HELM Capabilities, HELM Lite, MedHELM, HELM Safety
Installation	`pip install crfm-helm`
How it is invoked	`helm-run --conf-paths run_specs.conf --suite v1 --max-eval-instances 1000`
Output	JSON + web UI with a reproducible leaderboard and per-prompt transparency

HELM is the tool that introduced holistic evaluation (multiple metrics over the same scenario instead of a single number). In 2026, with HELM in maintenance mode, the ecosystem has fragmented into specialised leaderboards; but HELM remains the methodological reference for evaluations that need fairness and toxicity as well as accuracy.

HELM Capabilities (published March 2025): an active leaderboard with full prompt transparency, reproducible with the HELM framework. Available at crfm.stanford.edu/helm/capabilities.

LiveBench

Field	Data
Repository	github.com/LiveBench/LiveBench
Official site	livebench.ai
Licence	MIT
Maintainer	NYU, UT Austin, UC Santa Barbara
Paper	arXiv:2406.19314
Size	1,000 active questions (monthly refresh, same total size)
Categories	6: math, coding, reasoning, language, instruction following, data analysis
Tasks	18 subtasks
What it measures	General capabilities over questions with objectively verifiable answers
Anti-contamination	Questions generated from recent sources: arXiv papers, news, IMDb, new datasets
Scoring	No LLM-as-judge: automatically verifiable answer (exact ground truth)
Cadence	New questions monthly; old ones retired to avoid accumulated memorisation

LiveBench is the architectural answer to the contamination problem: if the benchmark changes every month with questions based on sources published after the cutoff of the model under evaluation, memorisation of the training set cannot inflate the result. The limitation is size (1,000 questions) and that it does not cover every domain with the same depth as MMLU.

Execution:

git clone https://github.com/LiveBench/LiveBench
cd LiveBench
pip install -e .
python livebench/gen_model_answer.py \
 --model-path meta-llama/Llama-3.1-70B-Instruct \
 --model-id llama3-70b
python livebench/gen_ground_truth_judgment.py \
 --model-id llama3-70b
python livebench/show_livebench_result.py

lighteval (Hugging Face)

Field	Data
Repository	github.com/huggingface/lighteval
Licence	MIT
Maintainer	Hugging Face (Leaderboard & Evals Team)
Available tasks	1,000+ tasks across multiple domains and languages
Supported backends	Hugging Face Accelerate (CPU/GPU/multi-GPU), Nanotron, vLLM, TGI, OpenAI-compatible endpoints
What it measures	Accuracy, BERT score, BLEU, exact match; same as the harness + HF’s own tasks
Differentiator	Designed to be the engine behind HF’s leaderboards; easy extension with custom tasks
Installation	`pip install lighteval`
How it is invoked	`lighteval accelerate –model_args “pretrained=” –tasks “lighteval
Output	JSON + direct integration with the Hugging Face Hub to publish results

lighteval is the internal successor to lm-evaluation-harness in the HF ecosystem: more modern, better vLLM and distributed backend support, and designed to run without leaving the HF stack. For teams already using the Hub to manage models, it is the lowest-friction option.

OpenLLM Leaderboard v2 (archived 2025) → OpenEvals

The OpenLLM Leaderboard v2 evaluated more than 13,000 models between June 2024 and its closure in 2025, using six fixed benchmarks: IFEval, MuSR, GPQA, MATH, BBH and MMLU-Pro, all run with lm-evaluation-harness. Its closure was a direct consequence of the saturation and gaming problem: as the benchmarks became well known, models were optimised for them.

Successor: OpenEvals (huggingface.co/spaces/OpenEvals/find-a-leaderboard)

Aspect	OpenLLM Leaderboard v2	OpenEvals
Architecture	Single centralised leaderboard	200+ specialised community leaderboards
Benchmarks	Fixed (6 tasks)	Variable per leaderboard (math, code, medical, safety, etc.)
Models evaluated	13,000+	Distributed across leaderboards
Eval engine	lm-evaluation-harness	lighteval (most) or harness
Status	Archived with an updated UI (Mar 2025)	Active; you can create your own leaderboard
Advantage	Historical consistency	Specialisation, speed of adaptation

The archived collection is available for reference at huggingface.co/collections/OpenEvals/archived-open-llm-leaderboard-2024-2025.

LMArena / Chatbot Arena

Field	Data
Site	lmarena.ai (rebrand of LMSys Chatbot Arena, January 2026)
Methodology	Anonymous pairwise comparisons: the user votes for the answer they prefer without knowing the model
Statistical model	Bradley-Terry (maximum likelihood over the full history of match-ups; replaced classic Elo for greater stability)
Accumulated votes	> 6 million (Jun 2026)
Variants	Overall, Coding, Math, Hard Prompts, Multimodal — the specialised ones are more reliable
Licence/access	Free (web); vote data available under a CC licence
Limitations	Overall is biased by labs’ selective disclosure and by format gaming; Bradley-Terry was recalibrated in mid-2025 (Style Control), shifting some models by ±20–40 Elo

LMArena is the only benchmark based on human preference at scale. Its advantage: it captures real perceived quality, not just MCQ accuracy. Its limitation: it is hard to reproduce on-premise (it requires a user interface and a panel of human voters). Useful as an external validation signal, not as a CI gate.

Decision table: which tool for which objective

Objective	Tool	Why
Reproduce a standard academic eval	lm-evaluation-harness	De facto engine; maximum compatibility with papers
Holistic evaluation (accuracy + fairness + toxicity)	HELM (+ HELM Capabilities)	The only framework measuring every axis at once
Avoid contamination with recent questions	LiveBench	Monthly refresh; verifiable answers without a judge
Native integration with HF Hub and vLLM	lighteval	vLLM backend, 1,000+ tasks, direct publishing to the Hub
Compare against the public state of the art (OSS)	OpenEvals + lighteval	The successor ecosystem to the OpenLLM Leaderboard
Human preference at scale	LMArena	Bradley-Terry over millions of real comparisons
Custom domain eval (medical, legal, etc.)	lighteval with a custom task	Simpler to extend than the harness
Fast CI gate (<10 min)	lm-eval harness (subset)	Individual tasks with `--tasks ifeval,gpqa` + vLLM batching

The data contamination trap

What it is

A model that has seen a benchmark’s test data during pretraining memorises answers instead of generalising. The benchmark measures memorisation, not capability. This is called data contamination or benchmark leakage.

Scale of the problem (2024–2025 data)

Finding	Source	Figure
N-gram overlap in Llama-2-70B	“Data Contamination or Genuine Generalization?” (2025)	18.1% overlap with eval datasets
Out-of-distribution drop in contaminated large models	Ibid.	−9.4 pp on reformulated prompts vs the original test
HumanEval saturated	Artificial Analysis / multiple 2026 reports	Top models > 93%; separation ≤ 2 pp between the best 5
GPQA Diamond: appearance of indirect contamination	mindstudio.ai analysis 2026	“Data laundering”: questions leaked via online forums

Detection mechanisms

1. N-gram overlap (the classic method)

Compute the fraction of test set n-grams that appear in the model’s pretraining corpus. A practical threshold: if more than 13% of the test’s 8-grams match the corpus, the result is compromised.

\[ \text{contamination ratio} = \frac{|\{g \in \text{test n-grams}\} \cap \{g \in \text{train n-grams}\}|}{|\text{test n-grams}|} \]

Limit: it only works if you have access to the pretraining corpus. For closed models, it is not applicable.

2. Canary strings (decoy insertion)

Unique random strings are inserted into the evaluation dataset while it is being created. If the model reproduces those strings without having seen them in context, it confirms that it memorised them from pretraining. The technique was proposed in “Extracting Training Data from Large Language Models” (Carlini et al., 2021) and adopted in recent benchmarks.

3. Perturbation and reformulation

Semantically equivalent variants of the test are generated (paraphrasing, changing proper nouns, translation and back-translation). A model that generalises keeps its accuracy; a model that memorises drops. The average drop detected in contaminated models is 9–15 pp on equivalent reformulated prompts.

4. LiveBench (monthly refresh)

The architectural solution: if the questions are generated from sources published after the model’s cutoff (recent arXiv papers, news, new data), memorisation from pretraining cannot help. The model has to reason over information it could never have seen.

5. Kernel divergence analysis (Savelka et al., 2025)

A more sophisticated statistical method: it compares the embedding distribution of the test set with the distribution of the training corpus. An abnormally low divergence indicates that the benchmark and the training corpus come from the same source.

Why static leaderboards saturate

The cycle is predictable:

A new benchmark is published (MMLU in 2021, GPQA in 2023).
The labs use it as an evaluation target during pretraining and fine-tuning.
Performance on that benchmark rises quickly, faster than real capability.
The benchmark stops discriminating at the high end.
The community needs a new benchmark.

The average time between the publication of a benchmark and its saturation in frontier models has gone from ~36 months (MMLU: 2021–2024) to ~18 months (HumanEval: 2021–2023, GPQA: 2023–2025). Saturation is accelerating.

Honest methodology: reproducibility and sources of variance

The four parameters that must be pinned

An eval result is only reproducible if it declares:

Parameter	Example	Why it matters
Harness version	`lm-evaluation-harness==0.4.3`	Changes to the prompt template between versions invalidate the comparison
Exact model version	`meta-llama/Llama-3.1-70B-Instruct@sha256:abc…`	Silent model updates on the Hub change the results
Prompt template	`mmlu_pro_cot_0shot` vs `mmlu_pro_5shot`	The difference between 0-shot and 5-shot can be 5–12 pp
Number of shots	`--num_fewshot 5`	The published standard varies by benchmark; 0-shot and 5-shot are not comparable

Few-shot vs zero-shot: the numerical difference

Sensitivity to the number of shots varies dramatically by benchmark:

Benchmark	Typical 0-shot	Typical 5-shot	Difference
MMLU-Pro	62%	72%	+10 pp
GPQA Diamond	55%	65%	+10 pp
GSM8K	70%	82%	+12 pp
IFEval	68%	71%	+3 pp
HumanEval	75%	80%	+5 pp

Publishing an MMLU-Pro result in 0-shot without saying so, when the leaderboard standard is 5-shot, inflates the comparison by ~10 pp. Two results from the same model published with different numbers of shots are not comparable.

Prompt sensitivity

Minor perturbations in the prompt template produce variations of up to ±8 pp on MCQ and ±15 pp on generation tasks. Sources of variance identified in the literature:

Order of the options in MCQ (position bias): up to ±4 pp.
Wording of the question (paraphrasing): ±3–8 pp.
Presence or absence of “Let’s think step by step”: ±5–12 pp on reasoning.
Language of the system instructions: ±3–10 pp in non-multilingual models.

Why two evals give different results

A common scenario: the paper reports MMLU-Pro 74.3% and the reproduction gives 71.8%. Frequent causes:

Cause	Typical magnitude
Different harness version (changed template)	1–4 pp
Model with a silent update	0.5–3 pp
Different context truncation	0.5–2 pp
Sampling seed (for generation)	0.3–1.5 pp
Tokenisation with a chat template vs without one	2–8 pp

The sum of these sources explains gaps of 2–6 pp that are not noise but systematic protocol differences.

The confidence interval formula for comparing two models

To know whether the difference between model A (accuracy $p_A$) and model B ($p_B$) over a test set of size $n$ is statistically significant:

\[ \Delta p \pm 1.96 \cdot \sqrt{\frac{p_A(1-p_A) + p_B(1-p_B)}{n}} \]

Applied to GPQA Diamond ($n = 198$), if $p_A = 0.90$ and $p_B = 0.88$:

\[ \Delta p = 0.02 \quad;\quad \text{margin} = 1.96 \cdot \sqrt{\frac{0.09 + 0.1056}{198}} \approx 1.96 \cdot 0.0314 \approx 0.062 \]

The 95% confidence interval is $[{-0.042},\ {+0.082}]$: it crosses zero. A difference of 2 pp over 198 questions is not statistically distinguishable from noise. To distinguish 90% from 88% with 95% confidence you need:

\[ n \approx \frac{(1.96)^2 \cdot (p_A(1-p_A) + p_B(1-p_B))}{(\Delta p)^2} \approx \frac{3.84 \cdot 0.1956}{0.0004} \approx 1{,}878 \text{ questions} \]

GPQA Diamond is not that size. Differences of 1–2 pp on GPQA Diamond are statistically invisible.

The quality axis in the scorecard

The platform’s scorecard has three columns: performance (goodput under SLO), quality (score on a reference benchmark), energy (J/token). The quality column is built like this:

Field	Content
Primary benchmark	MMLU-Pro (5-shot) — separates in the 60–77% zone
Reasoning benchmark	GPQA Diamond (0-shot) — separates in the 55–90% zone
Instruction benchmark	IFEval (0-shot prompt-level accuracy)
Code benchmark	MBPP (0-shot, pass@1) — less saturated than HumanEval
Anti-contamination	LiveBench global score (monthly)
Tool	lm-evaluation-harness (primary) + lighteval (secondary, HF Hub)
Pinned version	`harness==0.4.3`, `lighteval==0.6.x`
Shots	Declared explicitly per benchmark
Reproducibility	Output JSON stored with the model SHA
Frequency	Per model release; not in the CI of every PR (GPU-hour cost)

A full eval over 4 benchmarks (MMLU-Pro 12,032 + GPQA 198 + IFEval 541 + MBPP 500) with a 70B model on a generic 2×H100 node takes on the order of 4–8 hours depending on batch size and number of shots. The GPU cost is real: at ~5 €/h per amortised card, a full eval costs on the order of 40–80 €. It is not run on every commit; it is run on every model release or major adapter release.

Sources

EleutherAI · lm-evaluation-harness — https://github.com/EleutherAI/lm-evaluation-harness
Stanford CRFM · HELM — https://github.com/stanford-crfm/helm
Stanford CRFM · HELM Capabilities (mar-2025) — https://crfm.stanford.edu/2025/03/20/helm-capabilities.html
Stanford CRFM · HELM (sitio oficial) — https://crfm.stanford.edu/helm/
LiveBench · paper arXiv:2406.19314 — https://arxiv.org/abs/2406.19314
LiveBench · sitio oficial — https://livebench.ai
LiveBench · GitHub — https://github.com/LiveBench/LiveBench
NYU Center for Data Science · LiveBench blog — https://nyudatascience.medium.com/livebench-challenging-language-models-with-contamination-free-questions-999b52967ec8
Hugging Face · lighteval — https://github.com/huggingface/lighteval
Hugging Face · OpenLLM Leaderboard (archivado) — https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard
Hugging Face · OpenEvals (sucesor) — https://huggingface.co/spaces/OpenEvals/find-a-leaderboard
Hugging Face · Archived Open LLM Leaderboard 2024-2025 — https://huggingface.co/collections/OpenEvals/archived-open-llm-leaderboard-2024-2025
LMArena · Chatbot Arena — https://lmarena.ai
LMSys · blog Chatbot Arena Elo — https://www.lmsys.org/blog/2023-05-03-arena/
“A Statistical Framework for Ranking LLM-Based Chatbots” · arXiv:2412.18407 — https://arxiv.org/pdf/2412.18407
“Data Contamination or Genuine Generalization?” (2025) — https://www.suaspress.org/ojs/index.php/AJNS/article/view/v2n2a03
“When Benchmarks Leak: Inference-Time Decontamination” (2025) · arXiv:2601.19334 — https://arxiv.org/pdf/2601.19334
“How Contaminated Is Your Benchmark?” (2025) · arXiv:2502.00678 — https://arxiv.org/pdf/2502.00678
“When AI Benchmarks Plateau” (2026) · arXiv:2602.16763 — https://arxiv.org/pdf/2602.16763
GPQA · arXiv:2311.12022 — https://arxiv.org/abs/2311.12022
GPQA Diamond leaderboard · Artificial Analysis — https://artificialanalysis.ai/evaluations/gpqa-diamond
“The Emperor’s New Clothes in Benchmarking?” (2025) · arXiv:2503.16402 — https://arxiv.org/pdf/2503.16402
lm-evaluation-harness · CLI reference — https://lm-evaluation-harness.readthedocs.io/running_evals/cli_reference/
crfm-helm · PyPI — https://pypi.org/project/crfm-helm/

LLM serving engines compared on the Pareto frontier: vLLM, SGLang, TRT-LLM and Dynamo

Tue, 16 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter). Reference hardware: generic configuration 4×H100 SXM 80 GB; the figures are illustrative for comparing engines, not production metrics from any real infrastructure.

TL;DR

Four OSS engines dominate on-premise LLM serving in 2026: vLLM, SGLang, NVIDIA TensorRT-LLM and NVIDIA Dynamo. None is superior on every axis. In raw throughput with a shared prompt, SGLang beats vLLM by 29 % on Llama-3.3-70B FP8 over H100 (16,200 vs 12,500 tok/s). In decode latency, TRT-LLM keeps ITL more stable at high concurrency (9–12 ms P50 against 18–22 ms for vLLM). Dynamo is not an inference engine: it is the orchestration layer on top of the previous three for multi-node clusters, and it reports up to 7× additional throughput in disaggregated serving on Blackwell. The right choice depends on three variables: load pattern (batch/interactive), deployment scale (single node/multi-node) and operational constraints (OSS maturity, ease of operation). The Pareto decision table is at the end of the article.

Framing the article: what this B8 is and is not

This article (B8 of the benchmarking pillar) takes for granted the metrics from the introduction to benchmarking and the tool catalogue from the card-by-card piece. It applies those concepts to engine selection. The protocol from GuideLLM for SLO validation and AIPerf in depth detail how to run the harness; here the result is used. The fundamentals of continuous batching, disaggregated serving, the attention backend and quantisation are assumed known.

Card for each engine

vLLM

Attribute	Value
Reference version	v0.9.x (June 2026)
Licence	Apache 2.0
Maintainer / governance	vLLM Project (Linux Foundation AI), led by UC Berkeley + Red Hat
Supported hardware	NVIDIA (Ampere/Hopper/Blackwell), AMD ROCm, Intel Gaudi, TPU, CPU
Repository	github.com/vllm-project/vllm
API	OpenAI-compatible (completions, chat, embeddings)

Origin: PagedAttention (Kwon et al., 2023). The first implementation of continuous batching with a paged KV cache. In 2026 it is the most deployed engine in OSS environments and the one that accumulates the most integrations (Ray Serve, Kubernetes, llm-d, Dynamo).

SGLang

Attribute	Value
Reference version	v0.4.x (June 2026)
Licence	Apache 2.0
Maintainer / governance	LMSYS Org (Berkeley, CMU, UCSD)
Supported hardware	NVIDIA (Ampere/Hopper/Blackwell), AMD ROCm
Repository	github.com/sgl-project/sglang
API	OpenAI-compatible + SGLang runtime API

Origin: the SGLang paper, 2024 (Zheng et al.). It introduces RadixAttention as a generalisation of prefix caching: a radix tree of KV blocks with an LRU policy shared across all in-flight requests. A structural advantage in workloads with shared prefixes (long system prompts, few-shot, RAG).

NVIDIA TensorRT-LLM (TRT-LLM)

Attribute	Value
Reference version	v0.18.x (June 2026)
Licence	Apache 2.0
Maintainer / governance	NVIDIA (proprietary in practice, OSS in name)
Supported hardware	NVIDIA only: Ampere (A100), Hopper (H100/H200/GH200), Ada (L40/L40S), Blackwell (B200/GB200)
Repository	github.com/NVIDIA/TensorRT-LLM
API	Python + C++ runtime; integrates with Triton Inference Server

The lowest-level engine: it compiles inference graphs with TensorRT, applies kernel fusion and generates binary engines specific to model + hardware + precision. Maximum performance on NVIDIA, with no portability to other hardware. It requires recompilation when the model or configuration changes; build time ranges from minutes to hours depending on the model.

NVIDIA Dynamo

Attribute	Value
Reference version	v1.1.1 (May 2026)
Licence	Apache 2.0
Maintainer / governance	NVIDIA + OSS community (ai-dynamo org on GitHub)
Supported hardware	NVIDIA (through the underlying engine: vLLM, SGLang or TRT-LLM)
Repository	github.com/ai-dynamo/dynamo
Role	Orchestration layer, NOT an inference engine

Dynamo does not run inference directly: it is the scheduler and router that coordinates pools of workers (vLLM, SGLang, TRT-LLM) across multiple nodes. Implemented in Rust (performance) + Python (extensibility). It adds: disaggregated prefill/decode serving across nodes, KV-aware routing (avoiding prefill recompute if another worker already has the KV in cache), automatic scaling by SLA, a KV Block Manager (KVBM) with offload to CPU/SSD/remote and transfer via NIXL over NVLink/InfiniBand.

Honourable mentions: LMDeploy and HF TGI

LMDeploy (OpenMMLab/InternLM, Apache 2.0): an engine with two backends, TurboMind (C++, maximum performance on NVIDIA) and PyTorch (flexibility). KV cache blocking and persistent batching; 4-bit inference up to 2.4× faster than FP16 in its own benchmark. A narrower ecosystem; no AMD support.

HF TGI (Hugging Face, Apache 2.0): the reference engine for the HuggingFace ecosystem. TGI v3.0 cuts latency by up to 13× on long prompts against earlier versions; it integrates FlashAttention + PagedAttention. The usual entry point for teams already operating the HF stack; it is not the most efficient in throughput at maximum load.

Capability matrix

The table uses this notation: Yes = supported and stable, Beta = available but not recommended for production, No = not available or only with a manual workaround.

Capability	vLLM	SGLang	TRT-LLM	Dynamo (via backend)
Continuous batching	Yes	Yes	Yes	Yes (delegates to the engine)
PagedAttention	Yes	Yes	Yes	Yes
RadixAttention / prefix caching	Yes (chunked prefill + prefix)	Yes (native RadixAttention)	Yes (prefix caching)	Yes + KV-aware routing
Speculative decoding	Yes (n-gram, EAGLE, DFlash, suffix)	Yes (draft+verify)	Yes (EAGLE-3, MTP)	Yes
FP8 quantisation	Yes	Yes	Yes	Yes
MXFP4/NVFP4 quantisation	Yes	Beta	Yes (Blackwell)	Yes
INT4 AWQ quantisation	Yes	Yes	Yes	Yes
GPTQ quantisation	Yes	Yes	Not native	Yes (via vLLM)
INT8 quantisation (SmoothQuant)	Yes	Yes	Yes	Yes
Structured output	Yes	Yes (compressed FSM)	Beta	Yes
Dynamic multi-LoRA	Yes	Yes	Beta	Yes
Disaggregated prefill/decode	Beta (experimental)	Yes	Yes	Yes (first class)
Chunked prefill	Yes	Yes	Yes	Yes
Multimodal support (VLM)	Yes	Yes	Yes	Yes
Multi-node tensor parallelism	Yes	Yes	Yes	Yes (+ pipeline across nodes)
OpenAI API compatible	Yes	Yes	Yes (via Triton)	Yes
Non-NVIDIA hardware	Yes (AMD/Intel/TPU)	Yes (AMD)	No	No

Sources for the matrix: docs.vllm.ai, docs.sglang.io, nvidia.github.io/TensorRT-LLM, docs.nvidia.com/dynamo, github.com/ai-dynamo/dynamo (feature matrix, June 2026).

Why they are not compared lightly: the variables that break a comparison

Before the table of figures, the four decisions that invalidate any cross-engine benchmark:

1. A different harness. A benchmark run with each engine’s native micro-bench (vllm bench serve vs SGLang bench) measures the client, not the engine. The discrepancy can be up to 7.2× at high concurrency. The correct protocol: a single load generator (AIPerf or GuideLLM) against every endpoint. The engine is the variable, the tool is the constant.

2. Different ISL/OSL. Input Sequence Length and Output Sequence Length determine the prefill/decode ratio and therefore which engine is favoured. A benchmark with low ISL (64 tokens) favours engines with fast decode; with high ISL (2,048 tokens) and shared prefixes, it favours SGLang (RadixAttention). Fixing ISL and OSL is mandatory for the comparison to be valid.

3. Different precision. FP16 and FP8 are not comparable: FP8 gives between 1.5× and 2× more throughput on Hopper. Comparing vLLM FP16 with TRT-LLM FP8 says nothing about the engine.

4. Not reporting P99. Maximum throughput is reached at the point where TTFT P99 already violates the SLO. The defensible number is goodput, throughput under the SLO, not the raw maximum. The end-to-end latency formula:

$$\text{latency}_{e2e} \approx \text{TTFT} + (N_{\text{out}} - 1) \times \text{TPOT}$$

where $N_{\text{out}}$ is the number of output tokens, makes it clear that a high TTFT with low TPOT has the same effect as the inverse case only for one specific $N_{\text{out}}$: both must be reported separately.

The complete fair-comparison protocol is in the tools article.

Performance on the Pareto frontier: illustrative figures

Reference configuration for every row: 4×H100 SXM 80 GB NVLink, model Llama-3.1-70B-Instruct FP8, harness AIPerf (formerly genai-perf, the multi-process successor), dataset ShareGPT (realistic length distribution), mean ISL ~512 tok, mean OSL ~256 tok, SLO TTFT P99 < 500 ms. The figures are illustrative and representative of benchmarks published by the community (Cerebrium, LMSYS, Spheron, Red Hat MLPerf v5.1); they are not measurements from any real infrastructure. Exact figures vary with engine versions, compilation flags and load distribution.

Throughput vs latency: Pareto table

Engine	Config	Throughput (tok/s)	TTFT P50 (ms)	TTFT P99 (ms)	ITL P50 (ms)	Goodput (@SLO)
vLLM v0.9	FP8, chunked prefill	12,500	160	420	18	~96 %
SGLang v0.4	FP8, RadixAttention	16,200	140	390	21	~97 %
TRT-LLM v0.18	FP8, compiled engine	14,800	190	480	10	~93 %
vLLM + Dynamo	FP8, disagg. P/D 2+2	18,500	120	310	19	~98 %
SGLang + Dynamo	FP8, disagg. P/D 2+2	21,000	110	280	22	~99 %
TRT-LLM + Dynamo	FP8, disagg. P/D 2+2	22,500	130	350	11	~97 %

Reading notes:

“Disagg. P/D 2+2” = 2 prefill GPUs + 2 decode GPUs (the same 4×H100 budget).
Goodput falls against raw throughput when the latency tail exceeds the SLO of 500 ms P99. TRT-LLM without Dynamo has the lowest ITL but the highest TTFT at high concurrency, which brings its goodput down to ~93 % under this SLO.
Dynamo adds ~30–40 % of effective throughput over 4 GPUs thanks to disaggregated serving and KV-aware routing, in the scenario with reusable prefixes (ShareGPT).
In workloads without shared prefixes (short ISL, unique prompts), Dynamo’s advantage shrinks to ~10–15 % over the base engine.

Saturation point (concurrency sweep, vLLM FP8, 4×H100)

Concurrency	TTFT P99 (ms)	ITL P50 (ms)	Throughput (tok/s)	Goodput (tok/s)
8	210	14	5,200	5,200
16	370	18	9,800	9,800
32	420	18	12,500	12,200
48	680	26	13,800	7,400
64	1,200	41	14,100	2,100

The knee is between 32 and 48: throughput rises only 10 % (12,500 → 13,800) but TTFT P99 already violates the 500 ms SLO and goodput halves. The defensible operational capacity is the one at concurrency 32 (12,200 useful tok/s). Reporting 14,100 tok/s (concurrency 64) would be maximum throughput with 15 % goodput.

Fair-comparison methodology: the harness as the constant

The protocol that makes numbers comparable across different engines. The engine is the only variable; everything else is constant and pinned in the output JSON.

Variables that are fixed (constants of the experiment)

Variable	Value fixed in the reference experiment
Load tool	AIPerf v2.x (multi-process)
Model	Llama-3.1-70B-Instruct
Precision	FP8
Hardware	4×H100 SXM 80 GB NVLink
Dataset	ShareGPT (realistic ISL/OSL distribution)
SLO	TTFT P99 < 500 ms, ITL P50 < 30 ms
Warm-up	200 requests discarded before measuring
Sweep	concurrencies 1, 4, 8, 16, 32, 48, 64
Primary metric	goodput (useful tok/s under SLO)

Harness steps

Deploy the engine with the exact config (pinned flags, fixed version).
Warm-up: 200 requests; discard the results.
Concurrency sweep: aiperf profile --concurrency-range 1:64:step with AIPerf.
Extend the sweep past the knee (until goodput falls below 50 %).
Collect the JSON: TTFT P50/P99, ITL P50, raw throughput, goodput.
Change only the engine; repeat steps 1–5.
Compare the goodput column across rows.

The sweep with GuideLLM is equivalent for SLO validation; AIPerf is preferable for engine comparisons because its automatic estimatedCapacity normalises the saturation point. Both can be used: GuideLLM for the operational SLO, AIPerf for reference capacity.

Reproducible output format

{
 "harness": "aiperf", "harness_version": "2.x",
 "model": "Llama-3.1-70B-Instruct", "precision": "FP8",
 "hardware": "4xH100_SXM_80GB_NVLink",
 "dataset": "sharegpt",
 "slo": {"ttft_p99_ms": 500, "itl_p50_ms": 30},
 "motor": "vllm", "motor_version": "0.9.x",
 "concurrency_sweep": [8, 16, 32, 48, 64],
 "results_at_knee": {
 "concurrency": 32,
 "ttft_p50_ms": 210, "ttft_p99_ms": 420,
 "itl_p50_ms": 18, "throughput_tok_s": 12500,
 "goodput_tok_s": 12200
 }
}

This versioned JSON is the auditable datum: anyone can reproduce the figure with the same tool, version, model, hardware and load.

State of the art 2026: relative maturity

Dimension	vLLM	SGLang	TRT-LLM	Dynamo
Production maturity	High (3+ years)	Medium-high (2 years)	High (NVIDIA)	Medium (1.0 GA Mar 2026)
Release cadence	Weekly	Weekly	Monthly	Fortnightly
Integration ecosystem	Very broad (Ray, k8s, llm-d, Dynamo, NIM)	Broad (Dynamo, k8s)	NVIDIA-centric (Triton, NIM)	NVIDIA-centric
Ease of operation	High (pip install, OpenAI API)	High	Medium (requires compiling engines)	Medium-low (multi-node, etcd/NATS)
Non-NVIDIA hardware	Yes (AMD, Intel, TPU)	Yes (AMD)	No	No
Disaggregated serving	Beta	Stable	Stable	First class
Dynamic multi-LoRA	Stable	Stable	Beta	Stable (via vLLM/SGLang)
Structured output	Stable	Stable (FSM)	Beta	Stable (via the engine)
OSS community support	Very high (>30 K GitHub stars)	High (~20 K stars)	High (~10 K stars)	Growing (6.8 K stars)

Pareto decision table: when to choose each engine

No prose; the decision as a table of criteria and outcome.

Selection criterion	Recommended engine	Quantitative justification
Single node, quick prototype, team with no LLM serving experience	vLLM	pip install + OpenAI API in minutes; the broadest community support
Workload with long shared prefixes (RAG, few-shot, system prompts > 512 tok)	SGLang	RadixAttention: up to 6.4× more throughput vs a baseline without prefix caching
Maximum throughput on NVIDIA, no portability, team with TRT experience	TRT-LLM	Fused kernels + native FP8: ITL 9–12 ms vs 18–22 ms for vLLM on the same hardware
Multi-node (> 8 GPUs), mixed prefill-intensive/decode-intensive traffic	Dynamo + SGLang	Disaggregated P/D: up to 7× additional throughput on Blackwell; 2× TTFT with KV-aware routing
AMD ROCm or Intel Gaudi hardware	vLLM	The only SOTA engine with stable non-NVIDIA support
Structured output + high concurrency	SGLang	Compressed FSM: guided decode with no appreciable overhead
Stable dynamic multi-LoRA in production	vLLM or SGLang	TRT-LLM multi-LoRA still in beta (Jun 2026)
Hugging Face ecosystem, models < 13B	HF TGI v3	Direct HF Hub integration; competitive latency on small models
Maximum energy efficiency (J/token) on NVIDIA	TRT-LLM	Lower-overhead kernels → fewer J/token at iso-throughput

Quantisation: effect on throughput and quality

The four main options in order of decreasing speed and increasing quality:

Format	Relative throughput (H100)	Quality degradation (perplexity)	Engines with stable support
NVFP4 / MXFP4	~2.2× vs FP16	1–3 % on MMLU	TRT-LLM (Blackwell), vLLM (experimental)
FP8 (W8A8)	~1.7× vs FP16	< 1 % on MMLU	vLLM, SGLang, TRT-LLM
INT4 AWQ	~1.5× vs FP16	1–2 % on MMLU	vLLM, SGLang, TRT-LLM
GPTQ (INT4)	~1.4× vs FP16	1–3 % on MMLU	vLLM, SGLang
FP16 (baseline)	1.0×	0 %	All

FP8 is the de facto standard on Hopper for production: 1.7× throughput with degradation below 1 %. The article on quantisation in depth develops the quality/speed trade-off by format and model architecture.

Disaggregated serving: when the overhead is worth it

Separating prefill/decode into independent pools introduces transfer latency for KV tensors between nodes (NIXL over NVLink or InfiniBand). The overhead pays off when:

Condition	Effect of disaggregated serving
Mean ISL > 1,024 tokens	Prefill dominates; separate pools stop it blocking decode
Prefill/decode ratio > 3:1 in time	The decode pool sits idle waiting for prefill
Mixed traffic (prefill burst + decode queue)	Independent scaling of each pool by SLA
< 512 tokens ISL, unique prompts	Transfer overhead exceeds the benefit; an aggregated engine is enough

Dynamo quantifies the break-even point: disaggregated serving pays off when the transfer time for KV tensors (a function of ISL and NVLink/IB bandwidth) is lower than the time the decode worker would spend waiting for the prefill worker in aggregated mode. On 4×H100 with an ISL of 512 tok, break-even occurs at a prefill/decode ratio > 2.5:1. The article on disaggregated serving develops the full formula.

MLPerf Inference v5.1: cross-vendor reference figures

MLPerf Inference v5.1 (September 2025) is the standard for cross-vendor comparability. Server scenario (Llama-3.1-70B, tokens/s):

Submitter	Hardware	Scenario	Throughput (tok/s)
Red Hat (vLLM)	1×H100 80 GB	Server	5,103
Red Hat (vLLM)	1×L40S	Server	1,207
NVIDIA (TRT-LLM)	8×H100 SXM	Server	~52,000 (estimated)

The MLPerf figures are not directly comparable with the internal benchmarks above (a different harness, MLPerf’s Server scenario with a Poisson distribution and fixed SLOs). They are useful for comparing vendors under the same rules; not for sizing a specific case. For that, run your own sweep with AIPerf/GuideLLM against the target endpoint.

Variables the benchmark does not capture

Omitted variable	Effect in production
Network latency (gateway, load balancer)	Can add 20–100 ms to the TTFT measured on the server
Engine compilation time (TRT-LLM)	Hours per model×precision×hardware; penalises cold starts and CI
Weight loading time on the GPU	Relevant in autoscaling; ModelExpress (Dynamo) cuts it 7×
Operational cost of the system (Dynamo vs a simple engine)	Dynamo requires etcd/NATS, a Rust runtime, a more specialised team
Response quality	Performance does not imply quality; measure with lm-evaluation-harness (article B7)
Energy (J/token)	Does not come out of a throughput benchmark; requires DCGM running alongside

Sources

vLLM · documentación oficial — https://docs.vllm.ai/en/latest/
vLLM · blog de rendimiento (v0.6.0, 2,7× throughput) — https://blog.vllm.ai/2024/09/05/perf-update.html
vLLM · disaggregated prefill (experimental) — https://docs.vllm.ai/en/latest/features/disagg_prefill/
SGLang · documentación oficial — https://docs.sglang.io/
SGLang · paper RadixAttention (LMSYS, 2024) — https://arxiv.org/pdf/2312.07104
SGLang · blog LMSYS sobre Llama-3 serving — https://www.lmsys.org/blog/2024-07-25-sglang-llama3/
NVIDIA TensorRT-LLM · GitHub — https://github.com/NVIDIA/TensorRT-LLM
NVIDIA TensorRT-LLM · overview y release notes — https://nvidia.github.io/TensorRT-LLM/overview.html
NVIDIA TensorRT-LLM · quantización — https://nvidia.github.io/TensorRT-LLM/features/quantization.html
NVIDIA Dynamo · GitHub (ai-dynamo/dynamo) — https://github.com/ai-dynamo/dynamo
NVIDIA Dynamo · documentación oficial — https://docs.nvidia.com/dynamo/latest
NVIDIA Dynamo · disaggregated serving — https://docs.dynamo.nvidia.com/dynamo/design-docs/disaggregated-serving
Cerebrium · benchmark vLLM vs SGLang vs TRT-LLM (Llama 3.1-70B) — https://cerebrium.ai/blog/benchmarking-vllm-sglang-tensorrt-for-llama-3-1-api
Spheron · vLLM vs TRT-LLM vs SGLang H100 benchmarks 2026 — https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/
MLCommons · MLPerf Inference v5.1 results — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
Red Hat · MLPerf Inference v5.1 con vLLM — https://www.redhat.com/en/blog/efficient-and-reproducible-llm-inference-red-hat-mlperf-inference-v51-results
arXiv 2605.24217 · sesgo sistemático en benchmarks de inferencia LLM — https://arxiv.org/html/2605.24217
IETF Draft · LLM Benchmarking Methodology (enero 2026) — https://www.ietf.org/archive/id/draft-gaikwad-llm-benchmarking-methodology-00.html

GenAI-Perf in depth: LLM inference profiling with real data

Mon, 15 Jun 2026 03:00:00 +0200

Notation: amounts in euros (N EUR), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

TL;DR

GenAI-Perf (the genai-perf package, part of NVIDIA’s Triton ecosystem) is the reference LLM inference profiler for OpenAI-compatible endpoints. Installable with pip install genai-perf. A single genai-perf profile command produces a console table with TTFT, ITL, request latency (avg/min/max/P75/P90/P99), output sequence length, output token throughput and request throughput, plus profile_export_genai_perf.json and .csv artefacts. The analyze subcommand automatically sweeps concurrencies from 1 to 256 (powers of 2) and generates a summary analyze_export_genai_perf.csv. GenAI-Perf was declared retired in April 2026 (replaced by AIPerf); it remains the reference benchmark for historical runs and documented comparisons, and the NVIDIA tool with the most public documentation for LLMs before 2026.

What GenAI-Perf is and where it fits

GenAI-Perf is a command-line tool for measuring the throughput and latency of generative AI models served through an inference server. It is part of the triton-inference-server/perf_analyzer GitHub repository and is distributed as a pip package (genai-perf) and inside the Triton SDK container (nvcr.io/nvidia/tritonserver:YY.MM-py3-sdk).

Position in the NVIDIA ecosystem:

Relationship with perf_analyzer: GenAI-Perf internally uses Triton’s perf_analyzer binary to generate load and measure latencies. perf_analyzer is the low-level load generator; GenAI-Perf is the high-level layer that adds prompt synthesis, OpenAI payload construction and the LLM-specific metrics (TTFT, ITL, sequence lengths).

Installation:

# Option 1: pip (requires CUDA 12, Ubuntu 24.04, Python 3.10+)
pip install genai-perf

# Option 2: Triton SDK container (recommended for reproducibility)
export RELEASE="25.01"
docker run -it --net=host --gpus=all \
 nvcr.io/nvidia/tritonserver:25.01-py3-sdk
# Inside the container:
genai-perf --help

Metrics GenAI-Perf produces

The metrics cover the full cycle of a streaming generation request. All of them are reported with avg, min, max, P75, P90 and P99 except the throughput ones (avg only).

Metrics glossary

Metric	Symbol	Exact definition (GenAI-Perf)	Unit
Time to First Token	TTFT	Time from sending the request to receiving the first token (includes queueing, prefill and network)	ms
Inter-Token Latency	ITL (= TPOT)	$\frac{e2e\_latency - TTFT}{output\_tokens - 1}$ — decode phase only, excludes the first token	ms/token
Request Latency	e2e_latency	$e2e = TTFT + Generation\_time$ — from the first request to the last response	ms
Output Token Throughput	OTT	$\frac{total\_output\_tokens}{T_y - T_x}$ where $T_x$ = first request, $T_y$ = last token received	tok/s
Request Throughput	RPS	$\frac{total\_completed\_requests}{T_y - T_x}$	req/s
Input Sequence Length	ISL	Mean length in tokens of the input prompt	tokens
Output Sequence Length	OSL	Mean length in tokens of the generated response	tokens

Methodological note on ITL versus LLMPerf: GenAI-Perf excludes TTFT from the ITL calculation (pure decode only). LLMPerf includes TTFT in its ITL mean. The numbers are not directly comparable.

Methodological note on TPS versus LLMPerf: GenAI-Perf measures throughput between the first request and the last token received ($T_y - T_x$). LLMPerf uses the total duration of the benchmark including prompt generation and response storage; in a concurrency-1 scenario, that difference can amount to up to 33% in the reported throughput figure.

Console output table (real example, TRT-LLM backend)

The table GenAI-Perf prints for a profile with ISL=200, OSL=100, concurrency=1:

 NVIDIA GenAI-Perf | LLM Metrics
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
┃ Statistic ┃ avg ┃ min ┃ max ┃ p99 ┃ p90 ┃ p75 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
│ Time to first token (ms) │ 13.68 │ 11.07 │ 21.50 │ 18.81 │ 14.29 │ 13.97 │
│ Inter token latency (ms) │ 1.86 │ 1.28 │ 2.11 │ 2.11 │ 2.01 │ 1.95 │
│ Request latency (ms) │ 203.70 │ 180.33 │ 228.30 │ 225.45 │ 216.48 │ 211.72 │
│ Output sequence length │ 103.46 │ 95.00 │ 134.00 │ 122.96 │ 108.00 │ 104.75 │
│ Input sequence length │ 200.00 │ 200.00 │ 200.00 │ 200.00 │ 200.00 │ 200.00 │
│ Output token throughput (per sec) │ 504.02 │ N/A │ N/A │ N/A │ N/A │ N/A │
│ Request throughput (per sec) │ 4.87 │ N/A │ N/A │ N/A │ N/A │ N/A │
└───────────────────────────────────┴────────┴────────┴────────┴────────┴────────┴────────┘

(Source: docs.nvidia.com/deeplearning/triton-inference-server — LLM tutorial)

Invocation against OpenAI-compatible endpoints

`profile` mode: one load point

The profile subcommand measures a fixed operating point (concurrency or request-rate, synthetic ISL/OSL).

Example 1: vLLM endpoint (chat, streaming)

# 1. Bring up the vLLM server
docker run -it --net=host --rm --gpus=all \
 vllm/vllm-openai:latest \
 --model meta-llama/Llama-3.1-70B-Instruct \
 --dtype float16

# 2. Profile with GenAI-Perf
genai-perf profile \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --concurrency 16 \
 --synthetic-input-tokens-mean 1024 \
 --synthetic-input-tokens-stddev 128 \
 --output-tokens-mean 256 \
 --output-tokens-stddev 0 \
 --output-tokens-mean-deterministic \
 --request-count 200 \
 --warmup-request-count 20 \
 --tokenizer meta-llama/Llama-3.1-70B-Instruct

Example 2: completions endpoint (no streaming)

genai-perf profile \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type completions \
 --concurrency 32 \
 --synthetic-input-tokens-mean 512 \
 --synthetic-input-tokens-stddev 64 \
 --output-tokens-mean 128 \
 --request-count 200 \
 --warmup-request-count 20

Example 3: NVIDIA NIM (TRT-LLM backend endpoint, via Triton)

genai-perf profile \
 -m meta/llama-3.1-70b-instruct \
 --backend tensorrtllm \
 --streaming \
 --concurrency 8 \
 --synthetic-input-tokens-mean 2048 \
 --synthetic-input-tokens-stddev 0 \
 --output-tokens-mean 512 \
 --output-tokens-mean-deterministic \
 --request-count 100 \
 --warmup-request-count 10 \
 --generate-plots

Example 4: request-rate instead of concurrency (Poisson)

genai-perf profile \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --request-rate 5.0 \
 --synthetic-input-tokens-mean 1024 \
 --output-tokens-mean 256 \
 --request-count 300 \
 --warmup-request-count 30

Main flags and what they do

Flag	Typical values	Function
`--endpoint-type`	`chat`, `completions`, `embeddings`	Type of OpenAI-compatible endpoint
`--streaming`	(boolean)	Enables SSE streaming; required to measure real TTFT and ITL
`--concurrency`	1, 8, 16, 32, 64	Number of concurrent requests maintained; GenAI-Perf guarantees N active at all times
`--request-rate`	1.0, 5.0, 10.0	Constant arrival rate (req/s); does not guarantee N active
`--synthetic-input-tokens-mean`	128–8192	Mean of synthetic ISL tokens
`--synthetic-input-tokens-stddev`	0–512	Standard deviation of the ISL (0 = fixed)
`--output-tokens-mean`	64–2048	Mean of target OSL tokens
`--output-tokens-mean-deterministic`	(boolean)	Sets the minimum output tokens = target mean (more precise with TRT-LLM)
`--request-count`	100–2000	Number of requests to benchmark
`--warmup-request-count`	10–50	Warm-up requests discarded from the metrics
`--generate-plots`	(boolean)	Generates PNG plots of TTFT vs ISL, ITL vs token position, etc.
`--tokenizer`	HF model id	Tokenizer for counting tokens; mandatory when ISL/OSL matter
`--backend`	`tensorrtllm`, `vllm`	Triton backend (for serving directly via Triton without an OpenAI endpoint)
`--extra-inputs`	`ignore_eos:true`	Extra request parameters; `ignore_eos:true` guarantees a consistent OSL

Note on --concurrency versus --request-rate: NVIDIA recommends using --concurrency. With --request-rate, if the rate exceeds the engine’s throughput, the queue grows without bound and the metrics stop being stable. With --concurrency, there are always exactly N active requests, which gives a clean latency-throughput curve.

Output artefacts

GenAI-Perf dumps all results into an artifacts/ directory:

Artefact	Format	Content
`profile_export_genai_perf.json`	JSON	Full metrics (avg, min, max, P75, P90, P99) + CLI arguments used
`profile_export_genai_perf.csv`	CSV	Exported console tables, ready to import into Excel/pandas
`profile_export.json`	JSON	Raw perf_analyzer data (per-request traces)
`inputs.json`	JSON	Synthetic payloads sent (for reproducibility)
PNG plots	PNG	TTFT analysis, Request latency, TTFT vs ISL, ITL vs position, ISL vs OSL (with `--generate-plots`)

The `analyze` subcommand: automatic concurrency sweep

The analyze subcommand sweeps multiple values of a parameter in a single command and generates a summary CSV.

Concurrency sweep (the most used)

# Sweeps concurrencies 1, 2, 4, 8, 16, 32, 64, 128, 256
genai-perf analyze \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --sweep-type concurrency \
 --sweep-range 1:256 \
 --synthetic-input-tokens-mean 1024 \
 --output-tokens-mean 256 \
 --request-count 200 \
 --warmup-request-count 20

Request-rate sweep

# Sweeps 2, 4, 6, 8, 10, 12 req/s
genai-perf analyze \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --sweep-type request_rate \
 --sweep-list 2,4,6,8,10,12 \
 --synthetic-input-tokens-mean 1024 \
 --output-tokens-mean 256 \
 --request-count 200 \
 --warmup-request-count 20

ISL sweep (to study the effect of prefill)

genai-perf analyze \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --sweep-type input_sequence_length \
 --sweep-list 256,512,1024,2048,4096 \
 --concurrency 16 \
 --output-tokens-mean 256 \
 --request-count 200 \
 --warmup-request-count 20

Artefact structure of `analyze`

For a sweep with --sweep-type concurrency --sweep-range 1:32:

artifacts/
llama70b-openai-chat-concurrency1/
inputs.json
profile_export.json
profile_export_genai_perf.json
profile_export_genai_perf.csv
llama70b-openai-chat-concurrency2/
...
llama70b-openai-chat-concurrency4/
...
[...]
analyze_export_genai_perf.csv ← summary of every scenario
checkpoint.json ← allows interrupted sweeps to be resumed

Summary CSV format

Config Name,Concurrency,ISL,p99 TTFT (ms),p99 ITL (ms),p99 Request Latency (ms),Avg. OTT (tok/s),RPS
llama70b_run_0,1,1024,45.2,6.8,1823.4,128.3,0.55
llama70b_run_1,4,1024,48.1,7.1,1901.2,510.7,2.18
llama70b_run_2,8,1024,61.3,7.9,2142.5,989.2,4.12
llama70b_run_3,16,1024,98.7,9.4,2891.3,1741.6,6.88
llama70b_run_4,32,1024,284.1,14.2,4320.8,2134.2,7.14
llama70b_run_5,64,1024,892.4,28.7,8741.2,2251.8,7.23

The summary CSV also includes a second table with GPU metrics (P99 power, energy, utilisation, memory) when they are captured via Triton’s telemetry URLs (--server-metrics-urls http://localhost:8002/metrics).

Comparison table: GenAI-Perf versus GuideLLM versus LLMPerf versus vLLM bench

The choice of tool changes the number. They are not comparable with each other without adjustment.

Dimension	GenAI-Perf	GuideLLM	LLMPerf	vLLM bench serve
Origin / maintainer	NVIDIA / Triton team (retired Apr 2026)	Red Hat / vLLM project	Anyscale / Ray	vLLM project
Class	multi-process load generator	multi-process load generator	multi-process load generator	single-process micro-bench
Supported endpoints	OpenAI-compatible + KServe + native Triton	OpenAI-compatible	OpenAI-compatible	native vLLM
Main load mode	fixed concurrency N (recommended) or constant request-rate	synchronous, concurrent, poisson, throughput, automatic sweep	batches of N concurrent (with a draining period at the end)	request-rate, concurrency
Draining period	NO — guarantees N active at all times	NO (poisson)	YES — at the end of each batch the system empties, concurrency drops to 0	N/A
Automatic sweep	`analyze` (concurrency, request-rate, ISL, OSL)	`--rate-type sweep` (idle to saturation, 10 rounds)	manual (several runs)	`vllm bench sweep serve`
LLM metrics	TTFT, ITL, e2e latency, OTT, RPS, ISL/OSL	TTFT, ITL/TPOT, throughput, goodput under SLO	TTFT, ITL (includes TTFT in the mean), TPS (total benchmark duration)	TTFT, TPOT, throughput
ITL difference	excludes TTFT: $(e2e - TTFT) / (N_{tok}-1)$	excludes TTFT (same as GenAI-Perf)	includes TTFT in the mean	excludes TTFT
TPS difference	$total\_tokens / (T_y - T_x)$	throughput + goodput under SLO	$total\_tokens / (T_{end} - T_{start})$ — up to 33% lower	$total\_tokens / (T_y - T_x)$
Goodput / SLO	NO	YES (the key differentiator)	NO	NO
Exports	JSON + CSV + PNG + checkpoint	JSON + YAML + CSV + interactive HTML	JSON	console / JSON
Built-in warm-up	YES (`--warmup-request-count`)	YES (warm-up requests)	NOT native	YES
Real datasets	OpenOrca, CNN DailyMail, JSONL file, moon_cake	file with a traffic distribution	JSONL file	sharegpt, random
GPU telemetry	YES (via Triton’s server-metrics-urls → summary CSV)	NOT native (integrate DCGM separately)	NO	NO
When to choose it	fine profiling of one operating point, comparing configs with the same harness, integration with NIM/Triton	SLO-driven sweep, finding the knee, sizing replicas	quick endpoint validation in the Ray ecosystem	iterating on vLLM flags

Fair-comparison rule: never cross numbers from different tools for the same conclusion. If vLLM and TRT-LLM are being compared, use GenAI-Perf against both, with the same ISL/OSL, the same concurrency and the same --warmup-request-count. The tool is the constant; the engine is the variable.

Profiling on 4×H100 SXM 80 GB: a full sweep example

Generic reference hardware: 4×H100 SXM 80 GB NVLink, model Llama-3.1-70B-Instruct FP16, ISL=1024 tokens, OSL=256 tokens, vLLM endpoint.

Illustrative result of a concurrency sweep (1→64)

Concurrency	TTFT P50 (ms)	TTFT P99 (ms)	ITL P50 (ms)	ITL P99 (ms)	OTT (tok/s)	RPS
1	38	45	6.4	7.1	130	0.51
4	40	51	6.8	8.2	510	2.00
8	46	62	7.3	9.4	980	3.83
16	68	98	8.9	12.1	1,720	6.72
32	145	284	13.7	21.4	2,130	8.32
48	380	740	22.3	38.6	2,280	8.90
64	890	1,820	41.2	78.3	2,340	9.14

How to read it: throughput saturates around concurrency 48–64, while TTFT P99 already exceeds 500 ms from concurrency 32 onwards. The knee (the last point with TTFT P99 < 500 ms) is at concurrency 16 with 1,720 useful tok/s. Anyone reporting “2,340 tok/s” (concurrency 64) is describing the maximum throughput, at a point where the system meets no interactive chat SLO at all.

Formulas for the key metrics

$$ \text{ITL} = \frac{e2e\_latency - TTFT}{N_{output\_tokens} - 1} $$ $$ \text{OTT} = \frac{\sum output\_tokens}{T_y - T_x} $$ $$ \text{RPS} = \frac{N_{requests}}{T_y - T_x} $$

where $T_x$ is the timestamp of the first request sent and $T_y$ the timestamp of the last token received from the last request.

Synthetic ISL/OSL profiling: what to configure for each use case

The ISL/OSL parameters determine which engine of the system is stressed: a long ISL stresses prefill (KV-cache, TTFT); a long OSL stresses decode (ITL, memory bandwidth).

Use case	Typical ISL	Typical OSL	`--synthetic-input-tokens-mean`	`--output-tokens-mean`
Short interactive chat	~300 tok	~100 tok	300	100
Code copilot	~800 tok	~200 tok	800	200
Document summarisation	~2000 tok	~256 tok	2000	256
RAG with long context	~4096 tok	~512 tok	4096	512
Batch without latency	~512 tok	~1024 tok	512	1024

The --extra-inputs ignore_eos:true option disables the engine’s EOS token, forcing exactly the requested tokens to be generated. Essential for measuring a consistent OSL on TRT-LLM; optional on vLLM.

Honest methodology: how not to measure badly

1. Warm-up is mandatory

Without warm-up, the first requests include the KV-cache loading latency, the engine’s autotuning and the initialisations. GenAI-Perf applies a sliding window to detect stability (max/min ratio within a margin over the last 3 measurements), but the explicit --warmup-request-count discards the first N requests from the metrics. Recommended minimum value: 10 requests (or 5% of the total if the total is large).

2. The concurrency knee: extend the sweep until you see it

Throughput saturates once the engine’s effective max batch size is reached. If the sweep stops before saturation is reached, the “maximum capacity” reported is the last concurrency tested, not the real knee. The rule: the sweep must reach a point where TTFT P99 has doubled relative to the low-load value (or where OTT stops growing by more than 5% between steps). Only then is the knee visible.

For a sweep with --sweep-range 1:256, GenAI-Perf sweeps 1, 2, 4, 8, 16, 32, 64, 128, 256 (powers of 2). On most 70B models over 4×H100, saturation occurs between concurrency 32 and 64; the range 1:128 covers the knee with room to spare.

3. Concurrency versus request-rate: choosing correctly

Mode	Behaviour	When to use
`--concurrency N`	Always N active requests; when one finishes, the next is sent	Clean latency-throughput curve; comparison between configs
`--request-rate R`	1 request is sent every 1/R seconds; the queue can grow without bound	Simulating traffic with arrivals at a constant rate

NVIDIA recommends --concurrency for benchmarking. With --request-rate, if R exceeds the engine’s maximum throughput, the queue grows indefinitely and the metrics become unstable.

4. Single-process versus multi-process bias

GenAI-Perf is multi-process: it maintains N concurrent requests using perf_analyzer’s internal manager. There is no risk of client saturation. This sets it apart from vllm bench serve (single-process), which can saturate on the client at high concurrency and give inflated throughput metrics.

5. `ignore_eos` and an inconsistent OSL

Without --extra-inputs ignore_eos:true, the engine may generate fewer tokens than requested (if the model produces an EOS earlier). This makes the real OSL vary between requests and the ITL become inconsistent between runs. For reproducible benchmarks with a controlled OSL: always enable ignore_eos:true.

6. The correct tokenizer

The --tokenizer flag must point to the model being served. If it is omitted, GenAI-Perf uses a default tokenizer that can give different ISL/OSL in the real model’s tokens, making the throughput numbers in tok/s not comparable between models with different vocabularies.

7. One run = a data point; three or more = a number

Variance between runs under identical conditions can be 5–10% on tail latency metrics (P99). A single profile is not a defensible number; the mean of three runs with the same config is.

Connection with the cost and energy track

GenAI-Perf can capture GPU metrics in the same summary CSV if --server-metrics-urls http://localhost:8002/metrics is passed (Triton’s metrics endpoint, which exposes power and utilisation via DCGM). The analyze summary CSV then includes a second table with P99 power (W), energy (MJ), utilisation (%) and memory (GB) per GPU and per scenario.

With that data, a single sweep yields the three axes of the scorecard:

Axis	Source	Metric
Performance	GenAI-Perf	OTT (tok/s), TTFT P99, ITL P99
Energy	DCGM via server-metrics-urls	Power (W) → $J/token = W / OTT$
Cost	node price	EUR/hour → $EUR/token = EUR_{hour} / (OTT \times 3600)$

The formula for cost per million tokens:

$$ CPM = \frac{EUR_{hour}}{OTT \times 3{.}6 \times 10^3} $$

where $OTT$ is the output token throughput in tok/s and $CPM$ is the cost per million tokens in EUR.

It connects with GPU observability with DCGM for the power capture, and with anatomy of an LLM request for the breakdown of phases (prefill versus decode) that TTFT and ITL quantify.

Status 2026: GenAI-Perf → AIPerf

NVIDIA announced in April 2026 that GenAI-Perf moves to passive maintenance mode (no new features) and that the active successor is AIPerf (github.com/ai-dynamo/aiperf). AIPerf adds automatic detection of the saturation point (estimatedCapacity) and is integrated into the NVIDIA Dynamo pipeline.

For new benchmarks in production: migrate to AIPerf. To reproduce historical runs documented with GenAI-Perf, or to use NVIDIA’s NIM benchmarking artefacts (which use GenAI-Perf): stay on GenAI-Perf 25.x.

The article LLM benchmark tools, one by one covers the full comparison including AIPerf. The article GuideLLM: SLO validation under load details the SLO-driven sweep that GenAI-Perf does not do natively. For the general methodology context: LLM benchmarking frameworks, state of the art.

Sources

NVIDIA · GenAI-Perf — README oficial — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/README.html
NVIDIA · Tutorial LLM con GenAI-Perf (comandos y tablas de salida reales) — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/docs/tutorial.html
NVIDIA · GenAI-Perf Analyze subcommand (sweep, CSV resumen, checkpoint) — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/docs/analyze.html
NVIDIA · NIM LLM Benchmarking — Métricas (definiciones TTFT, ITL, TPS, RPS, fórmulas exactas) — https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html
NVIDIA · NIM LLM Benchmarking — Parámetros y buenas prácticas (ISL/OSL, concurrencia vs request-rate, ignore_eos) — https://docs.nvidia.com/nim/benchmarking/llm/latest/parameters.html
NVIDIA Technical Blog · Measuring Generative AI Model Performance Using NVIDIA GenAI-Perf and an OpenAI-Compatible API — https://developer.nvidia.com/blog/measuring-generative-ai-model-performance-using-nvidia-genai-perf-and-an-openai-compatible-api/
GitHub · triton-inference-server/perf_analyzer (genai-perf) — https://github.com/triton-inference-server/perf_analyzer/blob/main/genai-perf/README.md
GitHub · ray-project/llmperf (métricas y diferencias metodológicas) — https://github.com/ray-project/llmperf
PyPI · nvidia-genai-perf-eval — https://pypi.org/project/nvidia-genai-perf-eval/
Macnica · Benchmarking LLM Applications Part 1: What is GenAI-Perf? — https://www.macnica.co.jp/en/business/semiconductor/articles/nvidia/145977/

GuideLLM in depth: validating the SLO under load and sizing from the knee

Sun, 14 Jun 2026 04:30:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Third article of the benchmarking track (B3). B2 covered the tool catalogue; here we go into GuideLLM in depth, because it answers the operational question that sizes a platform: how far can I load this engine without breaking the SLO? It is not a catalogue benchmark (“X tokens/s”), it is an SLO-driven sweep that finds the knee, the safe capacity, and from which the real replica count and cost per token come out. We will look at its load modes, how the SLO is defined, how to read the output and how to turn the knee into sizing and into euros. With real commands; no recommendations, only the mechanics.

Why GuideLLM

GuideLLM (a vLLM project) generates realistic, configurable traffic patterns and captures full distributions of TTFT, ITL and end-to-end behaviour, for SLO-driven evaluation, with reproducible sweeps that identify the safe operating range (Red Hat). Unlike a micro-benchmark, it measures the engine, not the client (multi-process load generation), and unlike MLPerf, it sizes your specific case (your model, your SLO, your load). It is the middle tool: not for tuning a flag, not for comparing vendors, but for deciding how many GPUs you need.

The load modes (rate-types)

GuideLLM offers several modes through --rate-type, and picking the right one is half the work:

`--rate-type`	What it does	When
synchronous	one request at a time	baseline latency, no concurrency
concurrent	keeps N fixed simultaneous requests	measuring at a specific concurrency
poisson	requests per second following Poisson	simulating real traffic (random arrivals)
throughput	saturates the engine to the maximum	raw peak capacity
sweep	automatic ramp from idle to maximum	finding the knee (the most useful one)

In concurrent mode, --rate "1,2,4" launches three benchmarks at those concurrencies (Medium · GuideLLM on OpenShift). The poisson mode is the most realistic for online inference (requests do not arrive at regular intervals), and the sweep is the one that automates the search for the saturation point.

The automatic sweep: from idle to the knee

The flagship mode. When running a sweep profile, GuideLLM automatically increases the request rate from idle to maximum throughput across 10 rounds, and produces an interactive HTML report with detailed latency and throughput (Red Hat · GuideLLM on Kubernetes). During the sweep, on detecting saturation it identifies the previous iteration, the knee, and returns it as estimated capacity; if it does not detect saturation, the sweep has to be extended beyond the knee.

A worked example: reading a 10-round sweep

An illustrative output from a sweep on a 70B on 8×H100 (SLO: TTFT P99 < 500 ms), per round:

Round	Rate (req/s)	TTFT P99 (ms)	ITL P50 (ms)	Throughput (tok/s)	Goodput (tok/s)
1	2	95	19	480	480
3	8	140	20	1,900	1,900
5	14	240	22	3,100	3,060
6	17	460	24	3,400	3,330
7	20	760	31	3,700	2,520
9	26	1,500	48	3,950	900
10	30	2,400	71	4,000	380

How to read it: up to round 6 (17 req/s) the TTFT P99 meets the SLO (460 ms) and goodput ≈ throughput (3,330 ≈ 3,400). In round 7 P99 already breaks (760 ms) and goodput drops to 2,520. The knee sits between 6 and 7: the safe capacity is that of round 6, ~3,330 tok/s of goodput. Rounds 9 and 10 give more raw throughput (3,950, 4,000) but with goodput of 900 and 380; the system is “performing” by mostly serving requests that miss the target. Whoever reports “4,000 tok/s” is describing round 10, where the system is broken. The defensible number is 3,330 tok/s under TTFT P99 < 500 ms, and that is the one that goes into the sizing. Note as well that the sweep had to reach round 10 (P99 of 2,400 ms) to see where it broke: that is why it has to be extended beyond the knee.

Defining the SLO: the number that decides the knee

The “knee” is not absolute: it depends on the SLO you define. A typical online inference SLO:

Metric	Example threshold	What it protects
TTFT P99	< 500 ms	the wait for the first token (interactivity)
TPOT / ITL P95	< 50 ms/token	the perceived “typing speed”
Error rate	< 0.1 %	reliability

Goodput, the throughput that meets that SLO, is the metric that defines the knee: the safe capacity is the maximum load where goodput ≈ throughput. Changing the SLO moves the knee: a stricter TTFT SLO (200 ms) gives an earlier knee (less safe capacity) than a loose one (1 s). That is why the SLO is fixed before the sweep, and reported alongside the result: a capacity “of 3,330 tok/s” without saying under which SLO means nothing.

Execution: the command

An SLO-driven sweep against a vLLM endpoint:

guidellm benchmark \
 --target "http://vllm:8000" \
 --rate-type sweep \
 --max-seconds 120 \
 --data "prompt_tokens=1024,output_tokens=256" \
 --output-path resultados.json

Key parameters: --rate-type sweep (the automatic ramp), --data (the length distribution of the load, which must resemble your real traffic), --max-seconds (duration per round), --target (the endpoint). For the SLO, GuideLLM allows the latency constraints that define goodput to be set. The output goes to --output-path in JSON, YAML or CSV, in addition to the interactive HTML report.

Defining the load: `--data` decides the result

The --data parameter (the distribution of prompt and output lengths) changes the knee as much as the SLO does. Three forms, from least to most faithful:

Load	`--data`	Fidelity
Fixed length	`prompt_tokens=1024,output_tokens=256`	low: real traffic is not fixed
Synthetic distribution	length ranges	medium
Real traces	a dataset of your traffic	high: the knee that applies

The trap: a sweep with short fixed-length prompts gives an optimistic knee that does not resemble production, where long prompts dominate the prefill cost and bring the knee forward. To size properly, feed GuideLLM the real distribution of your traffic (prompt and output lengths measured in production). The knee of a sweep is only as realistic as the load you put into it; with an unrealistic --data, you are sizing for traffic that does not exist.

The arrival profile matters too: --rate-type poisson simulates random arrivals (like real online traffic), while concurrent keeps N fixed (like a batch). An online service measured with concurrent can give a knee different from the real one; for interactive inference, poisson is the honest profile.

The output: what to read

GuideLLM produces standardised, exportable reports for dashboards, analysis and regression tracking, in JSON, YAML and CSV (Red Hat). What matters to read, per sweep round:

TTFT (P50, P95, P99) — the latency to the first token.
ITL/TPOT (P50, P95, P99) — between tokens.
Throughput (req/s and tok/s) — the raw figure.
Goodput — the part that meets the SLO (the honest number).

The interactive HTML report lets you see the full distributions (not just means), which is where the latency tail that a mean hides becomes visible. For sizing and comparison, the JSON is what gets versioned and fed into the reproducible harness.

From the knee to sizing and cost

This is where GuideLLM connects with the rest of the series. The replica count and the cost per token come out of the knee:

The calculation: if the knee gives a goodput of 3,330 tok/s per replica and your peak target load is 20,000 tok/s, you need 6 replicas (20,000 ÷ 3,330 ≈ 6.0). And the cost per token comes from the replica cost (from OpenCost, ~11 €/h) divided by its goodput: ~0.92 €/1M tokens. It is the bridge to capacity planning and to cost per token: the correct denominator is the goodput at the knee, not catalogue throughput. Sizing with raw throughput leaves you short of useful capacity and breaks the SLO in production.

Deployment as a Kubernetes Job

GuideLLM runs as a Kubernetes Job inside the cluster to benchmark models served by the orchestration platform (Red Hat). This matters for fidelity: the load generator runs inside the cluster, on the same network as the endpoint, so it measures the engine without the network latency of an external client. The Job is parameterised with the target, the rate-type and the load, and dumps the JSON to a volume or a store for versioning. Running it as a Job also makes automation easier (a CronJob for periodic benchmarking, or triggered from CI).

SLO validation in CI: catching regressions

The most valuable use in the medium term: a Job that runs a short sweep on every change (a vLLM release, a config change) and compares the goodput against the baseline. If goodput drops by more than a threshold or the knee moves earlier, the pipeline fails. Since GuideLLM exports standardised JSON for regression tracking, comparing two runs is trivial. That way a performance regression, which is a cost and capacity regression, is caught at the commit, not in production. The full sweep is not needed on every commit: a short sweep covering the knee is enough to catch the regression; the exhaustive one is for releases.

Comparing configurations with sweeps

GuideLLM shines for deciding between configurations by running the same sweep against each one. The fair protocol: same load (--data), same SLO, same hardware, varying only the config. Examples of decisions that a sweep settles with data:

Decision	What to compare	What the knee reveals
FP16 vs FP8	two deployments of the same model	FP8 usually gives a higher knee (more goodput)
`max-num-seqs`	different values of the flag	the one that maximises goodput under the SLO
vLLM vs SGLang	two engines, same model	which engine gives more goodput at your SLO
KV size/precision	KV cache configs	the effect on capacity

Each sweep gives a knee; the highest knee under the same SLO wins. It is the controlled experiment that fills the scorecard row (article B8): not “X is faster” in the abstract, but “X gives Y tok/s more goodput under this specific SLO, which translates into Z fewer replicas and W € less per million tokens”.

Watching the three axes during the sweep

A trick that connects with the cost and energy track: while the sweep runs, capture the GPU metrics with DCGM. That way each round yields the goodput (GuideLLM), the average power (DCGM → J/token) and the cost (node price → €/token) at once. In a single sweep campaign you measure the three axes of the scorecard for every operating point, and they come out coherent by construction (same instant, same load). Instead of an isolated performance benchmark, you get the complete row, cost, performance, energy, for the knee, which is exactly what the proposal needs. Align the time windows (the DCGM power and the GuideLLM metrics must cover the same interval) or the J/token does not correspond to the measured goodput.

Interpreting the distributions, not just the knee

The knee is the headline, but the GuideLLM HTML report gives full distributions, and there is information there that the knee summarises away. Three additional readings:

The shape of the tail. Two configs with the same P99 can have different tails: one with P99 at 500 ms and P99.9 at 600 ms is stable; another with P99 at 500 ms and P99.9 at 3,000 ms has a long tail that will hit some users severely. The mean, and even P99, hide it; the distribution shows it.
The TTFT/ITL split. Seeing the two distributions separately tells you whether the bottleneck is prefill (high TTFT) or decode (high ITL), which steers the optimisation (more concurrency vs chunked prefill, and so on).
The spread across rounds. If goodput varies a lot between rounds at the same rate, the system is unstable under load, something a single capacity number does not reveal.

The knee sizes; the distributions diagnose. For a proposal, the knee is the datum; for operating and optimising, the distributions are where you see what to fix.

GuideLLM in the reproducible harness

GuideLLM fits as the load engine of the reproducible harness (article S4). The pattern:

A script/Job that deploys the engine with the pinned config, warms up, runs the sweep and dumps the JSON with all the metadata (model, version, hardware, --data, SLO).
Named by date and config to version the runs.
A store (a git repo of JSONs or a bucket) to reproduce and compare.
DCGM capturing in parallel to add the energy to each point.

That way “reproduce the knee of this config” is one command, not an afternoon, and comparing two releases is a JSON diff. It is what turns benchmarking from a one-off activity into a continuous capability, and what lets the capacity figure in the proposal come with the bench to reproduce it.

The cost of a sweep (in euros)

A 10-round sweep at ~2 minutes per round occupies the node for ~20–30 minutes. At an amortised cost of ~11 €/h, that is ~4–5.5 € per full sweep of a model. Not much per run, but a continuous benchmarking programme (every release, every config, several models) adds up; hence the value of automating the sweep as a reproducible Job and running short sweeps in CI (only around the knee), reserving the exhaustive one for releases. The cost of measuring is part of the cost of the platform, and small next to the cost of sizing badly.

The SLO by use case: the knee moves with it

Since the SLO defines the knee, different use cases give different capacities on the same hardware. Worth keeping in mind when sizing:

Use case	Typical SLO	Effect on the knee
Interactive chat	TTFT P99 < 500 ms, ITL < 50 ms	early knee (latency rules)
Code copilot	TTFT P99 < 300 ms	even earlier knee
Batch / summarisation	no TTFT SLO, maximise throughput	late knee (almost pure throughput)
Agent (multi-step)	bounded per-step latency	depends on the number of steps

The same GPU serves more batch load than interactive chat load, because the batch SLO is loose. That is why sizing requires one sweep per load profile: there is no “the node’s capacity”, there is “the node’s capacity for this SLO”. Mixing loads with different SLOs in the same pool without separating them is a classic source of broken SLOs, with batch saturating and interactive chat paying for it. The fix connects with scheduling: separate pools or priorities by SLO.

GuideLLM vs the others: when to use which

To place it against the tools from B2:

Question	Tool	Why
“How far can I load without breaking the SLO?”	GuideLLM	SLO-driven sweep, the knee
“What is the endpoint’s maximum capacity?”	AIPerf	automatic saturation detection
“Does my vLLM flag change help?”	vllm bench serve	fast, native to the engine
“Which hardware/engine is better in the abstract?”	read MLPerf	cross-vendor comparability

GuideLLM is the tool for sizing under an SLO: when the question is operational (how many replicas, what cost per token at my SLO), it is the most direct answer. AIPerf and GuideLLM overlap (both multi-process); the practical difference is that GuideLLM is more oriented to the SLO and the reproducible report, and AIPerf to maximum capacity with automatic detection. Many teams use both; what must not be done is to compare a result from one with a result from the other as if they were the same measurement.

Checklist for a defensible sweep

For the knee of a sweep to support a sizing decision:

Step	Check
Realistic load	`--data` with your traffic’s distribution (or poisson)
Declared SLO	TTFT/TPOT with a fixed percentile and threshold
Extended sweep	reaches the point where P99 breaks (past the knee)
Goodput, not throughput	capacity is the goodput under the SLO
Job in the cluster	no network latency from an external client
Versioned output	JSON with all the metadata
DCGM in parallel	energy per point, optional but recommended

If you can tick the seven boxes, the knee is an auditable datum; if any is missing, it is an anecdote. The sizing in the proposal hangs on these seven being green.

Limits and traps (data-driven)

Not extending the sweep beyond the knee. If saturation is not detected, the estimated capacity is the last round tested, not the real knee. Extend it until P99 breaks.
Unrealistic load. A fixed-length --data does not resemble your traffic; use the real distribution (or poisson) for a knee that applies.
Reporting throughput, not goodput. The knee is defined by goodput under the SLO; raw throughput overestimates capacity.
Undeclared SLO. A capacity without the SLO it was measured under is neither comparable nor defensible.
Client outside the cluster. Running GuideLLM from outside puts network latency into the measurement; run it as a Job inside the cluster.

With GuideLLM mastered, you have the tool that turns performance into an actionable datum: the knee, the sizing and the cost per token, all from a reproducible sweep. The next article (B4) goes into NVIDIA’s AIPerf; this one closes SLO validation, which is what sizes the proposal.

Closing

GuideLLM answers the only performance question that sizes a platform: how far can I load without breaking the SLO? And it answers it with the right discipline: a sweep from idle to saturation that finds the knee, defined by goodput under a declared SLO, over a load that resembles the real one. Everything the proposal needs comes out of that knee: the replica count, the cost per token (with goodput as the denominator, not catalogue throughput) and, if you capture DCGM during the sweep, the energy per token as well. The mistake that invalidates the exercise is the usual one: reporting peak throughput instead of goodput, measuring with an unrealistic load, or not extending the sweep until you see where it breaks. Done properly, as a Job inside the cluster, with the real distribution, the SLO fixed and the output in versioned JSON, GuideLLM turns performance from a marketing figure into a reproducible datum that survives an audit and sizes a sovereign architecture with numbers. The knee is not the fastest point; it is the point where your platform still delivers what it promised.

Sources

Red Hat · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
Red Hat · desplegar y benchmarkear vLLM con GuideLLM en Kubernetes — https://developers.redhat.com/articles/2025/12/24/how-deploy-and-benchmark-vllm-guidellm-kubernetes
GuideLLM · PyPI — https://pypi.org/project/guidellm/
GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
Medium · GuideLLM en OpenShift (rate-types, ejemplo) — https://medium.com/@jajodia.nirjhar/exploring-guidellm-benchmarking-a-live-llm-on-openshift-ccc2d0841794

LLM benchmark tool catalogue: the practical deep dive

Sun, 14 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

What this article covers

Second article in the benchmarking track (B2). The B1 introduction pinned the metrics and the methodology; this article is the practical manual: for each tool, what it measures, how it is actually invoked (with the command), what output format it produces and when to choose it. The goal is that, after reading it, you know what to run to measure your engine and you can reproduce the number. No universal recommendations; only the mechanics of each tool and the criterion for choosing.

How to choose: the tool map

Recalling the split from B1, the tools are ordered on two axes: micro-bench (single-process, for tuning an engine) versus load generator (multi-process, for measuring real capacity), and engine-native versus endpoint-agnostic.

Datasets: the synthetic load matters as much as the tool

Before the profiles, one point that changes results without anyone noticing: what load you feed it. The tools accept different kinds of dataset, and each one measures something different:

Dataset	What it simulates	Bias it introduces
random (fixed lengths)	controlled uniform load	unrealistic: traffic is not fixed length
sharegpt (real conversations)	realistic prompt distribution	the de facto standard for comparison
your own traces	your real traffic	the most faithful, but specific to your case

The trap: a benchmark with short, fixed-length prompts gives a very high throughput that looks nothing like production, where lengths vary and long prompts dominate the prefill cost. For a defensible figure, use sharegpt (comparability) or, better, traces from your own traffic (fidelity). And always declare the length distribution (prompt/output) alongside the number, because two benchmarks with different datasets are not comparable even if they use the same tool.

vLLM bench serve

What it measures: TTFT, TPOT, throughput and latencies of the vLLM server under a synthetic load. Class: micro-bench (single-process). It is the tool for tuning vLLM and seeing the effect of its optimisations (decode, prefill).

Typical invocation (vLLM · benchmark CLI):

vllm bench serve \
 --backend vllm \
 --model meta-llama/Llama-3.1-8B-Instruct \
 --endpoint /v1/completions \
 --dataset-name sharegpt \
 --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
 --num-prompts 1000

Key parameters: --num-prompts (load), --dataset-name (sharegpt, random, etc.), --request-rate (requests/s). The output is a console summary with TTFT, TPOT, throughput and percentiles, and it can be dumped to JSON. To sweep concurrencies there is vllm bench sweep serve, which automates the sweep (vLLM · sweep).

Limit: single-process, it saturates on the client at high concurrency; less flexible than GuideLLM in datasets and load patterns. Use it to iterate quickly on the vLLM config, not to measure maximum capacity at scale.

SGLang bench

What it measures: the equivalent for the SGLang engine (TTFT, TPOT, throughput). Class: micro-bench. Use: tuning SGLang and comparing it against itself across configurations. The mechanics are analogous to vllm bench serve: a synthetic load, console/JSON output with the same metrics. If you are evaluating SGLang against vLLM, do not compare them with their respective micro-benches (same class but different implementations): use an endpoint-agnostic load generator (GuideLLM/AIPerf) against both endpoints, so that the tool is not the variable.

AIPerf (NVIDIA, formerly genai-perf)

What it measures: TTFT, ITL, throughput and latency against any compatible endpoint (vLLM, NIM, TGI, SGLang). Class: multi-process load generator. It is the successor to genai-perf (retired on 15 Apr 2026). Its distinguishing feature: during the sweep it detects GPU saturation and returns the previous iteration as estimatedCapacity, which is why the sweep has to extend beyond the knee (AIPerf).

Output: structured metrics (JSON) with TTFT/ITL distributions and the estimatedCapacity. Use it when you want the real capacity of an endpoint, whatever the engine, with automatic detection of the saturation point.

GuideLLM (vLLM project)

What it measures: full distributions of TTFT, ITL and end-to-end behaviour, for SLO-driven evaluation. Class: multi-process load generator. It is the recommended tool for benchmarking vLLM servers in production: more flexible than vllm bench serve in dataset loading, request format and traffic patterns, with live progress and automatic report generation (Red Hat).

Typical invocation:

guidellm benchmark \
 --target "http://localhost:8000" \
 --rate-type throughput \
 --max-requests 1000 \
 --data "samples=1000,prompt_tokens=1024,output_tokens=256"

Key parameters: --rate-type (synchronous, concurrent, throughput, or rate-based), --data (the load specification: number of samples and prompt/output lengths), --target (the endpoint). It generates reproducible sweeps to find the safe operating range under SLO, with full distributions (not just means). Output: a report with percentiles and, often, exportable to JSON/HTML. It is the default option for answering “how hard can I load this engine without breaking the SLO?”.

LLMPerf (Anyscale/Ray)

What it measures: throughput and latency at the inference level. Class: load generator. Use: endpoint validation, historically very widespread in the Ray/Anyscale ecosystem. It is a solid and well-known option, though less focused on distributions and sweeps than GuideLLM/AIPerf. It fits if you already operate on Ray or want a simple tool to validate an endpoint.

inference-benchmarker (Hugging Face)

What it measures: latency and throughput of inference endpoints, aimed at producing comparable reports. Class: load generator. Use: an OSS alternative inside the Hugging Face ecosystem, useful if you already work with TGI or the HF stack. Like the others in its class, its value lies in measuring real capacity with distributed load; the choice between this one, GuideLLM and AIPerf usually comes down to the ecosystem you already operate in rather than to deep differences in what they measure.

GuideLLM versus AIPerf: which of the two load generators

They are the two serious multi-process load options, and they overlap a great deal. The practical differences that tip the choice:

Criterion	GuideLLM	AIPerf
Origin	vLLM project (Red Hat)	NVIDIA (successor to genai-perf)
Focus	SLO-driven evaluation, reproducible sweeps	real capacity, automatic estimatedCapacity
Load patterns	synchronous, concurrent, rate-based	sweep with saturation detection
Report	live progress + automatic report	structured metrics (JSON)
Ecosystem	vLLM / OpenShift AI	NVIDIA NIM / Triton / vLLM

In practice: if your question is “how hard can I load it without breaking the SLO?”, GuideLLM answers it more directly with its SLO-driven sweeps; if your question is “what is the maximum capacity of this endpoint?”, AIPerf gives it with its automatic knee detection. Many teams use both: GuideLLM for the operational SLO, AIPerf for the reference capacity. What you must not do is compare a GuideLLM result with an AIPerf one as if they were the same measurement: although both are multi-process, their sweep methodology differs; pick one for a given comparison and stick to it.

MLPerf Inference (MLCommons)

What it measures: performance under standardised scenarios (Offline, Server, Interactive) with strict rules. Unlike the previous ones, MLPerf is not “run” for your day-to-day case: it is a vendor comparison suite whose results are read (published by NVIDIA, AMD, Intel and others). Use it to compare hardware and engines against each other under identical rules, not to size your specific workload. For that, GuideLLM/AIPerf against your endpoint.

Practical comparison table

Tool	Class	Base command	Output	When
vllm bench serve	micro	`vllm bench serve`	console/JSON	tune vLLM
vllm bench sweep	micro (sweep)	`vllm bench sweep serve`	console/JSON	sweep concurrency in vLLM
SGLang bench	micro	bench from the SGLang repo	console/JSON	tune SGLang
AIPerf	load	`aiperf profile …`	JSON + estimatedCapacity	real capacity, multi-endpoint
GuideLLM	load	`guidellm benchmark …`	report + JSON/HTML	SLO, reproducible sweep
LLMPerf	load	Ray/LLMPerf script	JSON	validate endpoint (Ray)
MLPerf	suite	(results are read)	official results	compare vendors

Summary: what you run for each question

So as not to get lost in the catalogue, the direct mapping from question to tool:

Your question	Tool	Why
“Does my vLLM config change help?”	`vllm bench serve` (+ sweep)	fast, engine-native
“How hard can I load without breaking the SLO?”	GuideLLM	SLO-driven sweeps
“What is the maximum capacity of the endpoint?”	AIPerf	automatic knee detection
“vLLM or SGLang for my workload?”	GuideLLM/AIPerf against both	same tool, engine as the variable
“Which hardware/engine is better in the abstract?”	read MLPerf	cross-vendor comparability
“Is there a regression in this release?”	short sweep in CI against a baseline	continuous detection

The rule underlying the whole table: micro-bench to iterate on an engine, load generator to measure capacity and decide, MLPerf to compare vendors. And, for any comparison between systems, the same tool for every candidate. If these three categories are clear to you, the specific choice is secondary.

Step-by-step benchmark methodology

A reliable run is not “launching the command”: it is a procedure. The steps, with the load tool (GuideLLM/AIPerf) against your endpoint:

Deploy the engine with the exact config to be measured (model, precision, flags), pinned.
Warm up: send a few requests and discard them, so that the prefix cache and the autotuning do not inflate the first numbers.
Sweep: sweep rising concurrencies (1, 8, 16, 24, 32…) beyond the knee, to see where latency takes off.
Collect the output in JSON with percentiles (TTFT/ITL/throughput/goodput) and the metadata.
Compare: read the goodput under the SLO, not the maximum throughput.

Skipping step 2 (warm-up) or not going past the knee in step 3 are the two mistakes that bias the result the most.

Comparing two engines: the fair protocol

If the goal is to choose between vLLM and SGLang (or TRT-LLM), the protocol that avoids false conclusions:

Same load tool (GuideLLM or AIPerf) against both endpoints, never each engine’s own micro-bench.
Same dataset and length distribution.
Same hardware and precision.
Same SLO to compute the goodput of both.
Vary only the engine; everything else fixed.

Only then is the difference you measure down to the engine and not to the tool, the dataset or the hardware. It is the controlled experiment that supports the scorecard row (article B8).

Output format and comparability

What you record alongside the number is what makes it comparable. A useful output includes, in JSON so it can be versioned:

{
 "tool": "guidellm", "version": "x.y.z",
 "model": "Llama-3.1-70B", "precision": "FP16",
 "hardware": "8xH100 SXM NVLink",
 "load": {"prompt_tokens": 1024, "output_tokens": 256, "concurrency": 16},
 "results": {"ttft_p50_ms": 180, "ttft_p99_ms": 460,
 "itl_p50_ms": 22, "throughput_tok_s": 3400, "goodput_tok_s": 3330}
}

Saving this for every run turns a benchmark into an auditable datum: anyone can reproduce the figure with the same tool, version, model, hardware and load. It is the material of the reproducible harness (article S4), and the difference between a defensible number and a console screenshot.

Worked example: reading the output of a sweep

An illustrative output from a GuideLLM sweep on a 70B on 8×H100 (SLO: P99 TTFT < 500 ms), just as you would read it:

Concurrency	TTFT P99 (ms)	ITL P50 (ms)	Throughput (tok/s)	Goodput (tok/s)
8	240	20	2,100	2,100
16	460	22	3,400	3,330
24	980	31	3,900	2,420
32	1,800	54	4,000	800

How to read it: the knee sits between 16 and 24. At concurrency 16, P99 (460 ms) meets the SLO and goodput (3,330 tok/s) ≈ throughput. At 24, throughput rises little (3,400 → 3,900) but P99 already violates the SLO and goodput falls to 2,420. At 32, throughput is at its maximum (4,000) but goodput collapses to 800: the system “performs well” while serving requests that do not comply. The defensible capacity is the one at concurrency 16 (3,330 useful tok/s), and that is the number that goes into capacity planning and into cost per token. Anyone reporting “4,000 tok/s” is describing the point where the system no longer meets its SLO.

Automating the harness

A manual run does not scale to a benchmarking programme. The minimum automation:

An idempotent script that deploys the engine, warms it up, runs the sweep and saves the JSON with all the metadata (model, version, hardware, dataset, SLO).
Naming by date and config, to version the runs and compare over time.
A results store (a git repo of JSONs, or a bucket) so that anyone can reproduce and compare.

The goal is that reproducing a number is one command, not an afternoon. It is the basis of the harness in article S4, and what turns benchmarking from a one-off activity into a continuous platform capability.

CI integration: continuous benchmarking

The next level is measuring on every change: a CI job that, when the engine or the config is updated, launches a short sweep against a test environment and compares against the baseline. If goodput falls by more than a threshold, the pipeline fails. That way a performance regression is caught at the commit, not in production. Watch out for two things: benchmarking in CI consumes GPU-hours (budget for it) and it needs a stable environment (same hardware) for the comparison to be valid. You do not need the full sweep on every commit: a short sweep covering the knee is enough to detect regressions; the exhaustive sweep is for releases.

The cost of measuring (in euros)

Benchmarking consumes GPU-hours, and that has a cost worth budgeting for. A serious sweep on an 8×H100 node can occupy the cards for a couple of hours; at an amortised cost of ~11 €/h, that is ~22 € per full sweep, more if you sweep several models and precisions. It is not much per run, but a continuous benchmarking programme (every release, every config change) adds up. The practical rule: automate the harness so that each run is cheap and reproducible, and measure what you are going to use to decide, not for the sake of exhaustiveness. The cost of measuring is part of the cost of the platform, small next to the cost of serving but real.

Observing during the benchmark: one run, three axes

A trick that saves work and connects the series: while the sweep runs, capture the GPU metrics too. With DCGM exporting to Prometheus during the run, you record at the same time:

Axis	Source during the sweep	Metric
Performance	the tool (GuideLLM/AIPerf)	TTFT, ITL, throughput, goodput
Energy	DCGM	power (W) → J/token
Cost	node price (OpenCost)	€/hour → CPM per point

That way, from a single run you get the three numbers for the same operating point: at concurrency 16, the goodput (3,330 tok/s), the average power (from DCGM, which divided by the throughput gives J/token) and the cost per token (with the node price). Instead of three separate campaigns, you measure the three axes at once, and they end up coherent by construction because they correspond to the same instant and the same load. It is what the harness in article S4 does, and the reason to export DCGM during the benchmark even if you are only after performance: energy and cost come almost free if you capture them in the same window.

The methodological warning: align the time windows. The DCGM power figures and the tool’s metrics have to cover exactly the same interval (no warm-up, no shutdown), or the J/token does not correspond to the throughput measured. Same window, same three numbers.

The benchmark is also FinOps and energy

An idea that connects this article with the rest of the series: measuring performance is, in fact, measuring cost and energy. By the identity of the opening article, goodput is the denominator of cost per token and of energy per token. When a sweep reveals that config A gives 3,330 tok/s of goodput and config B gives 4,000, you are not only measuring speed: you are measuring that B costs fewer euros and fewer watts per token. That is why the output JSON of a benchmark should be accompanied by the iron cost (from OpenCost) to compute the real CPM of every point in the sweep: throughput × node price = cost per token. Benchmarking is not an isolated axis; it is the tool that, indirectly, moves the cost and the energy of the platform the most, and the one that fills the performance column of the scorecard with numbers that translate directly into euros.

The operational consequence: do not benchmark performance in a vacuum. Every run you save with its throughput and its goodput should be crossable with the node cost (€/hour) and the energy (J/token) to give the three faces of the same operating point. That is how the benchmarking track feeds the FinOps and energy tracks instead of living apart.

State of the art 2026 and limits

Migration to multi-process: AIPerf (formerly genai-perf) and GuideLLM consolidate the measurement of real capacity; micro-benches are left for tuning engines.
GuideLLM as the OSS standard for SLO-driven evaluation with an automatic report.
Careful with comparing across classes: a micro-bench and a load generator are not comparable; pin the class and the tool.
Version and dataset matter: the same command with a different dataset or version gives a different number; pin them.
MLPerf does not size your case: it compares vendors, it does not replace a sweep against your workload.

With the practical catalogue covered, the next article in the track (B3) goes into GuideLLM and SLO validation under load in depth. The tool is the means; the reproducible datum is the end.

Mistakes that invalidate a benchmark

To finish, the list of what turns a run into rubbish, by frequency:

Mistake	Effect	Fix
Comparing different classes	up to 7× of spurious difference	same tool for everyone
Not discarding the warm-up	artificially low TTFT	warm up and discard
Not going past the knee	you do not know the safe capacity	extend the sweep
Unrealistic dataset (fixed length)	throughput that does not apply	sharegpt or your own traces
Reporting the mean, not P99	hides the tail	percentiles always
Tokenizer foreign to the model	biased tok/s and cost per token	count with the model’s own
Not pinning version/config	irreproducible	save everything in the JSON

Any one of these is enough for the number not to be defensible. A benchmark is only as good as its methodology: the tool matters less than running it properly and recording everything.

Closing

The catalogue of benchmark tools boils down to a simple decision (micro-bench to tune an engine, load generator to measure capacity, MLPerf to compare vendors) and to a discipline that weighs more than the choice: the method. The same guidellm benchmark gives a golden datum or a useless one depending on the dataset, the warm-up, how far the sweep goes and what you record in the output. For a sovereign architecture proposal, performance only counts if it comes with its command, its version, its load and its goodput under SLO, and, crossed with cost in euros and energy per token, it becomes the scorecard column that decides which engine and which configuration hold up the platform. You choose the tool in five minutes; the methodology is what makes the number survive an audit.

Sources

vLLM · CLI de benchmark (bench serve) — https://docs.vllm.ai/en/latest/cli/bench/serve/
vLLM · bench sweep serve — https://docs.vllm.ai/en/latest/cli/bench/sweep/serve/
Red Hat · desplegar y benchmarkear vLLM con GuideLLM en Kubernetes — https://developers.redhat.com/articles/2025/12/24/how-deploy-and-benchmark-vllm-guidellm-kubernetes
GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
NVIDIA AIPerf · guía de benchmarking — https://lucaberton.com/blog/nvidia-aiperf-llm-inference-benchmarking-guide/
Medium · Benchmarking LLM Serving Performance (guía) — https://medium.com/@kimdoil1211/benchmarking-llm-serving-performance-a-comprehensive-guide-db94b1bfe8cf

Cost, performance and energy: the three axes of LLM inference and the identity that ties them together

Sat, 13 Jun 2026 03:30:00 +0200

Notation: amounts in euros (N €), decimals with a point. Cost and energy references are European (France, Germany, Spain), because the aim is to defend a sovereign European architecture; when a source quotes prices in dollars it is flagged as “USD”. The dollar symbol is not used (on this site it is a formula delimiter).

What this series is

This is the first instalment of a series of 28 data articles (not opinion pieces) about the three dimensions that decide whether a sovereign European LLM platform is sustainable: cost (FinOps), performance (benchmarking) and energy. The aim is to have a body of verifiable figures, with their methodology, to support architecture decisions with numbers rather than intuition. This first article sets the frame: what each axis measures, how they link to one another, an end-to-end worked example with European data and the 2026 references that will be used as a base.

The quantitative thesis, in one sentence: the three axes are not independent. Throughput (tokens per second) is the common denominator of cost per token and energy per token, and energy is, on top of that, between 30 % and 50 % of the cost itself. Optimising one moves the other two. And in Europe, where the cluster sits (France, Germany, Spain) changes both the electricity cost and the carbon per token at once.

The three axes and their metrics

Axis	Question	Key metrics	Unit	Tool family
Cost (FinOps)	How much does each unit of work cost?	cost/1M tokens (CPM), cost/request, €/GPU-hour, utilisation	€/1M tok, €/h	OpenCost, Kubecost, CloudZero
Performance	How much work per second and at what latency?	TTFT, TPOT/ITL, throughput, goodput, P50/P99	tok/s, ms	GuideLLM, AIPerf, LLMPerf, MLPerf Inference
Energy	How many watts per unit of work?	energy/token, average power, PUE, carbon	J/tok, Wh, kWh, gCO₂	Zeus, CodeCarbon, MLPerf Power, Kepler

Each axis has its own in-depth introductory article in this same batch; here only how they fit together matters. It is worth breaking each one down a little more before crossing them.

Cost axis in detail

Metric	Definition	Why it matters
CPM (cost/1M tokens)	cluster cost ÷ tokens produced	the comparable unit on-prem vs cloud
Cost per request	cost attributed to one complete request	billing by real usage
€/GPU-hour	hourly cost of a GPU (amortised or rented)	the basis of any calculation
Utilisation	fraction of the GPU actually used	the lever that moves everything else
Fixed vs variable cost	amortised capex vs energy/operations	defines the break-even point

Performance axis in detail

Metric	Definition	Dominant phase
TTFT	latency to the first token	prefill
TPOT / ITL	time between output tokens	decode
Throughput	tokens or requests per second of the system	both
Goodput	throughput that meets the SLO (not the raw figure)	the honest number
P95 / P99	latency percentiles (not the mean)	the real experience

Energy axis in detail

Metric	Definition	Unit
Average power	average instantaneous consumption	W
Energy per token	energy per generated token	J/token
Total energy	integral of power over time	Wh, kWh
PUE	datacenter overhead (cooling, losses)	ratio ≥ 1
Carbon	energy × grid intensity	gCO₂eq

The identity that ties the three axes together: throughput

The cost per million tokens (CPM) is the cluster’s cost divided by what it produces:

$$\text{CPM} = \frac{\text{cluster cost per hour}}{\text{throughput (tok/s)} \times 3600 / 10^6}$$

Energy per token has exactly the same form, with power in the numerator:

$$\text{energy per token (J)} = \frac{\text{average power (W)}}{\text{throughput (tok/s)}}$$

The two share the same denominator: throughput. Hence the key operational consequence: every performance improvement that raises tokens/s lowers both cost per token and energy per token at the same time. These are not three separate optimisations; throughput moves them together. That is why performance benchmarking (axis 2) is, at bottom, also a FinOps tool and an energy efficiency tool.

The CPM↔throughput relationship is direct: the higher the throughput, the lower the CPM. It is the identity on which the whole cost analysis of the series is built.

Worked example: unit economics of an 8×H100 node in Europe

To see the identity in action, an end-to-end calculation on an example node (8×H100 SXM, Llama 3.1 70B FP16 served with vLLM), with European prices. Order-of-magnitude figures; the methodology is what matters.

Starting data:

Parameter	Value	Source
European cloud rental (Scaleway, per H100)	2.73 €/h	Scaleway
→ 8×H100 node on-demand	~21.84 €/h	8 × 2.73
Amortised on-prem cost (estimate)	~10–13 €/h	capex 24–36 months + opex
Throughput	2,800 tok/s	reference benchmark
Power per H100 (TDP)	up to 700 W	NVIDIA
Reference PUE	1.4	typical of a datacenter

Cost per million tokens. At 2,800 tok/s, hourly throughput is 2,800 × 3,600 / 10⁶ = 10.08 million tokens/hour. Therefore:

$$\text{CPM (cloud)} = \frac{21{.}84}{10{.}08} \approx 2{.}17 \text{ €/1M tok}$$

With the amortised on-prem cost (~11 €/h) it drops to ~1.09 €/1M tok, the difference between renting from a provider and amortising your own iron.

Energy per token. The power of the 8 cards is 8 × 700 = 5,600 W (board only). Over 2,800 tok/s:

$$\text{energy per token} = \frac{5,600}{2,800} = 2 \text{ J/token (board)}$$

Over a million tokens that is 2 × 10⁶ J = 0.56 kWh (GPU only). Applying PUE 1.4: ~0.78 kWh per million tokens. At French wholesale prices (~0.058 €/kWh) that is ~0.045 € of electricity per million tokens; in Germany (~0.091 €/kWh), ~0.071 €, some 57 % more, because of the grid, not the model.

The link. If a serving optimisation raised throughput from 2,800 to 4,200 tok/s (+50 %), the CPM would fall to ~1.45 €/1M tok and energy per token to ~1.33 J, both at once, through the same denominator. That is the identity, with numbers.

Energy is inside the cost

The second link: energy is not an axis separate from cost, it is a large component of it. 2026 reference data:

Figure	Value	Source
Electricity as a share of TCO (5 years)	30–40 % (>50 % at high density 24/7)	Spheron
Day-ahead wholesale France (Mar-2026)	~58 €/MWh (0.058 €/kWh)	TradingEconomics
Day-ahead wholesale Spain	~54 €/MWh (0.054 €/kWh)	TradingEconomics
Day-ahead wholesale Germany	~91 €/MWh (0.091 €/kWh)	TradingEconomics
Electricity cost/token at 80 % util	¼ of that at 20 % util	Spheron

A methodological warning: the wholesale price (day-ahead) is not what a company pays; industrial retail adds grid charges and taxes (roughly ×1.5–2). But the relative ordering holds: France and Spain, cheap; Germany, expensive.

The third link, utilisation, closes the triangle: power is consumed even when the GPU is idle, so spreading the fixed cost (energy included) over more tokens (more throughput, more occupancy) lowers all three axes at once. The electricity cost per token at 80 % utilisation is a quarter of what it is at 20 %.

Sensitivity analysis: what moves the cost per token

Four levers dominate the CPM, and it is worth knowing their elasticity before defending any figure:

Lever	Change	Effect on CPM	Mechanism
Throughput	+50 % tok/s	−33 % CPM	direct denominator
Utilisation	20 % → 80 %	up to −75 % on the electricity portion	spreads fixed cost
Location (energy)	Germany → France	lowers the electricity portion (~37 % less €/kWh)	wholesale FR < DE
Amortise vs rent	21.84 → 11 €/h	−50 % CPM	removes the provider’s margin
Quantisation (FP8/INT4)	+throughput, −VRAM/KV	lowers CPM and J/token	same identity

The reading: the CPM is not a number, it is a function of utilisation, throughput, location (energy price) and ownership model. Any on-prem vs cloud comparison that does not fix those assumptions is not comparable, and that is exactly the job of the synthesis article.

Simplified TCO model (on-premise)

The real on-premise cost is not the GPU alone: it is amortised capex + opex, over a hardware cycle of 24–36 months. Reference structure of an inference node:

Component	Type	Typical weight	Notes
GPUs	capex	the bulk	amortise over 24–36 months
Server, network, BMC	capex	medium	NICs, switching, out-of-band management
Electricity	opex	30–50 % of TCO	depending on country (FR/ES cheap, DE expensive) and PUE
Cooling	opex	inside the PUE	PUE 1.2–1.6 typical
Maintenance and support	opex	medium	spares, contracts, RMA
Staff and operations	opex	variable	the hidden cost that cloud rental includes

The difference between cloud rental (21.84 €/h in the example) and the real amortised cost (~10–13 €/h) is this breakdown: the provider charges a margin for absorbing capex, cooling and operations. The break-even point appears when sustained volume exceeds ~2M tokens/day, with typical amortisation in 6–12 months (Spheron). Below that threshold, renting usually wins; above it, amortising. Article A8 develops the full model and synthesis S3 connects it with sizing.

Reference: the price of a GPU-hour in European sovereign cloud

The other side of the on-prem comparison is the price of renting the GPU. For a sovereign European platform, what matters is not AWS/GCP/Azure (subject to the US CLOUD Act), but the European providers:

Provider	H100 (€/hour)	Location	Sovereignty
Scaleway	from 2.73	Paris, Warsaw	EU jurisdiction, exempt from US CLOUD Act
OVHcloud	+40–45 % vs Scaleway	France, Germany	EU jurisdiction, exempt from US CLOUD Act
Hyperscalers (reference)	3–7 USD	global	subject to US CLOUD Act

The data: Scaleway offers the H100 from 2.73 €/h in European datacenters (Paris, Warsaw), with models and data under EU jurisdiction, immune to extraterritorial regulation; OVHcloud serves it from France and Germany, ~40–45 % more expensive than Scaleway as of March 2026. Both are exempt from the US CLOUD Act, the foundational sovereignty requirement for data subject to GDPR (Scaleway, DeployBase).

The implication for the proposal: the comparison is not only cost, it is cost + sovereignty. An on-prem cluster of your own or a European cloud keep the data under EU jurisdiction; a US hyperscaler does not, however cheap the GPU-hour comes out. That is the leg that the cost axis on its own does not capture, and that synthesis S2 quantifies.

Energy and carbon reference: France, Germany, Spain

Here the choice of European country changes both the electricity cost and the carbon per token. It is the most important table for a sovereign proposal:

Country	Wholesale (€/MWh, Mar-2026)	Grid carbon (gCO₂/kWh)	Profile
France	~58	~20–60	nuclear: cheap and very clean
Spain	~54	~150–170	renewables + gas: cheap, medium carbon
Germany	~91	~363 (2024 average)	coal/gas + renewables: expensive and dirty

Sources: day-ahead wholesale prices (TradingEconomics), German carbon ~363 gCO₂/kWh (2024 average, Statista via Wikipedia), nuclear ~17–35 gCO₂e/kWh (EEA), Spain ~167 gCO₂/kWh (2020). The PUE multiplies all of that energy: at 1.4, for every kWh of GPU the datacenter spends 1.4 kWh with cooling included.

The calculation, on the example node (0.78 kWh per million tokens, with PUE): in France (~40 gCO₂/kWh) it is ~31 gCO₂ per million tokens; in Germany (~363 gCO₂/kWh), ~283 gCO₂ per million, a factor of ~9×. That is, the same load served in France emits of the order of 9× less carbon per token than in Germany, and costs ~37 % less in electricity, without touching a single line of the stack, purely because of the country’s grid. Siting the cluster in France or Spain is at once a cost lever, a carbon lever and a compliance lever. Hourly grid intensity is obtained from ElectricityMaps, the reference source for these calculations.

Scale context: global datacenters consumed ~460 TWh in 2025 (≈1.8 % of the world’s electricity), and the IEA projects passing 800 TWh in 2028, which is why energy efficiency per token stops being a detail and becomes a design constraint, and why carbon enters the European regulatory equation.

Sovereignty and regulation: the axis that does not appear in the CPM

For a European platform there is a fourth factor that the three numerical axes do not capture, and that can invalidate the lowest cost figure: jurisdiction. Three fronts that turn “the cheapest GPU-hour” into “the cheapest GPU-hour that complies”:

Front	What it requires	Architectural implication
GDPR + US CLOUD Act	personal data under a jurisdiction accessible to US authorities is exposed	European cloud (Scaleway/OVHcloud) or on-prem keep the data in the EU
EU AI Act	obligations for high-risk systems (traceability, control, documentation)	on-prem control makes auditability and compliance easier
CSRD / emissions reporting	reporting the carbon footprint (scope 2)	a low-carbon grid (France) improves the reported figure

The key point for the proposal: a US hyperscaler can offer a competitive GPU-hour, but it puts the data under the US CLOUD Act, which for data subject to GDPR is a compliance risk, not a saving. That is why this series’ comparison is not made in USD against US providers: it is made in euros, against options that keep data sovereignty, on-prem of your own or European cloud. Cost, energy and carbon are measurable axes; sovereignty is the constraint that decides which options even enter the table. This links with the mapping of the EU AI Act onto the on-premise architecture and with the technical controls ENS × ISO 42001 × EU AI Act.

The operational conclusion: France and Spain not only come out cheaper and cleaner than Germany; being EU jurisdiction, they meet the sovereignty constraint that a US hyperscaler does not meet at any price. Three advantages that reinforce each other, and that the proposal can defend with the same table.

Methodology: how to read (and not fool yourself with) these figures

Five warnings that apply to all the data in this series:

Board vs node. The GPU’s power (700 W per H100) is the floor; the complete node (CPU, NICs, power supplies, fans) consumes more. Always compare the same boundary.
Wholesale vs retail. The day-ahead price is not what the company pays; industrial retail adds grid charges and taxes. Use whichever matches your contract.
Measured vs estimated. A wattmeter figure and an nvidia-smi estimate are not the same thing; the difference is quantified in the energy articles.
Amortisation assumptions. The on-prem cost/hour depends on the hardware cycle (24 vs 36 months) and on the assumed utilisation; changing those assumptions moves the CPM more than any serving optimisation.
Comparability. Two CPM or J/token figures are only comparable if model, precision, context length, load and country match. Most public comparisons (and in USD) do not fix those assumptions, which is why the series reproduces the ones that matter with its own harness and European data.

The combined scorecard

The destination of the series is a single board where each model/configuration/location is scored on all the axes at once, to decide on a Pareto frontier rather than on a single number. An illustrative example (order-of-magnitude figures) on the same 8×H100 node:

Candidate	CPM (€/1M)	Goodput (tok/s)	Energy (J/tok)	Carbon (gCO₂/1M)	Sovereignty
A · 70B FP16, on-prem France	~1.09	~2,800	~2.0	low (grid ~40 gCO₂/kWh)	EU
B · 70B FP8, on-prem France	~0.75	~4,000	~1.4	lower	EU
C · Scaleway on-demand	~2.17	~2,800	n/a	low (FR)	EU
D · US hyperscaler	~3.0	~2,800	n/a	depends on region	not EU

The reading: B dominates A on the three intrinsic axes thanks to FP8. C (European cloud) is more expensive per token but shifts capex and operations, while keeping sovereignty. D (hyperscaler) may be competitive on cost but loses sovereignty, an axis that for GDPR data is not negotiable. No isolated number decides: the choice comes from seeing the whole row.

The trap the board avoids: optimising a single axis. Each cell will be filled in by an article in the series with measured data.

Where each axis is measured: from design to production

Each axis is measured at three different moments, with different tools and increasing precision. Confusing them is a common source of figures that do not add up:

Axis	Design (estimate)	Benchmark (controlled)	Production (continuous)
Cost	datasheet, GPU-hour price	CPM measured over synthetic load	OpenCost/Kubecost per team and token
Performance	theoretical throughput (roofline)	GuideLLM/AIPerf with SLO	OTel/Langfuse over real traffic
Energy	TDP × hours	Zeus/MLPerf Power on the bench	Kepler/DCGM per pod in the cluster

The rule: a design figure serves for sizing, but not for defending a proposal; for that you need a reproducible benchmark figure, and to operate, a continuous production one. When someone disputes a number in the proposal, the answer is not “a blog says so”, but “here is the bench, here is the methodology, reproduce it”.

Glossary of units and conversions

Quantity	Unit	Conversion / reference
Energy	joule (J), watt-hour (Wh)	1 Wh = 3,600 J; 1 kWh = 3.6 MJ = 3,600,000 J
Power	watt (W)	1 W = 1 J/s; H100 ≈ 700 W TDP
Energy per token	J/token	= average power (W) ÷ throughput (tok/s)
Throughput	tok/s	tokens generated per second by the system
Cost per token	€/1M tok (CPM)	= cost/hour ÷ (tok/s × 3,600 / 10⁶)
Latency	ms	TTFT (first token), TPOT/ITL (between tokens)
PUE	ratio (≥1)	total DC energy ÷ compute energy
Carbon	gCO₂eq	= energy (kWh) × PUE × grid intensity (gCO₂/kWh)

A useful conversion to keep in your head: at a constant 700 W, an H100 consumes ~0.7 kWh per hour, ~16.8 kWh a day and ~6,130 kWh a year (board only, no PUE). Multiplied by the PUE and your country’s energy price (FR ~0.058, ES ~0.054, DE ~0.091 €/kWh wholesale), that number is the basis of the electricity portion of the TCO.

How the rest of the series fits together

The 28 articles group into four tracks; this is the map:

Track	What it contributes	Closes with
FinOps (8)	measuring and allocating cost, TCO model	cost/token comparable on-prem vs European cloud
LLM benchmarking (8)	measuring performance reproducibly	Pareto frontier of engines
Energy (8)	measuring watts and carbon per token	energy in the TCO and in European regulation
Synthesis (4)	crossing the three axes	scorecard, on-prem vs cloud, sizing, harness

The three articles that follow this one in the batch introduce each axis in depth: their frameworks and the state of the art. From there, each article adds a layer of data to the final scorecard.

The thread running through all 28 instalments is the one in this article: cost, performance and energy are a single problem with three faces, tied together by throughput and by the energy that lives inside the cost, and in Europe, by the country where the cluster sits. Anyone who optimises one face while ignoring the others will end up with a pretty number and a platform that does not hold up. The work of the series is to produce the figures, measured, dated, European and reproducible, that allow a sovereign architecture to be defended before any committee with the only answer that closes a technical debate: the data and the bench to reproduce it.

Sources

Scaleway · H100 GPU instance (€/h price, EU sovereignty) — https://www.scaleway.com/en/h100/
DeployBase · OVHcloud GPU Pricing (European sovereignty) — https://deploybase.ai/articles/ovhcloud-gpu-pricing
TradingEconomics · electricity price France / Germany / Spain — https://tradingeconomics.com/france/electricity-price
EEA · emission intensity of electricity in Europe — https://www.eea.europa.eu/en/analysis/indicators/greenhouse-gas-emission-intensity-of-1
Electricity sector in Germany (carbon ~363 gCO₂/kWh) — https://en.wikipedia.org/wiki/Electricity_sector_in_Germany
Spheron · AI Inference Power & Cost Economics (2026) — https://www.spheron.network/blog/ai-inference-cost-economics-2026/
NVIDIA · H100 (TDP/specs) — https://www.nvidia.com/en-us/data-center/h100/

LLM inference benchmarking: frameworks, metrics and the state of the art (tool by tool)

Sat, 13 Jun 2026 02:30:00 +0200

Notation: amounts in euros (N €), decimals with a point. Performance is not very sensitive to the country, but its associated cost (cost per token) is expressed in € and links back to the opening article.

What this introduction covers

Third article in the data series, a deep dive into the performance axis. Measuring the performance of an inference engine looks trivial (“how many tokens per second?”) and it is exactly where people fool themselves most: two tools can report results that differ by a factor of 7 for the same system. This article inventories the metrics that matter and how they are defined, why the architecture of the tool biases the figure, how the saturation point is found with a concurrency sweep, and the profile of each framework. No recommendations: the choice of engine is settled in the Pareto article (B8); here there are only the facts and the methodology, because a benchmark without a published methodology is not comparable.

The performance metrics

There is no single performance metric, there are five, and mixing them is the first source of error:

Metric	Definition	Unit	Dominant phase
TTFT (Time To First Token)	time from sending the prompt to the first token	ms	prefill
TPOT / ITL (Time Per Output Token / Inter-Token Latency)	average time between output tokens once generation has started	ms/token	decode
Request throughput	complete request-response cycles per second at the concurrency tested	req/s	both
Token throughput	total tokens (input + output) per second across all concurrent requests	tok/s	both
Goodput	share of requests that meet the defined SLO	useful tok/s	both
P50 / P95 / P99	latency percentiles (not the mean)	ms	—

Precise definitions, because every tool computes them its own way (Anyscale · latency and throughput metrics): TTFT is what a user waits before seeing the first character, dominated by prefill compute; ITL is the average time between successive output tokens and sets the perceived “typing speed” of the answer; request throughput is complete cycles per second; token throughput is total tokens (input plus output) per second across all concurrent requests.

The latency decomposition

The total latency of a request with $N$ output tokens decomposes as:

$$\text{latency} \approx \text{TTFT} + (N-1)\times \text{TPOT}$$

That is why TTFT and TPOT are reported separately: a single mean latency hides very different profiles. A system with high TTFT and low TPOT (expensive prefill, fast decode) and another one the other way round can have the same mean latency for one particular length, yet behave in opposite ways when the response size changes. For an interactive chat experience TTFT and TPOT rule; for a batch of long summaries, token throughput does. Measuring the mean hides both realities.

Goodput: the honest metric

Raw throughput (TPS, RPS) says how much work the system does; goodput says how much of that work meets your quality-of-service standards (SLO) (Anyscale). An engine can boast 10,000 aggregate tok/s, but if half the requests violate the P99 TTFT SLO, its goodput is 5,000. The figure you defend in a proposal is the goodput, not the catalogue throughput: it is the only one that translates into satisfied users and an honest cost per token.

How each metric is instrumented (and where the error creeps in)

Before comparing numbers it helps to know where each clock starts and stops, because two tools can call different things “TTFT”:

Client-side versus server-side TTFT. The TTFT measured by the client includes network latency and gateway queueing; the one measured by the server does not. For an engine comparison the server-side figure is the one that matters; for user experience, the client-side one. Mixing the two invalidates the comparison.
Streaming is required. TTFT and ITL can only be measured if the response arrives as a stream (token by token). If the tool measures complete responses, there is no real TTFT: there is total latency in disguise.
Token counting. Throughput in tok/s depends on which tokenizer counts the tokens. If the tool uses a tokenizer different from the model’s, the token count (and therefore the tok/s and the cost per token) is biased. You have to count with the tokenizer of the model being served.
Warm-up and prefix cache. The first requests of a benchmark benefit from a warm prefix cache and give artificially low TTFT; the warm-up has to be discarded or the result inflates reality.

These four instrumentation decisions explain a good share of the discrepancies between tools. A performance figure that does not say where the clock is measured and with which tokenizer is not comparable, however precise it may look.

The architecture of the tool biases the figure

Here is the trap that invalidates half the published benchmarks. Tools split into two classes by the architecture of the client that generates the load, and that architecture determines whether the measurement is reliable at high concurrency:

Single-process micro-bench (vLLM bench, SGLang bench, genai-perf): a Python client with asyncio in a single process. Useful for quick experiments on one engine, but the single-process architecture introduces a bottleneck on the client side that biases the data at high concurrency (genAI-perf and vLLM): the client cannot generate enough load and you measure the limit of the client, not the limit of the engine.
Multi-process load (GuideLLM, AIPerf): they spread load generation across several processes, avoiding that limit. This is the class that has emerged for measuring at real scale.

The size of the bias is enormous: at 1,000 QPS, a single-process benchmark processed 75,574 tokens against the 545,733 tokens of a distributed architecture, a discrepancy of 7.2× in measurement capacity for the same system ([search]). Anyone comparing two engines with tools from different classes is not comparing the engines: they are comparing the benchmark clients.

Frameworks, one by one

vLLM bench and SGLang bench — engine micro-bench

What they measure: TTFT, TPOT and throughput of the engine itself (vLLM or SGLang). Class: single-process micro-bench. Use: quick experiments to tune one engine and see the effect of its optimisations (see decode and prefill). Limit: they saturate on the client at high concurrency; they are no good for measuring real capacity at scale.

AIPerf — the NVIDIA one (formerly genai-perf), multi-process

What it measures: TTFT, ITL, throughput and latency against vLLM, NIM, TGI and any compatible endpoint. Class: multi-process load. State-of-the-art datum: NVIDIA retired genai-perf and replaced it with AIPerf on 15 April 2026. During the sweep, AIPerf detects GPU saturation and identifies the previous iteration, returning it as estimatedCapacity; if it detects no saturation, estimatedCapacity is the last iteration tested, which is why the sweep has to extend beyond the knee (AIPerf).

GuideLLM — from the vLLM project, SLO-oriented

What it measures: full distributions of TTFT, ITL and end-to-end behaviour, for SLO-driven evaluation. Class: multi-process load. Differentiator: it generates realistic, configurable traffic patterns in synchronous, concurrent and rate-based modes, including reproducible sweeps to identify safe operating ranges (Red Hat · GuideLLM, GuideLLM · GitHub). It is the tool for answering “how hard can I load this engine without breaking the SLO?”.

LLMPerf — the Anyscale/Ray classic

What it measures: throughput and latency at the inference level. Use: endpoint validation, historically very widespread. Class: load generator. Limit: less focused on distributions and sweeps than GuideLLM/AIPerf.

MLPerf Inference — the industry standard

What it measures: performance under standardised scenarios with strict rules, for comparability across vendors. Maintainer: MLCommons. It is the gold standard of cross-vendor comparability; it is expanded on below.

Comparison table

Tool	Class	What it measures	Maintainer	When to use it
vLLM bench	single-process micro	TTFT, TPOT, throughput of vLLM	vLLM (OSS)	tune vLLM, quick experiments
SGLang bench	single-process micro	metrics of the SGLang engine	SGLang (OSS)	tune SGLang
AIPerf (formerly genai-perf)	multi-process load	TTFT, ITL, throughput; estimatedCapacity	NVIDIA (OSS)	real capacity, multi-endpoint
GuideLLM	multi-process load	distributions, SLO, sweeps	vLLM (OSS)	validate SLO, find the knee
LLMPerf	load	throughput and latency	Anyscale/Ray (OSS)	endpoint validation
MLPerf Inference	standard suite	server/offline/interactive scenarios	MLCommons	cross-vendor comparability

The concurrency sweep: finding the knee

The most useful measurement for sizing is not a number, it is a curve: how latency and throughput change as concurrency rises. Raising concurrency keeps the GPU busier and raises RPS, but past a certain point it sends TTFT, ITL and end-to-end latency through the roof ([search]). The goal of the sweep is to find the knee: the maximum concurrency where throughput still rises without latency breaking the SLO.

That is why AIPerf extends the sweep beyond the knee: only by seeing where latency takes off can it return the safe capacity (estimatedCapacity). This curve is the raw material of capacity planning: the knee yields the replica count and the cost per token at the target load.

Worked example: reading a sweep

An illustrative sweep on an example node (a 70B on 8×H100, SLO of P99 TTFT < 500 ms), to see how the knee is read:

Concurrency	RPS	TTFT P50 (ms)	TTFT P99 (ms)	Token tput (tok/s)	Goodput
1	2	80	110	350	100 %
8	14	110	240	2,100	100 %
16	22	180	460	3,400	98 %
24	26	320	980	3,900	62 %
32	27	540	1,800	4,000	20 %

Reading: up to concurrency ~16 throughput grows and P99 stays under the SLO (goodput ~100 %). Between 16 and 24 sits the knee: throughput barely rises any more (3,400 → 3,900 tok/s) but P99 takes off (460 → 980 ms) and goodput collapses (98 % → 62 %). At concurrency 32 raw throughput is at its maximum (4,000 tok/s) but goodput is 20 %: the system “performs well” while serving mostly requests that violate the SLO. The defensible safe capacity is the one at concurrency ~16, not the one at maximum throughput. This is the number that goes into capacity planning and into cost per token: at 3,400 useful tok/s, not 4,000 raw tok/s.

MLPerf Inference: the comparability standard

To compare across vendors and engines with identical rules there is MLPerf Inference (MLCommons). The datacenter category focuses on two scenarios plus an optional one (MLCommons · datacenter):

Scenario	What it simulates	Metric
Offline	raw throughput processing the whole dataset in batch	maximum throughput
Server	interactive environment: requests one at a time following Poisson at a mean RPS	RPS under TTFT and TPOT limits
Interactive (optional)	like server but with stricter latency limits	RPS under a hard SLO

The Server scenario is the realistic one for online inference: the load generator sends requests following a Poisson distribution and demands that specific TTFT and TPOT bounds be met. MLPerf Inference v5.0 (April 2025) introduced a large-scale 405B benchmark and a low-latency interactive 70B one, offering language benchmarks at every scale (7B to 405B), architectural diversity (including MoE) and scenarios (MLCommons · v5.0); v5.1 (September 2025) widened the results with record participation (MLCommons · v5.1).

The value of MLPerf is comparability: everyone measures the same thing under the same rules. Its limit is that those rules may not match your workload (your length distribution, your specific SLO), so it serves to compare hardware and engines against each other, not necessarily to size your case. For that, your own sweep.

Measurement bias and reproducibility

That two benchmarks give very different results for the same system is no accident: systematic measurement bias in production benchmarks is characterised in the literature (arXiv 2605.24217), and there is work dedicated to the meta-metrics and good practices of system-level performance benchmarking (arXiv 2508.10251). The most common sources of bias:

Source of bias	Effect	Mitigation
Single-process client	underestimates throughput at high concurrency	use multi-process load
Unrealistic length distribution	results that do not apply to your traffic	use realistic traces
Measuring the mean instead of percentiles	hides the latency tail	report P95/P99
Uncontrolled warm-up	the prefix cache inflates the first results	discard the warm-up
Not pinning engine/model version	irreproducible	pin everything and publish it

The methodological conclusion of this article: a benchmark without a published methodology is not comparable. For a performance figure to support a proposal it has to come with the tool, its version, the model and precision, the load distribution and the SLO. Synthesis article S4 builds a reproducible harness that pins all of that.

Checklist for a reproducible benchmark

For a performance figure to be defensible before a committee, it has to come with everything that allows it to be reproduced. The minimum published alongside the result:

What to pin	Why
Tool + version	each one measures differently; the version changes behaviour
Model + precision (FP16/FP8/INT4)	precision changes throughput and quality
Hardware (GPU, count, interconnect)	an 8×H100 NVLink is not an 8×H100 PCIe
Engine + version + flags	vLLM/SGLang/TRT-LLM and their configuration
Length distribution (in/out)	real traffic is not fixed length
Concurrency levels of the sweep	you have to go past the knee
SLO (which percentile, which threshold)	it defines the goodput
Warm-up handling	discard it or it biases the result
Tokenizer used for counting	it affects tok/s and cost per token

The practical rule: if you cannot hand over this table alongside the figure, the figure is not a datum, it is an anecdote. The reproducible harness of article S4 automates the recording of all these parameters so that anyone, including whoever challenges the proposal, can reproduce the exact number.

Performance ≠ quality

One warning that avoids the most expensive mistake: these tools measure speed and throughput, not accuracy. An engine can be blisteringly fast while serving a model that answers badly. Quality is measured with another family of tools (lm-evaluation-harness, HELM, task leaderboards) which is another axis of the scorecard (article B7). Confusing “fast” with “good” is how you build a platform that serves bad answers very quickly. On the final Pareto frontier, performance and quality are two distinct axes that must be looked at together, never one instead of the other.

The other family of tools, for reference (expanded in B7):

Tool	What it measures	Common trap
lm-evaluation-harness	accuracy on hundreds of standardised tasks	contamination of the test dataset
HELM	holistic evaluation (accuracy, robustness, bias, efficiency)	heavy to run
LiveBench / dynamic leaderboards	tasks that rotate to avoid contamination	comparability over time

The data point: contamination, the model having seen the test during training, inflates quality metrics just as warm-up inflates performance ones. That is why dynamic leaderboards rotate their questions. Quality and performance share that lesson: the measurement method biases the result as much as the system being measured.

The latency versus throughput trade-off

A property the sweep reveals and which is worth keeping in mind: latency and throughput pull in opposite directions. Batching groups requests to amortise the cost of moving the weights out of VRAM, which raises throughput, but each request waits for the batch to form, which raises individual latency. There are two operating regimes, and the benchmark serves to place you in the right one:

Regime	Optimises	Configuration	Case
Latency	low TTFT/TPOT	small batch, low concurrency	interactive chat, copilots
Throughput	maximum tok/s	large batch, high concurrency	overnight batch, ingestion

There is no single “best” point: there is the best point for your SLO. A benchmark that reports only maximum throughput is describing the throughput regime and ignoring whether that point meets the latency your case needs. That is why goodput, throughput under the SLO, is the metric that reconciles the two regimes: it measures how much throughput you get without stepping outside acceptable latency. The sweep traverses the curve between both regimes; your SLO marks where on that curve your system sits.

The connection with cost and energy

Performance is not an isolated axis: by the identity of the opening article, throughput is the denominator of cost per token and of energy per token. A sweep that finds a knee at 4,000 tok/s rather than 2,800 is not just “faster”: it lowers the CPM from ~1.09 to ~0.76 €/1M tok and the energy per token in the same proportion, on the same iron. That is why performance benchmarking is the tool that, indirectly, moves cost the most: every goodput improvement translates into euros and into watts per token. The number that connects the three axes is goodput, the throughput that meets the SLO, not catalogue throughput.

State of the art 2026

genai-perf → AIPerf migration (15 Apr 2026): NVIDIA consolidates its benchmarking into a multi-process tool with saturation detection.
GuideLLM as the OSS standard for SLO-driven evaluation with reproducible sweeps.
MLPerf Inference v5.0/v5.1 extends to 405B, interactive 70B and MoE, with record participation: cross-vendor comparability is maturing.
Measurement bias characterised: the community accepts that the measurement method matters as much as the system measured; the emphasis on reproducibility and meta-metrics is growing.

Limits and traps (data-driven)

Comparing tools from different classes. A single-process micro-bench and a multi-process load generator are not comparable; the difference can be 7×. Pin the class.
Catalogue throughput instead of goodput. The honest number is the one that meets the SLO.
Means instead of percentiles. The mean hides the tail; report P95/P99.
Not extending the sweep beyond the knee. Without seeing where latency takes off you do not know the safe capacity.
Confusing performance with quality. They are distinct axes; fast is not good.
Not pinning versions. Engine, model, precision and load left unpinned = irreproducible = not defensible.

The next article in the track (B2) goes into the tool catalogue in depth; this one pins the metrics and the methodology. With performance measured reproducibly, the scorecard can cross it with cost (in €) and energy for the final decision.

Closing

Performance benchmarking looks like the most “objective” of the three axes (they are tokens per second, after all) and it is exactly where the most manipulation happens, almost always without bad intent: a single-process tool here, a mean instead of a P99 there, a catalogue throughput instead of the goodput. The difference between a marketing number and a defensible datum lies not in the engine measured but in the methodology: the class of tool, where the clock is measured, with which tokenizer, how far the sweep goes and which SLO defines the goodput. For a sovereign architecture proposal, performance only counts if it is delivered with that reproducibility record, and, crossed with cost in euros and energy per token, it becomes the column of the Pareto frontier that decides which engine and which configuration hold up the platform. The number you defend is not the highest one: it is the reproducible goodput.

Sources

Anyscale · métricas de latencia y throughput de LLM — https://docs.anyscale.com/llm/serving/benchmarking/metrics
Red Hat · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
NVIDIA AIPerf · guía de benchmarking (ex genai-perf) — https://lucaberton.com/blog/nvidia-aiperf-llm-inference-benchmarking-guide/
MLCommons · MLPerf Inference datacenter (escenarios) — https://mlcommons.org/benchmarks/inference-datacenter/
MLCommons · MLPerf Inference v5.0 (405B + 70B interactivo) — https://mlcommons.org/2025/04/llm-inference-v5/
MLCommons · MLPerf Inference v5.1 — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
arXiv 2605.24217 · sesgo sistemático de medición en benchmarks de inferencia LLM — https://arxiv.org/html/2605.24217
arXiv 2508.10251 · meta-métricas y buenas prácticas de benchmarking de rendimiento — https://arxiv.org/pdf/2508.10251
Medium · LLM Inference Benchmarking (genAI-perf y vLLM) — https://kchandan.medium.com/llm-inference-benchmarking-genai-perf-and-vllm-5dd06b57428e

Batch sizing in vLLM: the two-hour grid search worth weeks of hardware

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

max-num-seqs and max-num-batched-tokens are the two dials that control how much work vLLM processes on each scheduler iteration. Their default values are calibrated to be safe on any hardware, not to maximise throughput on yours. A systematic grid search over 25 configurations, runnable in two hours, identifies the combination that, for your specific workload and hardware, can double throughput without changing a single line of the model or adding a GPU. The OTel metrics that confirm you found the optimum are vllm:num_waiting_seqs, vllm:num_preemptions_total and vllm:time_per_output_token_seconds.

The analogy

An industrial kitchen with one chef and ten stoves. If the maître d’ only sends one order at a time, the chef works at 10% of capacity. If he sends a hundred simultaneous orders but there are ingredients for only twenty, the chef spends half his time waiting for restocking. The optimum is the point where every stove is lit and the resupply never runs dry.

max-num-seqs is how many orders the chef can have in preparation at once. max-num-batched-tokens is how many ingredients he can process in a single movement of the wok. Getting either of them wrong leaves stoves empty.

The problem: the defaults are not for your hardware

In vLLM V1 (≥ 0.6), the defaults are:

max-num-seqs = 1024 (V1) / 256 (V0)
max-num-batched-tokens = 8192

These values guarantee that vLLM starts on any GPU without OOM. They do not guarantee optimal throughput. The reason: the optimal point depends on three variables vLLM does not know at start-up:

The length distribution of your real workload — a RAG system with 2K-token prompts needs a different budget from a chat with 50-token messages.
VRAM available for KV cache — determined by the model, the quantisation and --gpu-memory-utilization.
Real expected concurrency — how many simultaneous users arrive at the 95th percentile.

The interaction between these three factors makes it impossible for a universal default to be optimal in specific cases.

The arithmetic that matters

On each iteration, vLLM’s scheduler decides which tokens to process. The total budget available per step is max-num-batched-tokens. That budget is split between:

Decode tokens: 1 for each active request in the generation phase. With 64 requests in decode, 64 tokens of budget are consumed.
Prefill tokens (in chunks): the rest of the budget goes to processing new prompts.

$$\text{prefill tokens per step} = \text{max num batched tokens} - \text{num requests decode}$$

If max-num-batched-tokens = 8192 and you have 512 requests in decode, each step can only process 8192 - 512 = 7680 prefill tokens. With 2000-token prompts, that is ~3.8 new prompts per iteration.

The problem appears when max-num-seqs is very high relative to the available KV cache. Each active request in decode occupies KV cache blocks. If the blocks run out, vLLM performs preemption: it pauses a request, frees its KV cache and re-queues it. Each preemption costs additional latency for the paused request and complexity for the scheduler.

$$\text{KV budget} = \frac{\text{free VRAM} \times \text{gpu memory utilization}}{\text{bytes per token} \times \text{max model len}}$$

For a Qwen2.5-14B on an RTX 4090 with Q4_K_M (9 GB of model, 15 GB free):

$$\text{KV budget} = \frac{15 \times 0.92 \times 10^9}{40\,000} \approx 345\,000 \text{ tokens}$$

With max-model-len = 8192, the maximum number of simultaneous requests with a full context is:

$$\text{real max seqs} = \frac{345\,000}{8192} \approx 42 \text{ requests}$$

Setting max-num-seqs = 1024 with those numbers guarantees constant preemptions. The optimum is at 40-50.

The grid search: methodology

Step 1: measure the real workload

Before hunting for the optimum, you need to know your traffic’s percentiles. From Langfuse or the vLLM logs:

# Extract the length distribution from Langfuse
import langfuse
client = langfuse.Langfuse()

traces = client.fetch_traces(limit=1000).data
prompt_lens = [t.input_tokens for t in traces if t.input_tokens]
output_lens = [t.output_tokens for t in traces if t.output_tokens]

import numpy as np
print(f"Prompt p50={np.percentile(prompt_lens,50):.0f} p95={np.percentile(prompt_lens,95):.0f} p99={np.percentile(prompt_lens,99):.0f}")
print(f"Output p50={np.percentile(output_lens,50):.0f} p95={np.percentile(output_lens,95):.0f} p99={np.percentile(output_lens,99):.0f}")

Step 2: work out the KV budget

Run once with --dry-run, or read vLLM’s start-up log:

INFO: # GPU blocks: 4521, # CPU blocks: 512

Each block is 16 tokens. 4521 × 16 = 72,336 tokens of total KV budget.

Step 3: the grid

With the KV budget known and the p95 of prompt/output length:

# grid_search_batch.py
import subprocess, json, time

MODEL = "Qwen/Qwen2.5-14B-Instruct-AWQ"
PROMPT_LEN = 512 # p50 of your workload
OUTPUT_LEN = 256
CONCURRENCY = 32 # simultaneous users expected at peak

seqs_values = [32, 64, 128, 256, 512]
tokens_values = [4096, 8192, 16384, 32768, 65536]

results = []
for seqs in seqs_values:
 for tokens in tokens_values:
 cmd = [
 "python", "-m", "vllm.entrypoints.benchmark_throughput",
 "--model", MODEL,
 "--max-num-seqs", str(seqs),
 "--max-num-batched-tokens", str(tokens),
 "--num-prompts", "200",
 "--input-len", str(PROMPT_LEN),
 "--output-len", str(OUTPUT_LEN),
 ]
 out = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
 # Parse throughput from the output
 for line in out.stdout.splitlines():
 if "Throughput" in line:
 tps = float(line.split(":")[1].strip().split()[0])
 results.append({"seqs": seqs, "tokens": tokens, "tps": tps})
 print(f"seqs={seqs} tokens={tokens} → {tps:.1f} tok/s")

# Save for analysis
with open("grid_results.json", "w") as f:
 json.dump(results, f, indent=2)

25 configurations × ~5 min = ~2 hours. Real execution time, not waiting time.

Step 4: read the surface

The result is a 5×5 throughput matrix. The typical shape:

max-num-batched-tokens → 4K 8K 16K 32K 64K
max-num-seqs ↓
32 180 310 380 390 385 ← max-num-seqs too low
64 185 350 480 510 508 ← optimal point for this workload
128 182 340 450 480 475
256 178 320 400 410 402 ← KV cache runs out, preemptions
512 170 290 360 370 368 ← high preemptions

The optimum in this example: max-num-seqs=64, max-num-batched-tokens=32768. Above that, preemptions cancel out the concurrency gain.

Confirmation with OTel

Once the optimal configuration is deployed, three Prometheus metrics confirm it is well calibrated:

# 1. Queued requests — should stay close to 0
# If it grows steadily: max-num-seqs too low or max-num-batched-tokens insufficient
vllm:num_waiting_seqs

# 2. Preemptions — should be 0 or very occasional (<1/min)
# If it grows: max-num-seqs too high for the available KV cache
rate(vllm:num_preemptions_total[5m]) * 60

# 3. ITL (inter-token latency) — should be stable, with no spikes
# Bimodality = badly calibrated batch size (some requests outside the CUDA graph bucket)
histogram_quantile(0.99, rate(vllm:time_per_output_token_seconds_bucket[5m]))

The optimal configuration produces:

num_waiting_seqs ≈ 0 under normal operation
num_preemptions_total stable (not growing)
time_per_output_token unimodal

If num_waiting_seqs is high with gpu_cache_usage_perc low: raise max-num-batched-tokens to process prefills faster. If num_preemptions_total grows: lower max-num-seqs or enable FP8 KV cache to free up blocks.

Reference configurations by profile

Based on the grid search for mid-range hardware (generic 4×H100, 14B-70B model):

Profile	Prompt p50	Output p50	max-num-seqs	max-num-batched-tokens
Conversational chat	150 tok	300 tok	256	16384
Enterprise RAG	1500 tok	200 tok	64	32768
Coding (completion)	800 tok	500 tok	128	32768
Summarisation	2500 tok	400 tok	32	65536
Batch processing	4000 tok	800 tok	16	65536

None of these is universal. They are starting points for the grid search on your real hardware and workload.

When not to touch the defaults

If your system sits below 50% KV cache utilisation (vllm:gpu_cache_usage_perc < 0.50) under real demand and with no num_waiting_seqs, the defaults are enough for your current load. The grid search contributes more when you are close to maximum capacity or when you want to extract the full performance of fixed hardware.

Benchmarking on lo0 — Tech Blog

MLPerf Inference: how to read it, what LoadGen enforces and what comparability it offers

TL;DR

What it is and how it is organised

The four scenarios

LoadGen: what it measures and what it does not

Duration, query count and early stopping

The compliance tests

Divisions, categories and what can be touched

The three divisions

The availability categories

What Closed allows and forbids

The SLOs almost nobody cites: TTFT and TPOT

Quality: the 99 % and the token count

The current Datacenter suite

How to read a results row

Per-accelerator is not an official metric

The comparison rules

How much audit sits behind a row

The recent rounds, in figures

What MLPerf Inference does not measure

Usage flow for an on-premise platform

Cross-links from the benchmarking track

See also

Sources

The reproducible harness: measuring cost, performance and energy in a single auditable experiment

TL;DR

Why the three axes must be measured together

Architecture of the integrated bench

The Kubernetes Job: the full YAML

Performance layer: GuideLLM and AIPerf

GuideLLM — the SLO-driven sweep

AIPerf — NVIDIA’s concurrency profiler

Cost layer: OpenCost and LiteLLM

OpenCost — the denominator in euros per GPU-hour

LiteLLM — the per-request token counter

Energy layer: DCGM and Kepler

DCGM Exporter — GPU power and energy

Kepler — pod-level energy

The MLPerf Power reference

Unifying the metrics: the PromQL queries

The 3-axis scorecard: table and interpretation

The multi-objective Pareto frontier

Reproducible design: the twelve mandatory metadata

Idempotence and versioning of results

How it connects with the other posts in the series

Full flow of a benchmarking session

Cost of the experiment

What the harness does not measure (and why)

Sources

Measurement bias and reproducibility: why two benchmarks of the same model give figures that differ by up to 7×

TL;DR

The sources of bias: a catalogue with magnitudes

The dominant bias: single-process client saturation

Tokenizer bias: tok/s are not comparable across vocabularies

The bias of ignore_eos and an inconsistent OSL

The bias of the ISL/OSL distribution

The bias of warmup and the prefix cache

The bias of concurrency vs request-rate

The bias of the measurement window and the thermal state

How the tools compute ITL differently

Comparability across tools: why their numbers do not cross

The honest harness: a reproducibility checklist

Hardware

Software and model

Load

SLO and reported metrics

Methodological defence of the datum before an audit

A numerical example: the same node, four different measurements

Decomposing the total bias

State of the art 2026: what has changed

Cross-links from the benchmarking track

Sources

LLM quality benchmarks: the contamination trap and the OSS tools for not fooling yourself

TL;DR

Quality ≠ performance: the independent Pareto axis

Reference benchmarks: what each one measures

OSS evaluation tools: data sheets

lm-evaluation-harness (EleutherAI)

HELM (Stanford CRFM)

The bias of `concurrency` vs `request-rate`

`profile` mode: one load point

The `analyze` subcommand: automatic concurrency sweep

Artefact structure of `analyze`