Ttft on lo0 — Tech Blog

MLPerf Inference: how to read it, what LoadGen enforces and what comparability it offers

Sat, 29 Aug 2026 09:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

MLPerf Inference is maintained by MLCommons on a six-monthly cadence. It imposes three things no home-made benchmark has: a mandatory load generator (LoadGen, compiled from a tagged revision and unmodified), model-equivalence rules that bound what can be optimised, and an audit process that reviews up to two submissions per round. The four scenarios (SingleStream, MultiStream, Server, Offline) have a minimum duration of 600 s and different metrics: Server reports the sustained Poisson rate and is the only one with a latency constraint. For the LLM benchmarks that constraint is not a request latency but two simultaneous SLOs at the 99th percentile: TTFT and TPOT, with per-model figures ranging from 2000 ms / 200 ms on Llama 2 70B to 450 ms / 40 ms on its Interactive variant. Quality is required relative to the reference model (99 % or 99.9 %) and the generation length is audited as well (Llama 2 70B: more than 90 % of 294.45 tokens per sample). Three reading rules sum up the real comparability: results are only compared within the same version, division, category and scenario; per-accelerator is not an official MLCommons metric, it is derived; and neither cost nor energy enters the primary metric, to the point that round v5.1 had 2 power submissions against 27 participating organisations.

What it is and how it is organised

MLPerf Inference is maintained by the MLCommons Association through the MLPerf Inference Working Group, whose participation is limited to MLCommons members and affiliates (MLPerf Inference Working Group). The declared cadence for publishing results is roughly six months.

Round	Results published
v4.0	27 March 2024
v4.1	28 August 2024
v5.0	2 April 2025
v5.1	9 September 2025
v6.0	1 April 2026
v6.1	in preparation as of this data sheet

The suite splits into two system categories: Datacenter and Edge, the latter defined in the rules as everything that is not datacenter (inference_rules.adoc). A Datacenter system has two requirements that do not apply in Edge:

Mandatory ECC on DRAM and HBM, active during all performance and accuracy runs. There is no requirement on SRAM.
Mandatory networking since round v3.0, with a minimum bandwidth computed from the achieved throughput. For Llama 3.1 405B the minimum ingress is throughput × 20000 × dtype_size bytes/s; for Llama 2 70B the minimum egress is throughput × 1024 × dtype_size.

Edge additionally allows inferred results: a MultiStream derived from SingleStream counts as 8 times the p99 latency; an Offline derived from MultiStream, 8000 divided by the mean latency in milliseconds.

The four scenarios

This is the normative table, transcribed from the rules:

Scenario	Query generation	Duration	Samples/query	Latency constraint	Tail latency	Reported metric
SingleStream	The next query is issued when the SUT completes the previous one	600 s	1	None	90 %	p90 latency with early stopping
Server / Interactive	LoadGen issues queries following a Poisson distribution	600 s	1	Benchmark-specific	99 %	Maximum Poisson parameter supported
Offline	All samples are delivered at the start in a single query	1 query, 600 s	At least 24,576	None	N/A	Measured throughput
MultiStream	The next query is issued when the SUT completes the previous one	600 s	8	None	99 %	p99 query latency with early stopping

Four operational clarifications that change how each number is interpreted:

In Server, LoadGen does not measure a latency: it runs a binary search over the target rate. For a given value it generates queries at that QPS with Poisson arrivals; if the run fails the latency gate, it lowers the value and repeats. The final published metric is, per the official FAQ, scheduled samples per second, not completed samples nor the target entered by the user.
In MultiStream, the latency of a query is the maximum of the latencies of its samples, and the 8 samples of each query are contiguous in the order in which they were loaded. The value multi_stream_samples_per_query = 8 is fixed in the official configuration.
In Offline there is no latency constraint at all. The rules themselves forbid techniques that only work in fixed-length experiments except in the Offline scenario, which is a normative acknowledgement that an Offline result says nothing about behaviour under an SLO.
The percentiles in the table are the lower theoretical bound for runs with very many queries. With early stopping, the percentile actually computed is somewhat above the nominal one, as a penalty for processing few queries.

The mandatory scenarios depend on the benchmark. In Datacenter, the summarisation, question answering, text generation and VLM tasks require (Server | Interactive), Offline, that is, the submitter chooses between Server and Interactive. Reasoning and recommendation require Server, Offline. Medical segmentation, node classification, speech-to-text and RAG require only Offline.

LoadGen: what it measures and what it does not

LoadGen is the module that generates the load and computes the metrics. Its use is mandatory for all submissions, and it must be compiled from a tagged and approved revision of the repository, unaltered; the README is explicit that local modifications of the C++ library are not accepted.

Its four declared responsibilities are generating the queries according to the scenario, tracking the latency of each query, validating the accuracy of the results and computing the final metrics. Latency is defined as the time from when LoadGen was scheduled to pass a query to the system under test until it receives the response.

Equally important is what it declares out of scope: LoadGen does not know the model, does not know the data formats, cannot score accuracy and does not know the scenario constraints of the MLPerf rules. The consequence is written in its own README: being model-agnostic, it cannot enforce the MLPerf requirements, for example the target percentiles and latencies. Those limits are injected from mlperf.conf and user.conf. The justification for making it mandatory appears in the founding ISCA 2020 paper: it establishes a clear boundary between the components that belong to the submitter and those that belong to MLPerf, and it measures the performance of the complete system instead of that of a single piece.

Duration, query count and early stopping

The minimum duration is 600,000 ms in all four scenarios. The minimum number of queries comes from a confidence interval table that the rules publish:

Tail percentile	Confidence	Margin of error	Inferences	Rounded
90 %	99 %	0.50 %	23,886	24,576
95 %	99 %	0.25 %	50,425	57,344
97 %	99 %	0.15 %	85,811	90,112
99 %	99 %	0.05 %	262,742	270,336

The submission checker applies Server: 270336, SingleStream: 1024 and MultiStream: 662. For Offline, the minimum is set per benchmark: 24,576 on Llama 2 70B, 15,000 on Mixtral 8x7B, 13,368 on Llama 3.1 8B, 8,313 on Llama 3.1 405B, 4,388 on DeepSeek-R1, 1,633 on Whisper and 43 on 3D UNet.

The early stopping mechanism allows runs to be shortened while keeping the statistical guarantee. With tolerance d = 0 and confidence c = 0.99, the algorithm resolves by binary search the smallest number of queries below the latency threshold that satisfies the criterion, given the number of queries observed above it.

LoadGen’s modes are SubmissionRun (accuracy followed by performance), AccuracyOnly, PerformanceOnly and FindPeakPerformance. The last, applicable only to Server, takes target_qps as a lower bound if it passes, estimates the upper bound at double and doubles it until failure, and then does a binary search. In performance mode LoadGen selects queries uniformly at random with replacement from a set of size QSL; in accuracy mode it uses a copy of the validation dataset, each sample exactly once. One accuracy run must be executed for every performance result submitted, and the same code in both modes.

The seeds are announced four weeks before the deadline and the mandatory generator is Mersenne Twister 19937. They change every round: in v5.1, qsl_rng_seed = 1780908523862526354; in v5.0 it was 6023615788873153749.

The compliance tests

They are activated by placing an audit.config file in the working directory, whose parameters override those of mlperf.conf and user.conf. Their logs are mandatory in the submission package. The declared purpose is to detect anomalies, not to diagnose their cause.

For the LLM benchmarks the required test is TEST06, specifically against the end-of-sequence token exploit. It runs 100 samples and demands three conditions: that the first token reported separately matches the first token of the model’s output, that the output ends with zero or a single EOS token, and that the number of tokens reported matches the real length of the generated sequence.

Divisions, categories and what can be touched

The three divisions

Closed: requires preprocessing, postprocessing and model equivalent to the reference implementation. It allows calibration for quantisation and does not allow retraining. It is the only one that can use the MLPerf name unqualified.
Open: allows arbitrary preprocessing, postprocessing and model, including retraining. The accuracy, latency and scenario constraints do not apply: instead, the accuracy obtained and the latency constraints under which the performance was obtained must be reported. The model may have any origin, be quantised in any way and be sparsified in any way. It must use the same validation dataset as the corresponding Closed and use all of it.
Network: inherits all the Closed requirements, applies only to Datacenter, and separates the LoadGen node from the system under test through a fabric. The submitter’s dispatch library cannot preprocess, postprocess, batch, pad or cache. Bus interconnects are forbidden by name (PCIe, CXL, CCIX, HyperTransport, NVLink, QPI, UPI, ICI) and only Ethernet, IEEE 802.11, InfiniBand and 3GPP are admitted, with a requirement to work chassis to chassis at more than ten metres.

The availability categories

Category	Hardware	Software
Available in cloud	Available for cloud rental	Available
Available on premise	Available for purchase	Available
Preview	Must be available for the next submission, or the one after 140 days, whichever is longer	Available except what is needed for substantially new hardware
RDI	Does not meet the above	Does not meet the above

Available demands four cumulative conditions: price available, having been rented or shipped to at least one third party, public evidence of availability and reasonable availability to additional third parties on the submission date. A Preview result obliges a resubmission as Available with equal or better performance, tolerating up to 2 % degradation from noise; if there is no resubmission, the Preview result is marked invalid. RDI components cannot be submitted as Available until the cycle after the next one, or 221 days, whichever is longer.

What Closed allows and forbids

MLPerf delivers the weights in fp16 or fp32. The central rule is that the submitter may perform purely mathematical and reproducible quantisation, using only the calibration data and the tensors of the delivered model, to any numerical format that reaches the required quality, and that the method must be described publicly at a level that allows it to be reproduced. The test against back doors is elegant: the description of the quantisation method must be much smaller than the non-zero weights it produces.

Allowed, among other things: any framework or runtime, arbitrary data layout, varying the matrix multiplication algorithm, mathematically equivalent transformations, polynomial approximations of transcendentals, processing queries out of order within what the scenario admits, replacing dense operations with mathematically equivalent sparse operations, hand-picking different precisions per operation, fusing and unfusing, dynamic batching, and mixtures of experts combining weights with different quantisation.

Forbidden: replacing or supplementing weights wholesale, discarding non-zero weights, including pruning, caching queries or responses, coalescing identical queries, modifying weights during the timed portion, quantisation algorithms of size comparable to the weights they produce, hardcoding the total number of queries, using knowledge of the LoadGen implementation to predict peaks or troughs in the Server scenario, changing the number of search beams, and incorporating statistics from the performance or accuracy sets.

Two clarifications that matter to anyone serving LLMs in production: the KV cache is allowed just as in the reference model provided it is not applied across queries; PagedAttention is admitted if the block is reused only within the batch; continuous or dynamic batching is allowed; and KV cache entries are treated as activations for quantisation purposes, and cannot be pruned. Speculative decoding is only admitted in the benchmark and scenario combinations explicitly listed, with the reference head at the same precision, and implementations that artificially manipulate the acceptance rate are forbidden.

The SLOs almost nobody cites: TTFT and TPOT

This is the point where an MLPerf table is most often misread. In the LLM benchmarks of the Server scenario, the constraint is not a request latency: the rules set target_latency = 0 and activate use_token_latencies = 1, so that the gate is formed by two simultaneous metrics, both evaluated at the 99th percentile:

TTFT (time to first token): latency of the first token.
TPOT (time per output token): mean interval between all generated tokens.

The figures are in mlperf.conf and change per round and per benchmark:

Benchmark (Server scenario)	TTFT	TPOT	Round
`llama2-70b`	2000 ms	200 ms	v5.0, v5.1, v6.x
`llama2-70b-interactive`	450 ms	40 ms	v5.1, v6.x
`mixtral-8x7b`	2000 ms	200 ms	v5.0, v5.1, v6.x
`llama3_1-405b`	6000 ms	175 ms	v5.0, v5.1, v6.x
`llama3_1-405b-interactive`	4500 ms	80 ms	v6.x
`llama3_1-8b`	2000 ms	100 ms	v5.1, v6.x
`llama3_1-8b-interactive`	500 ms	30 ms	v6.x
`deepseek-r1`	2000 ms	80 ms	v5.1, v6.x
`deepseek-r1-interactive`	1500 ms	15 ms	v6.x
`gpt-oss-120b`	3000 ms	80 ms	v6.x

A TPOT of 40 ms is equivalent to 25 tokens per second per user, which is the order of magnitude of a conversational interface; one of 200 ms is equivalent to 5 tokens per second, which is a batch use case dressed up as a service. Two results for the same model under the Server and Interactive labels are not the same measurement. Two documented discrepancies between primary sources remain: the llama2-70b-interactive variant does not appear in the mlperf.conf tagged as v5.0 despite having been announced in that round, and in the v6.x cycle the TPOT of gpt-oss-120b-interactive appears as 15 ms in mlperf.conf and as 20 ms in the text of the rules.

For contrast, the constraints of the non-LLM benchmarks in the Server scenario are conventional request latencies: ResNet-50 15 ms, RetinaNet 100 ms, BERT 130 ms, DLRMv2 60 ms, DLRMv3 80 ms, RNN-T 1000 ms, GPT-J and Stable Diffusion XL 20,000 ms.

Quality: the 99 % and the token count

The quality targets are expressed relative to the reference model, never in absolute terms, and each benchmark demands one variant, the other or both. In the current Datacenter suite, 3D UNet, Llama 3.1 8B and Whisper ask for 99 % and 99.9 % of FP32, which generates two distinct results; Llama 2 70B asks only for 99.9 %; Llama 3.1 405B, Mixtral, DeepSeek-R1, GPT-OSS-120B, Qwen3-VL, RGAT and WAN-2.2 ask only for 99 %; and E2E-RAG is the exception at 97 %. In the checker this materialises as separate benchmarks (bert-99 and bert-99.9), each with its computed numerical threshold.

On top of that metric there is a second LLM-specific constraint that is cited very rarely: the generation length is audited as well.

Benchmark	Length constraint
Llama 2 70B	tokens per sample > 90 % of 294.45
Llama 3.1 405B	tokens per sample between 90 % and 110 % of 684.68
Mixtral 8x7B	tokens per sample between 90 % and 110 % of 144.84
Llama 3.1 8B	total generated length > 90 % of 8,167,644

The FAQ settles the obvious shortcut: reducing the maximum output length below the reference is not allowed, and truncating tokens to improve performance or reach the accuracy target is not permitted. The inference parameters are fixed in Closed (max_new_tokens = 1024 on Llama 2 70B, 20000 on Llama 3.1 405B and DeepSeek-R1), as is the decoding algorithm: greedy search in the Llama family, Mixtral and DeepSeek-R1; sampling with temperature = 1.0 and top_p = 1.0 in GPT-OSS-120B and Qwen3-VL. Accuracy is reported to five significant figures with round-half-to-even.

The current Datacenter suite

Area	Task	Model	Dataset	QSL	Required quality
Vision	Medical segmentation	3D UNet	KiTS 2019	42	99 % and 99.9 % of FP32 (DICE 0.86330)
Language	Summarisation	Llama 3.1 8B	CNN DailyMail v3.0.0	13,368	99 % and 99.9 % of FP32
Language	Question answering	Llama 2 70B	OpenOrca	24,576	99.9 % of FP32
Language	Text generation	Llama 3.1 405B	LongBench, Ruler, GovReport	8,313	99 % of FP16
Language	QA, maths and code	Mixtral 8x7B	OpenOrca, GSM8K, MBXP	15,000	99 % of FP16
Language	Reasoning	DeepSeek-R1	`mlperf_deepseek_r1`	4,388	99 % of FP16 (exact match 81.9132 %)
Language	QA, maths and code	GPT-OSS-120B	AIME25, GPQA Diamond, LiveCodeBench v6	6,396	99 % of 83.13 %
Vision	Vision-language model	Qwen3-VL-235B-A22B	Shopify product catalogue	48,289	99 % of BF16 (hierarchical F1 0.7824)
Commerce	Recommendation	DLRMv3	Synthetic Streaming 100B	34,996	99.9 % of FP32
Generative	Text to video	WAN-2.2-T2V-A14B	VBench prompts	248	99 % of BF16 (VBench 69.7752)
Graphs	Node classification	RGAT	IGBH	788,379	99 % of FP32 (72.86 %)
Audio	Speech to text	Whisper	LibriSpeech	1,633	99 % and 99.9 % of FP32 (WER 2.0671 %)
Language	End-to-end RAG	E2E-RAG	FRAMES	824	97 % of FP32

The E2E-RAG benchmark deserves attention from anyone designing a RAG platform, because it fixes the whole pipeline: a FAISS HNSW index with mandatory parameters M = 32, efConstruction = 200 and efSearch = 100; e5-base-v2 embeddings; ColBERTv2 reranking; generation and query rewriting with GPT-OSS-120B; evaluation with Llama 3.1 8B as judge; and a maximum of 5 retrieval iterations, mandatory for all submitters. Model loading is not timed; building the vector database is.

The definition of sample changes per model and is a frequent source of confusion: in the LLMs it is a sequence, in DLRMv3 it is a request with a user history and 2048 candidates, in WAN-2.2 it is a pair of positive and negative prompts, and in PointPainting it is five images and a lidar point cloud.

How to read a results row

The columns of the official table in the Closed division are Submitter, Software, System, Benchmark Results, Processor/Count, Details, Accelerator/Count and Code. The Open division adds Model Used and Notes. Rows with power measurement add System Power in Server and Offline, or Energy Per Stream in the stream scenarios. Any citation must use the identifier in the format major-version.minor-version.entry.benchmark, of the style 5.1-0053, and carry a footnote with suite, version, division, benchmark, scenario, date and source.

Per-accelerator is not an official metric

There is no per-accelerator column. It is a derived metric, and the MLCommons messaging guidelines are blunt: any comparison based on a different or derived metric, such as power, cost, model size or accuracy, must make the basis of comparison clear in the text and in a footnote, and secondary and derived metrics cannot be presented as official or verified MLPerf metrics. NVIDIA itself states this with the standard formula in its round blogs: per-GPU performance is not a primary MLPerf Inference metric and is computed by dividing the reported throughput by the number of reported accelerators. The same applies to combining results from several benchmarks: MLCommons allows it but does not endorse it, and the composite cannot be presented as an official result.

The comparison rules

MLPerf results can only be compared with compatible MLPerf results: same benchmark, same scenario and compatible versions per the normative compatibility table.
MLPerf results cannot be compared with non-MLPerf results.
When comparing, any difference in version, division, category, verification status, scenario or number of chips must be clearly identified. When comparing Open with Closed, it must be identified in what sense the Open result would not qualify as Closed.
Submitters cannot publish results of a version before its official date; non-submitters must wait two weeks from that date.
The sanctions regime goes as far as barring an infringer from submitting results in the future and marking their results as non-compliant permanently in the database, with a deadline of three working days to withdraw the infringing content.

How much audit sits behind a row

In each round up to two submissions are audited: one chosen at random from all of them and zero or one chosen by the review committee. And there is a constraint that conditions the reading of the whole table: only Available submissions in the Closed division are auditable. Preview, RDI and Open fall outside the process. There is an exemption from the random audit if the system is equivalent to another already audited and neither the aggregate performance nor the per-accelerator performance differ by more than 10 % from those of the previous audit. The deadlines are 28 days to select an auditor, 30 for the report after signing the confidentiality agreements and around 90 days for the complete process, with two days of hardware access for the auditor. The rules close with three sentences that define the standard of proof: results that cannot be replicated are not valid results, benchmark detection is not allowed and input-based optimisation is not allowed either.

The recent rounds, in figures

Round	Organisations	Performance results	Power results	New benchmarks
v4.0	23	more than 8,500	900	2
v4.1	22	964	31	1
v5.0	23	17,457	n/a	4
v5.1	27	n/a	2	3
v6.0	24	n/a	n/a	5 of 11 Datacenter tests

These counts do not form a time series: the counting criteria differ between press releases and none documents the criterion, to the point that v4.0 and v4.1 differ by a factor close to nine.

The published performance jumps are comparable within each announcement. Between v4.0 and v5.0, twelve months apart, the number of Llama 2 70B submissions multiplied by 2.5, the median score doubled and the best score was 3.3 times faster. Between v5.0 and v5.1, in six months, the best systems improved by up to 50 % in some scenarios. Round v6.0 changed axis and its press release talks about scale rather than speed: 30 % more multi-node submissions, 10 % of the systems with more than ten nodes against 2 % in the previous round, and a system larger than 72 nodes and 288 accelerators, quadrupling the previous maximum node count.

Hardware that debuted in each round: MI300X, TPU v6e, Xeon Granite Rapids and B200 in v4.1; MI325X, Xeon 6980P, GB200 and Jetson AGX Thor in v5.0; MI355X, Intel Arc Pro B60, GB300 and RTX Pro 6000 Blackwell Server Edition in v5.1.

What MLPerf Inference does not measure

Cost. There is no cost or TCO metric, and cost is explicitly classified as a derived metric that cannot be presented as official.
Energy, except as an optional extension. MLPerf Power is regulated in a separate document, requires measuring at the wall with the power group’s workflow and PTDaemon, and forbids any other method. Its real adoption is marginal: 2 power submissions in v5.1 against 27 participating organisations. In addition, MLCommons bounds what that figure means: the measured power is only valid for the benchmark it accompanies, and any other reference to power, such as a TDP setting or a power supply rating, is neither measured nor validated by MLCommons. The messaging rule forbids submitters from publishing per-watt normalised comparisons using any proxy other than the measured power.
Configurations not optimised by the vendor. The bias is acknowledged in the audit rule itself, which contemplates submissions whose performance is not consistent with the known characteristics of the hardware, or where the committee lacks visibility into how it was achieved, or where hardware and software are not reasonably available to the public.
Barrier to entry. Only MLCommons members and test partners can submit results for review. Anyone measuring with the same code without submitting must label every figure as unverified with the corresponding legend.

Usage flow for an on-premise platform

MLPerf Inference is useful in a purchasing decision if it is used for what it is, a test bench with written rules, and not as a ranking.

Filter by division and category before looking at any number: keep Closed and Available. A Preview or RDI result describes hardware that cannot yet be bought, and neither of the two is auditable.
Choose the scenario that matches the load pattern: Server or Interactive if there are users waiting, Offline if the work is batch. An Offline result does not authorise promising a latency.
Read the SLO before the throughput: in any LLM benchmark, the number of queries per second only means something alongside its TTFT/TPOT pair. The same machine publishes very different figures under Server and under Interactive.
Check the quality variant: 99 % and 99.9 % are different results for the same model, and the second usually costs performance.
Translate to your own configuration carefully: dividing by the number of accelerators is a derived metric, not an official one, and it has to be declared as such.
Close the gap MLPerf leaves open with your own measurement: cost, energy and behaviour under real traffic. That is exactly the function of the reproducible harness and of the track B bench tools, and the reason no platform decision stands on a public table alone.

Cross-links from the benchmarking track

B1/B2 — LLM inference benchmarking: frameworks, metrics and state of the art: the TTFT, TPOT and goodput metrics outside the MLPerf corset, and which tool measures each one.
B2 — LLM benchmark tool catalogue: the tools with which you reproduce at home what MLPerf standardises.
B3 — GuideLLM and SLO validation under load: the practical equivalent of LoadGen’s binary search in the Server scenario.
B6 — Measurement bias and reproducibility: why LoadGen exists, told from the side of what happens when there is none.
B8 — Serving engines compared on the Pareto frontier: the comparison MLPerf does not make, with engines instead of systems.
C4 — MLPerf Power: the energy extension of this same machinery, with its certified measurement at the wall.

Sources

MLCommons, MLPerf Inference Working Group — https://mlcommons.org/working-groups/benchmarks/inference/
MLCommons, MLPerf Inference Rules (inference_rules.adoc) — https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc
MLCommons, MLPerf Inference Power Measurement (power_measurement.adoc) — https://github.com/mlcommons/inference_policies/blob/master/power_measurement.adoc
MLCommons, MLPerf Submission Rules (submission_rules.adoc) — https://github.com/mlcommons/policies/blob/master/submission_rules.adoc
MLCommons, MLPerf Results Messaging Guidelines — https://github.com/mlcommons/policies/blob/master/MLPerf_Results_Messaging_Guidelines.adoc
MLCommons, LoadGen README — https://github.com/mlcommons/inference/blob/master/loadgen/README.md
MLCommons, loadgen/test_settings.h — https://github.com/mlcommons/inference/blob/master/loadgen/test_settings.h
MLCommons, loadgen/mlperf.conf (master) — https://github.com/mlcommons/inference/blob/master/loadgen/mlperf.conf
MLCommons, loadgen/mlperf.conf (tag v5.1) — https://github.com/mlcommons/inference/blob/v5.1/loadgen/mlperf.conf
MLCommons, loadgen/mlperf.conf (tag v5.0) — https://github.com/mlcommons/inference/blob/v5.0/loadgen/mlperf.conf
MLCommons, tools/submission/submission_checker.py (tag v5.0) — https://github.com/mlcommons/inference/blob/v5.0/tools/submission/submission_checker.py
MLCommons, compliance/nvidia TEST06 README (tag v5.1) — https://github.com/mlcommons/inference/blob/v5.1/compliance/nvidia/TEST06/README.md
MLCommons, Benchmark results: Inference Datacenter — https://mlcommons.org/benchmarks/inference-datacenter/
Reddi et al., MLPerf Inference Benchmark, ISCA 2020 — https://arxiv.org/abs/1911.02549
MLCommons, MLPerf Inference v5.0 Benchmark Results (abril 2025) — https://www.businesswire.com/news/home/20250402313932/en/MLCommons-Releases-New-MLPerf-Inference-v5.0-Benchmark-Results
MLCommons, MLPerf Inference v5.1 Benchmark Results (septiembre 2025) — https://www.globenewswire.com/news-release/2025/09/09/3147136/0/en/MLCommons-Releases-New-MLPerf-Inference-v5-1-Benchmark-Results.html
MLCommons, MLPerf Inference v6.0 Benchmark Results (abril 2026) — https://www.globenewswire.com/news-release/2026/04/01/3266801/0/en/MLCommons-Releases-New-MLPerf-Inference-v6-0-Benchmark-Results.html
MLCommons, MLPerf Inference v4.1 Benchmark Results (agosto 2024) — https://www.businesswire.com/news/home/20240828886616/en/New-MLPerf-Inference-v4.1-Benchmark-Results-Highlight-Rapid-Hardware-and-Software-Innovations-in-Generative-AI-Systems
NVIDIA Developer Blog, NVIDIA Blackwell delivers massive performance leaps in MLPerf Inference v5.0 — https://developer.nvidia.com/blog/nvidia-blackwell-delivers-massive-performance-leaps-in-mlperf-inference-v5-0/
NVIDIA Developer Blog, NVIDIA Blackwell Ultra sets new inference records in MLPerf debut — https://developer.nvidia.com/blog/nvidia-blackwell-ultra-sets-new-inference-records-in-mlperf-debut/
MLCommons, MLPerf Automotive (octubre 2025) — https://arxiv.org/html/2510.27065v1
Tschand et al., MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts — https://arxiv.org/abs/2410.12032

GenAI-Perf in depth: LLM inference profiling with real data

Mon, 15 Jun 2026 03:00:00 +0200

Notation: amounts in euros (N EUR), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

TL;DR

GenAI-Perf (the genai-perf package, part of NVIDIA’s Triton ecosystem) is the reference LLM inference profiler for OpenAI-compatible endpoints. Installable with pip install genai-perf. A single genai-perf profile command produces a console table with TTFT, ITL, request latency (avg/min/max/P75/P90/P99), output sequence length, output token throughput and request throughput, plus profile_export_genai_perf.json and .csv artefacts. The analyze subcommand automatically sweeps concurrencies from 1 to 256 (powers of 2) and generates a summary analyze_export_genai_perf.csv. GenAI-Perf was declared retired in April 2026 (replaced by AIPerf); it remains the reference benchmark for historical runs and documented comparisons, and the NVIDIA tool with the most public documentation for LLMs before 2026.

What GenAI-Perf is and where it fits

GenAI-Perf is a command-line tool for measuring the throughput and latency of generative AI models served through an inference server. It is part of the triton-inference-server/perf_analyzer GitHub repository and is distributed as a pip package (genai-perf) and inside the Triton SDK container (nvcr.io/nvidia/tritonserver:YY.MM-py3-sdk).

Position in the NVIDIA ecosystem:

Relationship with perf_analyzer: GenAI-Perf internally uses Triton’s perf_analyzer binary to generate load and measure latencies. perf_analyzer is the low-level load generator; GenAI-Perf is the high-level layer that adds prompt synthesis, OpenAI payload construction and the LLM-specific metrics (TTFT, ITL, sequence lengths).

Installation:

# Option 1: pip (requires CUDA 12, Ubuntu 24.04, Python 3.10+)
pip install genai-perf

# Option 2: Triton SDK container (recommended for reproducibility)
export RELEASE="25.01"
docker run -it --net=host --gpus=all \
 nvcr.io/nvidia/tritonserver:25.01-py3-sdk
# Inside the container:
genai-perf --help

Metrics GenAI-Perf produces

The metrics cover the full cycle of a streaming generation request. All of them are reported with avg, min, max, P75, P90 and P99 except the throughput ones (avg only).

Metrics glossary

Metric	Symbol	Exact definition (GenAI-Perf)	Unit
Time to First Token	TTFT	Time from sending the request to receiving the first token (includes queueing, prefill and network)	ms
Inter-Token Latency	ITL (= TPOT)	$\frac{e2e\_latency - TTFT}{output\_tokens - 1}$ — decode phase only, excludes the first token	ms/token
Request Latency	e2e_latency	$e2e = TTFT + Generation\_time$ — from the first request to the last response	ms
Output Token Throughput	OTT	$\frac{total\_output\_tokens}{T_y - T_x}$ where $T_x$ = first request, $T_y$ = last token received	tok/s
Request Throughput	RPS	$\frac{total\_completed\_requests}{T_y - T_x}$	req/s
Input Sequence Length	ISL	Mean length in tokens of the input prompt	tokens
Output Sequence Length	OSL	Mean length in tokens of the generated response	tokens

Methodological note on ITL versus LLMPerf: GenAI-Perf excludes TTFT from the ITL calculation (pure decode only). LLMPerf includes TTFT in its ITL mean. The numbers are not directly comparable.

Methodological note on TPS versus LLMPerf: GenAI-Perf measures throughput between the first request and the last token received ($T_y - T_x$). LLMPerf uses the total duration of the benchmark including prompt generation and response storage; in a concurrency-1 scenario, that difference can amount to up to 33% in the reported throughput figure.

Console output table (real example, TRT-LLM backend)

The table GenAI-Perf prints for a profile with ISL=200, OSL=100, concurrency=1:

 NVIDIA GenAI-Perf | LLM Metrics
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
┃ Statistic ┃ avg ┃ min ┃ max ┃ p99 ┃ p90 ┃ p75 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
│ Time to first token (ms) │ 13.68 │ 11.07 │ 21.50 │ 18.81 │ 14.29 │ 13.97 │
│ Inter token latency (ms) │ 1.86 │ 1.28 │ 2.11 │ 2.11 │ 2.01 │ 1.95 │
│ Request latency (ms) │ 203.70 │ 180.33 │ 228.30 │ 225.45 │ 216.48 │ 211.72 │
│ Output sequence length │ 103.46 │ 95.00 │ 134.00 │ 122.96 │ 108.00 │ 104.75 │
│ Input sequence length │ 200.00 │ 200.00 │ 200.00 │ 200.00 │ 200.00 │ 200.00 │
│ Output token throughput (per sec) │ 504.02 │ N/A │ N/A │ N/A │ N/A │ N/A │
│ Request throughput (per sec) │ 4.87 │ N/A │ N/A │ N/A │ N/A │ N/A │
└───────────────────────────────────┴────────┴────────┴────────┴────────┴────────┴────────┘

(Source: docs.nvidia.com/deeplearning/triton-inference-server — LLM tutorial)

Invocation against OpenAI-compatible endpoints

`profile` mode: one load point

The profile subcommand measures a fixed operating point (concurrency or request-rate, synthetic ISL/OSL).

Example 1: vLLM endpoint (chat, streaming)

# 1. Bring up the vLLM server
docker run -it --net=host --rm --gpus=all \
 vllm/vllm-openai:latest \
 --model meta-llama/Llama-3.1-70B-Instruct \
 --dtype float16

# 2. Profile with GenAI-Perf
genai-perf profile \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --concurrency 16 \
 --synthetic-input-tokens-mean 1024 \
 --synthetic-input-tokens-stddev 128 \
 --output-tokens-mean 256 \
 --output-tokens-stddev 0 \
 --output-tokens-mean-deterministic \
 --request-count 200 \
 --warmup-request-count 20 \
 --tokenizer meta-llama/Llama-3.1-70B-Instruct

Example 2: completions endpoint (no streaming)

genai-perf profile \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type completions \
 --concurrency 32 \
 --synthetic-input-tokens-mean 512 \
 --synthetic-input-tokens-stddev 64 \
 --output-tokens-mean 128 \
 --request-count 200 \
 --warmup-request-count 20

Example 3: NVIDIA NIM (TRT-LLM backend endpoint, via Triton)

genai-perf profile \
 -m meta/llama-3.1-70b-instruct \
 --backend tensorrtllm \
 --streaming \
 --concurrency 8 \
 --synthetic-input-tokens-mean 2048 \
 --synthetic-input-tokens-stddev 0 \
 --output-tokens-mean 512 \
 --output-tokens-mean-deterministic \
 --request-count 100 \
 --warmup-request-count 10 \
 --generate-plots

Example 4: request-rate instead of concurrency (Poisson)

genai-perf profile \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --request-rate 5.0 \
 --synthetic-input-tokens-mean 1024 \
 --output-tokens-mean 256 \
 --request-count 300 \
 --warmup-request-count 30

Main flags and what they do

Flag	Typical values	Function
`--endpoint-type`	`chat`, `completions`, `embeddings`	Type of OpenAI-compatible endpoint
`--streaming`	(boolean)	Enables SSE streaming; required to measure real TTFT and ITL
`--concurrency`	1, 8, 16, 32, 64	Number of concurrent requests maintained; GenAI-Perf guarantees N active at all times
`--request-rate`	1.0, 5.0, 10.0	Constant arrival rate (req/s); does not guarantee N active
`--synthetic-input-tokens-mean`	128–8192	Mean of synthetic ISL tokens
`--synthetic-input-tokens-stddev`	0–512	Standard deviation of the ISL (0 = fixed)
`--output-tokens-mean`	64–2048	Mean of target OSL tokens
`--output-tokens-mean-deterministic`	(boolean)	Sets the minimum output tokens = target mean (more precise with TRT-LLM)
`--request-count`	100–2000	Number of requests to benchmark
`--warmup-request-count`	10–50	Warm-up requests discarded from the metrics
`--generate-plots`	(boolean)	Generates PNG plots of TTFT vs ISL, ITL vs token position, etc.
`--tokenizer`	HF model id	Tokenizer for counting tokens; mandatory when ISL/OSL matter
`--backend`	`tensorrtllm`, `vllm`	Triton backend (for serving directly via Triton without an OpenAI endpoint)
`--extra-inputs`	`ignore_eos:true`	Extra request parameters; `ignore_eos:true` guarantees a consistent OSL

Note on --concurrency versus --request-rate: NVIDIA recommends using --concurrency. With --request-rate, if the rate exceeds the engine’s throughput, the queue grows without bound and the metrics stop being stable. With --concurrency, there are always exactly N active requests, which gives a clean latency-throughput curve.

Output artefacts

GenAI-Perf dumps all results into an artifacts/ directory:

Artefact	Format	Content
`profile_export_genai_perf.json`	JSON	Full metrics (avg, min, max, P75, P90, P99) + CLI arguments used
`profile_export_genai_perf.csv`	CSV	Exported console tables, ready to import into Excel/pandas
`profile_export.json`	JSON	Raw perf_analyzer data (per-request traces)
`inputs.json`	JSON	Synthetic payloads sent (for reproducibility)
PNG plots	PNG	TTFT analysis, Request latency, TTFT vs ISL, ITL vs position, ISL vs OSL (with `--generate-plots`)

The `analyze` subcommand: automatic concurrency sweep

The analyze subcommand sweeps multiple values of a parameter in a single command and generates a summary CSV.

Concurrency sweep (the most used)

# Sweeps concurrencies 1, 2, 4, 8, 16, 32, 64, 128, 256
genai-perf analyze \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --sweep-type concurrency \
 --sweep-range 1:256 \
 --synthetic-input-tokens-mean 1024 \
 --output-tokens-mean 256 \
 --request-count 200 \
 --warmup-request-count 20

Request-rate sweep

# Sweeps 2, 4, 6, 8, 10, 12 req/s
genai-perf analyze \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --sweep-type request_rate \
 --sweep-list 2,4,6,8,10,12 \
 --synthetic-input-tokens-mean 1024 \
 --output-tokens-mean 256 \
 --request-count 200 \
 --warmup-request-count 20

ISL sweep (to study the effect of prefill)

genai-perf analyze \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --sweep-type input_sequence_length \
 --sweep-list 256,512,1024,2048,4096 \
 --concurrency 16 \
 --output-tokens-mean 256 \
 --request-count 200 \
 --warmup-request-count 20

Artefact structure of `analyze`

For a sweep with --sweep-type concurrency --sweep-range 1:32:

artifacts/
llama70b-openai-chat-concurrency1/
inputs.json
profile_export.json
profile_export_genai_perf.json
profile_export_genai_perf.csv
llama70b-openai-chat-concurrency2/
...
llama70b-openai-chat-concurrency4/
...
[...]
analyze_export_genai_perf.csv ← summary of every scenario
checkpoint.json ← allows interrupted sweeps to be resumed

Summary CSV format

Config Name,Concurrency,ISL,p99 TTFT (ms),p99 ITL (ms),p99 Request Latency (ms),Avg. OTT (tok/s),RPS
llama70b_run_0,1,1024,45.2,6.8,1823.4,128.3,0.55
llama70b_run_1,4,1024,48.1,7.1,1901.2,510.7,2.18
llama70b_run_2,8,1024,61.3,7.9,2142.5,989.2,4.12
llama70b_run_3,16,1024,98.7,9.4,2891.3,1741.6,6.88
llama70b_run_4,32,1024,284.1,14.2,4320.8,2134.2,7.14
llama70b_run_5,64,1024,892.4,28.7,8741.2,2251.8,7.23

The summary CSV also includes a second table with GPU metrics (P99 power, energy, utilisation, memory) when they are captured via Triton’s telemetry URLs (--server-metrics-urls http://localhost:8002/metrics).

Comparison table: GenAI-Perf versus GuideLLM versus LLMPerf versus vLLM bench

The choice of tool changes the number. They are not comparable with each other without adjustment.

Dimension	GenAI-Perf	GuideLLM	LLMPerf	vLLM bench serve
Origin / maintainer	NVIDIA / Triton team (retired Apr 2026)	Red Hat / vLLM project	Anyscale / Ray	vLLM project
Class	multi-process load generator	multi-process load generator	multi-process load generator	single-process micro-bench
Supported endpoints	OpenAI-compatible + KServe + native Triton	OpenAI-compatible	OpenAI-compatible	native vLLM
Main load mode	fixed concurrency N (recommended) or constant request-rate	synchronous, concurrent, poisson, throughput, automatic sweep	batches of N concurrent (with a draining period at the end)	request-rate, concurrency
Draining period	NO — guarantees N active at all times	NO (poisson)	YES — at the end of each batch the system empties, concurrency drops to 0	N/A
Automatic sweep	`analyze` (concurrency, request-rate, ISL, OSL)	`--rate-type sweep` (idle to saturation, 10 rounds)	manual (several runs)	`vllm bench sweep serve`
LLM metrics	TTFT, ITL, e2e latency, OTT, RPS, ISL/OSL	TTFT, ITL/TPOT, throughput, goodput under SLO	TTFT, ITL (includes TTFT in the mean), TPS (total benchmark duration)	TTFT, TPOT, throughput
ITL difference	excludes TTFT: $(e2e - TTFT) / (N_{tok}-1)$	excludes TTFT (same as GenAI-Perf)	includes TTFT in the mean	excludes TTFT
TPS difference	$total\_tokens / (T_y - T_x)$	throughput + goodput under SLO	$total\_tokens / (T_{end} - T_{start})$ — up to 33% lower	$total\_tokens / (T_y - T_x)$
Goodput / SLO	NO	YES (the key differentiator)	NO	NO
Exports	JSON + CSV + PNG + checkpoint	JSON + YAML + CSV + interactive HTML	JSON	console / JSON
Built-in warm-up	YES (`--warmup-request-count`)	YES (warm-up requests)	NOT native	YES
Real datasets	OpenOrca, CNN DailyMail, JSONL file, moon_cake	file with a traffic distribution	JSONL file	sharegpt, random
GPU telemetry	YES (via Triton’s server-metrics-urls → summary CSV)	NOT native (integrate DCGM separately)	NO	NO
When to choose it	fine profiling of one operating point, comparing configs with the same harness, integration with NIM/Triton	SLO-driven sweep, finding the knee, sizing replicas	quick endpoint validation in the Ray ecosystem	iterating on vLLM flags

Fair-comparison rule: never cross numbers from different tools for the same conclusion. If vLLM and TRT-LLM are being compared, use GenAI-Perf against both, with the same ISL/OSL, the same concurrency and the same --warmup-request-count. The tool is the constant; the engine is the variable.

Profiling on 4×H100 SXM 80 GB: a full sweep example

Generic reference hardware: 4×H100 SXM 80 GB NVLink, model Llama-3.1-70B-Instruct FP16, ISL=1024 tokens, OSL=256 tokens, vLLM endpoint.

Illustrative result of a concurrency sweep (1→64)

Concurrency	TTFT P50 (ms)	TTFT P99 (ms)	ITL P50 (ms)	ITL P99 (ms)	OTT (tok/s)	RPS
1	38	45	6.4	7.1	130	0.51
4	40	51	6.8	8.2	510	2.00
8	46	62	7.3	9.4	980	3.83
16	68	98	8.9	12.1	1,720	6.72
32	145	284	13.7	21.4	2,130	8.32
48	380	740	22.3	38.6	2,280	8.90
64	890	1,820	41.2	78.3	2,340	9.14

How to read it: throughput saturates around concurrency 48–64, while TTFT P99 already exceeds 500 ms from concurrency 32 onwards. The knee (the last point with TTFT P99 < 500 ms) is at concurrency 16 with 1,720 useful tok/s. Anyone reporting “2,340 tok/s” (concurrency 64) is describing the maximum throughput, at a point where the system meets no interactive chat SLO at all.

Formulas for the key metrics

$$ \text{ITL} = \frac{e2e\_latency - TTFT}{N_{output\_tokens} - 1} $$ $$ \text{OTT} = \frac{\sum output\_tokens}{T_y - T_x} $$ $$ \text{RPS} = \frac{N_{requests}}{T_y - T_x} $$

where $T_x$ is the timestamp of the first request sent and $T_y$ the timestamp of the last token received from the last request.

Synthetic ISL/OSL profiling: what to configure for each use case

The ISL/OSL parameters determine which engine of the system is stressed: a long ISL stresses prefill (KV-cache, TTFT); a long OSL stresses decode (ITL, memory bandwidth).

Use case	Typical ISL	Typical OSL	`--synthetic-input-tokens-mean`	`--output-tokens-mean`
Short interactive chat	~300 tok	~100 tok	300	100
Code copilot	~800 tok	~200 tok	800	200
Document summarisation	~2000 tok	~256 tok	2000	256
RAG with long context	~4096 tok	~512 tok	4096	512
Batch without latency	~512 tok	~1024 tok	512	1024

The --extra-inputs ignore_eos:true option disables the engine’s EOS token, forcing exactly the requested tokens to be generated. Essential for measuring a consistent OSL on TRT-LLM; optional on vLLM.

Honest methodology: how not to measure badly

1. Warm-up is mandatory

Without warm-up, the first requests include the KV-cache loading latency, the engine’s autotuning and the initialisations. GenAI-Perf applies a sliding window to detect stability (max/min ratio within a margin over the last 3 measurements), but the explicit --warmup-request-count discards the first N requests from the metrics. Recommended minimum value: 10 requests (or 5% of the total if the total is large).

2. The concurrency knee: extend the sweep until you see it

Throughput saturates once the engine’s effective max batch size is reached. If the sweep stops before saturation is reached, the “maximum capacity” reported is the last concurrency tested, not the real knee. The rule: the sweep must reach a point where TTFT P99 has doubled relative to the low-load value (or where OTT stops growing by more than 5% between steps). Only then is the knee visible.

For a sweep with --sweep-range 1:256, GenAI-Perf sweeps 1, 2, 4, 8, 16, 32, 64, 128, 256 (powers of 2). On most 70B models over 4×H100, saturation occurs between concurrency 32 and 64; the range 1:128 covers the knee with room to spare.

3. Concurrency versus request-rate: choosing correctly

Mode	Behaviour	When to use
`--concurrency N`	Always N active requests; when one finishes, the next is sent	Clean latency-throughput curve; comparison between configs
`--request-rate R`	1 request is sent every 1/R seconds; the queue can grow without bound	Simulating traffic with arrivals at a constant rate

NVIDIA recommends --concurrency for benchmarking. With --request-rate, if R exceeds the engine’s maximum throughput, the queue grows indefinitely and the metrics become unstable.

4. Single-process versus multi-process bias

GenAI-Perf is multi-process: it maintains N concurrent requests using perf_analyzer’s internal manager. There is no risk of client saturation. This sets it apart from vllm bench serve (single-process), which can saturate on the client at high concurrency and give inflated throughput metrics.

5. `ignore_eos` and an inconsistent OSL

Without --extra-inputs ignore_eos:true, the engine may generate fewer tokens than requested (if the model produces an EOS earlier). This makes the real OSL vary between requests and the ITL become inconsistent between runs. For reproducible benchmarks with a controlled OSL: always enable ignore_eos:true.

6. The correct tokenizer

The --tokenizer flag must point to the model being served. If it is omitted, GenAI-Perf uses a default tokenizer that can give different ISL/OSL in the real model’s tokens, making the throughput numbers in tok/s not comparable between models with different vocabularies.

7. One run = a data point; three or more = a number

Variance between runs under identical conditions can be 5–10% on tail latency metrics (P99). A single profile is not a defensible number; the mean of three runs with the same config is.

Connection with the cost and energy track

GenAI-Perf can capture GPU metrics in the same summary CSV if --server-metrics-urls http://localhost:8002/metrics is passed (Triton’s metrics endpoint, which exposes power and utilisation via DCGM). The analyze summary CSV then includes a second table with P99 power (W), energy (MJ), utilisation (%) and memory (GB) per GPU and per scenario.

With that data, a single sweep yields the three axes of the scorecard:

Axis	Source	Metric
Performance	GenAI-Perf	OTT (tok/s), TTFT P99, ITL P99
Energy	DCGM via server-metrics-urls	Power (W) → $J/token = W / OTT$
Cost	node price	EUR/hour → $EUR/token = EUR_{hour} / (OTT \times 3600)$

The formula for cost per million tokens:

$$ CPM = \frac{EUR_{hour}}{OTT \times 3{.}6 \times 10^3} $$

where $OTT$ is the output token throughput in tok/s and $CPM$ is the cost per million tokens in EUR.

It connects with GPU observability with DCGM for the power capture, and with anatomy of an LLM request for the breakdown of phases (prefill versus decode) that TTFT and ITL quantify.

Status 2026: GenAI-Perf → AIPerf

NVIDIA announced in April 2026 that GenAI-Perf moves to passive maintenance mode (no new features) and that the active successor is AIPerf (github.com/ai-dynamo/aiperf). AIPerf adds automatic detection of the saturation point (estimatedCapacity) and is integrated into the NVIDIA Dynamo pipeline.

For new benchmarks in production: migrate to AIPerf. To reproduce historical runs documented with GenAI-Perf, or to use NVIDIA’s NIM benchmarking artefacts (which use GenAI-Perf): stay on GenAI-Perf 25.x.

The article LLM benchmark tools, one by one covers the full comparison including AIPerf. The article GuideLLM: SLO validation under load details the SLO-driven sweep that GenAI-Perf does not do natively. For the general methodology context: LLM benchmarking frameworks, state of the art.

Sources

NVIDIA · GenAI-Perf — README oficial — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/README.html
NVIDIA · Tutorial LLM con GenAI-Perf (comandos y tablas de salida reales) — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/docs/tutorial.html
NVIDIA · GenAI-Perf Analyze subcommand (sweep, CSV resumen, checkpoint) — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/docs/analyze.html
NVIDIA · NIM LLM Benchmarking — Métricas (definiciones TTFT, ITL, TPS, RPS, fórmulas exactas) — https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html
NVIDIA · NIM LLM Benchmarking — Parámetros y buenas prácticas (ISL/OSL, concurrencia vs request-rate, ignore_eos) — https://docs.nvidia.com/nim/benchmarking/llm/latest/parameters.html
NVIDIA Technical Blog · Measuring Generative AI Model Performance Using NVIDIA GenAI-Perf and an OpenAI-Compatible API — https://developer.nvidia.com/blog/measuring-generative-ai-model-performance-using-nvidia-genai-perf-and-an-openai-compatible-api/
GitHub · triton-inference-server/perf_analyzer (genai-perf) — https://github.com/triton-inference-server/perf_analyzer/blob/main/genai-perf/README.md
GitHub · ray-project/llmperf (métricas y diferencias metodológicas) — https://github.com/ray-project/llmperf
PyPI · nvidia-genai-perf-eval — https://pypi.org/project/nvidia-genai-perf-eval/
Macnica · Benchmarking LLM Applications Part 1: What is GenAI-Perf? — https://www.macnica.co.jp/en/business/semiconductor/articles/nvidia/145977/

LLM inference benchmarking: frameworks, metrics and the state of the art (tool by tool)

Sat, 13 Jun 2026 02:30:00 +0200

Notation: amounts in euros (N €), decimals with a point. Performance is not very sensitive to the country, but its associated cost (cost per token) is expressed in € and links back to the opening article.

What this introduction covers

Third article in the data series, a deep dive into the performance axis. Measuring the performance of an inference engine looks trivial (“how many tokens per second?”) and it is exactly where people fool themselves most: two tools can report results that differ by a factor of 7 for the same system. This article inventories the metrics that matter and how they are defined, why the architecture of the tool biases the figure, how the saturation point is found with a concurrency sweep, and the profile of each framework. No recommendations: the choice of engine is settled in the Pareto article (B8); here there are only the facts and the methodology, because a benchmark without a published methodology is not comparable.

The performance metrics

There is no single performance metric, there are five, and mixing them is the first source of error:

Metric	Definition	Unit	Dominant phase
TTFT (Time To First Token)	time from sending the prompt to the first token	ms	prefill
TPOT / ITL (Time Per Output Token / Inter-Token Latency)	average time between output tokens once generation has started	ms/token	decode
Request throughput	complete request-response cycles per second at the concurrency tested	req/s	both
Token throughput	total tokens (input + output) per second across all concurrent requests	tok/s	both
Goodput	share of requests that meet the defined SLO	useful tok/s	both
P50 / P95 / P99	latency percentiles (not the mean)	ms	—

Precise definitions, because every tool computes them its own way (Anyscale · latency and throughput metrics): TTFT is what a user waits before seeing the first character, dominated by prefill compute; ITL is the average time between successive output tokens and sets the perceived “typing speed” of the answer; request throughput is complete cycles per second; token throughput is total tokens (input plus output) per second across all concurrent requests.

The latency decomposition

The total latency of a request with $N$ output tokens decomposes as:

$$\text{latency} \approx \text{TTFT} + (N-1)\times \text{TPOT}$$

That is why TTFT and TPOT are reported separately: a single mean latency hides very different profiles. A system with high TTFT and low TPOT (expensive prefill, fast decode) and another one the other way round can have the same mean latency for one particular length, yet behave in opposite ways when the response size changes. For an interactive chat experience TTFT and TPOT rule; for a batch of long summaries, token throughput does. Measuring the mean hides both realities.

Goodput: the honest metric

Raw throughput (TPS, RPS) says how much work the system does; goodput says how much of that work meets your quality-of-service standards (SLO) (Anyscale). An engine can boast 10,000 aggregate tok/s, but if half the requests violate the P99 TTFT SLO, its goodput is 5,000. The figure you defend in a proposal is the goodput, not the catalogue throughput: it is the only one that translates into satisfied users and an honest cost per token.

How each metric is instrumented (and where the error creeps in)

Before comparing numbers it helps to know where each clock starts and stops, because two tools can call different things “TTFT”:

Client-side versus server-side TTFT. The TTFT measured by the client includes network latency and gateway queueing; the one measured by the server does not. For an engine comparison the server-side figure is the one that matters; for user experience, the client-side one. Mixing the two invalidates the comparison.
Streaming is required. TTFT and ITL can only be measured if the response arrives as a stream (token by token). If the tool measures complete responses, there is no real TTFT: there is total latency in disguise.
Token counting. Throughput in tok/s depends on which tokenizer counts the tokens. If the tool uses a tokenizer different from the model’s, the token count (and therefore the tok/s and the cost per token) is biased. You have to count with the tokenizer of the model being served.
Warm-up and prefix cache. The first requests of a benchmark benefit from a warm prefix cache and give artificially low TTFT; the warm-up has to be discarded or the result inflates reality.

These four instrumentation decisions explain a good share of the discrepancies between tools. A performance figure that does not say where the clock is measured and with which tokenizer is not comparable, however precise it may look.

The architecture of the tool biases the figure

Here is the trap that invalidates half the published benchmarks. Tools split into two classes by the architecture of the client that generates the load, and that architecture determines whether the measurement is reliable at high concurrency:

Single-process micro-bench (vLLM bench, SGLang bench, genai-perf): a Python client with asyncio in a single process. Useful for quick experiments on one engine, but the single-process architecture introduces a bottleneck on the client side that biases the data at high concurrency (genAI-perf and vLLM): the client cannot generate enough load and you measure the limit of the client, not the limit of the engine.
Multi-process load (GuideLLM, AIPerf): they spread load generation across several processes, avoiding that limit. This is the class that has emerged for measuring at real scale.

The size of the bias is enormous: at 1,000 QPS, a single-process benchmark processed 75,574 tokens against the 545,733 tokens of a distributed architecture, a discrepancy of 7.2× in measurement capacity for the same system ([search]). Anyone comparing two engines with tools from different classes is not comparing the engines: they are comparing the benchmark clients.

Frameworks, one by one

vLLM bench and SGLang bench — engine micro-bench

What they measure: TTFT, TPOT and throughput of the engine itself (vLLM or SGLang). Class: single-process micro-bench. Use: quick experiments to tune one engine and see the effect of its optimisations (see decode and prefill). Limit: they saturate on the client at high concurrency; they are no good for measuring real capacity at scale.

AIPerf — the NVIDIA one (formerly genai-perf), multi-process

What it measures: TTFT, ITL, throughput and latency against vLLM, NIM, TGI and any compatible endpoint. Class: multi-process load. State-of-the-art datum: NVIDIA retired genai-perf and replaced it with AIPerf on 15 April 2026. During the sweep, AIPerf detects GPU saturation and identifies the previous iteration, returning it as estimatedCapacity; if it detects no saturation, estimatedCapacity is the last iteration tested, which is why the sweep has to extend beyond the knee (AIPerf).

GuideLLM — from the vLLM project, SLO-oriented

What it measures: full distributions of TTFT, ITL and end-to-end behaviour, for SLO-driven evaluation. Class: multi-process load. Differentiator: it generates realistic, configurable traffic patterns in synchronous, concurrent and rate-based modes, including reproducible sweeps to identify safe operating ranges (Red Hat · GuideLLM, GuideLLM · GitHub). It is the tool for answering “how hard can I load this engine without breaking the SLO?”.

LLMPerf — the Anyscale/Ray classic

What it measures: throughput and latency at the inference level. Use: endpoint validation, historically very widespread. Class: load generator. Limit: less focused on distributions and sweeps than GuideLLM/AIPerf.

MLPerf Inference — the industry standard

What it measures: performance under standardised scenarios with strict rules, for comparability across vendors. Maintainer: MLCommons. It is the gold standard of cross-vendor comparability; it is expanded on below.

Comparison table

Tool	Class	What it measures	Maintainer	When to use it
vLLM bench	single-process micro	TTFT, TPOT, throughput of vLLM	vLLM (OSS)	tune vLLM, quick experiments
SGLang bench	single-process micro	metrics of the SGLang engine	SGLang (OSS)	tune SGLang
AIPerf (formerly genai-perf)	multi-process load	TTFT, ITL, throughput; estimatedCapacity	NVIDIA (OSS)	real capacity, multi-endpoint
GuideLLM	multi-process load	distributions, SLO, sweeps	vLLM (OSS)	validate SLO, find the knee
LLMPerf	load	throughput and latency	Anyscale/Ray (OSS)	endpoint validation
MLPerf Inference	standard suite	server/offline/interactive scenarios	MLCommons	cross-vendor comparability

The concurrency sweep: finding the knee

The most useful measurement for sizing is not a number, it is a curve: how latency and throughput change as concurrency rises. Raising concurrency keeps the GPU busier and raises RPS, but past a certain point it sends TTFT, ITL and end-to-end latency through the roof ([search]). The goal of the sweep is to find the knee: the maximum concurrency where throughput still rises without latency breaking the SLO.

That is why AIPerf extends the sweep beyond the knee: only by seeing where latency takes off can it return the safe capacity (estimatedCapacity). This curve is the raw material of capacity planning: the knee yields the replica count and the cost per token at the target load.

Worked example: reading a sweep

An illustrative sweep on an example node (a 70B on 8×H100, SLO of P99 TTFT < 500 ms), to see how the knee is read:

Concurrency	RPS	TTFT P50 (ms)	TTFT P99 (ms)	Token tput (tok/s)	Goodput
1	2	80	110	350	100 %
8	14	110	240	2,100	100 %
16	22	180	460	3,400	98 %
24	26	320	980	3,900	62 %
32	27	540	1,800	4,000	20 %

Reading: up to concurrency ~16 throughput grows and P99 stays under the SLO (goodput ~100 %). Between 16 and 24 sits the knee: throughput barely rises any more (3,400 → 3,900 tok/s) but P99 takes off (460 → 980 ms) and goodput collapses (98 % → 62 %). At concurrency 32 raw throughput is at its maximum (4,000 tok/s) but goodput is 20 %: the system “performs well” while serving mostly requests that violate the SLO. The defensible safe capacity is the one at concurrency ~16, not the one at maximum throughput. This is the number that goes into capacity planning and into cost per token: at 3,400 useful tok/s, not 4,000 raw tok/s.

MLPerf Inference: the comparability standard

To compare across vendors and engines with identical rules there is MLPerf Inference (MLCommons). The datacenter category focuses on two scenarios plus an optional one (MLCommons · datacenter):

Scenario	What it simulates	Metric
Offline	raw throughput processing the whole dataset in batch	maximum throughput
Server	interactive environment: requests one at a time following Poisson at a mean RPS	RPS under TTFT and TPOT limits
Interactive (optional)	like server but with stricter latency limits	RPS under a hard SLO

The Server scenario is the realistic one for online inference: the load generator sends requests following a Poisson distribution and demands that specific TTFT and TPOT bounds be met. MLPerf Inference v5.0 (April 2025) introduced a large-scale 405B benchmark and a low-latency interactive 70B one, offering language benchmarks at every scale (7B to 405B), architectural diversity (including MoE) and scenarios (MLCommons · v5.0); v5.1 (September 2025) widened the results with record participation (MLCommons · v5.1).

The value of MLPerf is comparability: everyone measures the same thing under the same rules. Its limit is that those rules may not match your workload (your length distribution, your specific SLO), so it serves to compare hardware and engines against each other, not necessarily to size your case. For that, your own sweep.

Measurement bias and reproducibility

That two benchmarks give very different results for the same system is no accident: systematic measurement bias in production benchmarks is characterised in the literature (arXiv 2605.24217), and there is work dedicated to the meta-metrics and good practices of system-level performance benchmarking (arXiv 2508.10251). The most common sources of bias:

Source of bias	Effect	Mitigation
Single-process client	underestimates throughput at high concurrency	use multi-process load
Unrealistic length distribution	results that do not apply to your traffic	use realistic traces
Measuring the mean instead of percentiles	hides the latency tail	report P95/P99
Uncontrolled warm-up	the prefix cache inflates the first results	discard the warm-up
Not pinning engine/model version	irreproducible	pin everything and publish it

The methodological conclusion of this article: a benchmark without a published methodology is not comparable. For a performance figure to support a proposal it has to come with the tool, its version, the model and precision, the load distribution and the SLO. Synthesis article S4 builds a reproducible harness that pins all of that.

Checklist for a reproducible benchmark

For a performance figure to be defensible before a committee, it has to come with everything that allows it to be reproduced. The minimum published alongside the result:

What to pin	Why
Tool + version	each one measures differently; the version changes behaviour
Model + precision (FP16/FP8/INT4)	precision changes throughput and quality
Hardware (GPU, count, interconnect)	an 8×H100 NVLink is not an 8×H100 PCIe
Engine + version + flags	vLLM/SGLang/TRT-LLM and their configuration
Length distribution (in/out)	real traffic is not fixed length
Concurrency levels of the sweep	you have to go past the knee
SLO (which percentile, which threshold)	it defines the goodput
Warm-up handling	discard it or it biases the result
Tokenizer used for counting	it affects tok/s and cost per token

The practical rule: if you cannot hand over this table alongside the figure, the figure is not a datum, it is an anecdote. The reproducible harness of article S4 automates the recording of all these parameters so that anyone, including whoever challenges the proposal, can reproduce the exact number.

Performance ≠ quality

One warning that avoids the most expensive mistake: these tools measure speed and throughput, not accuracy. An engine can be blisteringly fast while serving a model that answers badly. Quality is measured with another family of tools (lm-evaluation-harness, HELM, task leaderboards) which is another axis of the scorecard (article B7). Confusing “fast” with “good” is how you build a platform that serves bad answers very quickly. On the final Pareto frontier, performance and quality are two distinct axes that must be looked at together, never one instead of the other.

The other family of tools, for reference (expanded in B7):

Tool	What it measures	Common trap
lm-evaluation-harness	accuracy on hundreds of standardised tasks	contamination of the test dataset
HELM	holistic evaluation (accuracy, robustness, bias, efficiency)	heavy to run
LiveBench / dynamic leaderboards	tasks that rotate to avoid contamination	comparability over time

The data point: contamination, the model having seen the test during training, inflates quality metrics just as warm-up inflates performance ones. That is why dynamic leaderboards rotate their questions. Quality and performance share that lesson: the measurement method biases the result as much as the system being measured.

The latency versus throughput trade-off

A property the sweep reveals and which is worth keeping in mind: latency and throughput pull in opposite directions. Batching groups requests to amortise the cost of moving the weights out of VRAM, which raises throughput, but each request waits for the batch to form, which raises individual latency. There are two operating regimes, and the benchmark serves to place you in the right one:

Regime	Optimises	Configuration	Case
Latency	low TTFT/TPOT	small batch, low concurrency	interactive chat, copilots
Throughput	maximum tok/s	large batch, high concurrency	overnight batch, ingestion

There is no single “best” point: there is the best point for your SLO. A benchmark that reports only maximum throughput is describing the throughput regime and ignoring whether that point meets the latency your case needs. That is why goodput, throughput under the SLO, is the metric that reconciles the two regimes: it measures how much throughput you get without stepping outside acceptable latency. The sweep traverses the curve between both regimes; your SLO marks where on that curve your system sits.

The connection with cost and energy

Performance is not an isolated axis: by the identity of the opening article, throughput is the denominator of cost per token and of energy per token. A sweep that finds a knee at 4,000 tok/s rather than 2,800 is not just “faster”: it lowers the CPM from ~1.09 to ~0.76 €/1M tok and the energy per token in the same proportion, on the same iron. That is why performance benchmarking is the tool that, indirectly, moves cost the most: every goodput improvement translates into euros and into watts per token. The number that connects the three axes is goodput, the throughput that meets the SLO, not catalogue throughput.

State of the art 2026

genai-perf → AIPerf migration (15 Apr 2026): NVIDIA consolidates its benchmarking into a multi-process tool with saturation detection.
GuideLLM as the OSS standard for SLO-driven evaluation with reproducible sweeps.
MLPerf Inference v5.0/v5.1 extends to 405B, interactive 70B and MoE, with record participation: cross-vendor comparability is maturing.
Measurement bias characterised: the community accepts that the measurement method matters as much as the system measured; the emphasis on reproducibility and meta-metrics is growing.

Limits and traps (data-driven)

Comparing tools from different classes. A single-process micro-bench and a multi-process load generator are not comparable; the difference can be 7×. Pin the class.
Catalogue throughput instead of goodput. The honest number is the one that meets the SLO.
Means instead of percentiles. The mean hides the tail; report P95/P99.
Not extending the sweep beyond the knee. Without seeing where latency takes off you do not know the safe capacity.
Confusing performance with quality. They are distinct axes; fast is not good.
Not pinning versions. Engine, model, precision and load left unpinned = irreproducible = not defensible.

The next article in the track (B2) goes into the tool catalogue in depth; this one pins the metrics and the methodology. With performance measured reproducibly, the scorecard can cross it with cost (in €) and energy for the final decision.

Closing

Performance benchmarking looks like the most “objective” of the three axes (they are tokens per second, after all) and it is exactly where the most manipulation happens, almost always without bad intent: a single-process tool here, a mean instead of a P99 there, a catalogue throughput instead of the goodput. The difference between a marketing number and a defensible datum lies not in the engine measured but in the methodology: the class of tool, where the clock is measured, with which tokenizer, how far the sweep goes and which SLO defines the goodput. For a sovereign architecture proposal, performance only counts if it is delivered with that reproducibility record, and, crossed with cost in euros and energy per token, it becomes the column of the Pareto frontier that decides which engine and which configuration hold up the platform. The number you defend is not the highest one: it is the reproducible goodput.

Sources

Anyscale · métricas de latencia y throughput de LLM — https://docs.anyscale.com/llm/serving/benchmarking/metrics
Red Hat · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
NVIDIA AIPerf · guía de benchmarking (ex genai-perf) — https://lucaberton.com/blog/nvidia-aiperf-llm-inference-benchmarking-guide/
MLCommons · MLPerf Inference datacenter (escenarios) — https://mlcommons.org/benchmarks/inference-datacenter/
MLCommons · MLPerf Inference v5.0 (405B + 70B interactivo) — https://mlcommons.org/2025/04/llm-inference-v5/
MLCommons · MLPerf Inference v5.1 — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
arXiv 2605.24217 · sesgo sistemático de medición en benchmarks de inferencia LLM — https://arxiv.org/html/2605.24217
arXiv 2508.10251 · meta-métricas y buenas prácticas de benchmarking de rendimiento — https://arxiv.org/pdf/2508.10251
Medium · LLM Inference Benchmarking (genAI-perf y vLLM) — https://kchandan.medium.com/llm-inference-benchmarking-genai-perf-and-vllm-5dd06b57428e

Prefix cache: hit rate engineering to go from 15% to 75%

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

vLLM’s prefix cache stores the KV cache blocks of shared prefixes and reuses them in later requests. A hit avoids recomputing that prefix: TTFT drops to the cost of the variable suffix only. In enterprise workloads with fixed system prompts (RAG, domain chatbots, assistants with long instructions) the hit rate should be >70%. In practice it is 10-20% for entirely avoidable reasons. This article identifies them, fixes them, and gives the OTel queries to confirm the result.

The analogy

A simultaneous conference interpreter who has to translate the speeches of twenty speakers. All of them open with the same two-page protocol preamble: the conference declaration, the rules of conduct, the day’s programme. An interpreter with no memory rereads the two pages for each speaker before starting on their specific speech. An interpreter with good notes reads them once, files them, and when the second speaker starts goes straight to the speech.

The prefix cache is that file. The prefix hash is the reference that lets you skip to the new part. But if the preamble changes by even one word, because someone drops in the day’s date, the interpreter has to reread everything from the beginning.

How the prefix cache hash works

vLLM splits the KV cache into blocks of 16 tokens. Each block has a hash computed over its exact content. When a new request arrives, vLLM checks whether any initial block of the prompt is already in cache by comparing hashes.

The hash is computed over the byte-by-byte content of the tokens. Any difference, a space, a different character, one extra token, produces a completely different hash. There is no partial matching inside a block.

Direct consequence: if your system prompt has 512 tokens and token number 3 changes between requests (because you interpolate a date, an ID, a version number), no block hits even though 99% of the text is identical.

Block 0 (tokens 0-15): hash = a3f7... ← in cache?
Block 1 (tokens 16-31): hash = 9d2c... ← in cache?
...
Block 31 (tokens 496-511): hash = 7e1a... ← in cache?

If block 0 does not hit (because its content changed), blocks 1-31 are not even checked although they are identical. The prefix cache is sequential.

Audit: why your real hit rate is low

Before changing anything, you need to know what is breaking the hash. The most direct method: pull the last 1000 production prompts and work out what fraction of the prefix varies.

# audit_prefix_cache.py
import langfuse, hashlib, collections
from transformers import AutoTokenizer

client = langfuse.Langfuse()
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-14B-Instruct")

traces = client.fetch_traces(limit=1000).data
prompts = [t.input for t in traces if t.input]

# Tokenise and extract the first 512 tokens (the typical system prompt)
prefixes = []
for prompt in prompts:
 tokens = tokenizer.encode(prompt, add_special_tokens=False)
 prefix_tokens = tuple(tokens[:512])
 prefixes.append(prefix_tokens)

# How many unique prefixes are there?
unique = len(set(prefixes))
total = len(prefixes)
print(f"Unique prefixes: {unique}/{total} ({unique/total*100:.1f}%)")
print(f"Theoretical hit rate if they were all equal: {(1 - unique/total)*100:.1f}%")

# Find which token differs between the most common prefix and the rest
from collections import Counter
most_common_prefix = Counter(prefixes).most_common(1)[0][0]

divergence_positions = []
for prefix in prefixes:
 if prefix == most_common_prefix:
 continue
 for i, (a, b) in enumerate(zip(most_common_prefix, prefix)):
 if a != b:
 divergence_positions.append(i)
 break

if divergence_positions:
 pos = Counter(divergence_positions).most_common(1)[0][0]
 token_text = tokenizer.decode([most_common_prefix[pos]])
 print(f"\nMost frequent divergence at position {pos}: '{token_text}'")
 print("→ The token at that position varies between requests")

The most common culprits, in order of frequency:

1. Timestamps and dates:

# ❌ Breaks the hash on every request
system = f"Current date: {datetime.now().strftime('%Y-%m-%d %H:%M')}. You are an assistant..."

# ✅ Take the date out of the system prompt
system = "You are an assistant specialising in cloud infrastructure."
# Pass the date as part of the user message if it is needed

2. Session and user IDs:

# ❌
system = f"User ID: {user_id}. Preferences: {user_prefs}. You are an assistant..."

# ✅ Separate the static from the contextual
system = "You are a specialised assistant." # always the same
# Add the user context as the first message of the history

3. Interpolated prompt versions:

# ❌
system = f"[v{PROMPT_VERSION}] You are an assistant..." # changes with every deploy

# ✅ Do not version in the text, version in the prompt name in Langfuse
system = "You are an assistant..."

4. Dynamic few-shots:

# ❌ Examples retrieved at random from a pool
examples = random.sample(example_pool, k=3)
system = f"Examples:\n{format_examples(examples)}\n\nYou are an assistant..."

# ✅ Fixed few-shots always in the same order
FIXED_EXAMPLES = [example_pool[0], example_pool[1], example_pool[2]]
system = f"Examples:\n{format_examples(FIXED_EXAMPLES)}\n\nYou are an assistant..."

Template engineering: the structure that maximises hits

The principle is simple: everything static goes first, everything dynamic goes after. The prefix cache is sequential. Once a block fails to hit, the rest are not looked up either.

OPTIMAL STRUCTURE for maximising prefix cache:
┌──────────────────────────────────────────────┐
│ STATIC BLOCK (tokens 0-511) │ ← hit rate ~100%
│ Invariant system prompt │
│ Fixed instructions │
│ Few-shots always in the same order │
├──────────────────────────────────────────────┤
│ SEMI-STATIC BLOCK (tokens 512-1023) │ ← hit rate ~60-80%
│ RAG documents for this session │
│ Conversation history so far │
├──────────────────────────────────────────────┤
│ DYNAMIC BLOCK (tokens 1024+) │ ← hit rate ~0% (expected)
│ Current user message │
│ Context specific to this request │
└──────────────────────────────────────────────┘

For RAG specifically: if the retrieved documents are the same across a set of similar queries (very frequent in RAG over fixed corporate documents), ordering them always in the same order (by ID, by a fixed score, not by a variable score) multiplies the hit rate of the semi-static block.

Prefix-aware routing: the next level

With a single vLLM instance, the prefix cache works automatically. The problem shows up with multiple replicas: the load balancer distributes requests round-robin, and the prefix cached on replica A is no use at all when the request lands on replica B.

The solution is prefix-aware routing: send requests with the same prefix to the same node.

With Ray Serve (native integration):

# ray_serve_prefix_router.py
from ray import serve
from ray.serve.llm import LLMConfig, build_llm_deployment

@serve.deployment
class PrefixAwareRouter:
 def __init__(self, replicas):
 self.replicas = replicas # list of vLLM handles

 async def __call__(self, request):
 body = await request.json()
 messages = body.get("messages", [])

 # Compute the hash of the system prompt (static prefix)
 system_content = ""
 for msg in messages:
 if msg["role"] == "system":
 system_content = msg["content"]
 break

 prefix_hash = hash(system_content)
 # Deterministic routing: same hash → same node
 replica_idx = prefix_hash % len(self.replicas)
 return await self.replicas[replica_idx].remote(request)

With an L7 gateway (Nginx/Traefik):

# nginx.conf — routing by X-Prefix-Hash header
upstream vllm_backends {
 hash $http_x_prefix_hash consistent;
 server vllm-0:8000;
 server vllm-1:8000;
 server vllm-2:8000;
 server vllm-3:8000;
}

The client computes the hash of the static prefix and includes it as a header:

import hashlib, requests

def llm_request(messages, base_url):
 system_msg = next((m["content"] for m in messages if m["role"] == "system"), "")
 prefix_hash = hashlib.sha256(system_msg.encode()).hexdigest()[:16]

 return requests.post(
 f"{base_url}/v1/chat/completions",
 json={"messages": messages, "model": "my-model"},
 headers={"X-Prefix-Hash": prefix_hash}
 )

Measuring the impact with OTel

# Current hit rate (0.0 to 1.0) — target > 0.70 with enterprise workloads
vllm:gpu_prefix_cache_hit_rate

# TTFT by percentile — should fall when the hit rate rises
histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))
histogram_quantile(0.95, rate(vllm:time_to_first_token_seconds_bucket[5m]))

The inverse correlation between hit rate and TTFT is the proof that the cache is working. If the hit rate goes from 15% to 70% and the p50 TTFT does not change, there is a configuration problem: the cache may be disabled, or the routing may not be sending requests to the right node.

Correlation query in Grafana (two-axis panel):

# Left Y axis: hit rate
vllm:gpu_prefix_cache_hit_rate

# Right Y axis: TTFT p50 (inverted)
histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))

The inverse slope should be visible: when the hit rate drops (a spike of requests with new prompts), TTFT rises. When the hit rate stabilises (users repeating the same flow), TTFT falls.

The impact in numbers

For a system with 100 req/min, a 512-token system prompt and a before/after hit rate:

Metric	Hit rate 15%	Hit rate 75%	Difference
Prefill tokens per minute	5,100	12,800 — 50% cached → 6,400 effective	−37% load
TTFT p50 (512 prompt + 100 suffix)	~820 ms	~180 ms (suffix only)	−78%
Prefill capacity freed	—	+1,200 tok/min	available for more requests

The 75% hit rate in this example is equivalent to being able to serve 37% more requests on the same hardware, because the prefill work of 3 out of every 4 requests is already done.

When the prefix cache does not help

The prefix cache is ineffective in workloads where every request has a completely unique prompt: translating a different document every time, code analysis with always-different context, creative generation with no system prompt. In these cases the hit rate structurally cannot exceed 5-10% and the template engineering effort does not pay off.

The signal: if your p99 input length is greater than the p50, you have high prompt variance and the prefix cache contributes little. If the p50 and the p99 are similar (consistent prompts), the prefix cache is the cheapest lever available.

References

Capacity planning for on-premise LLM inference: how to size GPUs from an SLO

Mon, 01 Jun 2026 15:00:00 +0200

This post complements the ones on KV cache (the piece that dominates the VRAM budget), Continuous batching (what defines the effective utilisation of the compute) and Seven layers of the stack (the pieces the sizing takes for granted). Before reading this one, make sure your team has written down the SLOs it is going to chase; without that input the calculation is not defensible.

TL;DR

LLM inference capacity planning does not answer “how many tokens per second does a GPU give” — that question has no universal answer, because throughput depends on concurrency, the prefill/decode split, context length, the inference engine and quantisation. The right question has three inputs (SLO: TTFT P95, TPOT P95, sustained RPS), a hardware reference (GPU model, VRAM, HBM bandwidth, effective FLOPs) and a model (parameters, GQA/MHA/MoE architecture, weight format). The calculation resolves into two coupled budgets that cross each other. VRAM budget: from the GPU’s total you subtract model weights and activations, what remains is the KV cache budget, and from there you derive the maximum concurrency possible at the average context you expect. Time budget: the engine (vLLM, SGLang, TensorRT-LLM) has a tokens/s ceiling in decode given by HBM bandwidth and another in prefill given by useful FLOPs; from there you derive the expected TPOT and, dividing prefill_tokens by prefill throughput, the expected TTFT. Both budgets must meet the SLO simultaneously: whichever is tighter dictates the sizing. On the Llama 70B BF16 example with tensor parallel 4 on 4×H100 SXM, a single replica saturates at ~28 concurrent requests and delivers ~3,200 tokens/s of aggregate decode with a median TPOT of 35 ms; for 200 sustained RPS at a profile of 800 prompt tokens + 250 of output, you need between 4 and 5 replicas with a 25 % cushion over the observed peak. Quantisation (FP8 → INT4) divides VRAM cost and decode time by between 1.5 and 4×, but it degrades quality measurably — it is not assumed free, it is validated with evals. The five habitual traps: confusing the mean with P95, ignoring the prefill/decode split of the real workload, sizing without head-room for retraining or rollback, forgetting that a GPU at 100 % SM util means nothing if the HBM is saturated, and not documenting the assumptions behind the calculation (a sizing with no written assumptions is a throwaway calculation).

You are here: DEPLOY (with one foot in OBSERVE)

Capacity planning is a piece with dual residency. It lives in DEPLOY because without a valid sizing you do not buy hardware or configure the inference engine. But its input is real observations: the distribution of prompt and output lengths, the prefill/decode mix of the workload, the real P95s already being seen in pre-production. Without that data the calculation is a napkin sketch, defensible only until the first customer arrives who does not fit the assumed average.

The analogy: the hotel with variable-sized rooms

Imagine a hotel where the rooms have no fixed size: each guest pays for the square metres they need, and the building’s floor plan reorganises itself dynamically to accommodate whoever arrives. Management wants to maximise occupancy, but it has two real constraints and one quality metric.

Constraint 1 — physical space. The floor has 1,000 m² in total. If a family comes in needing 200 m², that family occupies that surface and it cannot be handed to the next guest. The largest room limits how many simultaneous guests fit.

Constraint 2 — service staff. There are 10 receptionists. Each can handle the check-in of one guest every two minutes. When 60 guests arrive within an hour, the last ones wait in a queue; the time from walking into reception to receiving their key depends on how many are ahead of them.

Quality metric — a promise about time. The brochure says “check-in in under 15 minutes”. If too many guests arrive at once, that promise breaks even if there is physical space free.

The physical space is the GPU’s VRAM. Each room is a request with its KV cache (larger the longer the conversation). The receptionists are the compute units (Streaming Multiprocessors + Tensor Cores). The check-in is the prefill phase; the nights the guest spends afterwards are the decode steps. The 15-minute promise is the TTFT P95 SLO.

The hotel’s capacity planning is exactly this: given the expected guest profile (how many arrive per hour, how much space they ask for on average, how many minutes of waiting they tolerate), work out how many floors and how many receptionists are needed. It is not done by estimating “rooms per hour” in the abstract; it is done by crossing the two budgets with the promise about time. The analogy holds up the calculation all the way to the end.

The three SLO inputs

Before putting a single number on the sheet, you have to write down the three dimensions of the SLO. Without this, the calculation is aesthetics, not engineering.

TTFT P95 (Time-To-First-Token). The time from the client sending the request until it receives the first token. It is dominated by the prefill phase (processing the whole prompt in one go) plus the scheduler queue. For conversational chat, a reasonable target sits between 0.5 and 2 seconds P95. For coding assistants with large prompts (5–10 K tokens of context), between 2 and 4 s P95. Below 500 ms you enter UX territory for voice-style conversations, but that demands serious architectural compromises.

TPOT P95 (Time-Per-Output-Token). The time between consecutive tokens during decode. It dominates the “perceived fluency” of streaming. Above 80 ms/token the human reader perceives pauses; below 30 ms/token the output flows faster than it can be read. The usual industry target: 40–60 ms P95.

Sustained RPS meeting the SLO. The throughput the system must support without violating TTFT or TPOT. This is DistServe’s key metric, called goodput (see Continuous batching). “200 RPS peak” is not the same as “200 RPS with TTFT P95 ≤ 1.5 s”. Without the SLO condition, the RPS number means nothing.

These three dimensions come with a workload profile: the distribution of prompt and output lengths. Medians are not enough; you need P50, P95, P99. A badly measured profile is the main reason sizings fail.

The central formula: two budgets that cross

The calculation resolves into two independent sums that are then crossed. The smaller of the two rules.

VRAM budget

For a GPU with total VRAM $V$, the space available for KV cache is:

$$V_{\text{kv}} = V - V_{\text{model}} - V_{\text{activations}} - V_{\text{overhead}}$$

where:

$V_{\text{model}}$ is the size of the weights: for a model of $P$ parameters in a format of $b$ bytes/parameter, $V_{\text{model}} = P \cdot b$. Llama 70B BF16 = $70 \times 10^9 \times 2 = 140$ GB. Under tensor parallel TP=4, each GPU carries $140 / 4 = 35$ GB.
$V_{\text{activations}}$ are the intermediate buffers of the forward pass. For vLLM with a reasonable batch, between 2 and 6 GB per GPU depending on batch size and maximum length.
$V_{\text{overhead}}$ is the CUDA context, NCCL buffers, the PagedAttention pool, reserved paged blocks. 2–4 GB typical.

The KV cache budget per GPU is what is left over. For an H100 SXM 80 GB with Llama 70B TP=4 BF16:

$$V_{\text{kv}} = 80 - 35 - 4 - 3 = 38 \text{ GB per GPU} = 152 \text{ GB aggregated over TP=4}$$

The KV cache cost per token for a model with $L$ layers, $H_{\text{kv}}$ KV heads (GQA), per-head dimension $d_h$, in a format of $b$ bytes:

$$\text{kv per token} = 2 \cdot L \cdot H_{\text{kv}} \cdot d_h \cdot b$$

The factor of 2 is because both K and V are stored. For Llama 70B (L=80, $H_{\text{kv}}$=8 with GQA, $d_h$=128, BF16 = 2 bytes):

$$\text{kv per token} = 2 \cdot 80 \cdot 8 \cdot 128 \cdot 2 = 327,680 \text{ bytes} = 320 \text{ KB/token}$$

And the maximum concurrency at average context $C$:

$$N_{\text{max}} = \frac{V_{\text{kv}}}{C \cdot \text{kv per token}}$$

With an aggregated $V_{\text{kv}}$ of 152 GB and an average context of 1,500 tokens (800 prompt + 700 generated at the worst moment of the conversation):

$$N_{\text{max}} = \frac{152 \times 10^9}{1,500 \cdot 320 \times 10^3} \approx 316 \text{ concurrent requests}$$

This is the physical ceiling of concurrency for that replica. It is not what you are going to use; it is what you cannot exceed without OOM. The operational number sits well below it (head-room for spikes).

Time budget

Two sub-calculations come in here: decode (memory-bound) and prefill (compute-bound).

Decode TPOT. For every token generated, the model weights (the ones relevant to that request) have to be walked through and the accumulated KV cache has to be read. The bottleneck is HBM bandwidth. For a GPU with bandwidth $B$ GB/s and a model of $V_{\text{model per gpu}}$ GB of weights:

$$\text{tpot}_{\text{theoretical}} \approx \frac{V_{\text{model per gpu}}}{B}$$

For an H100 SXM with HBM3 at 3.35 TB/s and Llama 70B TP=4 BF16 (35 GB/GPU):

$$\text{tpot}_{\text{theoretical}} \approx \frac{35}{3,350} \approx 10.4 \text{ ms/token}$$

This is the best theoretical case with batch=1 and 100 % HBM efficiency. In practice vLLM on H100 with Llama 70B TP=4 reaches 12–18 ms/token at low batch and 30–45 ms/token at high batch (at concurrency 32, tokens compete for the shared HBM). The defensible operational number: 35 ms/token at concurrency 24–32.

Prefill throughput. Prefill processes N prompt tokens in a single forward pass. It is compute-bound: the bottleneck is FLOPs. For an H100 SXM with 989 sustained BF16 TFLOPs and Llama 70B (each forward pass costs roughly $2 \cdot P \cdot N$ FLOPs per sequence of length N):

$$\text{prefill tps} = \frac{4 \cdot \text{TFLOPs} \cdot \eta}{2 \cdot P} = \frac{4 \cdot 989 \times 10^{12} \cdot 0.5}{2 \cdot 70 \times 10^9} \approx 14,000 \text{ tokens/s}$$

(the factor of 4 is the GPUs in TP, $\eta$ is real efficiency between 0.4 and 0.6 on H100). An 800-token prompt takes this long in prefill:

$$\text{prefill time} = \frac{800}{14,000} \approx 57 \text{ ms}$$

Adding a typical queue of 100–300 ms at high concurrency, TTFT P95 ≈ 350–500 ms for that profile. Well below the 1.5 s target; there is margin.

The crossing

The real operational concurrency $N_{\text{op}}$ is the minimum of the VRAM ceiling, the concurrency at which TPOT starts to degrade past the SLO, and the concurrency at which TTFT starts to degrade past the SLO (prefill queue). For the example:

VRAM ceiling: 316.
TPOT degrades to 80 ms (SLO) at around concurrency ~80–100 (measured empirically with a benchmark, not a closed formula).
TTFT degrades to 1.5 s at around concurrency ~40–60 because of the prefill queue.

The replica’s operational concurrency is ~50. Applying 25 % head-room for spikes and rebalancing, target concurrency per replica ≈ 35–40.

Step-by-step worksheet: Llama 70B BF16 on 4×H100 SXM

The exercise’s input:

SLO: TTFT P95 ≤ 1.5 s; TPOT P95 ≤ 60 ms; 200 sustained RPS.
Workload: prompt P50=600, P95=1,200, P99=2,500; output P50=180, P95=500, P99=900. Average prompt 800, output 250.
Generic hardware: 4×H100 SXM 80 GB with NVLink, vLLM v1 engine, tensor parallel 4, BF16.

Step 1 — VRAM per GPU. Weights 35 GB, activations 4 GB, overhead 3 GB → KV budget 38 GB/GPU = 152 GB aggregated. KV/token for Llama 70B GQA = 320 KB. Ceiling of live tokens in cache: $152 \times 10^9 / 320 \times 10^3 \approx 475,000$ tokens. At the operational average context (800 prompt + 200 already generated = 1,000 live tokens per request), concurrency ceiling $\approx 475$.

Step 2 — average duration of a request. Prefill 800 tokens / 14,000 tps = 57 ms. Decode 250 tokens × 35 ms/token = 8,750 ms. Total $\approx 8.8$ s per request.

Step 3 — replica throughput. If the replica sustains operational concurrency 40 and each request lasts 8.8 s, the replica delivers roughly $40 / 8.8 \approx 4.5$ requests/s in steady state.

Step 4 — number of replicas. For a 200 RPS target: $200 / 4.5 \approx 45$ replicas. That is 45 × 4 = 180 GPUs. Too much: this sizing does not work because the cost per request is high.

Step 5 — review the levers. Before buying more hardware, there are three levers to explore, in this order:

Quantisation. Dropping to FP8 reduces weights to 17.5 GB/GPU (leaving more VRAM for KV cache → more concurrency), roughly doubles decode tokens/s (HBM saturated by half), and degrades MMLU quality typically by 0.5–1.5 points on models like Llama 70B. Rewriting the calculation in FP8: TPOT drops to ~18 ms, total time per request to 4.7 s, RPS per replica rises to ~8.5, replicas needed ≈ 24, equivalent to 96 GPUs.
Speculative decoding. With a small drafter and 60–70 % acceptance, effective TPOT falls 30–40 %. RPS per replica rises to ~12, replicas ≈ 17 = 68 GPUs.
Disaggregated serving. Separating prefill workers and decode workers lets each be scaled to the real mix of the workload (see Disaggregated serving). It usually shaves off another 20–40 % under asymmetric workloads.

Step 6 — recommended sizing. For the example, with FP8 + speculative decoding and 25 % head-room: 20 vLLM replicas at TP=4 over 80 H100 SXM. If the team does not want to depend on aggressive quantisation (pure BF16 for maximum fidelity), the calculation rises to 30 replicas = 120 GPUs and forces a renegotiation of the SLO or the budget.

Step 7 — write down the assumptions. This is the part no valid sizing skips. In the team’s repo, next to the calculation:

# sizing/llama70b-prod.yaml
fecha: 2026-06-01
slo:
 ttft_p95_ms: 1500
 tpot_p95_ms: 60
 rps_target: 200
workload:
 prompt_tokens_p50: 600
 prompt_tokens_p95: 1200
 output_tokens_p50: 180
 output_tokens_p95: 500
 asunto: chat productivo con RAG ligero
modelo:
 arquitectura: llama-70b-instruct
 formato_pesos: fp8
 motor: vllm-v1
hardware:
 gpu: H100-SXM-80GB
 topologia: TP=4 con NVLink intra-nodo
 red_inter_replica: 25 GbE
optimizaciones:
 - paged_attention
 - chunked_prefill
 - speculative_decoding (drafter llama-1.1b, aceptación esperada 65%)
asunciones_criticas:
 - utilizacion_hbm_eficiente: 0.55
 - head_room_pico_sobre_p95: 0.25
 - aceptacion_speculative_min: 0.55
plan_validacion:
 - benchmark vllm bench serve antes de procurement
 - canary 10% durante 7 días post-deploy

Without this YAML, the calculation is not reproducible a month later.

MoE case: Mixtral 8×22B (~141 B total, 39 B active)

MoEs change the calculation along one key dimension: the total weights are large but the active weights per token are small. For Mixtral 8×22B with top-2 routing:

Weight VRAM: $141 \times 2 = 282$ GB BF16. With TP=4 → 70 GB/GPU. It does not fit in an H100 80 GB alongside KV cache + activations. You need TP=8 (~35 GB/GPU) or FP8 with TP=4 (~35 GB/GPU).
Decode TPOT: dominated by the weights read per token, which are $\sim 39 / 8 \cdot 2 \approx 9.75$ GB/GPU with TP=4 (one top-2 expert per token, divided across 4 GPUs). On an H100 with HBM at 3.35 TB/s, theoretical TPOT ≈ 3 ms/token. In practice, 10–20 ms at reasonable concurrency.
Prefill: similar to the dense model of the active weights, ~39 B FLOPs/token.

MoE sizing usually delivers more RPS per GPU than an equivalent dense model: the low cost per token compensates for the extra VRAM. See MoE inference for the detail of routing and why a high batch is decisive so that each expert sees enough tokens.

Sensitivity table: context and quantisation

For Llama 70B on 4×H100 SXM (TP=4), operational concurrency per replica with an SLO of TTFT 1.5 s / TPOT 60 ms:

Average context	BF16	FP8	INT4 (AWQ)
500 tokens	55	110	180
1,000 tokens	40	80	130
2,000 tokens	24	50	85
4,000 tokens	12	26	48
8,000 tokens	6	13	25

Approximate numbers from public vLLM benchmarks as of June 2026, with ±20 % variation depending on engine version and the headroom adopted. To validate on your own hardware: vllm bench serve with your profile of real prompts.

The five habitual traps

Trap 1 — confusing the mean with P95. Average throughput over an hour may be 50 RPS while the 5-minute peak reaches 180 RPS. Sizing against the mean guarantees breaking the SLO at every peak. Rule: size against the hourly P95, with 20–30 % head-room over P95.

Trap 2 — not measuring the real prefill/decode split. A “RAG with short answers” workload spends 70–80 % of GPU time in prefill; a “writing assistant generating essays” spends 80 % in decode. The useful optimisations (chunked prefill vs speculative decoding) change radically. Without measuring it, you buy badly balanced hardware.

Trap 3 — sizing with no head-room for retraining or rollback. The production cluster is not just the inference engine: there is a re-embedding batch when the embeddings model changes, continuous canary evals (see Canary, blue-green and shadow), light fine-tuning, hot stand-by for rollback. Reserve 15–25 % of capacity for those non-negotiable workloads.

Trap 4 — “GPU at 100 % SM utilization” as a target. SM occupancy of 95 % with saturated HBM produces the same throughput as SM at 60 % with saturated HBM. The bottleneck in decode is the HBM. Optimising for “GPU usage 100 %” without looking at HBM utilisation and arithmetic intensity makes you spend more on GPU without gaining throughput. See GPU observability for LLM inference for which metrics to actually watch.

Trap 5 — not documenting the assumptions. A sizing with no reproducible YAML (workload, model, engine, head-room, critical assumptions) leaves the team with no way of knowing what changed when the cluster stops meeting the SLO six months later. Documenting is cheap; losing a quarter to debugging is not.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB with intra-node NVLink and 25 GbE between nodes, the recurring configurations in May 2026 are:

Model	Format	TP	Replicas that fit	Typical RPS per node (ctx 1K)
Llama 8B	BF16	1	4 (one per GPU)	240–320
Llama 8B	FP8	1	4	450–600
Llama 70B	BF16	4	1	30–45
Llama 70B	FP8	4	1	60–90
Llama 70B	INT4 AWQ	2	2	90–130
Mixtral 8×22B	FP8	4	1	90–140
Qwen 72B	BF16	4	1	28–42

These numbers are orders of magnitude to start the conversation, not commitments. The definitive sizing is validated with vllm bench serve or genai-perf (NVIDIA) using the customer’s real prompt/output profile. The prefill/decode asymmetry of each case’s workload can move these numbers 30–50 % up or down.

For clusters of 8×H100 SXM (typical of DGX servers or equivalent replicas), the options open up to TP=8 for 405B-class models or multi-replica TP=2 for 70B models with greater density. The metric that decides is always the same: tokens meeting SLO per kW and per euro of amortised hardware.

How the sizing is validated before buying

The spreadsheet sizing is the first half. The second is the validation benchmark.

Stage 1 — back-of-the-envelope sizing. This post’s formulas applied to the SLO and the expected workload. Output: an approximate number of replicas and a topology.

Stage 2 — synthetic micro-benchmark. On a borrowed GPU or one rented by the day, bring up the engine with the chosen model and run vllm bench serve with prompts of representative lengths. Validate TPOT, prefill TPS and the concurrency ceiling. Calibrate the HBM efficiency factor ($\eta$) used in the formulas.

Stage 3 — load test with realistic traffic. Generate traffic following the real distribution of the customer’s workload (not Poisson, not constant: the real trace). Measure P50/P95/P99 of TTFT, TPOT, throughput. Confirm the head-room.

Stage 4 — canary in production. With the cluster sized, route 5–10 % of real traffic for 7–14 days before closing the procurement of additional hardware. See Canary, blue-green and shadow for the mechanics.

Jumping from Stage 1 to full procurement is the most frequent cause of a cluster that is 40 % oversized and 60 % undersized at the same time, in different regions of the workload. Four weeks of validation done well save four months of refactoring.

What we have not covered (upcoming articles)

The observability metrics that close the sizing loop in production, see GPU observability for LLM inference.
Autoscaling that adjusts replicas to the real traffic curve, see Autoscaling LLM on Kubernetes.
Detailed cost accounting per tenant (showback / chargeback) over the sized hardware.
Sizing for continuous fine-tuning (PEFT and light training) that shares a cluster with inference.

References

Kwon et al. — vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (SOSP 2023).
Zhong et al. — DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024).
Agrawal et al. — Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI 2024).
NVIDIA — H100 Tensor Core GPU Architecture Whitepaper (HBM3 memory, bandwidth, sustained FLOPs).
vLLM project — vllm bench serve reference (benchmarking CLI included in the repo).
NVIDIA — genai-perf (official tool for benchmarking LLM services).

Ttft on lo0 — Tech Blog

MLPerf Inference: how to read it, what LoadGen enforces and what comparability it offers

TL;DR

What it is and how it is organised

The four scenarios

LoadGen: what it measures and what it does not

Duration, query count and early stopping

The compliance tests

Divisions, categories and what can be touched

The three divisions

The availability categories

What Closed allows and forbids

The SLOs almost nobody cites: TTFT and TPOT

Quality: the 99 % and the token count

The current Datacenter suite

How to read a results row

Per-accelerator is not an official metric

The comparison rules

How much audit sits behind a row

The recent rounds, in figures

What MLPerf Inference does not measure

Usage flow for an on-premise platform

Cross-links from the benchmarking track

See also

Sources

GenAI-Perf in depth: LLM inference profiling with real data

TL;DR

What GenAI-Perf is and where it fits

Metrics GenAI-Perf produces

Metrics glossary

Console output table (real example, TRT-LLM backend)

Invocation against OpenAI-compatible endpoints

profile mode: one load point

Main flags and what they do

Output artefacts

The analyze subcommand: automatic concurrency sweep

Concurrency sweep (the most used)

Request-rate sweep

ISL sweep (to study the effect of prefill)

Artefact structure of analyze

Summary CSV format

Comparison table: GenAI-Perf versus GuideLLM versus LLMPerf versus vLLM bench

Profiling on 4×H100 SXM 80 GB: a full sweep example

Illustrative result of a concurrency sweep (1→64)

Formulas for the key metrics

Synthetic ISL/OSL profiling: what to configure for each use case

Honest methodology: how not to measure badly

1. Warm-up is mandatory

2. The concurrency knee: extend the sweep until you see it

3. Concurrency versus request-rate: choosing correctly

4. Single-process versus multi-process bias

5. ignore_eos and an inconsistent OSL

6. The correct tokenizer

7. One run = a data point; three or more = a number

Connection with the cost and energy track

Status 2026: GenAI-Perf → AIPerf

See also

Sources

LLM inference benchmarking: frameworks, metrics and the state of the art (tool by tool)

What this introduction covers

The performance metrics

The latency decomposition

Goodput: the honest metric

How each metric is instrumented (and where the error creeps in)

The architecture of the tool biases the figure

Frameworks, one by one

vLLM bench and SGLang bench — engine micro-bench

AIPerf — the NVIDIA one (formerly genai-perf), multi-process

GuideLLM — from the vLLM project, SLO-oriented

LLMPerf — the Anyscale/Ray classic

MLPerf Inference — the industry standard

Comparison table

The concurrency sweep: finding the knee

Worked example: reading a sweep

MLPerf Inference: the comparability standard

Measurement bias and reproducibility

Checklist for a reproducible benchmark

Performance ≠ quality

The latency versus throughput trade-off

The connection with cost and energy

`profile` mode: one load point

The `analyze` subcommand: automatic concurrency sweep

Artefact structure of `analyze`

5. `ignore_eos` and an inconsistent OSL