LLM inference benchmarking: frameworks, metrics and the state of the art (tool by tool)
Contents
Notation: amounts in euros (N €), decimals with a point. Performance is not very sensitive to the country, but its associated cost (cost per token) is expressed in € and links back to the opening article.
What this introduction covers
Third article in the data series, a deep dive into the performance axis. Measuring the performance of an inference engine looks trivial (“how many tokens per second?”) and it is exactly where people fool themselves most: two tools can report results that differ by a factor of 7 for the same system. This article inventories the metrics that matter and how they are defined, why the architecture of the tool biases the figure, how the saturation point is found with a concurrency sweep, and the profile of each framework. No recommendations: the choice of engine is settled in the Pareto article (B8); here there are only the facts and the methodology, because a benchmark without a published methodology is not comparable.
The performance metrics
There is no single performance metric, there are five, and mixing them is the first source of error:
| Metric | Definition | Unit | Dominant phase |
|---|---|---|---|
| TTFT (Time To First Token) | time from sending the prompt to the first token | ms | prefill |
| TPOT / ITL (Time Per Output Token / Inter-Token Latency) | average time between output tokens once generation has started | ms/token | decode |
| Request throughput | complete request-response cycles per second at the concurrency tested | req/s | both |
| Token throughput | total tokens (input + output) per second across all concurrent requests | tok/s | both |
| Goodput | share of requests that meet the defined SLO | useful tok/s | both |
| P50 / P95 / P99 | latency percentiles (not the mean) | ms | — |
Precise definitions, because every tool computes them its own way (Anyscale · latency and throughput metrics): TTFT is what a user waits before seeing the first character, dominated by prefill compute; ITL is the average time between successive output tokens and sets the perceived “typing speed” of the answer; request throughput is complete cycles per second; token throughput is total tokens (input plus output) per second across all concurrent requests.
The latency decomposition
The total latency of a request with \(N\) output tokens decomposes as:
$$\text{latency} \approx \text{TTFT} + (N-1)\times \text{TPOT}$$That is why TTFT and TPOT are reported separately: a single mean latency hides very different profiles. A system with high TTFT and low TPOT (expensive prefill, fast decode) and another one the other way round can have the same mean latency for one particular length, yet behave in opposite ways when the response size changes. For an interactive chat experience TTFT and TPOT rule; for a batch of long summaries, token throughput does. Measuring the mean hides both realities.
Goodput: the honest metric
Raw throughput (TPS, RPS) says how much work the system does; goodput says how much of that work meets your quality-of-service standards (SLO) (Anyscale). An engine can boast 10,000 aggregate tok/s, but if half the requests violate the P99 TTFT SLO, its goodput is 5,000. The figure you defend in a proposal is the goodput, not the catalogue throughput: it is the only one that translates into satisfied users and an honest cost per token.
How each metric is instrumented (and where the error creeps in)
Before comparing numbers it helps to know where each clock starts and stops, because two tools can call different things “TTFT”:
- Client-side versus server-side TTFT. The TTFT measured by the client includes network latency and gateway queueing; the one measured by the server does not. For an engine comparison the server-side figure is the one that matters; for user experience, the client-side one. Mixing the two invalidates the comparison.
- Streaming is required. TTFT and ITL can only be measured if the response arrives as a stream (token by token). If the tool measures complete responses, there is no real TTFT: there is total latency in disguise.
- Token counting. Throughput in tok/s depends on which tokenizer counts the tokens. If the tool uses a tokenizer different from the model’s, the token count (and therefore the tok/s and the cost per token) is biased. You have to count with the tokenizer of the model being served.
- Warm-up and prefix cache. The first requests of a benchmark benefit from a warm prefix cache and give artificially low TTFT; the warm-up has to be discarded or the result inflates reality.
These four instrumentation decisions explain a good share of the discrepancies between tools. A performance figure that does not say where the clock is measured and with which tokenizer is not comparable, however precise it may look.
The architecture of the tool biases the figure
Here is the trap that invalidates half the published benchmarks. Tools split into two classes by the architecture of the client that generates the load, and that architecture determines whether the measurement is reliable at high concurrency:
- Single-process micro-bench (vLLM bench, SGLang bench, genai-perf): a Python client with asyncio in a single process. Useful for quick experiments on one engine, but the single-process architecture introduces a bottleneck on the client side that biases the data at high concurrency (genAI-perf and vLLM): the client cannot generate enough load and you measure the limit of the client, not the limit of the engine.
- Multi-process load (GuideLLM, AIPerf): they spread load generation across several processes, avoiding that limit. This is the class that has emerged for measuring at real scale.
The size of the bias is enormous: at 1,000 QPS, a single-process benchmark processed 75,574 tokens against the 545,733 tokens of a distributed architecture, a discrepancy of 7.2× in measurement capacity for the same system ([search]). Anyone comparing two engines with tools from different classes is not comparing the engines: they are comparing the benchmark clients.
Frameworks, one by one
vLLM bench and SGLang bench — engine micro-bench
What they measure: TTFT, TPOT and throughput of the engine itself (vLLM or SGLang). Class: single-process micro-bench. Use: quick experiments to tune one engine and see the effect of its optimisations (see decode and prefill). Limit: they saturate on the client at high concurrency; they are no good for measuring real capacity at scale.
AIPerf — the NVIDIA one (formerly genai-perf), multi-process
What it measures: TTFT, ITL, throughput and latency against vLLM, NIM, TGI and any
compatible endpoint. Class: multi-process load. State-of-the-art datum: NVIDIA retired
genai-perf and replaced it with AIPerf on 15 April 2026. During the sweep, AIPerf
detects GPU saturation and identifies the previous iteration, returning it as
estimatedCapacity; if it detects no saturation, estimatedCapacity is the last iteration
tested, which is why the sweep has to extend beyond the knee
(AIPerf).
GuideLLM — from the vLLM project, SLO-oriented
What it measures: full distributions of TTFT, ITL and end-to-end behaviour, for SLO-driven evaluation. Class: multi-process load. Differentiator: it generates realistic, configurable traffic patterns in synchronous, concurrent and rate-based modes, including reproducible sweeps to identify safe operating ranges (Red Hat · GuideLLM, GuideLLM · GitHub). It is the tool for answering “how hard can I load this engine without breaking the SLO?”.
LLMPerf — the Anyscale/Ray classic
What it measures: throughput and latency at the inference level. Use: endpoint validation, historically very widespread. Class: load generator. Limit: less focused on distributions and sweeps than GuideLLM/AIPerf.
MLPerf Inference — the industry standard
What it measures: performance under standardised scenarios with strict rules, for comparability across vendors. Maintainer: MLCommons. It is the gold standard of cross-vendor comparability; it is expanded on below.
Comparison table
| Tool | Class | What it measures | Maintainer | When to use it |
|---|---|---|---|---|
| vLLM bench | single-process micro | TTFT, TPOT, throughput of vLLM | vLLM (OSS) | tune vLLM, quick experiments |
| SGLang bench | single-process micro | metrics of the SGLang engine | SGLang (OSS) | tune SGLang |
| AIPerf (formerly genai-perf) | multi-process load | TTFT, ITL, throughput; estimatedCapacity | NVIDIA (OSS) | real capacity, multi-endpoint |
| GuideLLM | multi-process load | distributions, SLO, sweeps | vLLM (OSS) | validate SLO, find the knee |
| LLMPerf | load | throughput and latency | Anyscale/Ray (OSS) | endpoint validation |
| MLPerf Inference | standard suite | server/offline/interactive scenarios | MLCommons | cross-vendor comparability |
The concurrency sweep: finding the knee
The most useful measurement for sizing is not a number, it is a curve: how latency and throughput change as concurrency rises. Raising concurrency keeps the GPU busier and raises RPS, but past a certain point it sends TTFT, ITL and end-to-end latency through the roof ([search]). The goal of the sweep is to find the knee: the maximum concurrency where throughput still rises without latency breaking the SLO.
That is why AIPerf extends the sweep beyond the knee: only by seeing where latency takes
off can it return the safe capacity (estimatedCapacity). This curve is the raw material of
capacity planning: the knee
yields the replica count and the cost per token at the target load.
Worked example: reading a sweep
An illustrative sweep on an example node (a 70B on 8×H100, SLO of P99 TTFT < 500 ms), to see how the knee is read:
| Concurrency | RPS | TTFT P50 (ms) | TTFT P99 (ms) | Token tput (tok/s) | Goodput |
|---|---|---|---|---|---|
| 1 | 2 | 80 | 110 | 350 | 100 % |
| 8 | 14 | 110 | 240 | 2,100 | 100 % |
| 16 | 22 | 180 | 460 | 3,400 | 98 % |
| 24 | 26 | 320 | 980 | 3,900 | 62 % |
| 32 | 27 | 540 | 1,800 | 4,000 | 20 % |
Reading: up to concurrency ~16 throughput grows and P99 stays under the SLO (goodput ~100 %). Between 16 and 24 sits the knee: throughput barely rises any more (3,400 → 3,900 tok/s) but P99 takes off (460 → 980 ms) and goodput collapses (98 % → 62 %). At concurrency 32 raw throughput is at its maximum (4,000 tok/s) but goodput is 20 %: the system “performs well” while serving mostly requests that violate the SLO. The defensible safe capacity is the one at concurrency ~16, not the one at maximum throughput. This is the number that goes into capacity planning and into cost per token: at 3,400 useful tok/s, not 4,000 raw tok/s.
MLPerf Inference: the comparability standard
To compare across vendors and engines with identical rules there is MLPerf Inference (MLCommons). The datacenter category focuses on two scenarios plus an optional one (MLCommons · datacenter):
| Scenario | What it simulates | Metric |
|---|---|---|
| Offline | raw throughput processing the whole dataset in batch | maximum throughput |
| Server | interactive environment: requests one at a time following Poisson at a mean RPS | RPS under TTFT and TPOT limits |
| Interactive (optional) | like server but with stricter latency limits | RPS under a hard SLO |
The Server scenario is the realistic one for online inference: the load generator sends requests following a Poisson distribution and demands that specific TTFT and TPOT bounds be met. MLPerf Inference v5.0 (April 2025) introduced a large-scale 405B benchmark and a low-latency interactive 70B one, offering language benchmarks at every scale (7B to 405B), architectural diversity (including MoE) and scenarios (MLCommons · v5.0); v5.1 (September 2025) widened the results with record participation (MLCommons · v5.1).
The value of MLPerf is comparability: everyone measures the same thing under the same rules. Its limit is that those rules may not match your workload (your length distribution, your specific SLO), so it serves to compare hardware and engines against each other, not necessarily to size your case. For that, your own sweep.
Measurement bias and reproducibility
That two benchmarks give very different results for the same system is no accident: systematic measurement bias in production benchmarks is characterised in the literature (arXiv 2605.24217), and there is work dedicated to the meta-metrics and good practices of system-level performance benchmarking (arXiv 2508.10251). The most common sources of bias:
| Source of bias | Effect | Mitigation |
|---|---|---|
| Single-process client | underestimates throughput at high concurrency | use multi-process load |
| Unrealistic length distribution | results that do not apply to your traffic | use realistic traces |
| Measuring the mean instead of percentiles | hides the latency tail | report P95/P99 |
| Uncontrolled warm-up | the prefix cache inflates the first results | discard the warm-up |
| Not pinning engine/model version | irreproducible | pin everything and publish it |
The methodological conclusion of this article: a benchmark without a published methodology is not comparable. For a performance figure to support a proposal it has to come with the tool, its version, the model and precision, the load distribution and the SLO. Synthesis article S4 builds a reproducible harness that pins all of that.
Checklist for a reproducible benchmark
For a performance figure to be defensible before a committee, it has to come with everything that allows it to be reproduced. The minimum published alongside the result:
| What to pin | Why |
|---|---|
| Tool + version | each one measures differently; the version changes behaviour |
| Model + precision (FP16/FP8/INT4) | precision changes throughput and quality |
| Hardware (GPU, count, interconnect) | an 8×H100 NVLink is not an 8×H100 PCIe |
| Engine + version + flags | vLLM/SGLang/TRT-LLM and their configuration |
| Length distribution (in/out) | real traffic is not fixed length |
| Concurrency levels of the sweep | you have to go past the knee |
| SLO (which percentile, which threshold) | it defines the goodput |
| Warm-up handling | discard it or it biases the result |
| Tokenizer used for counting | it affects tok/s and cost per token |
The practical rule: if you cannot hand over this table alongside the figure, the figure is not a datum, it is an anecdote. The reproducible harness of article S4 automates the recording of all these parameters so that anyone, including whoever challenges the proposal, can reproduce the exact number.
Performance ≠ quality
One warning that avoids the most expensive mistake: these tools measure speed and throughput, not accuracy. An engine can be blisteringly fast while serving a model that answers badly. Quality is measured with another family of tools (lm-evaluation-harness, HELM, task leaderboards) which is another axis of the scorecard (article B7). Confusing “fast” with “good” is how you build a platform that serves bad answers very quickly. On the final Pareto frontier, performance and quality are two distinct axes that must be looked at together, never one instead of the other.
The other family of tools, for reference (expanded in B7):
| Tool | What it measures | Common trap |
|---|---|---|
| lm-evaluation-harness | accuracy on hundreds of standardised tasks | contamination of the test dataset |
| HELM | holistic evaluation (accuracy, robustness, bias, efficiency) | heavy to run |
| LiveBench / dynamic leaderboards | tasks that rotate to avoid contamination | comparability over time |
The data point: contamination, the model having seen the test during training, inflates quality metrics just as warm-up inflates performance ones. That is why dynamic leaderboards rotate their questions. Quality and performance share that lesson: the measurement method biases the result as much as the system being measured.
The latency versus throughput trade-off
A property the sweep reveals and which is worth keeping in mind: latency and throughput pull in opposite directions. Batching groups requests to amortise the cost of moving the weights out of VRAM, which raises throughput, but each request waits for the batch to form, which raises individual latency. There are two operating regimes, and the benchmark serves to place you in the right one:
| Regime | Optimises | Configuration | Case |
|---|---|---|---|
| Latency | low TTFT/TPOT | small batch, low concurrency | interactive chat, copilots |
| Throughput | maximum tok/s | large batch, high concurrency | overnight batch, ingestion |
There is no single “best” point: there is the best point for your SLO. A benchmark that reports only maximum throughput is describing the throughput regime and ignoring whether that point meets the latency your case needs. That is why goodput, throughput under the SLO, is the metric that reconciles the two regimes: it measures how much throughput you get without stepping outside acceptable latency. The sweep traverses the curve between both regimes; your SLO marks where on that curve your system sits.
The connection with cost and energy
Performance is not an isolated axis: by the identity of the opening article, throughput is the denominator of cost per token and of energy per token. A sweep that finds a knee at 4,000 tok/s rather than 2,800 is not just “faster”: it lowers the CPM from ~1.09 to ~0.76 €/1M tok and the energy per token in the same proportion, on the same iron. That is why performance benchmarking is the tool that, indirectly, moves cost the most: every goodput improvement translates into euros and into watts per token. The number that connects the three axes is goodput, the throughput that meets the SLO, not catalogue throughput.
State of the art 2026
- genai-perf → AIPerf migration (15 Apr 2026): NVIDIA consolidates its benchmarking into a multi-process tool with saturation detection.
- GuideLLM as the OSS standard for SLO-driven evaluation with reproducible sweeps.
- MLPerf Inference v5.0/v5.1 extends to 405B, interactive 70B and MoE, with record participation: cross-vendor comparability is maturing.
- Measurement bias characterised: the community accepts that the measurement method matters as much as the system measured; the emphasis on reproducibility and meta-metrics is growing.
Limits and traps (data-driven)
- Comparing tools from different classes. A single-process micro-bench and a multi-process load generator are not comparable; the difference can be 7×. Pin the class.
- Catalogue throughput instead of goodput. The honest number is the one that meets the SLO.
- Means instead of percentiles. The mean hides the tail; report P95/P99.
- Not extending the sweep beyond the knee. Without seeing where latency takes off you do not know the safe capacity.
- Confusing performance with quality. They are distinct axes; fast is not good.
- Not pinning versions. Engine, model, precision and load left unpinned = irreproducible = not defensible.
The next article in the track (B2) goes into the tool catalogue in depth; this one pins the metrics and the methodology. With performance measured reproducibly, the scorecard can cross it with cost (in €) and energy for the final decision.
Closing
Performance benchmarking looks like the most “objective” of the three axes (they are tokens per second, after all) and it is exactly where the most manipulation happens, almost always without bad intent: a single-process tool here, a mean instead of a P99 there, a catalogue throughput instead of the goodput. The difference between a marketing number and a defensible datum lies not in the engine measured but in the methodology: the class of tool, where the clock is measured, with which tokenizer, how far the sweep goes and which SLO defines the goodput. For a sovereign architecture proposal, performance only counts if it is delivered with that reproducibility record, and, crossed with cost in euros and energy per token, it becomes the column of the Pareto frontier that decides which engine and which configuration hold up the platform. The number you defend is not the highest one: it is the reproducible goodput.
See also
- LLM serving engines compared (vLLM/SGLang/TRT-LLM/Dynamo) — choosing the engine after measuring goodput: from the methodology of this article to the Pareto frontier.
- MLPerf Inference: how to read it and what comparability it offers — the industry standard: the four scenarios, the TTFT/TPOT SLOs of the Server scenario and the rules for comparing results.
- Measurement bias and reproducibility — sources of systematic bias in performance benchmarks and how to control them to obtain defensible data.
Sources
- Anyscale · métricas de latencia y throughput de LLM — https://docs.anyscale.com/llm/serving/benchmarking/metrics
- Red Hat · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
- GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
- NVIDIA AIPerf · guía de benchmarking (ex genai-perf) — https://lucaberton.com/blog/nvidia-aiperf-llm-inference-benchmarking-guide/
- MLCommons · MLPerf Inference datacenter (escenarios) — https://mlcommons.org/benchmarks/inference-datacenter/
- MLCommons · MLPerf Inference v5.0 (405B + 70B interactivo) — https://mlcommons.org/2025/04/llm-inference-v5/
- MLCommons · MLPerf Inference v5.1 — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
- arXiv 2605.24217 · sesgo sistemático de medición en benchmarks de inferencia LLM — https://arxiv.org/html/2605.24217
- arXiv 2508.10251 · meta-métricas y buenas prácticas de benchmarking de rendimiento — https://arxiv.org/pdf/2508.10251
- Medium · LLM Inference Benchmarking (genAI-perf y vLLM) — https://kchandan.medium.com/llm-inference-benchmarking-genai-perf-and-vllm-5dd06b57428e