---
title: "LLM inference benchmarking: frameworks, metrics and the state of the art (tool by tool)"
url: https://blog.lo0.es/en/posts/llm-inference-benchmarking-frameworks-metrics-state-of-the-art/
markdown: https://blog.lo0.es/en/posts/llm-inference-benchmarking-frameworks-metrics-state-of-the-art/index.md
lang: en
date: 2026-06-13
lastmod: 2026-09-13
author: "José Manuel Román"
description: "A data introduction to LLM inference benchmarking tooling: the metrics (TTFT, TPOT/ITL, throughput, goodput) and how they decompose, the difference between a single-process micro-bench and multi-process load (and the bias it introduces), the concurrency sweep used to find the knee, the profiles of vLLM/SGLang bench, AIPerf (formerly genai-perf), GuideLLM, LLMPerf and MLPerf Inference, and the reproducible methodology. State of the art 2026, with sources."
tags: ["benchmarking","vllm","guidellm","aiperf","llmperf","mlperf","ttft","throughput","goodput","data-series"]
translations:
  es: https://blog.lo0.es/posts/benchmarking-llm-frameworks-estado-del-arte/index.md
---

# LLM inference benchmarking: frameworks, metrics and the state of the art (tool by tool)


> Notation: amounts in **euros (N €)**, decimals with a point. Performance is not very
> sensitive to the country, but its associated cost (cost per token) is expressed in € and
> links back to the [opening article](https://blog.lo0.es/en/posts/cost-performance-energy-three-axes-llm-inference/index.md).

## What this introduction covers

Third article in the data series, a deep dive into the performance axis. Measuring the
performance of an inference engine looks trivial ("how many tokens per second?") and it is
exactly where people fool themselves most: two tools can report results that differ by a
factor of 7 for the same system. This article inventories the metrics that matter and **how
they are defined**, why the architecture of the tool biases the figure, how the saturation
point is found with a concurrency *sweep*, and the profile of each framework. No
recommendations: the choice of engine is settled in the Pareto article (B8); here there are
only the facts and the methodology, because **a benchmark without a published methodology is
not comparable**.

---

## The performance metrics

There is no single performance metric, there are five, and mixing them is the first source
of error:

| Metric | Definition | Unit | Dominant phase |
|---|---|---|---|
| **TTFT** (Time To First Token) | time from sending the prompt to the first token | ms | prefill |
| **TPOT / ITL** (Time Per Output Token / Inter-Token Latency) | average time between output tokens once generation has started | ms/token | decode |
| **Request throughput** | complete request-response cycles per second at the concurrency tested | req/s | both |
| **Token throughput** | total tokens (input + output) per second across all concurrent requests | tok/s | both |
| **Goodput** | share of requests that **meet the defined SLO** | useful tok/s | both |
| **P50 / P95 / P99** | latency percentiles (not the mean) | ms | — |

Precise definitions, because every tool computes them its own way ([Anyscale · latency and
throughput metrics](https://docs.anyscale.com/llm/serving/benchmarking/metrics)): TTFT is
what a user waits before seeing the first character, dominated by prefill compute; ITL is
the average time between successive output tokens and sets the perceived "typing speed" of
the answer; request throughput is complete cycles per second; token throughput is
total tokens (input plus output) per second across all concurrent requests.

### The latency decomposition

The total latency of a request with \(N\) output tokens decomposes as:

$$\text{latency} \approx \text{TTFT} + (N-1)\times \text{TPOT}$$

That is why TTFT and TPOT are reported separately: a single mean latency hides very
different profiles. A system with high TTFT and low TPOT (expensive prefill, fast decode) and
another one the other way round can have the same mean latency for one particular length, yet
behave in opposite ways when the response size changes. For an interactive chat experience
TTFT and TPOT rule; for a batch of long summaries, token throughput does. Measuring the mean
hides both realities.

### Goodput: the honest metric

Raw throughput (TPS, RPS) says how much work the system does; **goodput** says how much
of that work meets your quality-of-service standards (SLO)
([Anyscale](https://docs.anyscale.com/llm/serving/benchmarking/metrics)). An engine can boast
10,000 aggregate tok/s, but if half the requests violate the P99 TTFT SLO, its goodput is
5,000. The figure you defend in a proposal is the **goodput**, not the catalogue throughput:
it is the only one that translates into satisfied users and an honest cost per token.

---

## How each metric is instrumented (and where the error creeps in)

Before comparing numbers it helps to know where each clock starts and stops, because two
tools can call different things "TTFT":

- **Client-side versus server-side TTFT.** The TTFT measured by the client includes network
  latency and gateway queueing; the one measured by the server does not. For an engine
  comparison the server-side figure is the one that matters; for user experience, the
  client-side one. Mixing the two invalidates the comparison.
- **Streaming is required.** TTFT and ITL can only be measured if the response arrives as a
  *stream* (token by token). If the tool measures complete responses, there is no real TTFT:
  there is total latency in disguise.
- **Token counting.** Throughput in tok/s depends on which tokenizer counts the tokens.
  If the tool uses a tokenizer different from the model's, the token count (and therefore the
  tok/s and the cost per token) is biased. You have to count with the tokenizer of the model
  being served.
- **Warm-up and prefix cache.** The first requests of a benchmark benefit from a warm prefix
  cache and give artificially low TTFT; the warm-up has to be discarded or the result
  inflates reality.

These four instrumentation decisions explain a good share of the discrepancies between tools.
A performance figure that does not say where the clock is measured and with which tokenizer
is not comparable, however precise it may look.

---

## The architecture of the tool biases the figure

Here is the trap that invalidates half the published benchmarks. Tools split into two classes
by the architecture of the client that generates the load, and that architecture
determines whether the measurement is reliable at high concurrency:

- **Single-process micro-bench** (vLLM bench, SGLang bench, genai-perf): a Python client with
  asyncio in a single process. Useful for quick experiments on one engine, but the
  single-process architecture introduces a bottleneck on the client side that **biases the
  data at high concurrency** ([genAI-perf and vLLM](https://kchandan.medium.com/llm-inference-benchmarking-genai-perf-and-vllm-5dd06b57428e)):
  the client cannot generate enough load and you measure the limit of the client, not the
  limit of the engine.
- **Multi-process load** (GuideLLM, AIPerf): they spread load generation across several
  processes, avoiding that limit. This is the class that has emerged for measuring at real
  scale.

The size of the bias is enormous: at 1,000 QPS, a single-process benchmark processed
75,574 tokens against the 545,733 tokens of a distributed architecture, a
discrepancy of 7.2× in measurement capacity for the same system ([search]). Anyone
comparing two engines with tools from different classes is not comparing the engines: they
are comparing the benchmark clients.

> **Diagram:** Two classes of benchmark tool: a single-process client that saturates at high concurrency versus a multi-process load generator that measures the engine — [see the HTML version](https://blog.lo0.es/en/posts/llm-inference-benchmarking-frameworks-metrics-state-of-the-art/)

---

## Frameworks, one by one

### vLLM bench and SGLang bench — engine micro-bench

What they measure: TTFT, TPOT and throughput of the engine itself (vLLM or SGLang). Class:
single-process micro-bench. Use: quick experiments to tune one engine and see the effect of
its optimisations (see [decode](https://blog.lo0.es/en/posts/optimising-decode-vllm-squeezing-every-token-small-hardware/index.md) and
[prefill](https://blog.lo0.es/en/posts/optimising-prefill-vllm-knobs-ttft-will-not-forgive/index.md)). Limit: they saturate on the client at
high concurrency; they are no good for measuring real capacity at scale.

### AIPerf — the NVIDIA one (formerly genai-perf), multi-process

What it measures: TTFT, ITL, throughput and latency against **vLLM, NIM, TGI and any
compatible endpoint**. Class: multi-process load. State-of-the-art datum: NVIDIA **retired
genai-perf and replaced it with AIPerf on 15 April 2026**. During the *sweep*, AIPerf
detects GPU saturation and identifies the previous iteration, returning it as
`estimatedCapacity`; if it detects no saturation, `estimatedCapacity` is the last iteration
tested, which is why the sweep has to extend beyond the knee
([AIPerf](https://lucaberton.com/blog/nvidia-aiperf-llm-inference-benchmarking-guide/)).

### GuideLLM — from the vLLM project, SLO-oriented

What it measures: full distributions of TTFT, ITL and end-to-end behaviour, for
SLO-driven evaluation. Class: multi-process load. Differentiator: it generates realistic,
configurable traffic patterns in synchronous, concurrent and rate-based modes, including
reproducible sweeps to identify safe operating ranges
([Red Hat · GuideLLM](https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference),
[GuideLLM · GitHub](https://github.com/vllm-project/guidellm)). It is the tool for answering
"how hard can I load this engine without breaking the SLO?".

### LLMPerf — the Anyscale/Ray classic

What it measures: throughput and latency at the inference level. Use: endpoint validation,
historically very widespread. Class: load generator. Limit: less focused on distributions and
sweeps than GuideLLM/AIPerf.

### MLPerf Inference — the industry standard

What it measures: performance under standardised scenarios with strict rules, for
comparability across vendors. Maintainer: **MLCommons**. It is the gold standard of
cross-vendor comparability; it is expanded on below.

### Comparison table

| Tool | Class | What it measures | Maintainer | When to use it |
|---|---|---|---|---|
| **vLLM bench** | single-process micro | TTFT, TPOT, throughput of vLLM | vLLM (OSS) | tune vLLM, quick experiments |
| **SGLang bench** | single-process micro | metrics of the SGLang engine | SGLang (OSS) | tune SGLang |
| **AIPerf** (formerly genai-perf) | multi-process load | TTFT, ITL, throughput; estimatedCapacity | NVIDIA (OSS) | real capacity, multi-endpoint |
| **GuideLLM** | multi-process load | distributions, SLO, sweeps | vLLM (OSS) | validate SLO, find the knee |
| **LLMPerf** | load | throughput and latency | Anyscale/Ray (OSS) | endpoint validation |
| **MLPerf Inference** | standard suite | server/offline/interactive scenarios | MLCommons | cross-vendor comparability |

---

## The concurrency sweep: finding the knee

The most useful measurement for sizing is not a number, it is a curve: how latency and
throughput change as concurrency rises. Raising concurrency keeps the GPU busier and raises
RPS, but past a certain point it sends TTFT, ITL and end-to-end latency through the roof
([search]). The goal of the *sweep* is to find the **knee**: the maximum concurrency where
throughput still rises without latency breaking the SLO.

> **Diagram:** Concurrency sweep curve: throughput rises and saturates while latency spikes beyond the knee, which marks the safe capacity under SLO — [see the HTML version](https://blog.lo0.es/en/posts/llm-inference-benchmarking-frameworks-metrics-state-of-the-art/)

That is why AIPerf extends the sweep beyond the knee: only by seeing where latency takes
off can it return the safe capacity (`estimatedCapacity`). This curve is the raw material of
[capacity planning](https://blog.lo0.es/en/posts/capacity-planning-on-premise-llm-inference/index.md): the knee
yields the replica count and the cost per token at the target load.

---

### Worked example: reading a sweep

An illustrative sweep on an example node (a 70B on 8×H100, SLO of P99 TTFT < 500 ms), to
see how the knee is read:

| Concurrency | RPS | TTFT P50 (ms) | TTFT P99 (ms) | Token tput (tok/s) | Goodput |
|---|---|---|---|---|---|
| 1 | 2 | 80 | 110 | 350 | 100 % |
| 8 | 14 | 110 | 240 | 2,100 | 100 % |
| 16 | 22 | 180 | 460 | 3,400 | 98 % |
| 24 | 26 | 320 | 980 | 3,900 | 62 % |
| 32 | 27 | 540 | 1,800 | 4,000 | 20 % |

Reading: up to concurrency ~16 throughput grows and P99 stays under the SLO (goodput
~100 %). Between 16 and 24 sits the **knee**: throughput barely rises any more (3,400 →
3,900 tok/s) but P99 takes off (460 → 980 ms) and goodput collapses (98 % → 62 %). At
concurrency 32 raw throughput is at its maximum (4,000 tok/s) but goodput is 20 %: the
system "performs well" while serving mostly requests that violate the SLO. The defensible
safe capacity is the one at concurrency ~16, not the one at maximum throughput. This is the
number that goes into [capacity planning](https://blog.lo0.es/en/posts/capacity-planning-on-premise-llm-inference/index.md)
and into cost per token: at 3,400 useful tok/s, not 4,000 raw tok/s.

---

## MLPerf Inference: the comparability standard

To compare across vendors and engines with identical rules there is MLPerf Inference
(MLCommons). The datacenter category focuses on two scenarios plus an optional one
([MLCommons · datacenter](https://mlcommons.org/benchmarks/inference-datacenter/)):

| Scenario | What it simulates | Metric |
|---|---|---|
| **Offline** | raw throughput processing the whole dataset in batch | maximum throughput |
| **Server** | interactive environment: requests one at a time following Poisson at a mean RPS | RPS under TTFT and TPOT limits |
| **Interactive** (optional) | like server but with **stricter latency limits** | RPS under a hard SLO |

The Server scenario is the realistic one for online inference: the load generator sends
requests following a Poisson distribution and demands that specific TTFT and TPOT bounds
be met. **MLPerf Inference v5.0** (April 2025) introduced a large-scale 405B benchmark
and a low-latency interactive 70B one, offering language benchmarks at every scale (7B to
405B), architectural diversity (including MoE) and scenarios
([MLCommons · v5.0](https://mlcommons.org/2025/04/llm-inference-v5/)); **v5.1** (September
2025) widened the results with record participation
([MLCommons · v5.1](https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/)).

The value of MLPerf is comparability: everyone measures the same thing under the same
rules. Its limit is that those rules may not match your workload (your length distribution,
your specific SLO), so it serves to compare hardware and engines against each other, not
necessarily to size your case. For that, your own sweep.

---

## Measurement bias and reproducibility

That two benchmarks give very different results for the same system is no accident:
systematic measurement bias in production benchmarks is characterised in the literature
([arXiv 2605.24217](https://arxiv.org/html/2605.24217)), and there is work dedicated to the
meta-metrics and good practices of system-level performance benchmarking
([arXiv 2508.10251](https://arxiv.org/pdf/2508.10251)). The most common sources of bias:

| Source of bias | Effect | Mitigation |
|---|---|---|
| Single-process client | underestimates throughput at high concurrency | use multi-process load |
| Unrealistic length distribution | results that do not apply to your traffic | use realistic traces |
| Measuring the mean instead of percentiles | hides the latency tail | report P95/P99 |
| Uncontrolled warm-up | the prefix cache inflates the first results | discard the warm-up |
| Not pinning engine/model version | irreproducible | pin everything and publish it |

The methodological conclusion of this article: **a benchmark without a published methodology
is not comparable**. For a performance figure to support a proposal it has to come with the
tool, its version, the model and precision, the load distribution and the SLO. Synthesis
article S4 builds a reproducible harness that pins all of that.

---

## Checklist for a reproducible benchmark

For a performance figure to be defensible before a committee, it has to come with everything
that allows it to be reproduced. The minimum published alongside the result:

| What to pin | Why |
|---|---|
| Tool + version | each one measures differently; the version changes behaviour |
| Model + precision (FP16/FP8/INT4) | precision changes throughput and quality |
| Hardware (GPU, count, interconnect) | an 8×H100 NVLink is not an 8×H100 PCIe |
| Engine + version + flags | vLLM/SGLang/TRT-LLM and their configuration |
| Length distribution (in/out) | real traffic is not fixed length |
| Concurrency levels of the sweep | you have to go past the knee |
| SLO (which percentile, which threshold) | it defines the goodput |
| Warm-up handling | discard it or it biases the result |
| Tokenizer used for counting | it affects tok/s and cost per token |

The practical rule: if you cannot hand over this table alongside the figure, the figure is
not a datum, it is an anecdote. The reproducible harness of article S4 automates the recording
of all these parameters so that anyone, including whoever challenges the proposal, can
reproduce the exact number.

---

## Performance ≠ quality

One warning that avoids the most expensive mistake: these tools measure **speed and
throughput, not accuracy**. An engine can be blisteringly fast while serving a model that
answers badly. Quality is measured with another family of tools
(**lm-evaluation-harness**, **HELM**, task *leaderboards*) which is another axis of the
scorecard (article B7). Confusing "fast" with "good" is how you build a platform that serves
bad answers very quickly. On the final Pareto frontier, performance and quality are two
distinct axes that must be looked at together, never one instead of the other.

The other family of tools, for reference (expanded in B7):

| Tool | What it measures | Common trap |
|---|---|---|
| **lm-evaluation-harness** | accuracy on hundreds of standardised tasks | contamination of the test dataset |
| **HELM** | holistic evaluation (accuracy, robustness, bias, efficiency) | heavy to run |
| **LiveBench / dynamic leaderboards** | tasks that rotate to avoid contamination | comparability over time |

The data point: contamination, the model having seen the test during training, inflates
quality metrics just as warm-up inflates performance ones. That is why dynamic leaderboards
rotate their questions. Quality and performance share that lesson: the measurement method
biases the result as much as the system being measured.

---

## The latency versus throughput trade-off

A property the sweep reveals and which is worth keeping in mind: **latency and throughput
pull in opposite directions**. Batching groups requests to amortise the cost of moving the
weights out of VRAM, which raises throughput, but each request waits for the batch to form,
which raises individual latency. There are two operating regimes, and the benchmark
serves to place you in the right one:

| Regime | Optimises | Configuration | Case |
|---|---|---|---|
| **Latency** | low TTFT/TPOT | small batch, low concurrency | interactive chat, copilots |
| **Throughput** | maximum tok/s | large batch, high concurrency | overnight batch, ingestion |

There is no single "best" point: there is the best point for your SLO. A benchmark that
reports only maximum throughput is describing the throughput regime and ignoring whether that
point meets the latency your case needs. That is why **goodput**, throughput under the SLO,
is the metric that reconciles the two regimes: it measures how much throughput you get
without stepping outside acceptable latency. The sweep traverses the curve between both
regimes; your SLO marks where on that curve your system sits.

---

## The connection with cost and energy

Performance is not an isolated axis: by the identity of the [opening article](https://blog.lo0.es/en/posts/cost-performance-energy-three-axes-llm-inference/index.md),
throughput is the denominator of cost per token and of energy per token. A sweep that finds a
knee at 4,000 tok/s rather than 2,800 is not just "faster": it lowers the CPM from ~1.09 to
~0.76 €/1M tok and the energy per token in the same proportion, on the same iron. That is
why performance benchmarking is the tool that, indirectly, moves cost the most: every goodput
improvement translates into euros and into watts per token. The number that connects the
three axes is **goodput**, the throughput that meets the SLO, not catalogue throughput.

---

## State of the art 2026

- **genai-perf → AIPerf migration** (15 Apr 2026): NVIDIA consolidates its benchmarking into
  a multi-process tool with saturation detection.
- **GuideLLM** as the OSS standard for SLO-driven evaluation with reproducible sweeps.
- **MLPerf Inference v5.0/v5.1** extends to 405B, interactive 70B and MoE, with record
  participation: cross-vendor comparability is maturing.
- **Measurement bias characterised**: the community accepts that the measurement method
  matters as much as the system measured; the emphasis on reproducibility and meta-metrics is
  growing.

---

## Limits and traps (data-driven)

1. **Comparing tools from different classes.** A single-process micro-bench and a
   multi-process load generator are not comparable; the difference can be 7×. Pin the class.
2. **Catalogue throughput instead of goodput.** The honest number is the one that meets the
   SLO.
3. **Means instead of percentiles.** The mean hides the tail; report P95/P99.
4. **Not extending the sweep beyond the knee.** Without seeing where latency takes off you do
   not know the safe capacity.
5. **Confusing performance with quality.** They are distinct axes; fast is not good.
6. **Not pinning versions.** Engine, model, precision and load left unpinned = irreproducible
   = not defensible.

The next article in the track (B2) goes into the tool catalogue in depth; this one pins the
metrics and the methodology. With performance measured reproducibly, the scorecard can cross
it with cost (in €) and energy for the final decision.

## Closing

Performance benchmarking looks like the most "objective" of the three axes (they are tokens
per second, after all) and it is exactly where the most manipulation happens, almost always
without bad intent: a single-process tool here, a mean instead of a P99 there, a catalogue
throughput instead of the goodput. The difference between a marketing number and a defensible
datum lies not in the engine measured but in the **methodology**: the class of tool, where
the clock is measured, with which tokenizer, how far the sweep goes and which SLO defines the
goodput. For a sovereign architecture proposal, performance only counts if it is delivered
with that reproducibility record, and, crossed with cost in euros and energy per token, it
becomes the column of the Pareto frontier that decides which engine and which configuration
hold up the platform. The number you defend is not the highest one: it is the **reproducible
goodput**.

## See also

- [LLM serving engines compared (vLLM/SGLang/TRT-LLM/Dynamo)](https://blog.lo0.es/en/posts/llm-serving-engines-pareto-frontier-comparison/index.md) — choosing the engine after measuring goodput: from the methodology of this article to the Pareto frontier.
- [MLPerf Inference: how to read it and what comparability it offers](https://blog.lo0.es/en/posts/mlperf-inference-reading-loadgen-comparability/index.md) — the industry standard: the four scenarios, the TTFT/TPOT SLOs of the Server scenario and the rules for comparing results.
- [Measurement bias and reproducibility](https://blog.lo0.es/en/posts/measurement-bias-reproducibility-llm-benchmarks/index.md) — sources of systematic bias in performance benchmarks and how to control them to obtain defensible data.

## Sources

- Anyscale · métricas de latencia y throughput de LLM — https://docs.anyscale.com/llm/serving/benchmarking/metrics
- Red Hat · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
- GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
- NVIDIA AIPerf · guía de benchmarking (ex genai-perf) — https://lucaberton.com/blog/nvidia-aiperf-llm-inference-benchmarking-guide/
- MLCommons · MLPerf Inference datacenter (escenarios) — https://mlcommons.org/benchmarks/inference-datacenter/
- MLCommons · MLPerf Inference v5.0 (405B + 70B interactivo) — https://mlcommons.org/2025/04/llm-inference-v5/
- MLCommons · MLPerf Inference v5.1 — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
- arXiv 2605.24217 · sesgo sistemático de medición en benchmarks de inferencia LLM — https://arxiv.org/html/2605.24217
- arXiv 2508.10251 · meta-métricas y buenas prácticas de benchmarking de rendimiento — https://arxiv.org/pdf/2508.10251
- Medium · LLM Inference Benchmarking (genAI-perf y vLLM) — https://kchandan.medium.com/llm-inference-benchmarking-genai-perf-and-vllm-5dd06b57428e