---
title: "LLM benchmark tool catalogue: the practical deep dive"
url: https://blog.lo0.es/en/posts/llm-benchmark-tool-catalogue-practical-deep-dive/
markdown: https://blog.lo0.es/en/posts/llm-benchmark-tool-catalogue-practical-deep-dive/index.md
lang: en
date: 2026-06-14
lastmod: 2026-09-13
author: "José Manuel Román"
description: "A practical deep dive into LLM inference benchmarking tooling: what each tool measures, how it is invoked (real commands), what output format it produces and when to choose which. vLLM bench serve, SGLang bench, AIPerf, GuideLLM, LLMPerf and MLPerf, with the cost in euros of measuring. State of the art 2026, with sources."
tags: ["benchmarking","vllm","guidellm","aiperf","llmperf","mlperf","sglang","data-series"]
translations:
  es: https://blog.lo0.es/posts/herramientas-benchmark-llm-ficha-a-ficha/index.md
---

# LLM benchmark tool catalogue: the practical deep dive


> Notation: amounts in **euros (N €)**, decimals with a point. The dollar symbol is not used
> (on this site it is a formula delimiter).

## What this article covers

Second article in the **benchmarking** track (B2). The [B1 introduction](https://blog.lo0.es/en/posts/llm-inference-benchmarking-frameworks-metrics-state-of-the-art/index.md)
pinned the metrics and the methodology; this article is the **practical manual**: for each
tool, what it measures, how it is actually invoked (with the command), what output format
it produces and when to choose it. The goal is that, after reading it, you know what to run to
measure your engine and you can reproduce the number. No universal recommendations; only the
mechanics of each tool and the criterion for choosing.

---

## How to choose: the tool map

Recalling the split from B1, the tools are ordered on two axes: **micro-bench**
(single-process, for tuning an engine) versus **load generator** (multi-process, for measuring
real capacity), and **engine-native** versus **endpoint-agnostic**.

> **Diagram:** Map of benchmark tools on two axes: single-process micro-bench versus multi-process load generator, and engine-native versus endpoint-agnostic — [see the HTML version](https://blog.lo0.es/en/posts/llm-benchmark-tool-catalogue-practical-deep-dive/)

---

## Datasets: the synthetic load matters as much as the tool

Before the profiles, one point that changes results without anyone noticing: **what load you
feed it**. The tools accept different kinds of dataset, and each one measures something
different:

| Dataset | What it simulates | Bias it introduces |
|---|---|---|
| **random** (fixed lengths) | controlled uniform load | unrealistic: traffic is not fixed length |
| **sharegpt** (real conversations) | realistic prompt distribution | the de facto standard for comparison |
| **your own traces** | your real traffic | the most faithful, but specific to your case |

The trap: a benchmark with short, fixed-length prompts gives a very high throughput that
looks nothing like production, where lengths vary and long prompts dominate the prefill
cost. For a defensible figure, use **sharegpt** (comparability) or, better, **traces from your
own traffic** (fidelity). And always declare the length distribution (prompt/output) alongside
the number, because two benchmarks with different datasets are not comparable even if they
use the same tool.

---

## vLLM bench serve

What it measures: TTFT, TPOT, throughput and latencies of the vLLM server under a
synthetic load. Class: micro-bench (single-process). It is the tool for **tuning vLLM** and
seeing the effect of its optimisations ([decode](https://blog.lo0.es/en/posts/optimising-decode-vllm-squeezing-every-token-small-hardware/index.md),
[prefill](https://blog.lo0.es/en/posts/optimising-prefill-vllm-knobs-ttft-will-not-forgive/index.md)).

Typical invocation ([vLLM · benchmark CLI](https://docs.vllm.ai/en/latest/cli/bench/serve/)):

```bash
vllm bench serve \
  --backend vllm \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --endpoint /v1/completions \
  --dataset-name sharegpt \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 1000
```

Key parameters: `--num-prompts` (load), `--dataset-name` (sharegpt, random, etc.),
`--request-rate` (requests/s). The output is a console summary with TTFT, TPOT, throughput
and percentiles, and it can be dumped to JSON. To sweep concurrencies there is **`vllm bench
sweep serve`**, which automates the sweep ([vLLM · sweep](https://docs.vllm.ai/en/latest/cli/bench/sweep/serve/)).

Limit: single-process, it saturates on the client at high concurrency; less flexible than
GuideLLM in datasets and load patterns. Use it to iterate quickly on the vLLM config, not to
measure maximum capacity at scale.

---

## SGLang bench

What it measures: the equivalent for the **SGLang** engine (TTFT, TPOT, throughput). Class:
micro-bench. Use: tuning SGLang and comparing it against itself across configurations. The
mechanics are analogous to `vllm bench serve`: a synthetic load, console/JSON output with the
same metrics. If you are evaluating SGLang against vLLM, do not compare them with their
respective micro-benches (same class but different implementations): use an endpoint-agnostic
load generator (GuideLLM/AIPerf) against both endpoints, so that the tool is not the variable.

---

## AIPerf (NVIDIA, formerly genai-perf)

What it measures: TTFT, ITL, throughput and latency against any compatible endpoint (vLLM,
NIM, TGI, SGLang). Class: **multi-process** load generator. It is the successor to genai-perf
(retired on 15 Apr 2026). Its distinguishing feature: during the sweep it **detects GPU
saturation** and returns the previous iteration as `estimatedCapacity`, which is why the sweep
has to extend beyond the knee
([AIPerf](https://lucaberton.com/blog/nvidia-aiperf-llm-inference-benchmarking-guide/)).

Output: structured metrics (JSON) with TTFT/ITL distributions and the `estimatedCapacity`. Use
it when you want the real capacity of an endpoint, whatever the engine, with automatic
detection of the saturation point.

---

## GuideLLM (vLLM project)

What it measures: full distributions of TTFT, ITL and end-to-end behaviour, for **SLO-driven**
evaluation. Class: multi-process load generator. It is the recommended tool for benchmarking
vLLM servers in production: more flexible than `vllm bench serve` in dataset loading, request
format and traffic patterns, with live progress and automatic report generation
([Red Hat](https://developers.redhat.com/articles/2025/12/24/how-deploy-and-benchmark-vllm-guidellm-kubernetes)).

Typical invocation:

```bash
guidellm benchmark \
  --target "http://localhost:8000" \
  --rate-type throughput \
  --max-requests 1000 \
  --data "samples=1000,prompt_tokens=1024,output_tokens=256"
```

Key parameters: `--rate-type` (`synchronous`, `concurrent`, `throughput`, or rate-based),
`--data` (the load specification: number of samples and prompt/output lengths), `--target`
(the endpoint). It generates reproducible sweeps to find the safe operating range under
SLO, with full distributions (not just means). Output: a report with percentiles and, often,
exportable to JSON/HTML. It is the default option for answering "how hard can I load this
engine without breaking the SLO?".

---

## LLMPerf (Anyscale/Ray)

What it measures: throughput and latency at the inference level. Class: load generator. Use:
endpoint validation, historically very widespread in the Ray/Anyscale ecosystem. It is a solid
and well-known option, though less focused on distributions and sweeps than GuideLLM/AIPerf.
It fits if you already operate on Ray or want a simple tool to validate an endpoint.

---

## inference-benchmarker (Hugging Face)

What it measures: latency and throughput of inference endpoints, aimed at producing comparable
reports. Class: load generator. Use: an OSS alternative inside the Hugging Face ecosystem,
useful if you already work with TGI or the HF stack. Like the others in its class, its value
lies in measuring real capacity with distributed load; the choice between this one, GuideLLM
and AIPerf usually comes down to the ecosystem you already operate in rather than to deep
differences in what they measure.

---

## GuideLLM versus AIPerf: which of the two load generators

They are the two serious multi-process load options, and they overlap a great deal. The
practical differences that tip the choice:

| Criterion | GuideLLM | AIPerf |
|---|---|---|
| Origin | vLLM project (Red Hat) | NVIDIA (successor to genai-perf) |
| Focus | **SLO**-driven evaluation, reproducible sweeps | real capacity, automatic **estimatedCapacity** |
| Load patterns | synchronous, concurrent, rate-based | sweep with saturation detection |
| Report | live progress + automatic report | structured metrics (JSON) |
| Ecosystem | vLLM / OpenShift AI | NVIDIA NIM / Triton / vLLM |

In practice: if your question is "how hard can I load it without breaking the SLO?",
**GuideLLM** answers it more directly with its SLO-driven sweeps; if your question is "what is
the maximum capacity of this endpoint?", **AIPerf** gives it with its automatic knee
detection. Many teams use both: GuideLLM for the operational SLO, AIPerf for the reference
capacity. What you must not do is compare a GuideLLM result with an AIPerf one as if they
were the same measurement: although both are multi-process, their sweep methodology differs;
pick one for a given comparison and stick to it.

---

## MLPerf Inference (MLCommons)

What it measures: performance under standardised scenarios (Offline, Server, Interactive)
with strict rules. Unlike the previous ones, MLPerf is not "run" for your day-to-day case:
it is a vendor comparison suite whose results are read (published by NVIDIA, AMD, Intel and
others). Use it to compare hardware and engines against each other under identical rules, not
to size your specific workload. For that, GuideLLM/AIPerf against your endpoint.

---

## Practical comparison table

| Tool | Class | Base command | Output | When |
|---|---|---|---|---|
| **vllm bench serve** | micro | `vllm bench serve` | console/JSON | tune vLLM |
| **vllm bench sweep** | micro (sweep) | `vllm bench sweep serve` | console/JSON | sweep concurrency in vLLM |
| **SGLang bench** | micro | bench from the SGLang repo | console/JSON | tune SGLang |
| **AIPerf** | load | `aiperf profile …` | JSON + estimatedCapacity | real capacity, multi-endpoint |
| **GuideLLM** | load | `guidellm benchmark …` | report + JSON/HTML | SLO, reproducible sweep |
| **LLMPerf** | load | Ray/LLMPerf script | JSON | validate endpoint (Ray) |
| **MLPerf** | suite | (results are read) | official results | compare vendors |

---

## Summary: what you run for each question

So as not to get lost in the catalogue, the direct mapping from question to tool:

| Your question | Tool | Why |
|---|---|---|
| "Does my vLLM config change help?" | `vllm bench serve` (+ sweep) | fast, engine-native |
| "How hard can I load without breaking the SLO?" | GuideLLM | SLO-driven sweeps |
| "What is the maximum capacity of the endpoint?" | AIPerf | automatic knee detection |
| "vLLM or SGLang for my workload?" | GuideLLM/AIPerf against both | same tool, engine as the variable |
| "Which hardware/engine is better in the abstract?" | read MLPerf | cross-vendor comparability |
| "Is there a regression in this release?" | short sweep in CI against a baseline | continuous detection |

The rule underlying the whole table: **micro-bench to iterate on an engine, load generator to
measure capacity and decide, MLPerf to compare vendors**. And, for any comparison between
systems, the same tool for every candidate. If these three categories are clear to you, the
specific choice is secondary.

---

## Step-by-step benchmark methodology

A reliable run is not "launching the command": it is a procedure. The steps, with the load
tool (GuideLLM/AIPerf) against your endpoint:

> **Diagram:** Benchmark flow: deploy the engine, warm up, sweep concurrency, collect metrics and compare — [see the HTML version](https://blog.lo0.es/en/posts/llm-benchmark-tool-catalogue-practical-deep-dive/)

1. **Deploy** the engine with the exact config to be measured (model, precision, flags),
   pinned.
2. **Warm up**: send a few requests and discard them, so that the prefix cache and the
   autotuning do not inflate the first numbers.
3. **Sweep**: sweep rising concurrencies (1, 8, 16, 24, 32…) beyond the knee, to see where
   latency takes off.
4. **Collect** the output in JSON with percentiles (TTFT/ITL/throughput/goodput) and the
   metadata.
5. **Compare**: read the goodput under the SLO, not the maximum throughput.

Skipping step 2 (warm-up) or not going past the knee in step 3 are the two mistakes that bias
the result the most.

---

## Comparing two engines: the fair protocol

If the goal is to choose between vLLM and SGLang (or TRT-LLM), the protocol that avoids false
conclusions:

- **Same load tool** (GuideLLM or AIPerf) against both endpoints, never each engine's own
  micro-bench.
- **Same dataset and length distribution.**
- **Same hardware and precision.**
- **Same SLO** to compute the goodput of both.
- Vary only the engine; everything else fixed.

Only then is the difference you measure down to the engine and not to the tool, the dataset or
the hardware. It is the controlled experiment that supports the scorecard row (article B8).

---

## Output format and comparability

What you record alongside the number is what makes it comparable. A useful output includes,
in JSON so it can be versioned:

```json
{
  "tool": "guidellm", "version": "x.y.z",
  "model": "Llama-3.1-70B", "precision": "FP16",
  "hardware": "8xH100 SXM NVLink",
  "load": {"prompt_tokens": 1024, "output_tokens": 256, "concurrency": 16},
  "results": {"ttft_p50_ms": 180, "ttft_p99_ms": 460,
              "itl_p50_ms": 22, "throughput_tok_s": 3400, "goodput_tok_s": 3330}
}
```

Saving this for every run turns a benchmark into an auditable datum: anyone can reproduce the
figure with the same tool, version, model, hardware and load. It is the material of the
reproducible harness (article S4), and the difference between a defensible number and a
console screenshot.

---

## Worked example: reading the output of a sweep

An illustrative output from a GuideLLM sweep on a 70B on 8×H100 (SLO: P99 TTFT < 500 ms), just
as you would read it:

| Concurrency | TTFT P99 (ms) | ITL P50 (ms) | Throughput (tok/s) | Goodput (tok/s) |
|---|---|---|---|---|
| 8 | 240 | 20 | 2,100 | 2,100 |
| 16 | 460 | 22 | 3,400 | 3,330 |
| 24 | 980 | 31 | 3,900 | 2,420 |
| 32 | 1,800 | 54 | 4,000 | 800 |

How to read it: the **knee** sits between 16 and 24. At concurrency 16, P99 (460 ms) meets the
SLO and goodput (3,330 tok/s) ≈ throughput. At 24, throughput rises little (3,400 → 3,900) but
P99 already violates the SLO and goodput falls to 2,420. At 32, throughput is at its
maximum (4,000) but goodput collapses to 800: the system "performs well" while serving requests
that do not comply. The defensible capacity is the one at concurrency 16 (3,330 useful
tok/s), and that is the number that goes into [capacity planning](https://blog.lo0.es/en/posts/capacity-planning-on-premise-llm-inference/index.md)
and into cost per token. Anyone reporting "4,000 tok/s" is describing the point where the
system no longer meets its SLO.

---

## Automating the harness

A manual run does not scale to a benchmarking programme. The minimum automation:

- An **idempotent script** that deploys the engine, warms it up, runs the sweep and saves the
  JSON with all the metadata (model, version, hardware, dataset, SLO).
- **Naming by date and config**, to version the runs and compare over time.
- A **results store** (a git repo of JSONs, or a bucket) so that anyone can reproduce and
  compare.

The goal is that reproducing a number is one command, not an afternoon. It is the basis of the
harness in article S4, and what turns benchmarking from a one-off activity into a continuous
platform capability.

---

## CI integration: continuous benchmarking

The next level is measuring on every change: a CI job that, when the engine or the config
is updated, launches a short sweep against a test environment and **compares against the
baseline**. If goodput falls by more than a threshold, the pipeline fails. That way a
performance regression is caught at the commit, not in production. Watch out for two things:
benchmarking in CI consumes GPU-hours (budget for it) and it needs a stable environment (same
hardware) for the comparison to be valid. You do not need the full sweep on every commit: a
short sweep covering the knee is enough to detect regressions; the exhaustive sweep is for
releases.

---

## The cost of measuring (in euros)

Benchmarking consumes GPU-hours, and that has a cost worth budgeting for. A serious sweep
on an 8×H100 node can occupy the cards for a couple of hours; at an amortised cost of ~11 €/h,
that is ~22 € per full sweep, more if you sweep several models and precisions. It is not much
per run, but a continuous benchmarking programme (every release, every config change) adds up.
The practical rule: automate the harness so that each run is cheap and reproducible, and
measure what you are going to use to decide, not for the sake of exhaustiveness. The cost of
measuring is part of the cost of the platform, small next to the cost of serving but real.

---

## Observing during the benchmark: one run, three axes

A trick that saves work and connects the series: **while the sweep runs, capture the GPU
metrics too**. With DCGM exporting to Prometheus during the run, you record at the same time:

| Axis | Source during the sweep | Metric |
|---|---|---|
| Performance | the tool (GuideLLM/AIPerf) | TTFT, ITL, throughput, goodput |
| Energy | DCGM | power (W) → J/token |
| Cost | node price (OpenCost) | €/hour → CPM per point |

That way, from a single run you get the three numbers for the same operating point: at
concurrency 16, the goodput (3,330 tok/s), the average power (from DCGM, which divided by the
throughput gives J/token) and the cost per token (with the node price). Instead of three
separate campaigns, you measure the three axes at once, and they end up **coherent by
construction** because they correspond to the same instant and the same load. It is what the
harness in article S4 does, and the reason to export DCGM during the benchmark even if you are
only after performance: energy and cost come almost free if you capture them in the same
window.

The methodological warning: align the time windows. The DCGM power figures and the tool's
metrics have to cover exactly the same interval (no warm-up, no shutdown), or the J/token does
not correspond to the throughput measured. Same window, same three numbers.

---

## The benchmark is also FinOps and energy

An idea that connects this article with the rest of the series: **measuring performance is, in
fact, measuring cost and energy**. By the identity of the [opening article](https://blog.lo0.es/en/posts/cost-performance-energy-three-axes-llm-inference/index.md),
goodput is the denominator of cost per token and of energy per token. When a sweep reveals
that config A gives 3,330 tok/s of goodput and config B gives 4,000, you are not only measuring
speed: you are measuring that B costs fewer euros and fewer watts per token. That is why the
output JSON of a benchmark should be accompanied by the iron cost (from OpenCost) to compute
the real CPM of every point in the sweep: throughput × node price = cost per token.
Benchmarking is not an isolated axis; it is the tool that, indirectly, moves the cost and the
energy of the platform the most, and the one that fills the performance column of the scorecard
with numbers that translate directly into euros.

The operational consequence: do not benchmark performance in a vacuum. Every run you save with
its throughput and its goodput should be crossable with the node cost (€/hour) and the energy
(J/token) to give the three faces of the same operating point. That is how the benchmarking
track feeds the FinOps and energy tracks instead of living apart.

---

## State of the art 2026 and limits

- **Migration to multi-process**: AIPerf (formerly genai-perf) and GuideLLM consolidate the
  measurement of real capacity; micro-benches are left for tuning engines.
- **GuideLLM as the OSS standard** for SLO-driven evaluation with an automatic report.
- **Careful with comparing across classes**: a micro-bench and a load generator are not
  comparable; pin the class and the tool.
- **Version and dataset matter**: the same command with a different dataset or version gives a
  different number; pin them.
- **MLPerf does not size your case**: it compares vendors, it does not replace a sweep against
  your workload.

With the practical catalogue covered, the next article in the track (B3) goes into GuideLLM
and SLO validation under load in depth. The tool is the means; the reproducible datum is the
end.

## Mistakes that invalidate a benchmark

To finish, the list of what turns a run into rubbish, by frequency:

| Mistake | Effect | Fix |
|---|---|---|
| Comparing different classes | up to 7× of spurious difference | same tool for everyone |
| Not discarding the warm-up | artificially low TTFT | warm up and discard |
| Not going past the knee | you do not know the safe capacity | extend the sweep |
| Unrealistic dataset (fixed length) | throughput that does not apply | sharegpt or your own traces |
| Reporting the mean, not P99 | hides the tail | percentiles always |
| Tokenizer foreign to the model | biased tok/s and cost per token | count with the model's own |
| Not pinning version/config | irreproducible | save everything in the JSON |

Any one of these is enough for the number not to be defensible. A benchmark is only as good as
its methodology: the tool matters less than running it properly and recording everything.

## Closing

The catalogue of benchmark tools boils down to a simple decision (micro-bench to tune an
engine, load generator to measure capacity, MLPerf to compare vendors) and to a discipline that
weighs more than the choice: **the method**. The same `guidellm benchmark` gives a golden datum
or a useless one depending on the dataset, the warm-up, how far the sweep goes and what you
record in the output. For a sovereign architecture proposal, performance only counts if it
comes with its command, its version, its load and its goodput under SLO, and, crossed with cost
in euros and energy per token, it becomes the scorecard column that decides which engine and
which configuration hold up the platform. You choose the tool in five minutes; the methodology
is what makes the number survive an audit.

## See also

- [GenAI-Perf in depth](https://blog.lo0.es/en/posts/genai-perf-llm-inference-profiling-in-depth/index.md) — an extended profile of the NVIDIA profiler: metrics (TTFT/TPOT/ISL/OSL), invocation against an OpenAI-compatible endpoint and comparison with GuideLLM/LLMPerf.
- [LLM serving engines compared (vLLM/SGLang/TRT-LLM/Dynamo)](https://blog.lo0.es/en/posts/llm-serving-engines-pareto-frontier-comparison/index.md) — once goodput has been measured with these tools, here is which engine wins at each point of the Pareto frontier.
- [Measurement bias and reproducibility](https://blog.lo0.es/en/posts/measurement-bias-reproducibility-llm-benchmarks/index.md) — the sources of bias that invalidate results even when the tool is correctly configured: warm-up, real versus synthetic dataset, shared environment.

## Sources

- vLLM · CLI de benchmark (`bench serve`) — https://docs.vllm.ai/en/latest/cli/bench/serve/
- vLLM · `bench sweep serve` — https://docs.vllm.ai/en/latest/cli/bench/sweep/serve/
- Red Hat · desplegar y benchmarkear vLLM con GuideLLM en Kubernetes — https://developers.redhat.com/articles/2025/12/24/how-deploy-and-benchmark-vllm-guidellm-kubernetes
- GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
- NVIDIA AIPerf · guía de benchmarking — https://lucaberton.com/blog/nvidia-aiperf-llm-inference-benchmarking-guide/
- Medium · Benchmarking LLM Serving Performance (guía) — https://medium.com/@kimdoil1211/benchmarking-llm-serving-performance-a-comprehensive-guide-db94b1bfe8cf