LLM benchmark tool catalogue: the practical deep dive

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

What this article covers

Second article in the benchmarking track (B2). The B1 introduction pinned the metrics and the methodology; this article is the practical manual: for each tool, what it measures, how it is actually invoked (with the command), what output format it produces and when to choose it. The goal is that, after reading it, you know what to run to measure your engine and you can reproduce the number. No universal recommendations; only the mechanics of each tool and the criterion for choosing.

How to choose: the tool map

Recalling the split from B1, the tools are ordered on two axes: micro-bench (single-process, for tuning an engine) versus load generator (multi-process, for measuring real capacity), and engine-native versus endpoint-agnostic.

Datasets: the synthetic load matters as much as the tool

Before the profiles, one point that changes results without anyone noticing: what load you feed it. The tools accept different kinds of dataset, and each one measures something different:

The trap: a benchmark with short, fixed-length prompts gives a very high throughput that looks nothing like production, where lengths vary and long prompts dominate the prefill cost. For a defensible figure, use sharegpt (comparability) or, better, traces from your own traffic (fidelity). And always declare the length distribution (prompt/output) alongside the number, because two benchmarks with different datasets are not comparable even if they use the same tool.

vLLM bench serve

What it measures: TTFT, TPOT, throughput and latencies of the vLLM server under a synthetic load. Class: micro-bench (single-process). It is the tool for tuning vLLM and seeing the effect of its optimisations (decode, prefill).

Typical invocation (vLLM · benchmark CLI):

vllm bench serve \
  --backend vllm \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --endpoint /v1/completions \
  --dataset-name sharegpt \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 1000

Key parameters: --num-prompts (load), --dataset-name (sharegpt, random, etc.), --request-rate (requests/s). The output is a console summary with TTFT, TPOT, throughput and percentiles, and it can be dumped to JSON. To sweep concurrencies there is vllm bench sweep serve, which automates the sweep (vLLM · sweep).

Limit: single-process, it saturates on the client at high concurrency; less flexible than GuideLLM in datasets and load patterns. Use it to iterate quickly on the vLLM config, not to measure maximum capacity at scale.

SGLang bench

What it measures: the equivalent for the SGLang engine (TTFT, TPOT, throughput). Class: micro-bench. Use: tuning SGLang and comparing it against itself across configurations. The mechanics are analogous to vllm bench serve: a synthetic load, console/JSON output with the same metrics. If you are evaluating SGLang against vLLM, do not compare them with their respective micro-benches (same class but different implementations): use an endpoint-agnostic load generator (GuideLLM/AIPerf) against both endpoints, so that the tool is not the variable.

AIPerf (NVIDIA, formerly genai-perf)

What it measures: TTFT, ITL, throughput and latency against any compatible endpoint (vLLM, NIM, TGI, SGLang). Class: multi-process load generator. It is the successor to genai-perf (retired on 15 Apr 2026). Its distinguishing feature: during the sweep it detects GPU saturation and returns the previous iteration as estimatedCapacity, which is why the sweep has to extend beyond the knee (AIPerf).

Output: structured metrics (JSON) with TTFT/ITL distributions and the estimatedCapacity. Use it when you want the real capacity of an endpoint, whatever the engine, with automatic detection of the saturation point.

GuideLLM (vLLM project)

What it measures: full distributions of TTFT, ITL and end-to-end behaviour, for SLO-driven evaluation. Class: multi-process load generator. It is the recommended tool for benchmarking vLLM servers in production: more flexible than vllm bench serve in dataset loading, request format and traffic patterns, with live progress and automatic report generation (Red Hat).

Typical invocation:

guidellm benchmark \
  --target "http://localhost:8000" \
  --rate-type throughput \
  --max-requests 1000 \
  --data "samples=1000,prompt_tokens=1024,output_tokens=256"

Key parameters: --rate-type (synchronous, concurrent, throughput, or rate-based), --data (the load specification: number of samples and prompt/output lengths), --target (the endpoint). It generates reproducible sweeps to find the safe operating range under SLO, with full distributions (not just means). Output: a report with percentiles and, often, exportable to JSON/HTML. It is the default option for answering “how hard can I load this engine without breaking the SLO?”.

LLMPerf (Anyscale/Ray)

What it measures: throughput and latency at the inference level. Class: load generator. Use: endpoint validation, historically very widespread in the Ray/Anyscale ecosystem. It is a solid and well-known option, though less focused on distributions and sweeps than GuideLLM/AIPerf. It fits if you already operate on Ray or want a simple tool to validate an endpoint.

inference-benchmarker (Hugging Face)

What it measures: latency and throughput of inference endpoints, aimed at producing comparable reports. Class: load generator. Use: an OSS alternative inside the Hugging Face ecosystem, useful if you already work with TGI or the HF stack. Like the others in its class, its value lies in measuring real capacity with distributed load; the choice between this one, GuideLLM and AIPerf usually comes down to the ecosystem you already operate in rather than to deep differences in what they measure.

GuideLLM versus AIPerf: which of the two load generators

They are the two serious multi-process load options, and they overlap a great deal. The practical differences that tip the choice:

In practice: if your question is “how hard can I load it without breaking the SLO?”, GuideLLM answers it more directly with its SLO-driven sweeps; if your question is “what is the maximum capacity of this endpoint?”, AIPerf gives it with its automatic knee detection. Many teams use both: GuideLLM for the operational SLO, AIPerf for the reference capacity. What you must not do is compare a GuideLLM result with an AIPerf one as if they were the same measurement: although both are multi-process, their sweep methodology differs; pick one for a given comparison and stick to it.

MLPerf Inference (MLCommons)

What it measures: performance under standardised scenarios (Offline, Server, Interactive) with strict rules. Unlike the previous ones, MLPerf is not “run” for your day-to-day case: it is a vendor comparison suite whose results are read (published by NVIDIA, AMD, Intel and others). Use it to compare hardware and engines against each other under identical rules, not to size your specific workload. For that, GuideLLM/AIPerf against your endpoint.

Practical comparison table

Summary: what you run for each question

So as not to get lost in the catalogue, the direct mapping from question to tool:

The rule underlying the whole table: micro-bench to iterate on an engine, load generator to measure capacity and decide, MLPerf to compare vendors. And, for any comparison between systems, the same tool for every candidate. If these three categories are clear to you, the specific choice is secondary.

Step-by-step benchmark methodology

A reliable run is not “launching the command”: it is a procedure. The steps, with the load tool (GuideLLM/AIPerf) against your endpoint:

1. Deploy the engine with the exact config to be measured (model, precision, flags), pinned.
2. Warm up: send a few requests and discard them, so that the prefix cache and the autotuning do not inflate the first numbers.
3. Sweep: sweep rising concurrencies (1, 8, 16, 24, 32…) beyond the knee, to see where latency takes off.
4. Collect the output in JSON with percentiles (TTFT/ITL/throughput/goodput) and the metadata.
5. Compare: read the goodput under the SLO, not the maximum throughput.

Skipping step 2 (warm-up) or not going past the knee in step 3 are the two mistakes that bias the result the most.

Comparing two engines: the fair protocol

If the goal is to choose between vLLM and SGLang (or TRT-LLM), the protocol that avoids false conclusions:

• Same load tool (GuideLLM or AIPerf) against both endpoints, never each engine’s own micro-bench.
• Same dataset and length distribution.
• Same hardware and precision.
• Same SLO to compute the goodput of both.
• Vary only the engine; everything else fixed.

Only then is the difference you measure down to the engine and not to the tool, the dataset or the hardware. It is the controlled experiment that supports the scorecard row (article B8).

Output format and comparability

What you record alongside the number is what makes it comparable. A useful output includes, in JSON so it can be versioned:

{
  "tool": "guidellm", "version": "x.y.z",
  "model": "Llama-3.1-70B", "precision": "FP16",
  "hardware": "8xH100 SXM NVLink",
  "load": {"prompt_tokens": 1024, "output_tokens": 256, "concurrency": 16},
  "results": {"ttft_p50_ms": 180, "ttft_p99_ms": 460,
              "itl_p50_ms": 22, "throughput_tok_s": 3400, "goodput_tok_s": 3330}
}

Saving this for every run turns a benchmark into an auditable datum: anyone can reproduce the figure with the same tool, version, model, hardware and load. It is the material of the reproducible harness (article S4), and the difference between a defensible number and a console screenshot.

Worked example: reading the output of a sweep

An illustrative output from a GuideLLM sweep on a 70B on 8×H100 (SLO: P99 TTFT < 500 ms), just as you would read it:

How to read it: the knee sits between 16 and 24. At concurrency 16, P99 (460 ms) meets the SLO and goodput (3,330 tok/s) ≈ throughput. At 24, throughput rises little (3,400 → 3,900) but P99 already violates the SLO and goodput falls to 2,420. At 32, throughput is at its maximum (4,000) but goodput collapses to 800: the system “performs well” while serving requests that do not comply. The defensible capacity is the one at concurrency 16 (3,330 useful tok/s), and that is the number that goes into capacity planning and into cost per token. Anyone reporting “4,000 tok/s” is describing the point where the system no longer meets its SLO.

Automating the harness

A manual run does not scale to a benchmarking programme. The minimum automation:

• An idempotent script that deploys the engine, warms it up, runs the sweep and saves the JSON with all the metadata (model, version, hardware, dataset, SLO).
• Naming by date and config, to version the runs and compare over time.
• A results store (a git repo of JSONs, or a bucket) so that anyone can reproduce and compare.

The goal is that reproducing a number is one command, not an afternoon. It is the basis of the harness in article S4, and what turns benchmarking from a one-off activity into a continuous platform capability.

CI integration: continuous benchmarking

The next level is measuring on every change: a CI job that, when the engine or the config is updated, launches a short sweep against a test environment and compares against the baseline. If goodput falls by more than a threshold, the pipeline fails. That way a performance regression is caught at the commit, not in production. Watch out for two things: benchmarking in CI consumes GPU-hours (budget for it) and it needs a stable environment (same hardware) for the comparison to be valid. You do not need the full sweep on every commit: a short sweep covering the knee is enough to detect regressions; the exhaustive sweep is for releases.

The cost of measuring (in euros)

Benchmarking consumes GPU-hours, and that has a cost worth budgeting for. A serious sweep on an 8×H100 node can occupy the cards for a couple of hours; at an amortised cost of ~11 €/h, that is ~22 € per full sweep, more if you sweep several models and precisions. It is not much per run, but a continuous benchmarking programme (every release, every config change) adds up. The practical rule: automate the harness so that each run is cheap and reproducible, and measure what you are going to use to decide, not for the sake of exhaustiveness. The cost of measuring is part of the cost of the platform, small next to the cost of serving but real.

Observing during the benchmark: one run, three axes

A trick that saves work and connects the series: while the sweep runs, capture the GPU metrics too. With DCGM exporting to Prometheus during the run, you record at the same time:

That way, from a single run you get the three numbers for the same operating point: at concurrency 16, the goodput (3,330 tok/s), the average power (from DCGM, which divided by the throughput gives J/token) and the cost per token (with the node price). Instead of three separate campaigns, you measure the three axes at once, and they end up coherent by construction because they correspond to the same instant and the same load. It is what the harness in article S4 does, and the reason to export DCGM during the benchmark even if you are only after performance: energy and cost come almost free if you capture them in the same window.

The methodological warning: align the time windows. The DCGM power figures and the tool’s metrics have to cover exactly the same interval (no warm-up, no shutdown), or the J/token does not correspond to the throughput measured. Same window, same three numbers.

The benchmark is also FinOps and energy

An idea that connects this article with the rest of the series: measuring performance is, in fact, measuring cost and energy. By the identity of the opening article, goodput is the denominator of cost per token and of energy per token. When a sweep reveals that config A gives 3,330 tok/s of goodput and config B gives 4,000, you are not only measuring speed: you are measuring that B costs fewer euros and fewer watts per token. That is why the output JSON of a benchmark should be accompanied by the iron cost (from OpenCost) to compute the real CPM of every point in the sweep: throughput × node price = cost per token. Benchmarking is not an isolated axis; it is the tool that, indirectly, moves the cost and the energy of the platform the most, and the one that fills the performance column of the scorecard with numbers that translate directly into euros.

The operational consequence: do not benchmark performance in a vacuum. Every run you save with its throughput and its goodput should be crossable with the node cost (€/hour) and the energy (J/token) to give the three faces of the same operating point. That is how the benchmarking track feeds the FinOps and energy tracks instead of living apart.

State of the art 2026 and limits

• Migration to multi-process: AIPerf (formerly genai-perf) and GuideLLM consolidate the measurement of real capacity; micro-benches are left for tuning engines.
• GuideLLM as the OSS standard for SLO-driven evaluation with an automatic report.
• Careful with comparing across classes: a micro-bench and a load generator are not comparable; pin the class and the tool.
• Version and dataset matter: the same command with a different dataset or version gives a different number; pin them.
• MLPerf does not size your case: it compares vendors, it does not replace a sweep against your workload.

With the practical catalogue covered, the next article in the track (B3) goes into GuideLLM and SLO validation under load in depth. The tool is the means; the reproducible datum is the end.

Mistakes that invalidate a benchmark

To finish, the list of what turns a run into rubbish, by frequency:

Any one of these is enough for the number not to be defensible. A benchmark is only as good as its methodology: the tool matters less than running it properly and recording everything.

Closing

The catalogue of benchmark tools boils down to a simple decision (micro-bench to tune an engine, load generator to measure capacity, MLPerf to compare vendors) and to a discipline that weighs more than the choice: the method. The same guidellm benchmark gives a golden datum or a useless one depending on the dataset, the warm-up, how far the sweep goes and what you record in the output. For a sovereign architecture proposal, performance only counts if it comes with its command, its version, its load and its goodput under SLO, and, crossed with cost in euros and energy per token, it becomes the scorecard column that decides which engine and which configuration hold up the platform. You choose the tool in five minutes; the methodology is what makes the number survive an audit.

See also

• GenAI-Perf in depth — an extended profile of the NVIDIA profiler: metrics (TTFT/TPOT/ISL/OSL), invocation against an OpenAI-compatible endpoint and comparison with GuideLLM/LLMPerf.
• LLM serving engines compared (vLLM/SGLang/TRT-LLM/Dynamo) — once goodput has been measured with these tools, here is which engine wins at each point of the Pareto frontier.
• Measurement bias and reproducibility — the sources of bias that invalidate results even when the tool is correctly configured: warm-up, real versus synthetic dataset, shared environment.

Sources

• vLLM · CLI de benchmark (bench serve) — https://docs.vllm.ai/en/latest/cli/bench/serve/
• vLLM · bench sweep serve — https://docs.vllm.ai/en/latest/cli/bench/sweep/serve/
• Red Hat · desplegar y benchmarkear vLLM con GuideLLM en Kubernetes — https://developers.redhat.com/articles/2025/12/24/how-deploy-and-benchmark-vllm-guidellm-kubernetes
• GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
• NVIDIA AIPerf · guía de benchmarking — https://lucaberton.com/blog/nvidia-aiperf-llm-inference-benchmarking-guide/
• Medium · Benchmarking LLM Serving Performance (guía) — https://medium.com/@kimdoil1211/benchmarking-llm-serving-performance-a-comprehensive-guide-db94b1bfe8cf