GuideLLM in depth: validating the SLO under load and sizing from the knee

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Third article of the benchmarking track (B3). B2 covered the tool catalogue; here we go into GuideLLM in depth, because it answers the operational question that sizes a platform: how far can I load this engine without breaking the SLO? It is not a catalogue benchmark (“X tokens/s”), it is an SLO-driven sweep that finds the knee, the safe capacity, and from which the real replica count and cost per token come out. We will look at its load modes, how the SLO is defined, how to read the output and how to turn the knee into sizing and into euros. With real commands; no recommendations, only the mechanics.

Why GuideLLM

GuideLLM (a vLLM project) generates realistic, configurable traffic patterns and captures full distributions of TTFT, ITL and end-to-end behaviour, for SLO-driven evaluation, with reproducible sweeps that identify the safe operating range (Red Hat). Unlike a micro-benchmark, it measures the engine, not the client (multi-process load generation), and unlike MLPerf, it sizes your specific case (your model, your SLO, your load). It is the middle tool: not for tuning a flag, not for comparing vendors, but for deciding how many GPUs you need.

The load modes (rate-types)

GuideLLM offers several modes through --rate-type, and picking the right one is half the work:

In concurrent mode, --rate "1,2,4" launches three benchmarks at those concurrencies (Medium · GuideLLM on OpenShift). The poisson mode is the most realistic for online inference (requests do not arrive at regular intervals), and the sweep is the one that automates the search for the saturation point.

The automatic sweep: from idle to the knee

The flagship mode. When running a sweep profile, GuideLLM automatically increases the request rate from idle to maximum throughput across 10 rounds, and produces an interactive HTML report with detailed latency and throughput (Red Hat · GuideLLM on Kubernetes). During the sweep, on detecting saturation it identifies the previous iteration, the knee, and returns it as estimated capacity; if it does not detect saturation, the sweep has to be extended beyond the knee.

A worked example: reading a 10-round sweep

An illustrative output from a sweep on a 70B on 8×H100 (SLO: TTFT P99 < 500 ms), per round:

How to read it: up to round 6 (17 req/s) the TTFT P99 meets the SLO (460 ms) and goodput ≈ throughput (3,330 ≈ 3,400). In round 7 P99 already breaks (760 ms) and goodput drops to 2,520. The knee sits between 6 and 7: the safe capacity is that of round 6, ~3,330 tok/s of goodput. Rounds 9 and 10 give more raw throughput (3,950, 4,000) but with goodput of 900 and 380; the system is “performing” by mostly serving requests that miss the target. Whoever reports “4,000 tok/s” is describing round 10, where the system is broken. The defensible number is 3,330 tok/s under TTFT P99 < 500 ms, and that is the one that goes into the sizing. Note as well that the sweep had to reach round 10 (P99 of 2,400 ms) to see where it broke: that is why it has to be extended beyond the knee.

Defining the SLO: the number that decides the knee

The “knee” is not absolute: it depends on the SLO you define. A typical online inference SLO:

Goodput, the throughput that meets that SLO, is the metric that defines the knee: the safe capacity is the maximum load where goodput ≈ throughput. Changing the SLO moves the knee: a stricter TTFT SLO (200 ms) gives an earlier knee (less safe capacity) than a loose one (1 s). That is why the SLO is fixed before the sweep, and reported alongside the result: a capacity “of 3,330 tok/s” without saying under which SLO means nothing.

Execution: the command

An SLO-driven sweep against a vLLM endpoint:

guidellm benchmark \
  --target "http://vllm:8000" \
  --rate-type sweep \
  --max-seconds 120 \
  --data "prompt_tokens=1024,output_tokens=256" \
  --output-path resultados.json

Key parameters: --rate-type sweep (the automatic ramp), --data (the length distribution of the load, which must resemble your real traffic), --max-seconds (duration per round), --target (the endpoint). For the SLO, GuideLLM allows the latency constraints that define goodput to be set. The output goes to --output-path in JSON, YAML or CSV, in addition to the interactive HTML report.

Defining the load: --data decides the result

The --data parameter (the distribution of prompt and output lengths) changes the knee as much as the SLO does. Three forms, from least to most faithful:

The trap: a sweep with short fixed-length prompts gives an optimistic knee that does not resemble production, where long prompts dominate the prefill cost and bring the knee forward. To size properly, feed GuideLLM the real distribution of your traffic (prompt and output lengths measured in production). The knee of a sweep is only as realistic as the load you put into it; with an unrealistic --data, you are sizing for traffic that does not exist.

The arrival profile matters too: --rate-type poisson simulates random arrivals (like real online traffic), while concurrent keeps N fixed (like a batch). An online service measured with concurrent can give a knee different from the real one; for interactive inference, poisson is the honest profile.

The output: what to read

GuideLLM produces standardised, exportable reports for dashboards, analysis and regression tracking, in JSON, YAML and CSV (Red Hat). What matters to read, per sweep round:

• TTFT (P50, P95, P99) — the latency to the first token.
• ITL/TPOT (P50, P95, P99) — between tokens.
• Throughput (req/s and tok/s) — the raw figure.
• Goodput — the part that meets the SLO (the honest number).

The interactive HTML report lets you see the full distributions (not just means), which is where the latency tail that a mean hides becomes visible. For sizing and comparison, the JSON is what gets versioned and fed into the reproducible harness.

From the knee to sizing and cost

This is where GuideLLM connects with the rest of the series. The replica count and the cost per token come out of the knee:

The calculation: if the knee gives a goodput of 3,330 tok/s per replica and your peak target load is 20,000 tok/s, you need 6 replicas (20,000 ÷ 3,330 ≈ 6.0). And the cost per token comes from the replica cost (from OpenCost, ~11 €/h) divided by its goodput: ~0.92 €/1M tokens. It is the bridge to capacity planning and to cost per token: the correct denominator is the goodput at the knee, not catalogue throughput. Sizing with raw throughput leaves you short of useful capacity and breaks the SLO in production.

Deployment as a Kubernetes Job

GuideLLM runs as a Kubernetes Job inside the cluster to benchmark models served by the orchestration platform (Red Hat). This matters for fidelity: the load generator runs inside the cluster, on the same network as the endpoint, so it measures the engine without the network latency of an external client. The Job is parameterised with the target, the rate-type and the load, and dumps the JSON to a volume or a store for versioning. Running it as a Job also makes automation easier (a CronJob for periodic benchmarking, or triggered from CI).

SLO validation in CI: catching regressions

The most valuable use in the medium term: a Job that runs a short sweep on every change (a vLLM release, a config change) and compares the goodput against the baseline. If goodput drops by more than a threshold or the knee moves earlier, the pipeline fails. Since GuideLLM exports standardised JSON for regression tracking, comparing two runs is trivial. That way a performance regression, which is a cost and capacity regression, is caught at the commit, not in production. The full sweep is not needed on every commit: a short sweep covering the knee is enough to catch the regression; the exhaustive one is for releases.

Comparing configurations with sweeps

GuideLLM shines for deciding between configurations by running the same sweep against each one. The fair protocol: same load (--data), same SLO, same hardware, varying only the config. Examples of decisions that a sweep settles with data:

Each sweep gives a knee; the highest knee under the same SLO wins. It is the controlled experiment that fills the scorecard row (article B8): not “X is faster” in the abstract, but “X gives Y tok/s more goodput under this specific SLO, which translates into Z fewer replicas and W € less per million tokens”.

Watching the three axes during the sweep

A trick that connects with the cost and energy track: while the sweep runs, capture the GPU metrics with DCGM. That way each round yields the goodput (GuideLLM), the average power (DCGM → J/token) and the cost (node price → €/token) at once. In a single sweep campaign you measure the three axes of the scorecard for every operating point, and they come out coherent by construction (same instant, same load). Instead of an isolated performance benchmark, you get the complete row, cost, performance, energy, for the knee, which is exactly what the proposal needs. Align the time windows (the DCGM power and the GuideLLM metrics must cover the same interval) or the J/token does not correspond to the measured goodput.

Interpreting the distributions, not just the knee

The knee is the headline, but the GuideLLM HTML report gives full distributions, and there is information there that the knee summarises away. Three additional readings:

• The shape of the tail. Two configs with the same P99 can have different tails: one with P99 at 500 ms and P99.9 at 600 ms is stable; another with P99 at 500 ms and P99.9 at 3,000 ms has a long tail that will hit some users severely. The mean, and even P99, hide it; the distribution shows it.
• The TTFT/ITL split. Seeing the two distributions separately tells you whether the bottleneck is prefill (high TTFT) or decode (high ITL), which steers the optimisation (more concurrency vs chunked prefill, and so on).
• The spread across rounds. If goodput varies a lot between rounds at the same rate, the system is unstable under load, something a single capacity number does not reveal.

The knee sizes; the distributions diagnose. For a proposal, the knee is the datum; for operating and optimising, the distributions are where you see what to fix.

GuideLLM in the reproducible harness

GuideLLM fits as the load engine of the reproducible harness (article S4). The pattern:

1. A script/Job that deploys the engine with the pinned config, warms up, runs the sweep and dumps the JSON with all the metadata (model, version, hardware, --data, SLO).
2. Named by date and config to version the runs.
3. A store (a git repo of JSONs or a bucket) to reproduce and compare.
4. DCGM capturing in parallel to add the energy to each point.

That way “reproduce the knee of this config” is one command, not an afternoon, and comparing two releases is a JSON diff. It is what turns benchmarking from a one-off activity into a continuous capability, and what lets the capacity figure in the proposal come with the bench to reproduce it.

The cost of a sweep (in euros)

A 10-round sweep at ~2 minutes per round occupies the node for ~20–30 minutes. At an amortised cost of ~11 €/h, that is ~4–5.5 € per full sweep of a model. Not much per run, but a continuous benchmarking programme (every release, every config, several models) adds up; hence the value of automating the sweep as a reproducible Job and running short sweeps in CI (only around the knee), reserving the exhaustive one for releases. The cost of measuring is part of the cost of the platform, and small next to the cost of sizing badly.

The SLO by use case: the knee moves with it

Since the SLO defines the knee, different use cases give different capacities on the same hardware. Worth keeping in mind when sizing:

The same GPU serves more batch load than interactive chat load, because the batch SLO is loose. That is why sizing requires one sweep per load profile: there is no “the node’s capacity”, there is “the node’s capacity for this SLO”. Mixing loads with different SLOs in the same pool without separating them is a classic source of broken SLOs, with batch saturating and interactive chat paying for it. The fix connects with scheduling: separate pools or priorities by SLO.

GuideLLM vs the others: when to use which

To place it against the tools from B2:

GuideLLM is the tool for sizing under an SLO: when the question is operational (how many replicas, what cost per token at my SLO), it is the most direct answer. AIPerf and GuideLLM overlap (both multi-process); the practical difference is that GuideLLM is more oriented to the SLO and the reproducible report, and AIPerf to maximum capacity with automatic detection. Many teams use both; what must not be done is to compare a result from one with a result from the other as if they were the same measurement.

Checklist for a defensible sweep

For the knee of a sweep to support a sizing decision:

If you can tick the seven boxes, the knee is an auditable datum; if any is missing, it is an anecdote. The sizing in the proposal hangs on these seven being green.

Limits and traps (data-driven)

1. Not extending the sweep beyond the knee. If saturation is not detected, the estimated capacity is the last round tested, not the real knee. Extend it until P99 breaks.
2. Unrealistic load. A fixed-length --data does not resemble your traffic; use the real distribution (or poisson) for a knee that applies.
3. Reporting throughput, not goodput. The knee is defined by goodput under the SLO; raw throughput overestimates capacity.
4. Undeclared SLO. A capacity without the SLO it was measured under is neither comparable nor defensible.
5. Client outside the cluster. Running GuideLLM from outside puts network latency into the measurement; run it as a Job inside the cluster.

With GuideLLM mastered, you have the tool that turns performance into an actionable datum: the knee, the sizing and the cost per token, all from a reproducible sweep. The next article (B4) goes into NVIDIA’s AIPerf; this one closes SLO validation, which is what sizes the proposal.

Closing

GuideLLM answers the only performance question that sizes a platform: how far can I load without breaking the SLO? And it answers it with the right discipline: a sweep from idle to saturation that finds the knee, defined by goodput under a declared SLO, over a load that resembles the real one. Everything the proposal needs comes out of that knee: the replica count, the cost per token (with goodput as the denominator, not catalogue throughput) and, if you capture DCGM during the sweep, the energy per token as well. The mistake that invalidates the exercise is the usual one: reporting peak throughput instead of goodput, measuring with an unrealistic load, or not extending the sweep until you see where it breaks. Done properly, as a Job inside the cluster, with the real distribution, the SLO fixed and the output in versioned JSON, GuideLLM turns performance from a marketing figure into a reproducible datum that survives an audit and sizes a sovereign architecture with numbers. The knee is not the fastest point; it is the point where your platform still delivers what it promised.

See also

• GenAI-Perf in depth — NVIDIA’s profiler and how it compares with GuideLLM data sheet by data sheet (rate-types, concurrency sweep, metrics).
• LLM serving engines compared (vLLM/SGLang/TRT-LLM/Dynamo) — after validating the SLOs with GuideLLM, this comparison decides which engine meets the goodput-latency Pareto frontier.
• Measurement bias and reproducibility — the setup errors that make a concurrency sweep non-reproducible, even when the tool is correctly configured.

Sources

• Red Hat · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
• Red Hat · desplegar y benchmarkear vLLM con GuideLLM en Kubernetes — https://developers.redhat.com/articles/2025/12/24/how-deploy-and-benchmark-vllm-guidellm-kubernetes
• GuideLLM · PyPI — https://pypi.org/project/guidellm/
• GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
• Medium · GuideLLM en OpenShift (rate-types, ejemplo) — https://medium.com/@jajodia.nirjhar/exploring-guidellm-benchmarking-a-live-llm-on-openshift-ccc2d0841794