Goodput on lo0 — Tech Blog

GuideLLM in depth: validating the SLO under load and sizing from the knee

Sun, 14 Jun 2026 04:30:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Third article of the benchmarking track (B3). B2 covered the tool catalogue; here we go into GuideLLM in depth, because it answers the operational question that sizes a platform: how far can I load this engine without breaking the SLO? It is not a catalogue benchmark (“X tokens/s”), it is an SLO-driven sweep that finds the knee, the safe capacity, and from which the real replica count and cost per token come out. We will look at its load modes, how the SLO is defined, how to read the output and how to turn the knee into sizing and into euros. With real commands; no recommendations, only the mechanics.

Why GuideLLM

GuideLLM (a vLLM project) generates realistic, configurable traffic patterns and captures full distributions of TTFT, ITL and end-to-end behaviour, for SLO-driven evaluation, with reproducible sweeps that identify the safe operating range (Red Hat). Unlike a micro-benchmark, it measures the engine, not the client (multi-process load generation), and unlike MLPerf, it sizes your specific case (your model, your SLO, your load). It is the middle tool: not for tuning a flag, not for comparing vendors, but for deciding how many GPUs you need.

The load modes (rate-types)

GuideLLM offers several modes through --rate-type, and picking the right one is half the work:

`--rate-type`	What it does	When
synchronous	one request at a time	baseline latency, no concurrency
concurrent	keeps N fixed simultaneous requests	measuring at a specific concurrency
poisson	requests per second following Poisson	simulating real traffic (random arrivals)
throughput	saturates the engine to the maximum	raw peak capacity
sweep	automatic ramp from idle to maximum	finding the knee (the most useful one)

In concurrent mode, --rate "1,2,4" launches three benchmarks at those concurrencies (Medium · GuideLLM on OpenShift). The poisson mode is the most realistic for online inference (requests do not arrive at regular intervals), and the sweep is the one that automates the search for the saturation point.

The automatic sweep: from idle to the knee

The flagship mode. When running a sweep profile, GuideLLM automatically increases the request rate from idle to maximum throughput across 10 rounds, and produces an interactive HTML report with detailed latency and throughput (Red Hat · GuideLLM on Kubernetes). During the sweep, on detecting saturation it identifies the previous iteration, the knee, and returns it as estimated capacity; if it does not detect saturation, the sweep has to be extended beyond the knee.

A worked example: reading a 10-round sweep

An illustrative output from a sweep on a 70B on 8×H100 (SLO: TTFT P99 < 500 ms), per round:

Round	Rate (req/s)	TTFT P99 (ms)	ITL P50 (ms)	Throughput (tok/s)	Goodput (tok/s)
1	2	95	19	480	480
3	8	140	20	1,900	1,900
5	14	240	22	3,100	3,060
6	17	460	24	3,400	3,330
7	20	760	31	3,700	2,520
9	26	1,500	48	3,950	900
10	30	2,400	71	4,000	380

How to read it: up to round 6 (17 req/s) the TTFT P99 meets the SLO (460 ms) and goodput ≈ throughput (3,330 ≈ 3,400). In round 7 P99 already breaks (760 ms) and goodput drops to 2,520. The knee sits between 6 and 7: the safe capacity is that of round 6, ~3,330 tok/s of goodput. Rounds 9 and 10 give more raw throughput (3,950, 4,000) but with goodput of 900 and 380; the system is “performing” by mostly serving requests that miss the target. Whoever reports “4,000 tok/s” is describing round 10, where the system is broken. The defensible number is 3,330 tok/s under TTFT P99 < 500 ms, and that is the one that goes into the sizing. Note as well that the sweep had to reach round 10 (P99 of 2,400 ms) to see where it broke: that is why it has to be extended beyond the knee.

Defining the SLO: the number that decides the knee

The “knee” is not absolute: it depends on the SLO you define. A typical online inference SLO:

Metric	Example threshold	What it protects
TTFT P99	< 500 ms	the wait for the first token (interactivity)
TPOT / ITL P95	< 50 ms/token	the perceived “typing speed”
Error rate	< 0.1 %	reliability

Goodput, the throughput that meets that SLO, is the metric that defines the knee: the safe capacity is the maximum load where goodput ≈ throughput. Changing the SLO moves the knee: a stricter TTFT SLO (200 ms) gives an earlier knee (less safe capacity) than a loose one (1 s). That is why the SLO is fixed before the sweep, and reported alongside the result: a capacity “of 3,330 tok/s” without saying under which SLO means nothing.

Execution: the command

An SLO-driven sweep against a vLLM endpoint:

guidellm benchmark \
 --target "http://vllm:8000" \
 --rate-type sweep \
 --max-seconds 120 \
 --data "prompt_tokens=1024,output_tokens=256" \
 --output-path resultados.json

Key parameters: --rate-type sweep (the automatic ramp), --data (the length distribution of the load, which must resemble your real traffic), --max-seconds (duration per round), --target (the endpoint). For the SLO, GuideLLM allows the latency constraints that define goodput to be set. The output goes to --output-path in JSON, YAML or CSV, in addition to the interactive HTML report.

Defining the load: `--data` decides the result

The --data parameter (the distribution of prompt and output lengths) changes the knee as much as the SLO does. Three forms, from least to most faithful:

Load	`--data`	Fidelity
Fixed length	`prompt_tokens=1024,output_tokens=256`	low: real traffic is not fixed
Synthetic distribution	length ranges	medium
Real traces	a dataset of your traffic	high: the knee that applies

The trap: a sweep with short fixed-length prompts gives an optimistic knee that does not resemble production, where long prompts dominate the prefill cost and bring the knee forward. To size properly, feed GuideLLM the real distribution of your traffic (prompt and output lengths measured in production). The knee of a sweep is only as realistic as the load you put into it; with an unrealistic --data, you are sizing for traffic that does not exist.

The arrival profile matters too: --rate-type poisson simulates random arrivals (like real online traffic), while concurrent keeps N fixed (like a batch). An online service measured with concurrent can give a knee different from the real one; for interactive inference, poisson is the honest profile.

The output: what to read

GuideLLM produces standardised, exportable reports for dashboards, analysis and regression tracking, in JSON, YAML and CSV (Red Hat). What matters to read, per sweep round:

TTFT (P50, P95, P99) — the latency to the first token.
ITL/TPOT (P50, P95, P99) — between tokens.
Throughput (req/s and tok/s) — the raw figure.
Goodput — the part that meets the SLO (the honest number).

The interactive HTML report lets you see the full distributions (not just means), which is where the latency tail that a mean hides becomes visible. For sizing and comparison, the JSON is what gets versioned and fed into the reproducible harness.

From the knee to sizing and cost

This is where GuideLLM connects with the rest of the series. The replica count and the cost per token come out of the knee:

The calculation: if the knee gives a goodput of 3,330 tok/s per replica and your peak target load is 20,000 tok/s, you need 6 replicas (20,000 ÷ 3,330 ≈ 6.0). And the cost per token comes from the replica cost (from OpenCost, ~11 €/h) divided by its goodput: ~0.92 €/1M tokens. It is the bridge to capacity planning and to cost per token: the correct denominator is the goodput at the knee, not catalogue throughput. Sizing with raw throughput leaves you short of useful capacity and breaks the SLO in production.

Deployment as a Kubernetes Job

GuideLLM runs as a Kubernetes Job inside the cluster to benchmark models served by the orchestration platform (Red Hat). This matters for fidelity: the load generator runs inside the cluster, on the same network as the endpoint, so it measures the engine without the network latency of an external client. The Job is parameterised with the target, the rate-type and the load, and dumps the JSON to a volume or a store for versioning. Running it as a Job also makes automation easier (a CronJob for periodic benchmarking, or triggered from CI).

SLO validation in CI: catching regressions

The most valuable use in the medium term: a Job that runs a short sweep on every change (a vLLM release, a config change) and compares the goodput against the baseline. If goodput drops by more than a threshold or the knee moves earlier, the pipeline fails. Since GuideLLM exports standardised JSON for regression tracking, comparing two runs is trivial. That way a performance regression, which is a cost and capacity regression, is caught at the commit, not in production. The full sweep is not needed on every commit: a short sweep covering the knee is enough to catch the regression; the exhaustive one is for releases.

Comparing configurations with sweeps

GuideLLM shines for deciding between configurations by running the same sweep against each one. The fair protocol: same load (--data), same SLO, same hardware, varying only the config. Examples of decisions that a sweep settles with data:

Decision	What to compare	What the knee reveals
FP16 vs FP8	two deployments of the same model	FP8 usually gives a higher knee (more goodput)
`max-num-seqs`	different values of the flag	the one that maximises goodput under the SLO
vLLM vs SGLang	two engines, same model	which engine gives more goodput at your SLO
KV size/precision	KV cache configs	the effect on capacity

Each sweep gives a knee; the highest knee under the same SLO wins. It is the controlled experiment that fills the scorecard row (article B8): not “X is faster” in the abstract, but “X gives Y tok/s more goodput under this specific SLO, which translates into Z fewer replicas and W € less per million tokens”.

Watching the three axes during the sweep

A trick that connects with the cost and energy track: while the sweep runs, capture the GPU metrics with DCGM. That way each round yields the goodput (GuideLLM), the average power (DCGM → J/token) and the cost (node price → €/token) at once. In a single sweep campaign you measure the three axes of the scorecard for every operating point, and they come out coherent by construction (same instant, same load). Instead of an isolated performance benchmark, you get the complete row, cost, performance, energy, for the knee, which is exactly what the proposal needs. Align the time windows (the DCGM power and the GuideLLM metrics must cover the same interval) or the J/token does not correspond to the measured goodput.

Interpreting the distributions, not just the knee

The knee is the headline, but the GuideLLM HTML report gives full distributions, and there is information there that the knee summarises away. Three additional readings:

The shape of the tail. Two configs with the same P99 can have different tails: one with P99 at 500 ms and P99.9 at 600 ms is stable; another with P99 at 500 ms and P99.9 at 3,000 ms has a long tail that will hit some users severely. The mean, and even P99, hide it; the distribution shows it.
The TTFT/ITL split. Seeing the two distributions separately tells you whether the bottleneck is prefill (high TTFT) or decode (high ITL), which steers the optimisation (more concurrency vs chunked prefill, and so on).
The spread across rounds. If goodput varies a lot between rounds at the same rate, the system is unstable under load, something a single capacity number does not reveal.

The knee sizes; the distributions diagnose. For a proposal, the knee is the datum; for operating and optimising, the distributions are where you see what to fix.

GuideLLM in the reproducible harness

GuideLLM fits as the load engine of the reproducible harness (article S4). The pattern:

A script/Job that deploys the engine with the pinned config, warms up, runs the sweep and dumps the JSON with all the metadata (model, version, hardware, --data, SLO).
Named by date and config to version the runs.
A store (a git repo of JSONs or a bucket) to reproduce and compare.
DCGM capturing in parallel to add the energy to each point.

That way “reproduce the knee of this config” is one command, not an afternoon, and comparing two releases is a JSON diff. It is what turns benchmarking from a one-off activity into a continuous capability, and what lets the capacity figure in the proposal come with the bench to reproduce it.

The cost of a sweep (in euros)

A 10-round sweep at ~2 minutes per round occupies the node for ~20–30 minutes. At an amortised cost of ~11 €/h, that is ~4–5.5 € per full sweep of a model. Not much per run, but a continuous benchmarking programme (every release, every config, several models) adds up; hence the value of automating the sweep as a reproducible Job and running short sweeps in CI (only around the knee), reserving the exhaustive one for releases. The cost of measuring is part of the cost of the platform, and small next to the cost of sizing badly.

The SLO by use case: the knee moves with it

Since the SLO defines the knee, different use cases give different capacities on the same hardware. Worth keeping in mind when sizing:

Use case	Typical SLO	Effect on the knee
Interactive chat	TTFT P99 < 500 ms, ITL < 50 ms	early knee (latency rules)
Code copilot	TTFT P99 < 300 ms	even earlier knee
Batch / summarisation	no TTFT SLO, maximise throughput	late knee (almost pure throughput)
Agent (multi-step)	bounded per-step latency	depends on the number of steps

The same GPU serves more batch load than interactive chat load, because the batch SLO is loose. That is why sizing requires one sweep per load profile: there is no “the node’s capacity”, there is “the node’s capacity for this SLO”. Mixing loads with different SLOs in the same pool without separating them is a classic source of broken SLOs, with batch saturating and interactive chat paying for it. The fix connects with scheduling: separate pools or priorities by SLO.

GuideLLM vs the others: when to use which

To place it against the tools from B2:

Question	Tool	Why
“How far can I load without breaking the SLO?”	GuideLLM	SLO-driven sweep, the knee
“What is the endpoint’s maximum capacity?”	AIPerf	automatic saturation detection
“Does my vLLM flag change help?”	vllm bench serve	fast, native to the engine
“Which hardware/engine is better in the abstract?”	read MLPerf	cross-vendor comparability

GuideLLM is the tool for sizing under an SLO: when the question is operational (how many replicas, what cost per token at my SLO), it is the most direct answer. AIPerf and GuideLLM overlap (both multi-process); the practical difference is that GuideLLM is more oriented to the SLO and the reproducible report, and AIPerf to maximum capacity with automatic detection. Many teams use both; what must not be done is to compare a result from one with a result from the other as if they were the same measurement.

Checklist for a defensible sweep

For the knee of a sweep to support a sizing decision:

Step	Check
Realistic load	`--data` with your traffic’s distribution (or poisson)
Declared SLO	TTFT/TPOT with a fixed percentile and threshold
Extended sweep	reaches the point where P99 breaks (past the knee)
Goodput, not throughput	capacity is the goodput under the SLO
Job in the cluster	no network latency from an external client
Versioned output	JSON with all the metadata
DCGM in parallel	energy per point, optional but recommended

If you can tick the seven boxes, the knee is an auditable datum; if any is missing, it is an anecdote. The sizing in the proposal hangs on these seven being green.

Limits and traps (data-driven)

Not extending the sweep beyond the knee. If saturation is not detected, the estimated capacity is the last round tested, not the real knee. Extend it until P99 breaks.
Unrealistic load. A fixed-length --data does not resemble your traffic; use the real distribution (or poisson) for a knee that applies.
Reporting throughput, not goodput. The knee is defined by goodput under the SLO; raw throughput overestimates capacity.
Undeclared SLO. A capacity without the SLO it was measured under is neither comparable nor defensible.
Client outside the cluster. Running GuideLLM from outside puts network latency into the measurement; run it as a Job inside the cluster.

With GuideLLM mastered, you have the tool that turns performance into an actionable datum: the knee, the sizing and the cost per token, all from a reproducible sweep. The next article (B4) goes into NVIDIA’s AIPerf; this one closes SLO validation, which is what sizes the proposal.

Closing

GuideLLM answers the only performance question that sizes a platform: how far can I load without breaking the SLO? And it answers it with the right discipline: a sweep from idle to saturation that finds the knee, defined by goodput under a declared SLO, over a load that resembles the real one. Everything the proposal needs comes out of that knee: the replica count, the cost per token (with goodput as the denominator, not catalogue throughput) and, if you capture DCGM during the sweep, the energy per token as well. The mistake that invalidates the exercise is the usual one: reporting peak throughput instead of goodput, measuring with an unrealistic load, or not extending the sweep until you see where it breaks. Done properly, as a Job inside the cluster, with the real distribution, the SLO fixed and the output in versioned JSON, GuideLLM turns performance from a marketing figure into a reproducible datum that survives an audit and sizes a sovereign architecture with numbers. The knee is not the fastest point; it is the point where your platform still delivers what it promised.

Sources

Red Hat · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
Red Hat · desplegar y benchmarkear vLLM con GuideLLM en Kubernetes — https://developers.redhat.com/articles/2025/12/24/how-deploy-and-benchmark-vllm-guidellm-kubernetes
GuideLLM · PyPI — https://pypi.org/project/guidellm/
GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
Medium · GuideLLM en OpenShift (rate-types, ejemplo) — https://medium.com/@jajodia.nirjhar/exploring-guidellm-benchmarking-a-live-llm-on-openshift-ccc2d0841794

LLM inference benchmarking: frameworks, metrics and the state of the art (tool by tool)

Sat, 13 Jun 2026 02:30:00 +0200

Notation: amounts in euros (N €), decimals with a point. Performance is not very sensitive to the country, but its associated cost (cost per token) is expressed in € and links back to the opening article.

What this introduction covers

Third article in the data series, a deep dive into the performance axis. Measuring the performance of an inference engine looks trivial (“how many tokens per second?”) and it is exactly where people fool themselves most: two tools can report results that differ by a factor of 7 for the same system. This article inventories the metrics that matter and how they are defined, why the architecture of the tool biases the figure, how the saturation point is found with a concurrency sweep, and the profile of each framework. No recommendations: the choice of engine is settled in the Pareto article (B8); here there are only the facts and the methodology, because a benchmark without a published methodology is not comparable.

The performance metrics

There is no single performance metric, there are five, and mixing them is the first source of error:

Metric	Definition	Unit	Dominant phase
TTFT (Time To First Token)	time from sending the prompt to the first token	ms	prefill
TPOT / ITL (Time Per Output Token / Inter-Token Latency)	average time between output tokens once generation has started	ms/token	decode
Request throughput	complete request-response cycles per second at the concurrency tested	req/s	both
Token throughput	total tokens (input + output) per second across all concurrent requests	tok/s	both
Goodput	share of requests that meet the defined SLO	useful tok/s	both
P50 / P95 / P99	latency percentiles (not the mean)	ms	—

Precise definitions, because every tool computes them its own way (Anyscale · latency and throughput metrics): TTFT is what a user waits before seeing the first character, dominated by prefill compute; ITL is the average time between successive output tokens and sets the perceived “typing speed” of the answer; request throughput is complete cycles per second; token throughput is total tokens (input plus output) per second across all concurrent requests.

The latency decomposition

The total latency of a request with $N$ output tokens decomposes as:

$$\text{latency} \approx \text{TTFT} + (N-1)\times \text{TPOT}$$

That is why TTFT and TPOT are reported separately: a single mean latency hides very different profiles. A system with high TTFT and low TPOT (expensive prefill, fast decode) and another one the other way round can have the same mean latency for one particular length, yet behave in opposite ways when the response size changes. For an interactive chat experience TTFT and TPOT rule; for a batch of long summaries, token throughput does. Measuring the mean hides both realities.

Goodput: the honest metric

Raw throughput (TPS, RPS) says how much work the system does; goodput says how much of that work meets your quality-of-service standards (SLO) (Anyscale). An engine can boast 10,000 aggregate tok/s, but if half the requests violate the P99 TTFT SLO, its goodput is 5,000. The figure you defend in a proposal is the goodput, not the catalogue throughput: it is the only one that translates into satisfied users and an honest cost per token.

How each metric is instrumented (and where the error creeps in)

Before comparing numbers it helps to know where each clock starts and stops, because two tools can call different things “TTFT”:

Client-side versus server-side TTFT. The TTFT measured by the client includes network latency and gateway queueing; the one measured by the server does not. For an engine comparison the server-side figure is the one that matters; for user experience, the client-side one. Mixing the two invalidates the comparison.
Streaming is required. TTFT and ITL can only be measured if the response arrives as a stream (token by token). If the tool measures complete responses, there is no real TTFT: there is total latency in disguise.
Token counting. Throughput in tok/s depends on which tokenizer counts the tokens. If the tool uses a tokenizer different from the model’s, the token count (and therefore the tok/s and the cost per token) is biased. You have to count with the tokenizer of the model being served.
Warm-up and prefix cache. The first requests of a benchmark benefit from a warm prefix cache and give artificially low TTFT; the warm-up has to be discarded or the result inflates reality.

These four instrumentation decisions explain a good share of the discrepancies between tools. A performance figure that does not say where the clock is measured and with which tokenizer is not comparable, however precise it may look.

The architecture of the tool biases the figure

Here is the trap that invalidates half the published benchmarks. Tools split into two classes by the architecture of the client that generates the load, and that architecture determines whether the measurement is reliable at high concurrency:

Single-process micro-bench (vLLM bench, SGLang bench, genai-perf): a Python client with asyncio in a single process. Useful for quick experiments on one engine, but the single-process architecture introduces a bottleneck on the client side that biases the data at high concurrency (genAI-perf and vLLM): the client cannot generate enough load and you measure the limit of the client, not the limit of the engine.
Multi-process load (GuideLLM, AIPerf): they spread load generation across several processes, avoiding that limit. This is the class that has emerged for measuring at real scale.

The size of the bias is enormous: at 1,000 QPS, a single-process benchmark processed 75,574 tokens against the 545,733 tokens of a distributed architecture, a discrepancy of 7.2× in measurement capacity for the same system ([search]). Anyone comparing two engines with tools from different classes is not comparing the engines: they are comparing the benchmark clients.

Frameworks, one by one

vLLM bench and SGLang bench — engine micro-bench

What they measure: TTFT, TPOT and throughput of the engine itself (vLLM or SGLang). Class: single-process micro-bench. Use: quick experiments to tune one engine and see the effect of its optimisations (see decode and prefill). Limit: they saturate on the client at high concurrency; they are no good for measuring real capacity at scale.

AIPerf — the NVIDIA one (formerly genai-perf), multi-process

What it measures: TTFT, ITL, throughput and latency against vLLM, NIM, TGI and any compatible endpoint. Class: multi-process load. State-of-the-art datum: NVIDIA retired genai-perf and replaced it with AIPerf on 15 April 2026. During the sweep, AIPerf detects GPU saturation and identifies the previous iteration, returning it as estimatedCapacity; if it detects no saturation, estimatedCapacity is the last iteration tested, which is why the sweep has to extend beyond the knee (AIPerf).

GuideLLM — from the vLLM project, SLO-oriented

What it measures: full distributions of TTFT, ITL and end-to-end behaviour, for SLO-driven evaluation. Class: multi-process load. Differentiator: it generates realistic, configurable traffic patterns in synchronous, concurrent and rate-based modes, including reproducible sweeps to identify safe operating ranges (Red Hat · GuideLLM, GuideLLM · GitHub). It is the tool for answering “how hard can I load this engine without breaking the SLO?”.

LLMPerf — the Anyscale/Ray classic

What it measures: throughput and latency at the inference level. Use: endpoint validation, historically very widespread. Class: load generator. Limit: less focused on distributions and sweeps than GuideLLM/AIPerf.

MLPerf Inference — the industry standard

What it measures: performance under standardised scenarios with strict rules, for comparability across vendors. Maintainer: MLCommons. It is the gold standard of cross-vendor comparability; it is expanded on below.

Comparison table

Tool	Class	What it measures	Maintainer	When to use it
vLLM bench	single-process micro	TTFT, TPOT, throughput of vLLM	vLLM (OSS)	tune vLLM, quick experiments
SGLang bench	single-process micro	metrics of the SGLang engine	SGLang (OSS)	tune SGLang
AIPerf (formerly genai-perf)	multi-process load	TTFT, ITL, throughput; estimatedCapacity	NVIDIA (OSS)	real capacity, multi-endpoint
GuideLLM	multi-process load	distributions, SLO, sweeps	vLLM (OSS)	validate SLO, find the knee
LLMPerf	load	throughput and latency	Anyscale/Ray (OSS)	endpoint validation
MLPerf Inference	standard suite	server/offline/interactive scenarios	MLCommons	cross-vendor comparability

The concurrency sweep: finding the knee

The most useful measurement for sizing is not a number, it is a curve: how latency and throughput change as concurrency rises. Raising concurrency keeps the GPU busier and raises RPS, but past a certain point it sends TTFT, ITL and end-to-end latency through the roof ([search]). The goal of the sweep is to find the knee: the maximum concurrency where throughput still rises without latency breaking the SLO.

That is why AIPerf extends the sweep beyond the knee: only by seeing where latency takes off can it return the safe capacity (estimatedCapacity). This curve is the raw material of capacity planning: the knee yields the replica count and the cost per token at the target load.

Worked example: reading a sweep

An illustrative sweep on an example node (a 70B on 8×H100, SLO of P99 TTFT < 500 ms), to see how the knee is read:

Concurrency	RPS	TTFT P50 (ms)	TTFT P99 (ms)	Token tput (tok/s)	Goodput
1	2	80	110	350	100 %
8	14	110	240	2,100	100 %
16	22	180	460	3,400	98 %
24	26	320	980	3,900	62 %
32	27	540	1,800	4,000	20 %

Reading: up to concurrency ~16 throughput grows and P99 stays under the SLO (goodput ~100 %). Between 16 and 24 sits the knee: throughput barely rises any more (3,400 → 3,900 tok/s) but P99 takes off (460 → 980 ms) and goodput collapses (98 % → 62 %). At concurrency 32 raw throughput is at its maximum (4,000 tok/s) but goodput is 20 %: the system “performs well” while serving mostly requests that violate the SLO. The defensible safe capacity is the one at concurrency ~16, not the one at maximum throughput. This is the number that goes into capacity planning and into cost per token: at 3,400 useful tok/s, not 4,000 raw tok/s.

MLPerf Inference: the comparability standard

To compare across vendors and engines with identical rules there is MLPerf Inference (MLCommons). The datacenter category focuses on two scenarios plus an optional one (MLCommons · datacenter):

Scenario	What it simulates	Metric
Offline	raw throughput processing the whole dataset in batch	maximum throughput
Server	interactive environment: requests one at a time following Poisson at a mean RPS	RPS under TTFT and TPOT limits
Interactive (optional)	like server but with stricter latency limits	RPS under a hard SLO

The Server scenario is the realistic one for online inference: the load generator sends requests following a Poisson distribution and demands that specific TTFT and TPOT bounds be met. MLPerf Inference v5.0 (April 2025) introduced a large-scale 405B benchmark and a low-latency interactive 70B one, offering language benchmarks at every scale (7B to 405B), architectural diversity (including MoE) and scenarios (MLCommons · v5.0); v5.1 (September 2025) widened the results with record participation (MLCommons · v5.1).

The value of MLPerf is comparability: everyone measures the same thing under the same rules. Its limit is that those rules may not match your workload (your length distribution, your specific SLO), so it serves to compare hardware and engines against each other, not necessarily to size your case. For that, your own sweep.

Measurement bias and reproducibility

That two benchmarks give very different results for the same system is no accident: systematic measurement bias in production benchmarks is characterised in the literature (arXiv 2605.24217), and there is work dedicated to the meta-metrics and good practices of system-level performance benchmarking (arXiv 2508.10251). The most common sources of bias:

Source of bias	Effect	Mitigation
Single-process client	underestimates throughput at high concurrency	use multi-process load
Unrealistic length distribution	results that do not apply to your traffic	use realistic traces
Measuring the mean instead of percentiles	hides the latency tail	report P95/P99
Uncontrolled warm-up	the prefix cache inflates the first results	discard the warm-up
Not pinning engine/model version	irreproducible	pin everything and publish it

The methodological conclusion of this article: a benchmark without a published methodology is not comparable. For a performance figure to support a proposal it has to come with the tool, its version, the model and precision, the load distribution and the SLO. Synthesis article S4 builds a reproducible harness that pins all of that.

Checklist for a reproducible benchmark

For a performance figure to be defensible before a committee, it has to come with everything that allows it to be reproduced. The minimum published alongside the result:

What to pin	Why
Tool + version	each one measures differently; the version changes behaviour
Model + precision (FP16/FP8/INT4)	precision changes throughput and quality
Hardware (GPU, count, interconnect)	an 8×H100 NVLink is not an 8×H100 PCIe
Engine + version + flags	vLLM/SGLang/TRT-LLM and their configuration
Length distribution (in/out)	real traffic is not fixed length
Concurrency levels of the sweep	you have to go past the knee
SLO (which percentile, which threshold)	it defines the goodput
Warm-up handling	discard it or it biases the result
Tokenizer used for counting	it affects tok/s and cost per token

The practical rule: if you cannot hand over this table alongside the figure, the figure is not a datum, it is an anecdote. The reproducible harness of article S4 automates the recording of all these parameters so that anyone, including whoever challenges the proposal, can reproduce the exact number.

Performance ≠ quality

One warning that avoids the most expensive mistake: these tools measure speed and throughput, not accuracy. An engine can be blisteringly fast while serving a model that answers badly. Quality is measured with another family of tools (lm-evaluation-harness, HELM, task leaderboards) which is another axis of the scorecard (article B7). Confusing “fast” with “good” is how you build a platform that serves bad answers very quickly. On the final Pareto frontier, performance and quality are two distinct axes that must be looked at together, never one instead of the other.

The other family of tools, for reference (expanded in B7):

Tool	What it measures	Common trap
lm-evaluation-harness	accuracy on hundreds of standardised tasks	contamination of the test dataset
HELM	holistic evaluation (accuracy, robustness, bias, efficiency)	heavy to run
LiveBench / dynamic leaderboards	tasks that rotate to avoid contamination	comparability over time

The data point: contamination, the model having seen the test during training, inflates quality metrics just as warm-up inflates performance ones. That is why dynamic leaderboards rotate their questions. Quality and performance share that lesson: the measurement method biases the result as much as the system being measured.

The latency versus throughput trade-off

A property the sweep reveals and which is worth keeping in mind: latency and throughput pull in opposite directions. Batching groups requests to amortise the cost of moving the weights out of VRAM, which raises throughput, but each request waits for the batch to form, which raises individual latency. There are two operating regimes, and the benchmark serves to place you in the right one:

Regime	Optimises	Configuration	Case
Latency	low TTFT/TPOT	small batch, low concurrency	interactive chat, copilots
Throughput	maximum tok/s	large batch, high concurrency	overnight batch, ingestion

There is no single “best” point: there is the best point for your SLO. A benchmark that reports only maximum throughput is describing the throughput regime and ignoring whether that point meets the latency your case needs. That is why goodput, throughput under the SLO, is the metric that reconciles the two regimes: it measures how much throughput you get without stepping outside acceptable latency. The sweep traverses the curve between both regimes; your SLO marks where on that curve your system sits.

The connection with cost and energy

Performance is not an isolated axis: by the identity of the opening article, throughput is the denominator of cost per token and of energy per token. A sweep that finds a knee at 4,000 tok/s rather than 2,800 is not just “faster”: it lowers the CPM from ~1.09 to ~0.76 €/1M tok and the energy per token in the same proportion, on the same iron. That is why performance benchmarking is the tool that, indirectly, moves cost the most: every goodput improvement translates into euros and into watts per token. The number that connects the three axes is goodput, the throughput that meets the SLO, not catalogue throughput.

State of the art 2026

genai-perf → AIPerf migration (15 Apr 2026): NVIDIA consolidates its benchmarking into a multi-process tool with saturation detection.
GuideLLM as the OSS standard for SLO-driven evaluation with reproducible sweeps.
MLPerf Inference v5.0/v5.1 extends to 405B, interactive 70B and MoE, with record participation: cross-vendor comparability is maturing.
Measurement bias characterised: the community accepts that the measurement method matters as much as the system measured; the emphasis on reproducibility and meta-metrics is growing.

Limits and traps (data-driven)

Comparing tools from different classes. A single-process micro-bench and a multi-process load generator are not comparable; the difference can be 7×. Pin the class.
Catalogue throughput instead of goodput. The honest number is the one that meets the SLO.
Means instead of percentiles. The mean hides the tail; report P95/P99.
Not extending the sweep beyond the knee. Without seeing where latency takes off you do not know the safe capacity.
Confusing performance with quality. They are distinct axes; fast is not good.
Not pinning versions. Engine, model, precision and load left unpinned = irreproducible = not defensible.

The next article in the track (B2) goes into the tool catalogue in depth; this one pins the metrics and the methodology. With performance measured reproducibly, the scorecard can cross it with cost (in €) and energy for the final decision.

Closing

Performance benchmarking looks like the most “objective” of the three axes (they are tokens per second, after all) and it is exactly where the most manipulation happens, almost always without bad intent: a single-process tool here, a mean instead of a P99 there, a catalogue throughput instead of the goodput. The difference between a marketing number and a defensible datum lies not in the engine measured but in the methodology: the class of tool, where the clock is measured, with which tokenizer, how far the sweep goes and which SLO defines the goodput. For a sovereign architecture proposal, performance only counts if it is delivered with that reproducibility record, and, crossed with cost in euros and energy per token, it becomes the column of the Pareto frontier that decides which engine and which configuration hold up the platform. The number you defend is not the highest one: it is the reproducible goodput.

Sources

Anyscale · métricas de latencia y throughput de LLM — https://docs.anyscale.com/llm/serving/benchmarking/metrics
Red Hat · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
NVIDIA AIPerf · guía de benchmarking (ex genai-perf) — https://lucaberton.com/blog/nvidia-aiperf-llm-inference-benchmarking-guide/
MLCommons · MLPerf Inference datacenter (escenarios) — https://mlcommons.org/benchmarks/inference-datacenter/
MLCommons · MLPerf Inference v5.0 (405B + 70B interactivo) — https://mlcommons.org/2025/04/llm-inference-v5/
MLCommons · MLPerf Inference v5.1 — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
arXiv 2605.24217 · sesgo sistemático de medición en benchmarks de inferencia LLM — https://arxiv.org/html/2605.24217
arXiv 2508.10251 · meta-métricas y buenas prácticas de benchmarking de rendimiento — https://arxiv.org/pdf/2508.10251
Medium · LLM Inference Benchmarking (genAI-perf y vLLM) — https://kchandan.medium.com/llm-inference-benchmarking-genai-perf-and-vllm-5dd06b57428e

Continuous batching: the salon with 8 chairs that does not wait for the slow customer — Orca, vLLM, chunked prefill and goodput

Sat, 30 May 2026 16:00:00 +0200

This post complements those on KV cache (the artefact continuous batching manages), PagedAttention (the memory piece that makes it viable), Disaggregated serving (the next layer of optimisation), Speculative decoding, Multi-LoRA and MoE (the three extensions that coexist with the scheduler in production).

You are here: DEPLOY

TL;DR

The static batching of the original HuggingFace Transformers (the pre-2022 era) systematically under-used the GPU for two structural reasons. First: the scheduling unit was the complete request; the longest one in the batch blocked all the others until it finished (severe head-of-line blocking, P95 TTFT five to ten times worse than reasonable). Second: each batch slot reserved memory for max_seq_len even though the real output was much shorter; documented padding waste sat between 60 % and 80 % and sustained GPU SM utilisation on real workloads fell to 20-40 %. Orca (Yu et al., OSDI 2022, FriendliAI + Seoul National University) introduced the idea that unjammed everything: the scheduling unit stops being the request and becomes one decoder iteration, a single token. After each iteration the scheduler can add new requests to the batch and remove the finished ones. vLLM (Kwon et al., SOSP 2023, UC Berkeley) materialised it open-source and production-grade thanks to PagedAttention, which solves the KV cache fragmentation that theoretical continuous batching would cause with dynamic allocation. SARATHI / Sarathi-Serve (Microsoft Research India, OSDI 2024) closed the last gap: the prefill stalls that paused active decodes when a new request arrived, via chunked prefill (splitting a long prefill into small chunks and mixing them with decodes in the same step) and stall-free batching. DistServe (Zhong et al., OSDI 2024) reformulated the key metric: what matters is goodput (requests/s meeting the TTFT and TPOT SLOs), not raw throughput. In May 2026, vLLM v1 brings a unified scheduler with chunked prefill always-on; SGLang adds RadixAttention, which gives cross-request prefix-cache hits; TensorRT-LLM calls it in-flight batching; llama.cpp supports it natively. The three operational tensions are speculative decoding (nested raggedness), multi-LoRA (each request with its adapter) and MoE (each expert sees very few tokens per step at a typical batch size). This post takes apart the mechanism, the maths (GPU utilisation, goodput vs throughput), the three variants (Orca → vLLM → Sarathi-Serve), the pitfalls (preempt-on-OOM, starvation, inverse HoL) and the real numbers with production configurations.

The analogy: the salon with 8 chairs

A hair salon with 8 chairs and a single brilliant stylist who moves between them. Customers arrive with very different needs: some want a 15-minute cut, others a 2-hour colour with a base, others a 90-minute straightening. The question is how to organise the flow.

The traditional strategy (what HuggingFace Transformers did in its original generate()) is to seat 8 customers at once, all at the same time, and accept nobody new until the last one finishes. If one of those 8 is a 2-hour job, the 7 who wanted the 15-minute cut sit there doing nothing for 1 hour and 45 minutes. The stylist finishes with the quick ones and is left staring at empty chairs until the colour is done. When everyone is ready, another 8 come in. That is static batching, and the only thing that stops it from being worse is that the GPU does not complain like a human customer.

The continuous strategy (Orca, vLLM) changes the scheduling unit. The stylist does not think “I am going to do one whole customer and then the next”; he thinks “on each tick I take one step of work at every occupied chair and, every time a chair frees up, I call the next customer in the queue without waiting for the others to finish”. The quick-cut customer leaves after 15 minutes, his chair is filled immediately with the next one, and the slow jobs proceed at their own pace without delaying anyone. The stylist is never idle.

The continuous strategy with chunked prefill (SARATHI / Sarathi-Serve) adds a subtler distinction. Some customers need a long initial phase (a 10-minute hair analysis before the cut; in LLM terms, the prefill of the prompt). Without chunked prefill, the stylist had to stop all the other chairs to do the new customer’s analysis in one go, and that was a stall visible in the TPOT of the active ones. With chunked prefill, the analysis is split into 2-minute pieces interleaved between the other customers’ active cuts. The customers already under way no longer notice pauses; the new customer takes slightly longer to start his cut proper, but the whole salon does not freeze.

And the metric that matters: the owner does not want to maximise “customers served per hour” at the cost of some leaving furious. He wants to maximise “customers served per hour within the time SLA”, and that is goodput, DistServe’s contribution.

The problem continuous batching solves

There are two structural pathologies of static batching that deserve to be explained with concrete numbers.

Padding waste. Each batch slot reserved memory for max_seq_len (prompt + max output), even though the real output ended in far fewer tokens. For a batch of 32 with heterogeneously distributed output lengths (half of them ≤50 tokens, a long tail up to 4,000), the typical memory waste was 60-80 %. That translated directly into wasted concurrency: with the same VRAM, instead of serving 32 requests with smart allocation, you served 8.

HoL blocking (Head-of-Line). The scheduling unit was the complete request. A batch containing one request of 500 tokens and 31 of ≤50 tokens ran 450 extra “empty” iterations (the GPU executed forward passes for all 32 slots, even though 31 had already finished). Wasted compute cost: ~84 % of the tail time in the example.

Measurable result. Sustained GPU SM utilisation on real workloads under static batching: 20-40 %. That is, ~70 % of the datacenter’s compute went unused. When the first Orca and vLLM benchmarks came out showing 10-24× throughput improvement, it was not marketing exaggeration; it was recovering all that wasted compute.

Orca (OSDI ‘22): the idea that changed everything

The paper by Yu, Jeong, Kim, Kim and Chun at OSDI 2022 (“Orca: A Distributed Serving System for Transformer-Based Generative Models”, from Seoul National University + FriendliAI) introduced two contributions that have remained the basis of everything that followed.

Iteration-level scheduling. Instead of scheduling at the level of the complete request, it schedules at the level of one decoder iteration: the step that generates ONE token. After each iteration the scheduler can (a) add new requests to the batch, (b) remove requests that have generated EOS or reached max_tokens, (c) reorder priorities. The compute engine executes exactly one iteration over the current batch.

Selective batching. Here is the non-obvious subtlety. The technical problem with batching requests of different lengths and different KV cache states is that some operations (the GEMMs of the Q, K, V projections and the FFN) are insensitive to position and can be batched by concatenating tokens, whereas attention is sensitive to per-request state (each request has its own KV cache of a different length). Orca’s solution: batch the GEMMs (concatenate all the step’s tokens into a [total_tokens, hidden] tensor) and run attention sequentially per request.

Paper result: up to 36.9× throughput vs FasterTransformer on GPT-3 175B at the same latency level. Orca is not open-source, it is only documented in the paper. FriendliAI commercialises it as the Friendli Engine. But the idea was published and everyone adopted it.

vLLM (SOSP ‘23): the open-source materialisation

What Orca described as a concept, vLLM materialised in production. The paper by Kwon, Li, Zhuang, Sheng et al. (UC Berkeley Sky Computing Lab, SOSP 2023) introduces PagedAttention, covered in detail in the PagedAttention deep dive, but it also consolidates continuous batching as universal practice.

The reason PagedAttention is a prerequisite for practical continuous batching is fragmentation. If you are going to insert and remove requests from the batch dynamically, and each request has a KV cache that grows on every iteration, traditional contiguous allocation fragments the HBM until it is unusable. PagedAttention splits the KV cache into fixed-size blocks (16 tokens by default) allocated on demand from a global pool. Memory waste falls from ~60-80 % to less than 4 % (only the last partially filled block per sequence).

vLLM paper metrics (2023):

vs HuggingFace Transformers (static, no continuous batching): up to 24× throughput.
vs HuggingFace TGI (which already had primitive continuous batching): ~3.5×.
vs FasterTransformer: 2-4× at the same latency.

What operationally distinguishes vLLM from Orca: vLLM runs attention in a single fused CUDA kernel (paged_attention_kernel) over non-contiguous blocks; Orca described attention as sequential, request by request. And vLLM exposes OpenAI-compatible APIs that let you drop the engine into existing stacks without touching the client.

Chunked prefill (SARATHI / Sarathi-Serve, OSDI ‘24)

There is a detail the original Orca/vLLM continuous batching did not solve: when a new request enters the batch, its prefill (processing the whole prompt at once) can take hundreds of milliseconds. During that time, the active decodes of the other requests are essentially paused, the GPU being dedicated to the new prefill. This showed up as spikes in TPOT (“inter-token latency”) every time a long request came in, which broke strict SLAs.

SARATHI (Agrawal et al., arXiv 2308.16369, August 2023) and later Sarathi-Serve (same group at Microsoft Research India, OSDI 2024, arXiv 2403.02310) introduce two combined ideas:

Chunked prefill. A long prefill (8,192 tokens, say) is split into chunks (2,048 tokens, say) processed one per iteration. Instead of one 200 ms step processing 8K tokens, four 50 ms steps processing 2K each.

Decode-maximal batching (“stall-free”). On each iteration, the scheduler first fills the batch with the active decodes (each costing 1 token), and only the leftover space goes to new prefill chunks. The result: active decodes keep advancing 1 token per iteration without pausing, while the new request completes its prefill in small bites.

The observation that justifies it: prefill is compute-bound (it processes N tokens at once and saturates FLOPs) while decode is memory-bound (1 token per step, under-using compute, with the GPU waiting on HBM). Mixing prefill chunks with decodes in the same step exploits the arithmetic-intensity slack: the decodes piggyback on the free compute of the prefill chunk.

Numbers:

Original SARATHI (LLaMA-13B on an A6000): decode throughput +10×, end-to-end +1.33×.
Sarathi-Serve (Mistral-7B on an A100): 2.6× serving capacity vs plain vLLM. Yi-34B on 2×A100: 3.7×. Falcon-180B with pipeline parallel: 5.6×.

Adoption in May 2026: always-on in vLLM v1 (default since v0.8.0, January 2025), SGLang, TensorRT-LLM. The key setting in vLLM is --max-num-batched-tokens (the token budget per step; default 2048). Raising it prioritises throughput, lowering it prioritises low TPOT.

Goodput: the metric that matters (DistServe, OSDI ‘24)

The classic “throughput” metric (requests/s or tokens/s) has a problem when there are SLOs. A server can report 1,000 req/s while P99 TTFT is 30 seconds and the SLO is 1 second, so only about 200 req/s actually meet the contract.

DistServe (Zhong et al., OSDI 2024) formalises goodput as the right metric: goodput = max sustained request rate meeting the SLOs (TTFT bound AND TPOT bound). The practical definition is usually: maximum rate with ≥90 % of requests within both SLOs.

Why it matters for the scheduler:

Optimising raw throughput leads to maximising batch size, which inflates P99 TPOT.
Optimising goodput limits the batch size when TPOT starts violating the SLO, prefers small requests if the batch already has a tail, and leaves resources available for new requests.

DistServe result: up to 7.4× more requests served or a 12.6× tighter SLO vs vLLM at the same SLO attainment. The gain comes from disaggregating prefill and decode onto different GPUs (removing the interference between phases), but the idea of optimising for goodput is independent and applicable to any scheduler.

Operationally this translates into monitoring:

goodput_proxy = histogram_quantile(0.95, vllm:time_to_first_token_seconds_bucket) < SLO_TTFT
AND histogram_quantile(0.95, vllm:time_per_output_token_seconds_bucket) < SLO_TPOT

The iterative scheduler in action

The maths that matter

Three formulas explain much of the operational behaviour.

GPU utilisation under static batching. With a batch of size B whose seq_len_i are the real lengths and max(seq_len_i) is the length that defines the padding:

$$U_{\text{static}} = \frac{\sum_i \text{seq len}_i}{B \cdot \max_i \text{seq len}_i}$$

For B=32, 30 sequences of 50 tokens and 2 of 500: U = (30·50 + 2·500) / (32·500) = 2500/16000 = 15.6 %. Four out of every five GPU cycles wasted.

GPU utilisation under continuous batching (idealised).

$$U_{\text{continuous}} \approx 1 - \frac{T_{\text{scheduler}}}{T_{\text{iteration}}}$$

With scheduler overhead of ~50-200 µs and an iteration time of ~10-30 ms: U > 95 %. The unit of loss is no longer padding, it is scheduling overhead, and that is negligible compared with the forward pass.

Goodput vs throughput.

$$\text{Goodput}(R) = R \cdot P(\text{latency} < \text{SLO})$$

where R is the offered request rate. Typical curve: goodput grows linearly with R up to the saturation knee, then falls because P(SLO) collapses when the system congests. The optimal point is just before the knee, not at peak throughput.

Example: at R=100 req/s with P(SLO)=0.99, goodput = 99. At R=200 req/s with P(SLO)=0.4, goodput = 80. More offered load, less useful goodput.

Chunked prefill token budget. On each vLLM step with chunked prefill active:

$$\text{prefill tokens this step} = \text{max num batched tokens} - \text{num decodes active}$$

Each active decode costs 1 token of the budget; the rest is filled with new prefill chunks. If max_num_batched_tokens = 2048 and num_decodes_active = 200, there are 1,848 tokens for prefill (one chunk of 1,848 or several small chunks).

Real implementations in May 2026

Engine	Current scheduler V	Chunked prefill default	Relevant notes
vLLM v1 (default ≥0.8.0)	unified V1	always-on	EngineCore isolated in a separate process; prefix caching with O(1) eviction; preempt-mode `recompute` by default; xgrammar/outlines backends.
SGLang	own (PyTorch ecosystem)	yes	RadixAttention gives cross-request prefix-cache hits; non-blocking CPU scheduler; leader in stable latency at high concurrency.
TensorRT-LLM	proprietary “in-flight batching”	yes	Policies `GUARANTEED_NO_EVICT` (conservative, default) and `MAX_UTILIZATION` (aggressive, risk of a pause when the KV is full). Compile-time vs runtime.
Triton + tensorrtllm_backend	`gpt_model_type: inflight_fused_batching`	yes	`max_queue_delay_microseconds` to group newly arrived requests. Decoupled mode for SSE streaming.
llama.cpp (llama-server)	own	`--cont-batching` ON since 2024	`-np N` parallel slots; no PagedAttention (contiguous KV per slot), so less flexible but simpler. Endpoint `:8080/metrics`.

A typical production-ready vLLM v1 configuration:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
 --tensor-parallel-size 4 \
 --max-num-batched-tokens 4096 \
 --max-num-seqs 256 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --preemption-mode recompute \
 --scheduling-policy fcfs \
 --gpu-memory-utilization 0.92

SGLang equivalent:

python -m sglang.launch_server \
 --model meta-llama/Llama-3.1-70B-Instruct \
 --tp 4 \
 --chunked-prefill-size 4096 \
 --max-running-requests 256 \
 --enable-radix-cache

The three operational tensions

Continuous batching + speculative decoding. Speculative decoding produces 1 to γ+1 tokens per step depending on the acceptance rate. The batch stops being uniform in tokens produced per iteration: nested raggedness. PagedAttention absorbs it (the KV cache can grow at different rates per request in the same step), but the scheduler loses symmetry. At low QPS (a conversational assistant) the combination is excellent: vLLM reports up to 2.8× speedup. At high QPS, the draft consumes slots from the decode pool and can reduce aggregate goodput. Rule of thumb: disable speculative decoding when gpu_cache_usage > 0.85. Full detail in Speculative decoding.

Continuous batching + multi-LoRA. Each request in the batch can use a different adapter (via SGMV, see Multi-LoRA serving). Worst case: every request in the batch with a different adapter and a different rank, and throughput falls by up to 50 % vs the base with no LoRA. Best case: all requests on the same adapter, equivalent to the base with no LoRA. Practical mitigation: group adapters by rank in the routing ahead of the engine; set --max-lora-rank to the maximum actually served, not generously above it.

Continuous batching + MoE. Each expert sees batch · k / N tokens per step. With DeepSeek-V3 (256 experts, k=8) and batch=32 in decode, each expert processes only 1 token on average: total compute starvation. To match the per-GPU throughput of a dense model, MoE needs batches »10× larger, which puts pressure on the KV cache. Wide-EP (see MoE inference) distributes the experts across many GPUs and allows larger effective batches per expert, at the cost of all-to-all comms that add milliseconds per step.

Metrics to monitor

The Prometheus metrics exposed by vLLM (prefix vllm:, vllm_ when scraped):

vllm:time_to_first_token_seconds (Histogram) — TTFT including queue time.
vllm:time_per_output_token_seconds (Histogram) — TPOT.
vllm:e2e_request_latency_seconds (Histogram) — end-to-end.
vllm:num_requests_running (Gauge) — active batch.
vllm:num_requests_waiting (Gauge) — queue depth.
vllm:num_requests_swapped (Gauge) — preempted to CPU.
vllm:gpu_cache_usage_perc (Gauge) — fraction of KV cache occupied.
vllm:gpu_prefix_cache_hit_rate (Gauge) — prefix cache hits.
vllm:num_preemptions_total (Counter) — preemptions. Any sustained value is a red flag.

Practical operational rules:

Stable zone under sustained load: gpu_cache_usage_perc ∈ [0.7, 0.9].
Warning at >0.95 (preemption imminent).
Critical if num_requests_waiting grows faster than num_requests_running: the server is not absorbing the load; scale out.

Operational pitfalls

Preempt-on-OOM. When gpu_cache_usage reaches ~1.0 with pending requests that need to grow their KV, vLLM preempts. V1 does RECOMPUTE by default (discards the KV, regenerates it on return); V0 did SWAP (moves it to CPU). RECOMPUTE is better for short sequences (cheap to regenerate); SWAP is better for long ones. The vllm:num_preemptions_total metric should be zero or near zero in steady state.

Inverse HoL blocking (memory monopoly). A very long request occupies many KV blocks, so small requests do not fit in the batch even though compute is free. Chunked prefill mitigates the compute blocking during a new prefill, but it does not solve the memory monopoly. Partial solution: per-request limit policies (an aggressive max_tokens) or priorities.

Starvation. FCFS can leave requests pending for a long time if the active ones do not finish. vLLM supports --scheduling-policy priority with an x-priority header. Recent work (NeurIPS 2024 Efficient LLM Scheduling by Learning to Rank, arXiv:2501.14312 Locality-aware Fair Scheduling) proposes schedulers with quantum-based starvation prevention; not yet integrated into vLLM mainline.

Badly calibrated chunk size. A small chunk (512) gives low TPOT, high TTFT and memory overhead from more KV accesses. A large chunk (8192+) gives low TTFT and TPOT spikes during the chunk. Rule: start at 2,048, measure P95 TPOT, adjust.

Batch size cap. A high --max-num-seqs gives more concurrency but P99 TPOT explodes. A low one wastes throughput. Rule of thumb: max_num_seqs ≈ HBM_for_KV / (avg_seq_len × bytes_per_token_KV).

Implications for on-premise hardware

On an RTX 4090 (24 GB). llama.cpp with --cont-batching -np 4-8 is the natural pattern. Typical models: Llama 3 8B Q4_K_M with ~8 parallel slots, aggregate throughput on the order of hundreds of tok/s. vLLM also works if the weights fit (Llama 3 8B BF16 does; the 70B does not fit whole), although PagedAttention on consumer hardware gives less return than in the datacenter.

On a generic 4×H100 SXM cluster (320 GB, NVLink). Here vLLM v1 / SGLang are the de facto standard. Typical configurations:

Llama 3 70B FP8 + TP=4: dozens of concurrent sessions with P95 TPOT under 50 ms, tens of thousands of aggregate tok/s at a moderate batch size.
Llama 3 70B AWQ-INT4 + TP=2 plus the rest of the cluster for additional concurrency or multi-LoRA with SGMV.
DeepSeek-V3 needs larger setups (8-16 H100) to fit whole in FP8; with Wide-EP, continuous batching starts operating on much larger batches and the economics change (see MoE).

The rule of thumb for May 2026: vLLM v1 with chunked prefill always-on and prefix caching enabled is the sensible default configuration for any dense model that fits comfortably; SGLang offers better stable latency at high concurrency thanks to the overlap between the CPU scheduler and the GPU step; TensorRT-LLM gives peak throughput at high concurrency with the rigidity of compile-time.

What we have not covered

Locality-aware fair scheduling (arXiv:2501.14312) and learning-to-rank schedulers (NeurIPS 2024): the next generation of algorithms closing the fairness vs prefix-cache locality trade-off.
Smooth goodput (arXiv:2410.14257): a refinement of the DistServe metric using max slowdown instead of a binary SLO.
Triton tensorrtllm_backend in production: decoupled mode for streaming, ensembles with pre/post-processing, autoscaling with KServe.
vLLM speculators v0.3.0 and the vLLM-compatible drafter training framework.

References

Yu, G.-I., Jeong, J., Kim, G.-W., Kim, S., Chun, B.-G. Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022. https://www.usenix.org/system/files/osdi22-yu.pdf
Kwon, W. et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. https://arxiv.org/abs/2309.06180
Agrawal, A. et al. SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. 2023. https://arxiv.org/abs/2308.16369
Agrawal, A. et al. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. OSDI 2024. https://arxiv.org/abs/2403.02310
Zhong, Y. et al. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024. https://arxiv.org/abs/2401.09670
Sheng, Y. et al. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. MLSys 2024. https://arxiv.org/abs/2311.03285
Efficient LLM Scheduling by Learning to Rank. NeurIPS 2024.
Locality-aware Fair Scheduling. 2025. https://arxiv.org/abs/2501.14312
vLLM V1 alpha release (Jan 2025): https://blog.vllm.ai/2025/01/27/v1-alpha-release.html
vLLM Anatomy (Sep 2025): https://blog.vllm.ai/2025/09/05/anatomy-of-vllm.html
vLLM Large-Scale Serving (Dec 2025): https://blog.vllm.ai/2025/12/17/large-scale-serving.html
vLLM speculators v0.3.0 (Dec 2025): https://blog.vllm.ai/2025/12/13/speculators-v030.html
Anyscale, Continuous Batching for LLM Inference (2023): https://www.anyscale.com/blog/continuous-batching-llm-inference
vLLM metrics docs: https://docs.vllm.ai/en/latest/design/metrics/
vLLM optimization: https://docs.vllm.ai/en/stable/configuration/optimization/
SGLang: https://github.com/sgl-project/sglang
TensorRT-LLM performance tuning: https://nvidia.github.io/TensorRT-LLM/performance/performance-tuning-guide/tuning-max-batch-size-and-max-num-tokens.html
Red Hat, 5 steps to triage vLLM performance (Mar 2026): https://developers.redhat.com/articles/2026/03/09/5-steps-triage-vllm-performance
NVIDIA blog, Chunked Prefill with TensorRT-LLM: https://developer.nvidia.com/blog/streamlining-ai-inference-performance-and-deployment-with-nvidia-tensorrt-llm-chunked-prefill/

Goodput on lo0 — Tech Blog

GuideLLM in depth: validating the SLO under load and sizing from the knee

What this article covers

Why GuideLLM

The load modes (rate-types)

The automatic sweep: from idle to the knee

A worked example: reading a 10-round sweep

Defining the SLO: the number that decides the knee

Execution: the command

Defining the load: --data decides the result

The output: what to read

From the knee to sizing and cost

Deployment as a Kubernetes Job

SLO validation in CI: catching regressions

Comparing configurations with sweeps

Watching the three axes during the sweep

Interpreting the distributions, not just the knee

GuideLLM in the reproducible harness

The cost of a sweep (in euros)

The SLO by use case: the knee moves with it

GuideLLM vs the others: when to use which

Checklist for a defensible sweep

Limits and traps (data-driven)

Closing

See also

Sources

LLM inference benchmarking: frameworks, metrics and the state of the art (tool by tool)

What this introduction covers

The performance metrics

The latency decomposition

Goodput: the honest metric

How each metric is instrumented (and where the error creeps in)

The architecture of the tool biases the figure

Frameworks, one by one

vLLM bench and SGLang bench — engine micro-bench

AIPerf — the NVIDIA one (formerly genai-perf), multi-process

GuideLLM — from the vLLM project, SLO-oriented

LLMPerf — the Anyscale/Ray classic

MLPerf Inference — the industry standard

Comparison table

The concurrency sweep: finding the knee

Worked example: reading a sweep

MLPerf Inference: the comparability standard

Measurement bias and reproducibility

Checklist for a reproducible benchmark

Performance ≠ quality

The latency versus throughput trade-off

The connection with cost and energy

State of the art 2026

Limits and traps (data-driven)

Closing

See also

Sources

Continuous batching: the salon with 8 chairs that does not wait for the slow customer — Orca, vLLM, chunked prefill and goodput

You are here: DEPLOY

TL;DR

The analogy: the salon with 8 chairs

The problem continuous batching solves

Orca (OSDI ‘22): the idea that changed everything

vLLM (SOSP ‘23): the open-source materialisation

Chunked prefill (SARATHI / Sarathi-Serve, OSDI ‘24)

Goodput: the metric that matters (DistServe, OSDI ‘24)

The iterative scheduler in action

The maths that matter

Real implementations in May 2026

The three operational tensions

Metrics to monitor

Operational pitfalls

Implications for on-premise hardware

What we have not covered

See also

References

Defining the load: `--data` decides the result