Measurement bias and reproducibility: why two benchmarks of the same model give figures that differ by up to 7×

Tue, 16 Jun 2026 07:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The mathematical symbol is reserved for formulas: amounts are expressed in € or USD. Thousands with a thin space ($1\,000$).

TL;DR

Running the same Llama-3-70B-Instruct model on the same 4×H100 SXM 80 GB node with two different tools can produce throughput results that differ by 7.2× without the engine changing a single line of code. The cause is not the engine: it is the measurement method. At 1,000 QPS, a single-process asyncio client (vLLM bench, SGLang bench, genai-perf pre-AIPerf) processed 75,574 tokens against the 545,733 tokens of a multi-process client, both measuring the same endpoint (arXiv 2605.24217). The bias is not random: it is systematic, directional and reproducible. This article catalogues the sources of bias, quantifies their magnitude and describes the minimum harness that turns a number into an auditable datum.

The sources of bias: a catalogue with magnitudes

Each source of bias moves the published number in a given direction and with a characteristic magnitude. The table below orders the sources from largest to smallest impact observed in the literature:

Source of bias	Direction of the bias	Documented magnitude	Reference
Single-process client saturation (Python GIL, asyncio)	understates throughput; overstates latency	up to 7.2× fewer tokens processed at 1,000 QPS	arXiv 2605.24217
Tokenizer different from the model’s (LLMPerf uses a universal LlamaTokenizer)	tok/s not comparable across different vocabularies	variable; up to 33 % of client overhead in the TPS denominator	NVIDIA blog (TPS = client overhead ÷ total duration)
ignore_eos missing or inconsistent OSL	underestimates throughput; artificially shortens the real OSL	up to premature termination; an unrealistic fixed-length benchmark	vLLM bench docs; NVIDIA fundamental concepts
Unrealistic ISL/OSL distribution (fixed length instead of a realistic distribution)	an optimistic knee that does not resemble production	the knee shifts; prefill understated with short prompts	AIPerf sequence-length-distributions docs
Missing warmup / warm prefix cache	overstates TTFT (artificially low on the first requests)	TTFT underestimated by an already populated KV cache	vLLM prefix caching docs; arXiv 2605.24217
`concurrency` vs `request-rate` as the load mode	concurrency saturates symmetrically; request-rate can build a queue	ITL overestimated if the queue grows without a ceiling with request-rate	NVIDIA fundamentals; AIPerf docs
Measurement window too short	does not capture the steady state; includes ramp-up	throughput overestimated; latency underestimated	meta-metrics arXiv 2508.10251
Thermal state and GPU clocks	thermal throttling reduces throughput without prior stabilisation	temperature > 75 °C triggers throttling; a difference of ~3 W of stable power	arXiv 2604.09048
Warm prefix cache between runs	overstates TTFT; unrealistic TTFT if the cache is pre-populated from the previous run	a severe effect in multi-prompt evaluations with shared prefixes	arXiv 2605.24217
Mean instead of percentiles	hides the latency tail	P99 can be 4–6× the median at high concurrency	NVIDIA fundamentals; Anyscale docs

The dominant bias: single-process client saturation

The largest bias is not in the engine but in the client that generates the load. The single-process tools (vLLM bench, SGLang bench, genai-perf before its replacement by AIPerf on 15 April 2026) use a single Python process with asyncio to manage the concurrent requests. The Python Global Interpreter Lock (GIL) prevents a single process from using more than one CPU core at a time, which introduces a bottleneck on the client side that becomes critical at high concurrency.

The mathematical effect: as the request rate rises, the client fails to dispatch requests at the configured pace, accumulates queueing time on the client side, and that time gets recorded as engine latency. The result is that TTFT and TPOT appear inflated and throughput appears depressed, without the engine having changed anything (arXiv 2605.24217).

The documented discrepancy at 1,000 QPS:

Client architecture	Tokens processed at 1,000 QPS	Ratio
Single-process (asyncio, Python GIL)	75,574	1× (baseline)
Multi-process (distributed load)	545,733	7.2×

Note: both clients point at the same endpoint; the difference is entirely attributable to the benchmark client, not to the engine.

Tokenizer bias: tok/s are not comparable across vocabularies

Tokens are not universal. Each model has its own tokenizer with its own vocabulary. Llama-3 uses a vocabulary of 128,256 tokens; Gemma uses 256,128; earlier models used 32,000–50,000. The same English text produces a different number of tokens depending on the model’s tokenizer.

The direct consequence for benchmarking: if the tool measures tok/s with a tokenizer different from that of the model being served, the token count, and therefore the throughput in tok/s and the cost per token, are biased. LLMPerf (archived in December 2025) used LlamaTokenizer universally for all models, which guaranteed internal consistency in the leaderboard but made the tok/s not comparable with measurements of other models with different vocabularies.

The TPS formula in LLMPerf additionally included the benchmark’s full denominator, the time spent generating prompts, preparing requests and storing responses, which NVIDIA estimated at up to 33 % of the total duration at concurrency 1. This makes LLMPerf’s TPS systematically lower than that of GenAI-Perf/AIPerf for the same system, without the engine being worse:

$$\text{TPS}_{\text{LLMPerf}} = \frac{\text{output tokens}}{T_{\text{end}} - T_{\text{start}}}$$ $$\text{TPS}_{\text{GenAI-Perf}} = \frac{\text{output tokens}}{T_y - T_x}$$

Where $T_{\text{start}}$ and $T_{\text{end}}$ include the client overheads, while $T_x$ and $T_y$ are the instant of the first request and that of the last token received, respectively (NVIDIA · Fundamental Concepts).

The operational rule: always count tokens with the tokenizer of the model being served, not with a proxy tokenizer.

The bias of ignore_eos and an inconsistent OSL

Most LLM models generate a special end-of-sequence token (EOS) when they consider the response complete. If the benchmark does not set ignore_eos=True, the real output length (OSL) varies from request to request according to the model’s natural length distribution, which produces:

Inconsistent OSL: two runs with the same seed can produce different OSL distributions if the model varies its natural output.
Spurious comparison: a “faster” model may be so simply because it generates shorter responses (it hits EOS sooner), not because it has more real throughput.
Underestimated throughput: if the benchmark expects OSL=256 but the model stops at OSL=80 on average, the throughput in tok/s appears higher but measures less work.

The ignore_eos parameter (or --ignore-eos in vLLM bench) instructs the engine to ignore the EOS token and continue until reaching max_tokens. It is mandatory for the OSL to be the configured one rather than the model’s natural one, and for two runs to be comparable (vLLM benchmark CLI docs).

The bias of the ISL/OSL distribution

The distribution of input (ISL) and output (OSL) lengths determines what proportion of the compute time goes to prefill (costly in TTFT) and how much to decode (costly in ITL). A benchmark with a fixed length, say ISL=128, OSL=128, produces results that do not resemble any real traffic.

Real use cases have very different distributions:

Use case	Typical ISL (tokens)	Typical OSL (tokens)	Dominated by
Translation	500–2,000	500–2,000	balanced
Generation (code, email)	~100	~1,000	decode (long OSL)
Summarisation / RAG	~1,000	~100	prefill (long ISL)
Reasoning (CoT)	~100	1,000–10,000	very long decode

A short-ISL benchmark with a model optimised for prefill will give an artificially low TTFT and an artificially high throughput. The knee of the sweep shifts: with ISL=64 the system admits more concurrency without breaking the TTFT SLO; with ISL=1,024 prefill saturates earlier and the knee appears earlier. Using the wrong distribution means sizing for traffic that does not exist.

AIPerf introduces sequence distributions with configurable per-component variance to reproduce realistic traffic mixes (AIPerf · Sequence Length Distributions):

--sequence-distribution "64|10,32|8:70;256|40,128|20:20;1024|100,512|50:10"

This creates 70 % of requests with $\text{ISL} \sim \mathcal{N}(64, 10)$ and $\text{OSL} \sim \mathcal{N}(32, 8)$, 20 % with medium ISL/OSL, and 10 % with long ISL/OSL, far more faithful to real chatbot traffic than a fixed length.

The bias of warmup and the prefix cache

Two related but distinct sources of bias:

Missing warmup. The first requests of a benchmark hit a “cold” engine: the GPU is in a low-clock state, the KV cache is empty, and the operating system may be paging memory. The TTFT of the first requests is structurally higher than that of the steady state. If the benchmark does not discard a warmup period, the mean TTFT includes these outliers and overestimates the real production latency. Some frameworks (GenAI-Perf/AIPerf) use a sliding window that excludes the ramp-up and ramp-down requests; others do not.

Warm prefix cache between runs. The prefix KV cache (prefix cache or prompt cache) stores the computed results of repeated prompt prefixes. If the benchmark runs multiple consecutive runs with the same prompts, the second and subsequent runs find the KV cache already populated and report an artificially low TTFT, that of the decode, not of the prefill. For a baseline benchmark, the prefix cache must be cold; for one that simulates production with repeated prompts, warm. The distinction must be made explicit (arXiv 2605.24217; vLLM · Automatic Prefix Caching).

The bias of `concurrency` vs `request-rate`

The two load control modes produce different latency distributions for the same system:

Mode	Semantics	When to use	Risk
concurrency N	keeps exactly N requests in flight; as soon as one finishes, it launches another	measuring the system under a fixed concurrency load	over-represents sustained load; does not simulate real arrivals
request-rate r (constant or Poisson)	launches one request every $1/r$ seconds (constant) or with interarrival $\sim \text{Exp}(1/r)$ (Poisson)	simulating real traffic (random arrivals)	if the engine cannot absorb r req/s, the queue grows without a ceiling

NVIDIA recommends the concurrency mode for most capacity benchmarks (NVIDIA · LLM Benchmarking Fundamental Concepts). The request-rate mode is more faithful for online traffic (a Poisson distribution of arrivals), but if the rate exceeds the engine’s capacity the queue grows indefinitely and the latency metrics include queueing time that can dominate the TTFT, mixing queue behaviour with engine behaviour.

A system measured at concurrency=16 and another at request-rate=16 req/s are not under the same load: fixed concurrency guarantees 16 simultaneous requests; the rate sets the arrival pace but not the instantaneous concurrency. Comparing their results without adjustment is incorrect.

The bias of the measurement window and the thermal state

Window too short. A 30-second benchmark may measure the system’s ramp-up, not the steady state. The recommendation of arXiv 2508.10251 is that the measurement window cover at least 3-5× the system’s ramp time under load, and that the metrics be computed only over the steady window, excluding warmup and cooldown.

Thermal state of the GPU. Datacenter GPUs (H100 SXM, A100) operate with thermal throttling above ~75 °C. If the GPU has not reached its steady-state temperature before the benchmark, the first measurements correspond to a higher-clock state than the sustainable one. Controlled experiments document that power must stabilise within a range of 3 W for at least 30 seconds before the measurements are representative (arXiv 2604.09048 · Watt Counts). The effect is especially severe in prefill benchmarks (TTFT), where the high clocks of the cold state produce an artificially low TTFT.

For a reproducible benchmark on 4×H100 SXM 80 GB: before measuring, run sustained load for at least 2–3 minutes until nvidia-smi reports a stable temperature and stable power (variation < 3 W over 30 s). Only then start the measurement window.

How the tools compute ITL differently

ITL (Inter-Token Latency) appears in every tool but its formula varies, and the differences are not small:

GenAI-Perf / AIPerf:

$$\text{ITL} = \frac{\text{e2e latency} - \text{TTFT}}{\text{output tokens} - 1}$$

TTFT is excluded from the numerator and the denominator discounts the first token. ITL is a pure decode metric, with no prefill contamination.

LLMPerf (archived Dec. 2025):

$$\text{ITL}_{\text{LLMPerf}} = \frac{\text{e2e latency}}{\text{output tokens}}$$

TTFT is included in the numerator. For short sequences (OSL < 50 tokens), TTFT can represent 50–80 % of the e2e_latency, so that LLMPerf’s ITL measures mainly prefill, not decode. Two systems with the same decode but different prefill will show different ITLs in LLMPerf even if they are identical in decode speed.

vLLM bench (benchmark_serving.py):

It computes ITL as the mean of the intervals between consecutive tokens of the output stream, including intra-request variance. It can reveal decode jitter that the others average away.

The practical consequence: never cross a GenAI-Perf ITL with an LLMPerf one as if they were the same metric. They are different formulas over the same signal.

Comparability across tools: why their numbers do not cross

The following table summarises the methodological differences that make one tool’s numbers structurally incompatible with another’s without an explicit adjustment:

Dimension	vLLM bench	LLMPerf (archived)	GenAI-Perf / AIPerf	GuideLLM
Client architecture	single-process asyncio	single-process	multi-process	multi-process
ITL includes TTFT	no	yes	no	no
TPS denominator	$T_y - T_x$	$T_{\text{end}} - T_{\text{start}}$ (overhead included)	$T_y - T_x$	per round
Tokenizer	that of the model served	universal LlamaTokenizer	that of the model served	that of the model served
Warmup / sliding window	not automatic	no	yes (sliding window)	per round
ignore_eos by default	no	no	explicitly recommended	configurable
ISL/OSL distribution	manual parameter	manual parameter	distributions with variance	configurable `--data`
Primary load mode	concurrency	concurrency (drained batches)	concurrency (recommended)	sweep + poisson
Structured output	basic JSON	JSON	JSON + CSV	JSON + HTML + CSV

An LLMPerf result and a GenAI-Perf result for the same endpoint can differ in TTFT, ITL and TPS simultaneously and in all of them for methodological reasons, not because of engine differences. The only way to cross them is to run both tools on the same system under the same conditions and compute the empirical conversion factor, which in practice amounts to re-measuring with a single tool.

The honest harness: a reproducibility checklist

A benchmark whose number can be defended in an audit has to come accompanied by all the metadata that allow it to be reproduced. The minimum to demand, organised by category:

Hardware

Metadata	Example 4×H100	Effect if not fixed
GPU (model, variant, count)	4× NVIDIA H100 SXM 80 GB	H100 PCIe vs SXM differ in memory BW and NVLink
GPU-GPU interconnect	NVLink 4.0	tensor parallelism depends on the interconnect BW
CPU, RAM, CPU-GPU bandwidth	2× Intel Xeon 8480+, 512 GB DDR5, PCIe 5.0	the tokenisation bottleneck can be on the CPU
Thermal state at the start	GPU temperature < 65 °C, stable power ± 3 W	throttling alters TPS and TTFT by up to ~15 %
GPU clocks	without `nvidia-smi -pm 1` they can vary	a difference of up to ~10 % in throughput

Software and model

Metadata	Example	Effect if not fixed
Engine + version + flags	vLLM 0.8.4, `--tensor-parallel-size 4 --gpu-memory-utilization 0.90`	each version changes the scheduler and the KV cache management
Model + precision	Llama-3-70B-Instruct, FP8	FP16 vs FP8 differ by ~1.4–1.8× in throughput
Tokenizer used for counting	tokenizer of the model served (HF tokenizer)	a universal LlamaTokenizer biases tok/s across vocabularies
Generation seed	`--seed 42`	without a fixed seed, the OSL varies run-to-run
`ignore_eos`	`True`	without it, the OSL varies with the prompt content
Sampling parameters	`temperature=1.0, top_p=0.95`	greedy vs sampling affect logit speed

Load

Metadata	Example	Effect if not fixed
Bench tool + version	AIPerf 0.2.1	each version changes formulas and warmup
ISL/OSL distribution	$\mathcal{N}(512, 64)$ ISL, $\mathcal{N}(128, 20)$ OSL	changing the distribution moves the knee
Load mode	concurrency, sweep 1–64	concurrency vs request-rate: different distributions
Concurrency levels	1, 2, 4, 8, 16, 32, 64 (full sweep, beyond the knee)	without passing the knee, the safe capacity is unknown
Duration per point	120 s minimum per level	short windows capture ramp-up, not the steady state
Warmup	30 s excluded from the metric computation	without warmup, the metrics include the cold state
Prefix cache treatment	cold (flush between runs) or warm (declared explicitly)	warm cache: unrealistic TTFT for a baseline

SLO and reported metrics

Metadata	Example	Effect if not declared
Declared SLO	TTFT P99 < 500 ms, ITL P95 < 50 ms	goodput depends on the SLO; without an SLO there is no goodput
Reported percentiles	P50, P95, P99 (not just the mean)	the mean hides the tail; irrelevant for an SLO
Throughput vs goodput	goodput under the SLO, not raw throughput	catalogue throughput can be 5–10× the goodput
Total number of requests	≥ 1,000 per concurrency level	small samples: high variance in percentiles

Methodological defence of the datum before an audit

A throughput number presented without the data sheet above is not a datum: it is an anecdote. The validation pattern for a technical audit:

1. Traceability of the tool. The tool and its version must be pinnable to a Git commit or a container image hash. AIPerf and GuideLLM export JSON with version metadata; vLLM bench omits them by default and they have to be captured manually.

2. Reproducibility of the environment. The engine deployment script (or the Kubernetes manifest) and the exact benchmark command must be enough for a third party to reproduce the number on the same hardware. GuideLLM exports the benchmark file as an authoritative record of the session: configuration, metadata, per-benchmark statistics and request entries with individual timings (GitHub vllm-project/guidellm).

3. Verification of the steady state. The metrics must come from the benchmark’s steady window (warmup excluded). The throughput vs concurrency curve must show the knee, the point where throughput saturates and latency blows up, and extend beyond it. Without a visible knee, the reported capacity may be the client’s limit, not the engine’s.

4. Goodput, not raw throughput. The auditable throughput is the goodput under the declared SLO, not the peak throughput. A system that reports 4,000 tok/s but whose goodput under TTFT P99 < 500 ms is 1,800 tok/s has a real capacity of 1,800 tok/s for interactive use cases.

5. Internal comparability. If two engines or two configurations are compared, the tool, the load distribution, the SLO, the hardware and the warmup treatment must be identical. Any difference in these dimensions contaminates the comparison.

The goodput formula as an auditable metric:

$$\text{goodput} = \text{throughput} \times \Pr[\text{TTFT} \leq \text{SLO}_{\text{TTFT}}] \times \Pr[\text{ITL} \leq \text{SLO}_{\text{ITL}}]$$

Where the probabilities are estimated over the empirical distribution of the run. An engine with a throughput of 4,000 tok/s and $\Pr[\text{TTFT} \leq 500\,\text{ms}] = 0.45$ has a goodput of $4{,}000 \times 0.45 = 1{,}800$ tok/s.

A numerical example: the same node, four different measurements

Reference hardware: 4×H100 SXM 80 GB, model Llama-3-70B-Instruct FP8, tensor parallel 4, SLO TTFT P99 < 500 ms.

Benchmark configuration	Reported throughput	Real goodput	Factor vs goodput
vLLM bench, concurrency=32, no ignore_eos, no warmup, fixed OSL=64	5,800 tok/s	~1,200 tok/s (short OSL inflates TPS)	4.8×
LLMPerf, concurrency=32, LlamaTokenizer, overhead included in the denominator	3,100 tok/s	~1,600 tok/s (ITL includes TTFT)	1.9×
AIPerf, concurrency=32, ignore_eos, sliding window, model’s tokenizer	3,400 tok/s	3,350 tok/s	1.01×
GuideLLM, sweep 1–64, poisson, realistic ISL/OSL distribution, declared SLO	3,330 tok/s goodput (knee at round 6)	3,330 tok/s	1.0× (honest baseline)

The range: from 1,200 tok/s to 5,800 tok/s for the same engine, the same hardware and the same model. The maximum factor is 4.8×. The cause is not the engine; it is the instrumentation decisions.

Decomposing the total bias

The total observable bias between two tools is the multiplicative composition of the individual biases. For a comparison of single-process vLLM bench vs multi-process GuideLLM on the same system:

$$\text{total factor} \approx \underbrace{f_{\text{client}}}_{\leq 7.2\times} \times \underbrace{f_{\text{tokenizer}}}_{1.0{-}1.3\times} \times \underbrace{f_{\text{ignore-eos}}}_{1.0{-}2.0\times} \times \underbrace{f_{\text{ISL/OSL}}}_{1.0{-}1.5\times} \times \underbrace{f_{\text{warmup}}}_{1.0{-}1.2\times}$$

The client saturation factor ($\leq 7.2\times$) dominates, but the other factors multiply. Under adverse conditions (single-process client + different tokenizer + no ignore_eos + unrealistic OSL + no warmup), the compound bias can exceed 15–20× for the same system.

State of the art 2026: what has changed

genai-perf → AIPerf migration (15 Apr 2026): NVIDIA retired genai-perf and replaced it with AIPerf, which is multi-process with automatic saturation detection, configurable ISL/OSL distributions and a built-in sliding window. The migration removes the single-process client bias from NVIDIA’s official tooling.
LLMPerf archived (Dec. 2025): the ray-project/llmperf repository is archived and in read-only mode. The historical results of the LLMPerf leaderboard are comparable only among themselves; they must not be crossed with modern measurements without adjustment.
GuideLLM 0.5.x (2025–2026): a complete architectural refactor, multimodal support, and authoritative JSON export with all the session metadata. It is the OSS standard for SLO-driven evaluation.
arXiv 2605.24217 (May 2026): the first formal characterisation of systematic measurement bias in production LLM benchmarks, with a mathematical demonstration of the GIL effect and a proposal for a multi-process harness.
arXiv 2508.10251: meta-metrics and good practice for system-level performance benchmarking; it establishes the minimum dimensions of the parameter space that must be declared for a result to be reproducible.
MLPerf Inference v5.1 (Sep. 2025): 27 participants, a participation record. The MLPerf rules require declaring the exact code, the dataset and the hardware, and admit public review; it is the only industry benchmark with a formal review process for reproducibility.

Cross-links from the benchmarking track

The biases described here apply to every tool in the catalogue. For the complete data sheet of each tool:

Full catalogue: LLM benchmarking tools, data sheet by data sheet
GuideLLM and SLO validation: GuideLLM in depth: validating the SLO under load
AIPerf (ex genai-perf) and saturation detection: NVIDIA GenAI-Perf in depth
State of the art of frameworks: LLM benchmarking: frameworks and state of the art
Engine decision on the Pareto axis: Serving engines compared: the Pareto frontier

Sources

arXiv 2605.24217 · Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks — https://arxiv.org/abs/2605.24217
arXiv 2508.10251 · Meta-Metrics and Best Practices for System-Level Inference Performance Benchmarking — https://arxiv.org/pdf/2508.10251
arXiv 2604.09048 · Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures — https://arxiv.org/html/2604.09048v1
NVIDIA Technical Blog · LLM Inference Benchmarking: Fundamental Concepts (ITL, TPS, ISL/OSL, ignore_eos, concurrency vs request-rate) — https://developer.nvidia.com/blog/llm-benchmarking-fundamental-concepts/
NVIDIA AIPerf Docs · Sequence Length Distributions for Advanced Benchmarking — https://docs.nvidia.com/aiperf/tutorials/datasets-inputs/sequence-length-distributions-for-advanced-benchmarking
NVIDIA AIPerf · Request Rate with Max Concurrency — https://docs.nvidia.com/aiperf/tutorials/load-patterns-scheduling/request-rate-with-max-concurrency
NVIDIA AIPerf · Comprehensive LLM Benchmarking Guide — https://lucaberton.com/blog/nvidia-aiperf-llm-inference-benchmarking-guide/
vLLM Documentation · Benchmark CLI (ignore_eos, métricas disponibles) — https://docs.vllm.ai/en/latest/benchmarking/cli/
vLLM Documentation · Automatic Prefix Caching — https://docs.vllm.ai/en/stable/design/prefix_caching/
GitHub · ray-project/llmperf (archivado dic. 2025) — https://github.com/ray-project/llmperf
GitHub · vllm-project/guidellm (exportación JSON autoritativa, metadatos de sesión) — https://github.com/vllm-project/guidellm
Red Hat Developer · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
MLCommons · MLPerf Inference v5.1 results (récord 27 participantes, reproducibilidad formal) — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
arXiv 2502.16721 · Speed and Conversational LLMs: Not All Is About Tokens per Second (tokenizer incompatibilidad y tok/s entre vocabularios) — https://arxiv.org/pdf/2502.16721
Medium · LLM Inference Benchmarking (genAI-perf y vLLM, discrepancia 7,2×) — https://kchandan.medium.com/llm-inference-benchmarking-genai-perf-and-vllm-5dd06b57428e

Warmup on lo0 — Tech Blog