Measurement bias and reproducibility: why two benchmarks of the same model give figures that differ by up to 7×

Contents

Notation: amounts in euros (N €), decimals with a point. The mathematical symbol is reserved for formulas: amounts are expressed in € or USD. Thousands with a thin space (\(1\,000\)).

TL;DR

Running the same Llama-3-70B-Instruct model on the same 4×H100 SXM 80 GB node with two different tools can produce throughput results that differ by 7.2× without the engine changing a single line of code. The cause is not the engine: it is the measurement method. At 1,000 QPS, a single-process asyncio client (vLLM bench, SGLang bench, genai-perf pre-AIPerf) processed 75,574 tokens against the 545,733 tokens of a multi-process client, both measuring the same endpoint (arXiv 2605.24217). The bias is not random: it is systematic, directional and reproducible. This article catalogues the sources of bias, quantifies their magnitude and describes the minimum harness that turns a number into an auditable datum.


The sources of bias: a catalogue with magnitudes

Each source of bias moves the published number in a given direction and with a characteristic magnitude. The table below orders the sources from largest to smallest impact observed in the literature:

Source of biasDirection of the biasDocumented magnitudeReference
Single-process client saturation (Python GIL, asyncio)understates throughput; overstates latencyup to 7.2× fewer tokens processed at 1,000 QPSarXiv 2605.24217
Tokenizer different from the model’s (LLMPerf uses a universal LlamaTokenizer)tok/s not comparable across different vocabulariesvariable; up to 33 % of client overhead in the TPS denominatorNVIDIA blog (TPS = client overhead ÷ total duration)
ignore_eos missing or inconsistent OSLunderestimates throughput; artificially shortens the real OSLup to premature termination; an unrealistic fixed-length benchmarkvLLM bench docs; NVIDIA fundamental concepts
Unrealistic ISL/OSL distribution (fixed length instead of a realistic distribution)an optimistic knee that does not resemble productionthe knee shifts; prefill understated with short promptsAIPerf sequence-length-distributions docs
Missing warmup / warm prefix cacheoverstates TTFT (artificially low on the first requests)TTFT underestimated by an already populated KV cachevLLM prefix caching docs; arXiv 2605.24217
concurrency vs request-rate as the load modeconcurrency saturates symmetrically; request-rate can build a queueITL overestimated if the queue grows without a ceiling with request-rateNVIDIA fundamentals; AIPerf docs
Measurement window too shortdoes not capture the steady state; includes ramp-upthroughput overestimated; latency underestimatedmeta-metrics arXiv 2508.10251
Thermal state and GPU clocksthermal throttling reduces throughput without prior stabilisationtemperature > 75 °C triggers throttling; a difference of ~3 W of stable powerarXiv 2604.09048
Warm prefix cache between runsoverstates TTFT; unrealistic TTFT if the cache is pre-populated from the previous runa severe effect in multi-prompt evaluations with shared prefixesarXiv 2605.24217
Mean instead of percentileshides the latency tailP99 can be 4–6× the median at high concurrencyNVIDIA fundamentals; Anyscale docs

The dominant bias: single-process client saturation

The largest bias is not in the engine but in the client that generates the load. The single-process tools (vLLM bench, SGLang bench, genai-perf before its replacement by AIPerf on 15 April 2026) use a single Python process with asyncio to manage the concurrent requests. The Python Global Interpreter Lock (GIL) prevents a single process from using more than one CPU core at a time, which introduces a bottleneck on the client side that becomes critical at high concurrency.

The mathematical effect: as the request rate rises, the client fails to dispatch requests at the configured pace, accumulates queueing time on the client side, and that time gets recorded as engine latency. The result is that TTFT and TPOT appear inflated and throughput appears depressed, without the engine having changed anything (arXiv 2605.24217).

The documented discrepancy at 1,000 QPS:

Client architectureTokens processed at 1,000 QPSRatio
Single-process (asyncio, Python GIL)75,5741× (baseline)
Multi-process (distributed load)545,7337.2×

Note: both clients point at the same endpoint; the difference is entirely attributable to the benchmark client, not to the engine.

SINGLE-PROCESS (vLLM bench, SGLang bench, genai-perf pre-AIPerf)1 Python proc.asyncio + GILclient queue(≠ engine queue)Engine (vLLM…)TTFT/TPOToverestimatedTPS underestimatedMULTI-PROCESS (AIPerf, GuideLLM)N processesno shared GILEngine (vLLM…)measures the engine;7.2× more throughputcapturedThe client architecture determines what gets measured: the client bottleneck or the engine's.

Tokenizer bias: tok/s are not comparable across vocabularies

Tokens are not universal. Each model has its own tokenizer with its own vocabulary. Llama-3 uses a vocabulary of 128,256 tokens; Gemma uses 256,128; earlier models used 32,000–50,000. The same English text produces a different number of tokens depending on the model’s tokenizer.

The direct consequence for benchmarking: if the tool measures tok/s with a tokenizer different from that of the model being served, the token count, and therefore the throughput in tok/s and the cost per token, are biased. LLMPerf (archived in December 2025) used LlamaTokenizer universally for all models, which guaranteed internal consistency in the leaderboard but made the tok/s not comparable with measurements of other models with different vocabularies.

The TPS formula in LLMPerf additionally included the benchmark’s full denominator, the time spent generating prompts, preparing requests and storing responses, which NVIDIA estimated at up to 33 % of the total duration at concurrency 1. This makes LLMPerf’s TPS systematically lower than that of GenAI-Perf/AIPerf for the same system, without the engine being worse:

$$\text{TPS}_{\text{LLMPerf}} = \frac{\text{output tokens}}{T_{\text{end}} - T_{\text{start}}}$$ $$\text{TPS}_{\text{GenAI-Perf}} = \frac{\text{output tokens}}{T_y - T_x}$$

Where \(T_{\text{start}}\) and \(T_{\text{end}}\) include the client overheads, while \(T_x\) and \(T_y\) are the instant of the first request and that of the last token received, respectively (NVIDIA · Fundamental Concepts).

The operational rule: always count tokens with the tokenizer of the model being served, not with a proxy tokenizer.


The bias of ignore_eos and an inconsistent OSL

Most LLM models generate a special end-of-sequence token (EOS) when they consider the response complete. If the benchmark does not set ignore_eos=True, the real output length (OSL) varies from request to request according to the model’s natural length distribution, which produces:

  1. Inconsistent OSL: two runs with the same seed can produce different OSL distributions if the model varies its natural output.
  2. Spurious comparison: a “faster” model may be so simply because it generates shorter responses (it hits EOS sooner), not because it has more real throughput.
  3. Underestimated throughput: if the benchmark expects OSL=256 but the model stops at OSL=80 on average, the throughput in tok/s appears higher but measures less work.

The ignore_eos parameter (or --ignore-eos in vLLM bench) instructs the engine to ignore the EOS token and continue until reaching max_tokens. It is mandatory for the OSL to be the configured one rather than the model’s natural one, and for two runs to be comparable (vLLM benchmark CLI docs).


The bias of the ISL/OSL distribution

The distribution of input (ISL) and output (OSL) lengths determines what proportion of the compute time goes to prefill (costly in TTFT) and how much to decode (costly in ITL). A benchmark with a fixed length, say ISL=128, OSL=128, produces results that do not resemble any real traffic.

Real use cases have very different distributions:

Use caseTypical ISL (tokens)Typical OSL (tokens)Dominated by
Translation500–2,000500–2,000balanced
Generation (code, email)~100~1,000decode (long OSL)
Summarisation / RAG~1,000~100prefill (long ISL)
Reasoning (CoT)~1001,000–10,000very long decode

A short-ISL benchmark with a model optimised for prefill will give an artificially low TTFT and an artificially high throughput. The knee of the sweep shifts: with ISL=64 the system admits more concurrency without breaking the TTFT SLO; with ISL=1,024 prefill saturates earlier and the knee appears earlier. Using the wrong distribution means sizing for traffic that does not exist.

AIPerf introduces sequence distributions with configurable per-component variance to reproduce realistic traffic mixes (AIPerf · Sequence Length Distributions):

--sequence-distribution "64|10,32|8:70;256|40,128|20:20;1024|100,512|50:10"

This creates 70 % of requests with \(\text{ISL} \sim \mathcal{N}(64, 10)\) and \(\text{OSL} \sim \mathcal{N}(32, 8)\), 20 % with medium ISL/OSL, and 10 % with long ISL/OSL, far more faithful to real chatbot traffic than a fixed length.


The bias of warmup and the prefix cache

Two related but distinct sources of bias:

Missing warmup. The first requests of a benchmark hit a “cold” engine: the GPU is in a low-clock state, the KV cache is empty, and the operating system may be paging memory. The TTFT of the first requests is structurally higher than that of the steady state. If the benchmark does not discard a warmup period, the mean TTFT includes these outliers and overestimates the real production latency. Some frameworks (GenAI-Perf/AIPerf) use a sliding window that excludes the ramp-up and ramp-down requests; others do not.

Warm prefix cache between runs. The prefix KV cache (prefix cache or prompt cache) stores the computed results of repeated prompt prefixes. If the benchmark runs multiple consecutive runs with the same prompts, the second and subsequent runs find the KV cache already populated and report an artificially low TTFT, that of the decode, not of the prefill. For a baseline benchmark, the prefix cache must be cold; for one that simulates production with repeated prompts, warm. The distinction must be made explicit (arXiv 2605.24217; vLLM · Automatic Prefix Caching).


The bias of concurrency vs request-rate

The two load control modes produce different latency distributions for the same system:

ModeSemanticsWhen to useRisk
concurrency Nkeeps exactly N requests in flight; as soon as one finishes, it launches anothermeasuring the system under a fixed concurrency loadover-represents sustained load; does not simulate real arrivals
request-rate r (constant or Poisson)launches one request every \(1/r\) seconds (constant) or with interarrival \(\sim \text{Exp}(1/r)\) (Poisson)simulating real traffic (random arrivals)if the engine cannot absorb r req/s, the queue grows without a ceiling

NVIDIA recommends the concurrency mode for most capacity benchmarks (NVIDIA · LLM Benchmarking Fundamental Concepts). The request-rate mode is more faithful for online traffic (a Poisson distribution of arrivals), but if the rate exceeds the engine’s capacity the queue grows indefinitely and the latency metrics include queueing time that can dominate the TTFT, mixing queue behaviour with engine behaviour.

A system measured at concurrency=16 and another at request-rate=16 req/s are not under the same load: fixed concurrency guarantees 16 simultaneous requests; the rate sets the arrival pace but not the instantaneous concurrency. Comparing their results without adjustment is incorrect.


The bias of the measurement window and the thermal state

Window too short. A 30-second benchmark may measure the system’s ramp-up, not the steady state. The recommendation of arXiv 2508.10251 is that the measurement window cover at least 3-5× the system’s ramp time under load, and that the metrics be computed only over the steady window, excluding warmup and cooldown.

Thermal state of the GPU. Datacenter GPUs (H100 SXM, A100) operate with thermal throttling above ~75 °C. If the GPU has not reached its steady-state temperature before the benchmark, the first measurements correspond to a higher-clock state than the sustainable one. Controlled experiments document that power must stabilise within a range of 3 W for at least 30 seconds before the measurements are representative (arXiv 2604.09048 · Watt Counts). The effect is especially severe in prefill benchmarks (TTFT), where the high clocks of the cold state produce an artificially low TTFT.

For a reproducible benchmark on 4×H100 SXM 80 GB: before measuring, run sustained load for at least 2–3 minutes until nvidia-smi reports a stable temperature and stable power (variation < 3 W over 30 s). Only then start the measurement window.


How the tools compute ITL differently

ITL (Inter-Token Latency) appears in every tool but its formula varies, and the differences are not small:

GenAI-Perf / AIPerf:

$$\text{ITL} = \frac{\text{e2e latency} - \text{TTFT}}{\text{output tokens} - 1}$$

TTFT is excluded from the numerator and the denominator discounts the first token. ITL is a pure decode metric, with no prefill contamination.

LLMPerf (archived Dec. 2025):

$$\text{ITL}_{\text{LLMPerf}} = \frac{\text{e2e latency}}{\text{output tokens}}$$

TTFT is included in the numerator. For short sequences (OSL < 50 tokens), TTFT can represent 50–80 % of the e2e_latency, so that LLMPerf’s ITL measures mainly prefill, not decode. Two systems with the same decode but different prefill will show different ITLs in LLMPerf even if they are identical in decode speed.

vLLM bench (benchmark_serving.py):

It computes ITL as the mean of the intervals between consecutive tokens of the output stream, including intra-request variance. It can reveal decode jitter that the others average away.

The practical consequence: never cross a GenAI-Perf ITL with an LLMPerf one as if they were the same metric. They are different formulas over the same signal.


Comparability across tools: why their numbers do not cross

The following table summarises the methodological differences that make one tool’s numbers structurally incompatible with another’s without an explicit adjustment:

DimensionvLLM benchLLMPerf (archived)GenAI-Perf / AIPerfGuideLLM
Client architecturesingle-process asynciosingle-processmulti-processmulti-process
ITL includes TTFTnoyesnono
TPS denominator\(T_y - T_x\)\(T_{\text{end}} - T_{\text{start}}\) (overhead included)\(T_y - T_x\)per round
Tokenizerthat of the model serveduniversal LlamaTokenizerthat of the model servedthat of the model served
Warmup / sliding windownot automaticnoyes (sliding window)per round
ignore_eos by defaultnonoexplicitly recommendedconfigurable
ISL/OSL distributionmanual parametermanual parameterdistributions with varianceconfigurable --data
Primary load modeconcurrencyconcurrency (drained batches)concurrency (recommended)sweep + poisson
Structured outputbasic JSONJSONJSON + CSVJSON + HTML + CSV

An LLMPerf result and a GenAI-Perf result for the same endpoint can differ in TTFT, ITL and TPS simultaneously and in all of them for methodological reasons, not because of engine differences. The only way to cross them is to run both tools on the same system under the same conditions and compute the empirical conversion factor, which in practice amounts to re-measuring with a single tool.


The honest harness: a reproducibility checklist

A benchmark whose number can be defended in an audit has to come accompanied by all the metadata that allow it to be reproduced. The minimum to demand, organised by category:

Hardware

MetadataExample 4×H100Effect if not fixed
GPU (model, variant, count)4× NVIDIA H100 SXM 80 GBH100 PCIe vs SXM differ in memory BW and NVLink
GPU-GPU interconnectNVLink 4.0tensor parallelism depends on the interconnect BW
CPU, RAM, CPU-GPU bandwidth2× Intel Xeon 8480+, 512 GB DDR5, PCIe 5.0the tokenisation bottleneck can be on the CPU
Thermal state at the startGPU temperature < 65 °C, stable power ± 3 Wthrottling alters TPS and TTFT by up to ~15 %
GPU clockswithout nvidia-smi -pm 1 they can varya difference of up to ~10 % in throughput

Software and model

MetadataExampleEffect if not fixed
Engine + version + flagsvLLM 0.8.4, --tensor-parallel-size 4 --gpu-memory-utilization 0.90each version changes the scheduler and the KV cache management
Model + precisionLlama-3-70B-Instruct, FP8FP16 vs FP8 differ by ~1.4–1.8× in throughput
Tokenizer used for countingtokenizer of the model served (HF tokenizer)a universal LlamaTokenizer biases tok/s across vocabularies
Generation seed--seed 42without a fixed seed, the OSL varies run-to-run
ignore_eosTruewithout it, the OSL varies with the prompt content
Sampling parameterstemperature=1.0, top_p=0.95greedy vs sampling affect logit speed

Load

MetadataExampleEffect if not fixed
Bench tool + versionAIPerf 0.2.1each version changes formulas and warmup
ISL/OSL distribution\(\mathcal{N}(512, 64)\) ISL, \(\mathcal{N}(128, 20)\) OSLchanging the distribution moves the knee
Load modeconcurrency, sweep 1–64concurrency vs request-rate: different distributions
Concurrency levels1, 2, 4, 8, 16, 32, 64 (full sweep, beyond the knee)without passing the knee, the safe capacity is unknown
Duration per point120 s minimum per levelshort windows capture ramp-up, not the steady state
Warmup30 s excluded from the metric computationwithout warmup, the metrics include the cold state
Prefix cache treatmentcold (flush between runs) or warm (declared explicitly)warm cache: unrealistic TTFT for a baseline

SLO and reported metrics

MetadataExampleEffect if not declared
Declared SLOTTFT P99 < 500 ms, ITL P95 < 50 msgoodput depends on the SLO; without an SLO there is no goodput
Reported percentilesP50, P95, P99 (not just the mean)the mean hides the tail; irrelevant for an SLO
Throughput vs goodputgoodput under the SLO, not raw throughputcatalogue throughput can be 5–10× the goodput
Total number of requests≥ 1,000 per concurrency levelsmall samples: high variance in percentiles

Methodological defence of the datum before an audit

A throughput number presented without the data sheet above is not a datum: it is an anecdote. The validation pattern for a technical audit:

1. Traceability of the tool. The tool and its version must be pinnable to a Git commit or a container image hash. AIPerf and GuideLLM export JSON with version metadata; vLLM bench omits them by default and they have to be captured manually.

2. Reproducibility of the environment. The engine deployment script (or the Kubernetes manifest) and the exact benchmark command must be enough for a third party to reproduce the number on the same hardware. GuideLLM exports the benchmark file as an authoritative record of the session: configuration, metadata, per-benchmark statistics and request entries with individual timings (GitHub vllm-project/guidellm).

3. Verification of the steady state. The metrics must come from the benchmark’s steady window (warmup excluded). The throughput vs concurrency curve must show the knee, the point where throughput saturates and latency blows up, and extend beyond it. Without a visible knee, the reported capacity may be the client’s limit, not the engine’s.

4. Goodput, not raw throughput. The auditable throughput is the goodput under the declared SLO, not the peak throughput. A system that reports 4,000 tok/s but whose goodput under TTFT P99 < 500 ms is 1,800 tok/s has a real capacity of 1,800 tok/s for interactive use cases.

5. Internal comparability. If two engines or two configurations are compared, the tool, the load distribution, the SLO, the hardware and the warmup treatment must be identical. Any difference in these dimensions contaminates the comparison.

The goodput formula as an auditable metric:

$$\text{goodput} = \text{throughput} \times \Pr[\text{TTFT} \leq \text{SLO}_{\text{TTFT}}] \times \Pr[\text{ITL} \leq \text{SLO}_{\text{ITL}}]$$

Where the probabilities are estimated over the empirical distribution of the run. An engine with a throughput of 4,000 tok/s and \(\Pr[\text{TTFT} \leq 500\,\text{ms}] = 0.45\) has a goodput of \(4{,}000 \times 0.45 = 1{,}800\) tok/s.


A numerical example: the same node, four different measurements

Reference hardware: 4×H100 SXM 80 GB, model Llama-3-70B-Instruct FP8, tensor parallel 4, SLO TTFT P99 < 500 ms.

Benchmark configurationReported throughputReal goodputFactor vs goodput
vLLM bench, concurrency=32, no ignore_eos, no warmup, fixed OSL=645,800 tok/s~1,200 tok/s (short OSL inflates TPS)4.8×
LLMPerf, concurrency=32, LlamaTokenizer, overhead included in the denominator3,100 tok/s~1,600 tok/s (ITL includes TTFT)1.9×
AIPerf, concurrency=32, ignore_eos, sliding window, model’s tokenizer3,400 tok/s3,350 tok/s1.01×
GuideLLM, sweep 1–64, poisson, realistic ISL/OSL distribution, declared SLO3,330 tok/s goodput (knee at round 6)3,330 tok/s1.0× (honest baseline)

The range: from 1,200 tok/s to 5,800 tok/s for the same engine, the same hardware and the same model. The maximum factor is 4.8×. The cause is not the engine; it is the instrumentation decisions.


Decomposing the total bias

The total observable bias between two tools is the multiplicative composition of the individual biases. For a comparison of single-process vLLM bench vs multi-process GuideLLM on the same system:

$$\text{total factor} \approx \underbrace{f_{\text{client}}}_{\leq 7.2\times} \times \underbrace{f_{\text{tokenizer}}}_{1.0{-}1.3\times} \times \underbrace{f_{\text{ignore-eos}}}_{1.0{-}2.0\times} \times \underbrace{f_{\text{ISL/OSL}}}_{1.0{-}1.5\times} \times \underbrace{f_{\text{warmup}}}_{1.0{-}1.2\times}$$

The client saturation factor (\(\leq 7.2\times\)) dominates, but the other factors multiply. Under adverse conditions (single-process client + different tokenizer + no ignore_eos + unrealistic OSL + no warmup), the compound bias can exceed 15–20× for the same system.


State of the art 2026: what has changed

  • genai-perf → AIPerf migration (15 Apr 2026): NVIDIA retired genai-perf and replaced it with AIPerf, which is multi-process with automatic saturation detection, configurable ISL/OSL distributions and a built-in sliding window. The migration removes the single-process client bias from NVIDIA’s official tooling.
  • LLMPerf archived (Dec. 2025): the ray-project/llmperf repository is archived and in read-only mode. The historical results of the LLMPerf leaderboard are comparable only among themselves; they must not be crossed with modern measurements without adjustment.
  • GuideLLM 0.5.x (2025–2026): a complete architectural refactor, multimodal support, and authoritative JSON export with all the session metadata. It is the OSS standard for SLO-driven evaluation.
  • arXiv 2605.24217 (May 2026): the first formal characterisation of systematic measurement bias in production LLM benchmarks, with a mathematical demonstration of the GIL effect and a proposal for a multi-process harness.
  • arXiv 2508.10251: meta-metrics and good practice for system-level performance benchmarking; it establishes the minimum dimensions of the parameter space that must be declared for a result to be reproducible.
  • MLPerf Inference v5.1 (Sep. 2025): 27 participants, a participation record. The MLPerf rules require declaring the exact code, the dataset and the hardware, and admit public review; it is the only industry benchmark with a formal review process for reproducibility.

The biases described here apply to every tool in the catalogue. For the complete data sheet of each tool:


Sources