Measurement bias and reproducibility: why two benchmarks of the same model give figures that differ by up to 7×
Contents
Notation: amounts in euros (N €), decimals with a point. The mathematical symbol is reserved for formulas: amounts are expressed in € or USD. Thousands with a thin space (\(1\,000\)).
TL;DR
Running the same Llama-3-70B-Instruct model on the same 4×H100 SXM 80 GB node with two different tools can produce throughput results that differ by 7.2× without the engine changing a single line of code. The cause is not the engine: it is the measurement method. At 1,000 QPS, a single-process asyncio client (vLLM bench, SGLang bench, genai-perf pre-AIPerf) processed 75,574 tokens against the 545,733 tokens of a multi-process client, both measuring the same endpoint (arXiv 2605.24217). The bias is not random: it is systematic, directional and reproducible. This article catalogues the sources of bias, quantifies their magnitude and describes the minimum harness that turns a number into an auditable datum.
The sources of bias: a catalogue with magnitudes
Each source of bias moves the published number in a given direction and with a characteristic magnitude. The table below orders the sources from largest to smallest impact observed in the literature:
| Source of bias | Direction of the bias | Documented magnitude | Reference |
|---|---|---|---|
| Single-process client saturation (Python GIL, asyncio) | understates throughput; overstates latency | up to 7.2× fewer tokens processed at 1,000 QPS | arXiv 2605.24217 |
| Tokenizer different from the model’s (LLMPerf uses a universal LlamaTokenizer) | tok/s not comparable across different vocabularies | variable; up to 33 % of client overhead in the TPS denominator | NVIDIA blog (TPS = client overhead ÷ total duration) |
| ignore_eos missing or inconsistent OSL | underestimates throughput; artificially shortens the real OSL | up to premature termination; an unrealistic fixed-length benchmark | vLLM bench docs; NVIDIA fundamental concepts |
| Unrealistic ISL/OSL distribution (fixed length instead of a realistic distribution) | an optimistic knee that does not resemble production | the knee shifts; prefill understated with short prompts | AIPerf sequence-length-distributions docs |
| Missing warmup / warm prefix cache | overstates TTFT (artificially low on the first requests) | TTFT underestimated by an already populated KV cache | vLLM prefix caching docs; arXiv 2605.24217 |
concurrency vs request-rate as the load mode | concurrency saturates symmetrically; request-rate can build a queue | ITL overestimated if the queue grows without a ceiling with request-rate | NVIDIA fundamentals; AIPerf docs |
| Measurement window too short | does not capture the steady state; includes ramp-up | throughput overestimated; latency underestimated | meta-metrics arXiv 2508.10251 |
| Thermal state and GPU clocks | thermal throttling reduces throughput without prior stabilisation | temperature > 75 °C triggers throttling; a difference of ~3 W of stable power | arXiv 2604.09048 |
| Warm prefix cache between runs | overstates TTFT; unrealistic TTFT if the cache is pre-populated from the previous run | a severe effect in multi-prompt evaluations with shared prefixes | arXiv 2605.24217 |
| Mean instead of percentiles | hides the latency tail | P99 can be 4–6× the median at high concurrency | NVIDIA fundamentals; Anyscale docs |
The dominant bias: single-process client saturation
The largest bias is not in the engine but in the client that generates the load. The single-process tools (vLLM bench, SGLang bench, genai-perf before its replacement by AIPerf on 15 April 2026) use a single Python process with asyncio to manage the concurrent requests. The Python Global Interpreter Lock (GIL) prevents a single process from using more than one CPU core at a time, which introduces a bottleneck on the client side that becomes critical at high concurrency.
The mathematical effect: as the request rate rises, the client fails to dispatch requests at the configured pace, accumulates queueing time on the client side, and that time gets recorded as engine latency. The result is that TTFT and TPOT appear inflated and throughput appears depressed, without the engine having changed anything (arXiv 2605.24217).
The documented discrepancy at 1,000 QPS:
| Client architecture | Tokens processed at 1,000 QPS | Ratio |
|---|---|---|
| Single-process (asyncio, Python GIL) | 75,574 | 1× (baseline) |
| Multi-process (distributed load) | 545,733 | 7.2× |
Note: both clients point at the same endpoint; the difference is entirely attributable to the benchmark client, not to the engine.
Tokenizer bias: tok/s are not comparable across vocabularies
Tokens are not universal. Each model has its own tokenizer with its own vocabulary. Llama-3 uses a vocabulary of 128,256 tokens; Gemma uses 256,128; earlier models used 32,000–50,000. The same English text produces a different number of tokens depending on the model’s tokenizer.
The direct consequence for benchmarking: if the tool measures tok/s with a tokenizer different from that of the model being served, the token count, and therefore the throughput in tok/s and the cost per token, are biased. LLMPerf (archived in December 2025) used LlamaTokenizer universally for all models, which guaranteed internal consistency in the leaderboard but made the tok/s not comparable with measurements of other models with different vocabularies.
The TPS formula in LLMPerf additionally included the benchmark’s full denominator, the time spent generating prompts, preparing requests and storing responses, which NVIDIA estimated at up to 33 % of the total duration at concurrency 1. This makes LLMPerf’s TPS systematically lower than that of GenAI-Perf/AIPerf for the same system, without the engine being worse:
$$\text{TPS}_{\text{LLMPerf}} = \frac{\text{output tokens}}{T_{\text{end}} - T_{\text{start}}}$$ $$\text{TPS}_{\text{GenAI-Perf}} = \frac{\text{output tokens}}{T_y - T_x}$$Where \(T_{\text{start}}\) and \(T_{\text{end}}\) include the client overheads, while \(T_x\) and \(T_y\) are the instant of the first request and that of the last token received, respectively (NVIDIA · Fundamental Concepts).
The operational rule: always count tokens with the tokenizer of the model being served, not with a proxy tokenizer.
The bias of ignore_eos and an inconsistent OSL
Most LLM models generate a special end-of-sequence token (EOS) when
they consider the response complete. If the benchmark does not set ignore_eos=True, the real
output length (OSL) varies from request to request according to the model’s natural length
distribution, which produces:
- Inconsistent OSL: two runs with the same seed can produce different OSL distributions if the model varies its natural output.
- Spurious comparison: a “faster” model may be so simply because it generates shorter responses (it hits EOS sooner), not because it has more real throughput.
- Underestimated throughput: if the benchmark expects OSL=256 but the model stops at OSL=80 on average, the throughput in tok/s appears higher but measures less work.
The ignore_eos parameter (or --ignore-eos in vLLM bench) instructs the engine to ignore the
EOS token and continue until reaching max_tokens. It is mandatory for the OSL to be the
configured one rather than the model’s natural one, and for two runs to be comparable
(vLLM benchmark CLI docs).
The bias of the ISL/OSL distribution
The distribution of input (ISL) and output (OSL) lengths determines what proportion of the compute time goes to prefill (costly in TTFT) and how much to decode (costly in ITL). A benchmark with a fixed length, say ISL=128, OSL=128, produces results that do not resemble any real traffic.
Real use cases have very different distributions:
| Use case | Typical ISL (tokens) | Typical OSL (tokens) | Dominated by |
|---|---|---|---|
| Translation | 500–2,000 | 500–2,000 | balanced |
| Generation (code, email) | ~100 | ~1,000 | decode (long OSL) |
| Summarisation / RAG | ~1,000 | ~100 | prefill (long ISL) |
| Reasoning (CoT) | ~100 | 1,000–10,000 | very long decode |
A short-ISL benchmark with a model optimised for prefill will give an artificially low TTFT and an artificially high throughput. The knee of the sweep shifts: with ISL=64 the system admits more concurrency without breaking the TTFT SLO; with ISL=1,024 prefill saturates earlier and the knee appears earlier. Using the wrong distribution means sizing for traffic that does not exist.
AIPerf introduces sequence distributions with configurable per-component variance to reproduce realistic traffic mixes (AIPerf · Sequence Length Distributions):
--sequence-distribution "64|10,32|8:70;256|40,128|20:20;1024|100,512|50:10"
This creates 70 % of requests with \(\text{ISL} \sim \mathcal{N}(64, 10)\) and \(\text{OSL} \sim \mathcal{N}(32, 8)\), 20 % with medium ISL/OSL, and 10 % with long ISL/OSL, far more faithful to real chatbot traffic than a fixed length.
The bias of warmup and the prefix cache
Two related but distinct sources of bias:
Missing warmup. The first requests of a benchmark hit a “cold” engine: the GPU is in a low-clock state, the KV cache is empty, and the operating system may be paging memory. The TTFT of the first requests is structurally higher than that of the steady state. If the benchmark does not discard a warmup period, the mean TTFT includes these outliers and overestimates the real production latency. Some frameworks (GenAI-Perf/AIPerf) use a sliding window that excludes the ramp-up and ramp-down requests; others do not.
Warm prefix cache between runs. The prefix KV cache (prefix cache or prompt cache) stores the computed results of repeated prompt prefixes. If the benchmark runs multiple consecutive runs with the same prompts, the second and subsequent runs find the KV cache already populated and report an artificially low TTFT, that of the decode, not of the prefill. For a baseline benchmark, the prefix cache must be cold; for one that simulates production with repeated prompts, warm. The distinction must be made explicit (arXiv 2605.24217; vLLM · Automatic Prefix Caching).
The bias of concurrency vs request-rate
The two load control modes produce different latency distributions for the same system:
| Mode | Semantics | When to use | Risk |
|---|---|---|---|
| concurrency N | keeps exactly N requests in flight; as soon as one finishes, it launches another | measuring the system under a fixed concurrency load | over-represents sustained load; does not simulate real arrivals |
| request-rate r (constant or Poisson) | launches one request every \(1/r\) seconds (constant) or with interarrival \(\sim \text{Exp}(1/r)\) (Poisson) | simulating real traffic (random arrivals) | if the engine cannot absorb r req/s, the queue grows without a ceiling |
NVIDIA recommends the concurrency mode for most capacity benchmarks (NVIDIA · LLM Benchmarking Fundamental Concepts). The request-rate mode is more faithful for online traffic (a Poisson distribution of arrivals), but if the rate exceeds the engine’s capacity the queue grows indefinitely and the latency metrics include queueing time that can dominate the TTFT, mixing queue behaviour with engine behaviour.
A system measured at concurrency=16 and another at request-rate=16 req/s are not under the same load: fixed concurrency guarantees 16 simultaneous requests; the rate sets the arrival pace but not the instantaneous concurrency. Comparing their results without adjustment is incorrect.
The bias of the measurement window and the thermal state
Window too short. A 30-second benchmark may measure the system’s ramp-up, not the steady state. The recommendation of arXiv 2508.10251 is that the measurement window cover at least 3-5× the system’s ramp time under load, and that the metrics be computed only over the steady window, excluding warmup and cooldown.
Thermal state of the GPU. Datacenter GPUs (H100 SXM, A100) operate with thermal throttling above ~75 °C. If the GPU has not reached its steady-state temperature before the benchmark, the first measurements correspond to a higher-clock state than the sustainable one. Controlled experiments document that power must stabilise within a range of 3 W for at least 30 seconds before the measurements are representative (arXiv 2604.09048 · Watt Counts). The effect is especially severe in prefill benchmarks (TTFT), where the high clocks of the cold state produce an artificially low TTFT.
For a reproducible benchmark on 4×H100 SXM 80 GB: before measuring, run sustained load
for at least 2–3 minutes until nvidia-smi reports a stable temperature and stable
power (variation < 3 W over 30 s). Only then start the measurement window.
How the tools compute ITL differently
ITL (Inter-Token Latency) appears in every tool but its formula varies, and the differences are not small:
GenAI-Perf / AIPerf:
$$\text{ITL} = \frac{\text{e2e latency} - \text{TTFT}}{\text{output tokens} - 1}$$TTFT is excluded from the numerator and the denominator discounts the first token. ITL is a pure decode metric, with no prefill contamination.
LLMPerf (archived Dec. 2025):
$$\text{ITL}_{\text{LLMPerf}} = \frac{\text{e2e latency}}{\text{output tokens}}$$TTFT is included in the numerator. For short sequences (OSL < 50 tokens), TTFT can represent 50–80 % of the e2e_latency, so that LLMPerf’s ITL measures mainly prefill, not decode. Two systems with the same decode but different prefill will show different ITLs in LLMPerf even if they are identical in decode speed.
vLLM bench (benchmark_serving.py):
It computes ITL as the mean of the intervals between consecutive tokens of the output stream, including intra-request variance. It can reveal decode jitter that the others average away.
The practical consequence: never cross a GenAI-Perf ITL with an LLMPerf one as if they were the same metric. They are different formulas over the same signal.
Comparability across tools: why their numbers do not cross
The following table summarises the methodological differences that make one tool’s numbers structurally incompatible with another’s without an explicit adjustment:
| Dimension | vLLM bench | LLMPerf (archived) | GenAI-Perf / AIPerf | GuideLLM |
|---|---|---|---|---|
| Client architecture | single-process asyncio | single-process | multi-process | multi-process |
| ITL includes TTFT | no | yes | no | no |
| TPS denominator | \(T_y - T_x\) | \(T_{\text{end}} - T_{\text{start}}\) (overhead included) | \(T_y - T_x\) | per round |
| Tokenizer | that of the model served | universal LlamaTokenizer | that of the model served | that of the model served |
| Warmup / sliding window | not automatic | no | yes (sliding window) | per round |
| ignore_eos by default | no | no | explicitly recommended | configurable |
| ISL/OSL distribution | manual parameter | manual parameter | distributions with variance | configurable --data |
| Primary load mode | concurrency | concurrency (drained batches) | concurrency (recommended) | sweep + poisson |
| Structured output | basic JSON | JSON | JSON + CSV | JSON + HTML + CSV |
An LLMPerf result and a GenAI-Perf result for the same endpoint can differ in TTFT, ITL and TPS simultaneously and in all of them for methodological reasons, not because of engine differences. The only way to cross them is to run both tools on the same system under the same conditions and compute the empirical conversion factor, which in practice amounts to re-measuring with a single tool.
The honest harness: a reproducibility checklist
A benchmark whose number can be defended in an audit has to come accompanied by all the metadata that allow it to be reproduced. The minimum to demand, organised by category:
Hardware
| Metadata | Example 4×H100 | Effect if not fixed |
|---|---|---|
| GPU (model, variant, count) | 4× NVIDIA H100 SXM 80 GB | H100 PCIe vs SXM differ in memory BW and NVLink |
| GPU-GPU interconnect | NVLink 4.0 | tensor parallelism depends on the interconnect BW |
| CPU, RAM, CPU-GPU bandwidth | 2× Intel Xeon 8480+, 512 GB DDR5, PCIe 5.0 | the tokenisation bottleneck can be on the CPU |
| Thermal state at the start | GPU temperature < 65 °C, stable power ± 3 W | throttling alters TPS and TTFT by up to ~15 % |
| GPU clocks | without nvidia-smi -pm 1 they can vary | a difference of up to ~10 % in throughput |
Software and model
| Metadata | Example | Effect if not fixed |
|---|---|---|
| Engine + version + flags | vLLM 0.8.4, --tensor-parallel-size 4 --gpu-memory-utilization 0.90 | each version changes the scheduler and the KV cache management |
| Model + precision | Llama-3-70B-Instruct, FP8 | FP16 vs FP8 differ by ~1.4–1.8× in throughput |
| Tokenizer used for counting | tokenizer of the model served (HF tokenizer) | a universal LlamaTokenizer biases tok/s across vocabularies |
| Generation seed | --seed 42 | without a fixed seed, the OSL varies run-to-run |
ignore_eos | True | without it, the OSL varies with the prompt content |
| Sampling parameters | temperature=1.0, top_p=0.95 | greedy vs sampling affect logit speed |
Load
| Metadata | Example | Effect if not fixed |
|---|---|---|
| Bench tool + version | AIPerf 0.2.1 | each version changes formulas and warmup |
| ISL/OSL distribution | \(\mathcal{N}(512, 64)\) ISL, \(\mathcal{N}(128, 20)\) OSL | changing the distribution moves the knee |
| Load mode | concurrency, sweep 1–64 | concurrency vs request-rate: different distributions |
| Concurrency levels | 1, 2, 4, 8, 16, 32, 64 (full sweep, beyond the knee) | without passing the knee, the safe capacity is unknown |
| Duration per point | 120 s minimum per level | short windows capture ramp-up, not the steady state |
| Warmup | 30 s excluded from the metric computation | without warmup, the metrics include the cold state |
| Prefix cache treatment | cold (flush between runs) or warm (declared explicitly) | warm cache: unrealistic TTFT for a baseline |
SLO and reported metrics
| Metadata | Example | Effect if not declared |
|---|---|---|
| Declared SLO | TTFT P99 < 500 ms, ITL P95 < 50 ms | goodput depends on the SLO; without an SLO there is no goodput |
| Reported percentiles | P50, P95, P99 (not just the mean) | the mean hides the tail; irrelevant for an SLO |
| Throughput vs goodput | goodput under the SLO, not raw throughput | catalogue throughput can be 5–10× the goodput |
| Total number of requests | ≥ 1,000 per concurrency level | small samples: high variance in percentiles |
Methodological defence of the datum before an audit
A throughput number presented without the data sheet above is not a datum: it is an anecdote. The validation pattern for a technical audit:
1. Traceability of the tool. The tool and its version must be pinnable to a Git commit or a container image hash. AIPerf and GuideLLM export JSON with version metadata; vLLM bench omits them by default and they have to be captured manually.
2. Reproducibility of the environment. The engine deployment script (or the Kubernetes manifest) and the exact benchmark command must be enough for a third party to reproduce the number on the same hardware. GuideLLM exports the benchmark file as an authoritative record of the session: configuration, metadata, per-benchmark statistics and request entries with individual timings (GitHub vllm-project/guidellm).
3. Verification of the steady state. The metrics must come from the benchmark’s steady window (warmup excluded). The throughput vs concurrency curve must show the knee, the point where throughput saturates and latency blows up, and extend beyond it. Without a visible knee, the reported capacity may be the client’s limit, not the engine’s.
4. Goodput, not raw throughput. The auditable throughput is the goodput under the declared SLO, not the peak throughput. A system that reports 4,000 tok/s but whose goodput under TTFT P99 < 500 ms is 1,800 tok/s has a real capacity of 1,800 tok/s for interactive use cases.
5. Internal comparability. If two engines or two configurations are compared, the tool, the load distribution, the SLO, the hardware and the warmup treatment must be identical. Any difference in these dimensions contaminates the comparison.
The goodput formula as an auditable metric:
$$\text{goodput} = \text{throughput} \times \Pr[\text{TTFT} \leq \text{SLO}_{\text{TTFT}}] \times \Pr[\text{ITL} \leq \text{SLO}_{\text{ITL}}]$$Where the probabilities are estimated over the empirical distribution of the run. An engine with a throughput of 4,000 tok/s and \(\Pr[\text{TTFT} \leq 500\,\text{ms}] = 0.45\) has a goodput of \(4{,}000 \times 0.45 = 1{,}800\) tok/s.
A numerical example: the same node, four different measurements
Reference hardware: 4×H100 SXM 80 GB, model Llama-3-70B-Instruct FP8, tensor parallel 4, SLO TTFT P99 < 500 ms.
| Benchmark configuration | Reported throughput | Real goodput | Factor vs goodput |
|---|---|---|---|
| vLLM bench, concurrency=32, no ignore_eos, no warmup, fixed OSL=64 | 5,800 tok/s | ~1,200 tok/s (short OSL inflates TPS) | 4.8× |
| LLMPerf, concurrency=32, LlamaTokenizer, overhead included in the denominator | 3,100 tok/s | ~1,600 tok/s (ITL includes TTFT) | 1.9× |
| AIPerf, concurrency=32, ignore_eos, sliding window, model’s tokenizer | 3,400 tok/s | 3,350 tok/s | 1.01× |
| GuideLLM, sweep 1–64, poisson, realistic ISL/OSL distribution, declared SLO | 3,330 tok/s goodput (knee at round 6) | 3,330 tok/s | 1.0× (honest baseline) |
The range: from 1,200 tok/s to 5,800 tok/s for the same engine, the same hardware and the same model. The maximum factor is 4.8×. The cause is not the engine; it is the instrumentation decisions.
Decomposing the total bias
The total observable bias between two tools is the multiplicative composition of the individual biases. For a comparison of single-process vLLM bench vs multi-process GuideLLM on the same system:
$$\text{total factor} \approx \underbrace{f_{\text{client}}}_{\leq 7.2\times} \times \underbrace{f_{\text{tokenizer}}}_{1.0{-}1.3\times} \times \underbrace{f_{\text{ignore-eos}}}_{1.0{-}2.0\times} \times \underbrace{f_{\text{ISL/OSL}}}_{1.0{-}1.5\times} \times \underbrace{f_{\text{warmup}}}_{1.0{-}1.2\times}$$The client saturation factor (\(\leq 7.2\times\)) dominates, but the other factors multiply. Under adverse conditions (single-process client + different tokenizer + no ignore_eos + unrealistic OSL + no warmup), the compound bias can exceed 15–20× for the same system.
State of the art 2026: what has changed
- genai-perf → AIPerf migration (15 Apr 2026): NVIDIA retired genai-perf and replaced it with AIPerf, which is multi-process with automatic saturation detection, configurable ISL/OSL distributions and a built-in sliding window. The migration removes the single-process client bias from NVIDIA’s official tooling.
- LLMPerf archived (Dec. 2025): the
ray-project/llmperfrepository is archived and in read-only mode. The historical results of the LLMPerf leaderboard are comparable only among themselves; they must not be crossed with modern measurements without adjustment. - GuideLLM 0.5.x (2025–2026): a complete architectural refactor, multimodal support, and authoritative JSON export with all the session metadata. It is the OSS standard for SLO-driven evaluation.
- arXiv 2605.24217 (May 2026): the first formal characterisation of systematic measurement bias in production LLM benchmarks, with a mathematical demonstration of the GIL effect and a proposal for a multi-process harness.
- arXiv 2508.10251: meta-metrics and good practice for system-level performance benchmarking; it establishes the minimum dimensions of the parameter space that must be declared for a result to be reproducible.
- MLPerf Inference v5.1 (Sep. 2025): 27 participants, a participation record. The MLPerf rules require declaring the exact code, the dataset and the hardware, and admit public review; it is the only industry benchmark with a formal review process for reproducibility.
Cross-links from the benchmarking track
The biases described here apply to every tool in the catalogue. For the complete data sheet of each tool:
- Full catalogue: LLM benchmarking tools, data sheet by data sheet
- GuideLLM and SLO validation: GuideLLM in depth: validating the SLO under load
- AIPerf (ex genai-perf) and saturation detection: NVIDIA GenAI-Perf in depth
- State of the art of frameworks: LLM benchmarking: frameworks and state of the art
- Engine decision on the Pareto axis: Serving engines compared: the Pareto frontier
Sources
- arXiv 2605.24217 · Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks — https://arxiv.org/abs/2605.24217
- arXiv 2508.10251 · Meta-Metrics and Best Practices for System-Level Inference Performance Benchmarking — https://arxiv.org/pdf/2508.10251
- arXiv 2604.09048 · Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures — https://arxiv.org/html/2604.09048v1
- NVIDIA Technical Blog · LLM Inference Benchmarking: Fundamental Concepts (ITL, TPS, ISL/OSL, ignore_eos, concurrency vs request-rate) — https://developer.nvidia.com/blog/llm-benchmarking-fundamental-concepts/
- NVIDIA AIPerf Docs · Sequence Length Distributions for Advanced Benchmarking — https://docs.nvidia.com/aiperf/tutorials/datasets-inputs/sequence-length-distributions-for-advanced-benchmarking
- NVIDIA AIPerf · Request Rate with Max Concurrency — https://docs.nvidia.com/aiperf/tutorials/load-patterns-scheduling/request-rate-with-max-concurrency
- NVIDIA AIPerf · Comprehensive LLM Benchmarking Guide — https://lucaberton.com/blog/nvidia-aiperf-llm-inference-benchmarking-guide/
- vLLM Documentation · Benchmark CLI (ignore_eos, métricas disponibles) — https://docs.vllm.ai/en/latest/benchmarking/cli/
- vLLM Documentation · Automatic Prefix Caching — https://docs.vllm.ai/en/stable/design/prefix_caching/
- GitHub · ray-project/llmperf (archivado dic. 2025) — https://github.com/ray-project/llmperf
- GitHub · vllm-project/guidellm (exportación JSON autoritativa, metadatos de sesión) — https://github.com/vllm-project/guidellm
- Red Hat Developer · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
- MLCommons · MLPerf Inference v5.1 results (récord 27 participantes, reproducibilidad formal) — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
- arXiv 2502.16721 · Speed and Conversational LLMs: Not All Is About Tokens per Second (tokenizer incompatibilidad y tok/s entre vocabularios) — https://arxiv.org/pdf/2502.16721
- Medium · LLM Inference Benchmarking (genAI-perf y vLLM, discrepancia 7,2×) — https://kchandan.medium.com/llm-inference-benchmarking-genai-perf-and-vllm-5dd06b57428e