Reproducibilidad on lo0 — Tech Blog

The reproducible harness: measuring cost, performance and energy in a single auditable experiment

Tue, 16 Jun 2026 12:00:00 +0200

Notation: amounts in euros (N €), decimals with a point, thousands with a thin space. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

The “data” series has produced three independent measurement axes: cost per million tokens (OpenCost + LiteLLM), performance under an SLO (GuideLLM + AIPerf) and energy per token (DCGM + Kepler). The problem is that the three have been measured in different articles, with different loads and at different times: they are not comparable with each other. This closing article describes the integrated harness that runs the three axes in the same experiment, on the same node (4×H100 SXM, a generic reference), with all the metadata fixed, the output in versioned JSON/CSV and an idempotent Kubernetes Job. The result is the 3-axis scorecard (€/1M tok, Wh/token, TTFT/ITL P99) that allows configurations to be compared on a multi-objective Pareto frontier and any figure to be audited with the bench to reproduce it.

Why the three axes must be measured together

The series’ opening post (The three axes) established the identity:

$$\text{CPM} = \frac{\text{cost/h}}{\text{throughput (tok/s)} \times 3.6 \times 10^{-3}}$$ $$\text{energy/token (Wh)} = \frac{\text{mean power (W)}}{\text{throughput (tok/s)} \times 3\,600}$$

Throughput is the common denominator. If it is measured in different experiments, different time, different load, different GPU temperature, then CPM and energy/token do not share a denominator: they are three anecdotes, not a scorecard. The harness captures them in the same time window, over the same load, with Prometheus windows aligned to the second. Only then is the scorecard row coherent by construction.

The second reason is reproducibility. The post Bias and reproducibility in benchmarking listed twelve biases that invalidate comparisons: an unpinned engine, an undeclared tokeniser, unfixed input/output lengths, missing warmup, a client outside the cluster. The harness removes all of them because the metadata are part of the Job, not of the documentation.

Architecture of the integrated bench

The harness has four layers. Each one is OSS, exports to Prometheus and lives in the same Kubernetes namespace:

Layer	Tool(s)	Primary metric	Export protocol
Performance	GuideLLM (SLO sweep) + AIPerf	TTFT P99, ITL P99, goodput (tok/s)	native JSON/CSV + `/metrics` OpenMetrics
Cost	OpenCost + LiteLLM proxy	CPM (€/1M tok), cost/request	REST API + Prometheus scrape
Energy (GPU)	DCGM Exporter	`DCGM_FI_DEV_POWER_USAGE` (W), `DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION` (mJ)	Prometheus DaemonSet
Energy (pod)	Kepler	`kepler_container_joules_total`, energy per pod	Prometheus DaemonSet

MLPerf Power is used as an external comparability reference: its published results, with hardware documented in detail, allow the harness figures to be calibrated for plausibility. The in-house bench does not aim to be an MLPerf submission, but to be reproducible on your own cluster.

The Kubernetes Job: the full YAML

The experiment runs as a versioned Kubernetes Job. All the relevant metadata are environment variables declared in the manifest: not in ad-hoc scripts, not in external documentation. The Job is idempotent (same name = same run) and leaves traces in the pod log and in the output volume.

apiVersion: batch/v1
kind: Job
metadata:
 name: bench-llama3-70b-fp8-h100x4-20260616
 namespace: benchmark
 labels:
 bench/model: llama3-70b
 bench/precision: fp8
 bench/engine: vllm-0.9.1
 bench/hardware: h100x4-sxm
 bench/isl: "1024"
 bench/osl: "256"
 bench/concurrency: "32"
 bench/tokenizer: meta-llama-3-tokenizer-v3
 bench/run-id: "20260616T1200"
spec:
 backoffLimit: 0
 template:
 spec:
 restartPolicy: Never
 serviceAccountName: bench-runner
 volumes:
 - name: results
 persistentVolumeClaim:
 claimName: bench-results-pvc
 initContainers:
 # Warmup: 60 s of traffic before the experiment
 - name: warmup
 image: ghcr.io/vllm-project/guidellm:0.4.2
 command:
 - guidellm
 - benchmark
 - --target
 - http://vllm-svc.inference.svc.cluster.local:8000
 - --rate-type
 - concurrent
 - --rate
 - "4"
 - --max-seconds
 - "60"
 - --data
 - prompt_tokens=1024,output_tokens=256
 env:
 - name: GUIDELLM_ENV
 value: production
 containers:
 - name: guidellm-sweep
 image: ghcr.io/vllm-project/guidellm:0.4.2
 command:
 - guidellm
 - benchmark
 - --target
 - http://vllm-svc.inference.svc.cluster.local:8000
 - --rate-type
 - sweep
 - --max-seconds
 - "120"
 - --data
 - prompt_tokens=1024,output_tokens=256
 - --output-path
 - /results/guidellm-$(BENCH_RUN_ID).json
 env:
 - name: BENCH_RUN_ID
 valueFrom:
 fieldRef:
 fieldPath: metadata.labels['bench/run-id']
 volumeMounts:
 - name: results
 mountPath: /results

 - name: aiperf-concurrent
 image: nvcr.io/nvidia/aiperf:0.2.0
 command:
 - aiperf
 - profile
 - --url
 - http://vllm-svc.inference.svc.cluster.local:8000/v1
 - --model
 - meta-llama/Meta-Llama-3-70B-Instruct
 - --concurrency
 - "4,8,16,32"
 - --input-tokens
 - "1024"
 - --output-tokens
 - "256"
 - --num-requests
 - "200"
 - --output
 - /results/aiperf-$(BENCH_RUN_ID).json
 volumeMounts:
 - name: results
 mountPath: /results

 - name: scorecard-exporter
 image: python:3.12-slim
 command:
 - python
 - /scripts/export_scorecard.py
 - --run-id
 - "$(BENCH_RUN_ID)"
 - --prometheus
 - http://prometheus.monitoring.svc.cluster.local:9090
 - --output
 - /results/scorecard-$(BENCH_RUN_ID).json
 env:
 - name: BENCH_RUN_ID
 valueFrom:
 fieldRef:
 fieldPath: metadata.labels['bench/run-id']
 volumeMounts:
 - name: results
 mountPath: /results

The Job name (bench-llama3-70b-fp8-h100x4-20260616) is the unique identifier of the run. Changing it is enough to register a variant. The labels are the metadata that the scorecard-exporter reads to enrich the output JSON.

Performance layer: GuideLLM and AIPerf

GuideLLM — the SLO-driven sweep

GuideLLM (a vLLM project) generates realistic traffic patterns, synchronous, concurrent, poisson, throughput, sweep, and captures full TTFT and ITL distributions (Red Hat Developer). The sweep mode ramps from idle to saturation over 10 rounds and identifies the knee: the maximum load where goodput ≈ throughput under the declared SLO. The output is JSON/CSV with all the percentiles per round, ready to be versioned.

For the harness, the bench’s reference SLO is:

Metric	Threshold	Percentile
TTFT	500 ms	P99
ITL (TPOT)	50 ms/tok	P95
Error rate	0.5 %	—

The Job command already appears in the YAML above. The JSON output includes, per round: rate (req/s), TTFT (P50/P95/P99), ITL (P50/P95/P99), throughput (tok/s) and goodput (tok/s). The goodput under the SLO at the knee is the value that enters the scorecard.

For the cross-verification layer of the data and comparability with published results, see the post GuideLLM in depth.

AIPerf — NVIDIA’s concurrency profiler

AIPerf (the successor to GenAI-Perf, repository ai-dynamo/aiperf) measures TTFT, ITL, throughput and latency distributions at fixed concurrencies (GitHub ai-dynamo/aiperf). Where GuideLLM gives the automatic sweep up to the knee, AIPerf gives the detailed profile at specific concurrencies (4, 8, 16, 32 in the example): it allows the throughput-latency curve to be characterised point by point.

The two are complementary: GuideLLM finds the knee automatically; AIPerf confirms it and characterises the behaviour in the neighbourhood of that knee. Both outputs go to the results volume with the same BENCH_RUN_ID. The deep analysis of AIPerf/GenAI-Perf is in GenAI-Perf in depth.

Metrics the harness extracts from AIPerf for the scorecard:

# TTFT P99 at concurrency 16 (the closest to the sweep's knee):
aiperf profile ... --concurrency 16 \
 | jq '.results[] | select(.concurrency==16) | .ttft_ms.p99'

# Goodput (tok/s) at concurrency 16:
aiperf profile ... --concurrency 16 \
 | jq '.results[] | select(.concurrency==16) | .output_token_throughput'

Cost layer: OpenCost and LiteLLM

OpenCost — the denominator in euros per GPU-hour

OpenCost (CNCF incubating, opencost.io) allocates Kubernetes cost to namespace, label, pod and container in real time (GitHub opencost/opencost). For the harness, OpenCost answers the question: how much does the vllm-svc pod cost in euros per hour during the experiment’s window?

The harness calls the OpenCost REST API when the experiment finishes:

# Cost of the inference namespace over the experiment window (1 hour)
curl -s "http://opencost.monitoring.svc.cluster.local:9003/allocation" \
 --data-urlencode 'window=2026-06-16T12:00:00Z,2026-06-16T13:00:00Z' \
 --data-urlencode 'aggregate=namespace' \
 --data-urlencode 'namespace=inference' \
 | jq '.data[0].inference.totalCost'

The returned cost (in EUR, configured with the node’s real prices) is divided by the throughput measured in that same window to obtain the CPM:

$$\text{CPM} = \frac{\text{cost}_\text{namespace/h} \times 10^6}{\text{goodput (tok/s)} \times 3\,600}$$

The post OpenCost: cost allocation in Kubernetes details the price configuration and the allocation by GPU label.

LiteLLM — the per-request token counter

LiteLLM (litellm.ai, GitHub BerriAI/litellm) acts as an OpenAI-compatible proxy with token accounting per request, model and team. In the harness, GuideLLM and AIPerf point at the LiteLLM endpoint (which in turn forwards to vLLM): every request is recorded with prompt_tokens, completion_tokens and cost (using the custom pricing configured for the on-prem node).

# litellm-config.yaml (fragment of custom on-prem pricing)
model_list:
 - model_name: llama3-70b-fp8
 litellm_params:
 model: openai/meta-llama/Meta-Llama-3-70B-Instruct
 api_base: http://vllm-svc.inference.svc.cluster.local:8000/v1
 api_key: sk-dummy
 input_cost_per_token: 0.00000109 # 1.09 EUR/1M tok on-prem
 output_cost_per_token: 0.00000109

The LiteLLM records are exported to Prometheus as litellm_request_total_tokens and litellm_spend_metric_total, which the scorecard exporter consumes to compute the real CPM per token type (input vs output).

Energy layer: DCGM and Kepler

DCGM Exporter — GPU power and energy

DCGM Exporter (GitHub NVIDIA/dcgm-exporter, docs) exposes GPU metrics on /metrics for Prometheus as a DaemonSet on the GPU nodes. The two energy metrics the harness uses:

DCGM metric	Type	Unit	Use in the harness
`DCGM_FI_DEV_POWER_USAGE`	gauge	W	instantaneous power per GPU
`DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION`	counter	mJ	energy accumulated since boot

To compute the energy consumed during the experiment’s window (without the idle baseline), the harness takes the difference of the counter before and after the sweep:

# Deployment as a DaemonSet (fragment of the official chart)
helm repo add gpu-helm-charts \
 https://nvidia.github.io/dcgm-exporter/helm-charts
helm install dcgm-exporter gpu-helm-charts/dcgm-exporter \
 --namespace monitoring \
 --set serviceMonitor.enabled=true \
 --set serviceMonitor.interval=15s

The DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION field (in mJ) allows the experiment’s energy to be computed with hardware-counter accuracy, not with an estimate:

$$\text{experiment energy (J)} = \left(\text{counter}_\text{end} - \text{counter}_\text{start}\right) \times 10^{-3}$$ $$\text{Wh/token} = \frac{\text{experiment energy (J)}}{3\,600 \times \text{generated tokens}}$$

Kepler — pod-level energy

Kepler (CNCF sandbox, GitHub sustainable-computing-io/kepler) uses eBPF to estimate energy consumption at container and pod level, exporting kepler_container_joules_total to Prometheus (Red Hat Emerging Technologies). It combines RAPL (CPU/DRAM), NVML (GPU) and regression models when no sensors are available.

In the harness, Kepler complements DCGM: DCGM gives the hardware measurement of the GPU (more precise for GPU-intensive loads such as LLM inference), while Kepler attributes the energy to the specific vLLM pod (including the node’s CPU contribution). The main metric the harness consumes:

# Energy of the vllm-svc pod during the experiment (J)
increase(
 kepler_container_joules_total{
 container_namespace="inference",
 container_name="vllm"
 }[${BENCH_DURATION}]
)

The comparative analysis of DCGM vs Kepler vs Zeus is in Energy tools: deployment, precision and overhead.

The MLPerf Power reference

MLPerf Power (MLCommons, IEEE HPCA 2025 paper) establishes the protocol for measuring the energy efficiency of ML systems with high-precision external measurement. The in-house bench is not an MLPerf submission (that requires external power meters and committee review), but its published results are the calibration reference: if the harness gives a result of the same order as the MLPerf submission for the same hardware with the same model, the measurement is plausible. If it differs by more than a factor of 2, there is a methodological problem. See the post MLPerf Power: energy efficiency for the current reference data.

Unifying the metrics: the PromQL queries

The scorecard-exporter is the Job container that, once GuideLLM and AIPerf have finished, collects all the metrics from Prometheus and builds the JSON of the scorecard row. The key queries:

# Mean GPU power during the sweep (W) — 4×H100 node
avg_over_time(
 sum(DCGM_FI_DEV_POWER_USAGE{Hostname=~"gpu-node-.*"})[${BENCH_DURATION}:15s]
)

# Total GPU energy during the sweep (mJ → convert to J)
(
 sum(DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION{Hostname=~"gpu-node-.*"}) offset 0
 -
 sum(DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION{Hostname=~"gpu-node-.*"}) offset ${BENCH_DURATION}
) * 0.001

# Tokens generated during the sweep (from LiteLLM)
increase(
 litellm_request_total_tokens{model="llama3-70b-fp8", token_type="completion"}[${BENCH_DURATION}]
)

# Cost of the inference namespace over the window (from the OpenCost API)
# → REST call at the start and at the end of the Job, difference of accrual

The ${BENCH_DURATION} variable is the real duration of the sweep (in Prometheus format, e.g. 22m), which the exporter computes as end_ts - start_ts and substitutes into every query.

The output JSON of each run has this structure:

{
 "run_id": "20260616T1200",
 "metadata": {
 "model": "meta-llama/Meta-Llama-3-70B-Instruct",
 "engine": "vllm-0.9.1",
 "precision": "fp8",
 "hardware": "4xH100-SXM-80GB",
 "isl_tokens": 1024,
 "osl_tokens": 256,
 "tokenizer": "meta-llama-3-tokenizer-v3",
 "slo_ttft_p99_ms": 500,
 "slo_itl_p95_ms": 50,
 "bench_tool_guidellm": "0.4.2",
 "bench_tool_aiperf": "0.2.0",
 "dcgm_exporter": "3.3.9",
 "kepler": "0.10.2"
 },
 "performance": {
 "goodput_tok_s": 3120,
 "ttft_p99_ms": 487,
 "itl_p95_ms": 42,
 "throughput_peak_tok_s": 3890,
 "elbow_concurrency": 28
 },
 "cost": {
 "cpm_eur_1m": 0.97,
 "gpu_cost_eur_h": 10.8,
 "cost_window_eur": 3.24
 },
 "energy": {
 "wh_per_token": 0.00044,
 "j_per_token": 1.58,
 "power_mean_w": 4924,
 "energy_total_kwh": 0.287,
 "pue": 1.4,
 "wh_per_token_pue_adjusted": 0.000616
 },
 "carbon": {
 "grid_intensity_gco2_kwh": 40,
 "co2_per_1m_tokens_g": 24.6
 }
}

All the fields are computed from the same time windows. The JSON is versioned in git alongside the harness code. Reproducing any scorecard row is: git checkout <run-id> && kubectl apply -f job.yaml.

The 3-axis scorecard: table and interpretation

The following table illustrates how five configurations of the same model (Llama 3 70B) on the same reference node (4×H100 SXM) compare with the harness. The figures are illustrative of the order of magnitude (the bench fills them in with real measurements):

Config	Engine	Precision	CPM (€/1M)	Goodput (tok/s)	Wh/tok	TTFT P99 (ms)	ITL P95 (ms)	CO2 /1M (g, FR)
A	vLLM 0.9	FP16	1.64	1,890	0.00074	498	49	29.6
B	vLLM 0.9	FP8	0.97	3,120	0.00044	487	42	17.6
C	SGLang 0.4	FP8	0.88	3,410	0.00041	421	38	16.4
D	vLLM 0.9	FP8, ISL 512	1.31	2,240	0.00056	294	31	22.4
E	vLLM 0.9	FP8, max-batch 128	0.84	3,590	0.00040	721	55	16.0

How to read the table:

B vs A: FP8 over FP16 cuts CPM by 41 % and energy per token by 41 % thanks to the same higher throughput. Both meet the SLO (TTFT P99 < 500 ms, ITL P95 < 50 ms). B dominates A on all three axes: it is Pareto-superior.
C vs B: SGLang gives 9 % more goodput and 9 % lower CPM with FP8. Both meet the SLO. C is Pareto-superior to B (if the engine is indifferent for the stack).
D vs B: a shorter ISL (512 tok) lowers latency (TTFT P99 294 ms vs 487 ms) but lowers goodput and raises CPM. If the use case demands TTFT < 300 ms, D is the candidate; if TTFT < 500 ms is enough, B is better on cost and energy.
E vs C: max-batch 128 raises goodput (+5 %) and lowers CPM, but the TTFT P99 breaks the SLO (721 ms > 500 ms) and so does the ITL P95 (55 ms > 50 ms). E has the best raw throughput but is outside the SLO: it is not a valid candidate for the interactive chat use case.

The scorecard’s Pareto frontier, under the declared SLO, includes only A, B, C and D (E falls out for breaking the SLO). Of those four, C dominates B which dominates A. D only enters the frontier if the use case demands TTFT P99 < 300 ms. The decision is not a number; it is the whole row plus the SLO.

The multi-objective Pareto frontier

With three minimisation metrics, CPM (€/1M tok), Wh/tok and TTFT P99 (ms), the Pareto frontier is defined as the set of configurations where none dominates another on all three axes simultaneously. Formally, config $ i $ dominates config $ j $ if:

$$\text{CPM}_i \leq \text{CPM}_j \;\land\; \text{Wh/tok}_i \leq \text{Wh/tok}_j \;\land\; \text{TTFT}_{i} \leq \text{TTFT}_{j}$$

with at least one strict inequality. The scorecard allows it to be computed over the set of runs with a simple vector operation. In the table’s example, the frontier under the SLO (with E excluded) is $\{C, D\}$: C dominates on cost/energy/goodput; D dominates on latency. Between C and D the choice depends on the TTFT SLO of the specific use case.

Reproducible design: the twelve mandatory metadata

The checklist that closes the bias dossier (Bias and reproducibility). For the harness JSON to be auditable, the twelve fields must be present in every run:

#	Field	Example	Why it is mandatory
1	Serving engine version	`vllm-0.9.1`	the same model can perform differently across versions
2	Model version (commit/tag)	`meta-llama/Meta-Llama-3-70B-Instruct@sha256:abc`	avoids ambiguity between variants of the same name
3	Precision	`fp8`	FP16 vs FP8 changes throughput and latency
4	Tokeniser and version	`meta-llama-3-tokenizer-v3`	the ISL/OSL in tokens depends on the tokeniser
5	ISL (input sequence length)	`1024` tok	changes the prefill time and the knee
6	OSL (output sequence length)	`256` tok	changes the decode time
7	Hardware (model and count)	`4×H100-SXM-80GB`	the basis of any comparison
8	Maximum concurrency tested	`32`	defines the sweep’s range
9	Bench tool version	`guidellm-0.4.2`, `aiperf-0.2.0`	different versions can give different metrics
10	Declared SLO	`TTFT P99 < 500 ms, ITL P95 < 50 ms`	the knee depends on the SLO; without it there is no goodput
11	Warmup duration	`60 s`	without warmup the KV cache is cold and the first rounds are biased
12	Harness version / run-id	`20260616T1200`	identifies the run so it can be reproduced with `git checkout`

A scorecard missing any of these twelve fields is an anecdote. With all twelve, it is an auditable datum. The run-id in the Job name (bench-llama3-70b-fp8-h100x4-20260616) implicitly incorporates fields 1-3 and 7, forcing the Job name to change with every variant, which makes it impossible to overlap runs in the same namespace.

Idempotence and versioning of results

The Kubernetes Job is idempotent: if the same manifest is applied again (same name), the Job already exists and Kubernetes does not relaunch it (backoffLimit: 0 policy). This avoids accidental duplicate runs. To relaunch, the Job has to be deleted (kubectl delete job <name>) or the run-id changed.

Results are versioned with the following directory convention in the harness’s git repository:

results/
20260616T1200/
guidellm-20260616T1200.json
aiperf-20260616T1200.json
scorecard-20260616T1200.json
20260617T0900/
guidellm-20260617T0900.json
aiperf-20260617T0900.json
scorecard-20260617T0900.json
scorecard-aggregate.csv # all rows, for analysis and charts

The scorecard-aggregate.csv is the scorecard table in flat format: each row is a run, each column a field of the JSON. It is generated with:

# Generates the aggregate CSV from all the JSON files in results/
python scripts/aggregate_scorecards.py results/ > scorecard-aggregate.csv

The CSV is what feeds the Pareto charts, the reports and the architecture decisions. It is versioned in git; its diff is the configuration change.

How it connects with the other posts in the series

The harness is not a standalone article: it is the synthesis of the twenty-eight articles. Each tool has its deep dive:

Performance: GuideLLM in depth — sweep, SLO, goodput, knee; GenAI-Perf / AIPerf in depth — concurrency, rate sweep, distribution metrics.
Cost: OpenCost: cost allocation in Kubernetes — allocation by namespace/label, custom on-prem prices.
Energy: Energy tools: deployment, precision and overhead — DCGM vs Kepler vs Zeus, instrumentation overhead, relative precision; MLPerf Power: energy efficiency — the external calibration reference.
Theoretical framework: The three axes: cost, performance and energy — the CPM/energy/throughput identity that justifies measuring them together.
Bias: Bias and reproducibility in benchmarking — the twelve biases the harness removes.

The harness closes the series’ dossier because it makes the opening article’s promise possible: “when someone disputes a number in the proposal, the answer is not ‘a blog says so’, but ’this is the bench, this is the methodology, reproduce it’”. With the run-id and the git repo, reproducing is one command.

Full flow of a benchmarking session

The harness’s standard operating procedure, from start to finish:

# 1. Make sure the inference endpoint is up
kubectl get svc vllm-svc -n inference

# 2. Verify that DCGM and Kepler are scraping
curl -s http://prometheus.monitoring.svc.cluster.local:9090/api/v1/query \
 --data-urlencode 'query=DCGM_FI_DEV_POWER_USAGE' | jq '.data.result | length'

# 3. Record the energy counter BEFORE the experiment
PRE_ENERGY=$(kubectl exec -n monitoring dcgm-exporter-xxx -- \
 curl -s localhost:9400/metrics \
 | grep DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION | awk '{sum+=$2} END{print sum}')

# 4. Launch the Job (unique name per run)
kubectl apply -f jobs/bench-llama3-70b-fp8-h100x4-20260616.yaml

# 5. Wait for it to finish
kubectl wait --for=condition=complete job/bench-llama3-70b-fp8-h100x4-20260616 \
 -n benchmark --timeout=3600s

# 6. Record the energy counter AFTER
POST_ENERGY=$(kubectl exec -n monitoring dcgm-exporter-xxx -- \
 curl -s localhost:9400/metrics \
 | grep DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION | awk '{sum+=$2} END{print sum}')

# 7. Collect the results from the volume
kubectl cp benchmark/bench-run-pod:/results/ ./results/20260616T1200/

# 8. Compute the experiment's energy (mJ → kWh)
python scripts/calc_energy.py \
 --pre $PRE_ENERGY --post $POST_ENERGY \
 --run-id 20260616T1200

# 9. Add a row to the aggregate CSV
python scripts/aggregate_scorecards.py results/ > scorecard-aggregate.csv

# 10. Version it
git add results/20260616T1200/ scorecard-aggregate.csv
git commit -m "bench: llama3-70b fp8 4xH100 ISL1024 OSL256 (20260616T1200)"

Steps 6-9 are candidates for automation as a second post-processing Job launched when the main one completes (with ownerReferences or a clean-up CronJob). In the harness’s basic state, the manual steps are precisely the ones that force a review of the result before versioning it.

Cost of the experiment

A GuideLLM sweep of 10 rounds at 120 seconds per round occupies the node for ~22-26 minutes. The AIPerf profile at 4 concurrencies (200 requests each) adds ~8-12 minutes. Total per run: ~35-40 minutes of a 4×H100 node.

At the reference amortised cost (~10.8 €/h for 4×H100 on-prem):

$$\text{cost per run} \approx 10.8 \times \frac{38}{60} \approx 6.84 \text{ EUR}$$

At the reference European cloud price (4 × 2.73 = 10.92 €/h):

$$\text{cost per run (cloud)} \approx 10.92 \times \frac{38}{60} \approx 6.92 \text{ EUR}$$

A continuous benchmarking programme (10 configurations × 3 models × 2 sweeps/week) adds up to ~415 EUR/week. That is why the harness includes short sweeps (5 rounds, ~15 minutes) for CI and full sweeps for releases.

What the harness does not measure (and why)

Missing dimension	Reason	Where it is covered
Output quality (MMLU, HumanEval)	requires a quality evaluation bench, not a serving one	LLM quality benchmarks
Training throughput	the harness is exclusively for inference	outside the scope of the “data” series
Network and storage cost	OpenCost can attribute it, but it requires additional configuration	OpenCost: cost allocation
Hourly carbon intensity	use the ElectricityMaps API + the measured energy	From the watt to carbon: PUE and grid mix
Client-server network latency	the Job runs inside the cluster; WAN latency requires an external client	document as additional metadata

Sources

GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
GuideLLM · PyPI — https://pypi.org/project/guidellm/
Red Hat Developer · desplegar y benchmarkear vLLM con GuideLLM en Kubernetes — https://developers.redhat.com/articles/2025/12/24/how-deploy-and-benchmark-vllm-guidellm-kubernetes
Red Hat Developer · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
AIPerf · GitHub (ai-dynamo/aiperf) — https://github.com/ai-dynamo/aiperf
GenAI-Perf · documentación NVIDIA Triton — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/README.html
NVIDIA · LLM performance benchmarking con GenAI-Perf — https://developer.nvidia.com/blog/llm-performance-benchmarking-measuring-nvidia-nim-performance-with-genai-perf
NVIDIA NIM Benchmarking · métricas (TTFT, ITL, throughput) — https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html
OpenCost · web oficial — https://opencost.io/
OpenCost · GitHub (opencost/opencost) — https://github.com/opencost/opencost
OpenCost · CNCF blog (sandbox → incubating) — https://www.cncf.io/blog/2022/12/06/opencost-a-new-cncf-sandbox-project-for-real-time-kubernetes-cost-monitoring/
LiteLLM · web oficial — https://www.litellm.ai/
LiteLLM · GitHub (BerriAI/litellm) — https://github.com/BerriAI/litellm
LiteLLM · Spend Tracking docs — https://docs.litellm.ai/docs/proxy/cost_tracking
NVIDIA DCGM Exporter · documentación oficial — https://docs.nvidia.com/datacenter/dcgm/latest/gpu-telemetry/dcgm-exporter.html
NVIDIA DCGM Exporter · GitHub — https://github.com/NVIDIA/dcgm-exporter
NVIDIA DCGM Exporter · métricas CSV — https://github.com/NVIDIA/dcgm-exporter/blob/main/etc/dcp-metrics-included.csv
Kepler (Kubernetes-based Efficient Power Level Exporter) · GitHub — https://github.com/sustainable-computing-io/kepler
Red Hat Emerging Technologies · Introducing Kepler — https://next.redhat.com/2023/08/22/introducing-kepler-efficient-power-monitoring-for-kubernetes/
MLCommons · MLPerf Inference Datacenter benchmark — https://mlcommons.org/benchmarks/inference-datacenter/
MLCommons · MLPerf Power presentado en IEEE HPCA 2025 — https://mlcommons.org/2025/03/ml-commons-power-hpca/
IEEE Xplore · MLPerf Power paper (μWatts to MWatts) — https://ieeexplore.ieee.org/document/10946778/
MLCommons · MLPerf Inference v5.1 results (septiembre 2025) — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
Red Hat · Efficient and reproducible LLM inference: MLPerf Inference v5.1 — https://www.redhat.com/en/blog/efficient-and-reproducible-llm-inference-red-hat-mlperf-inference-v51-results
BentoML · Beyond tokens-per-second: cost, speed and quality in LLM inference — https://www.bentoml.com/blog/beyond-tokens-per-second-how-to-balance-speed-cost-and-quality-in-llm-inference

Measurement bias and reproducibility: why two benchmarks of the same model give figures that differ by up to 7×

Tue, 16 Jun 2026 07:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The mathematical symbol is reserved for formulas: amounts are expressed in € or USD. Thousands with a thin space ($1\,000$).

TL;DR

Running the same Llama-3-70B-Instruct model on the same 4×H100 SXM 80 GB node with two different tools can produce throughput results that differ by 7.2× without the engine changing a single line of code. The cause is not the engine: it is the measurement method. At 1,000 QPS, a single-process asyncio client (vLLM bench, SGLang bench, genai-perf pre-AIPerf) processed 75,574 tokens against the 545,733 tokens of a multi-process client, both measuring the same endpoint (arXiv 2605.24217). The bias is not random: it is systematic, directional and reproducible. This article catalogues the sources of bias, quantifies their magnitude and describes the minimum harness that turns a number into an auditable datum.

The sources of bias: a catalogue with magnitudes

Each source of bias moves the published number in a given direction and with a characteristic magnitude. The table below orders the sources from largest to smallest impact observed in the literature:

Source of bias	Direction of the bias	Documented magnitude	Reference
Single-process client saturation (Python GIL, asyncio)	understates throughput; overstates latency	up to 7.2× fewer tokens processed at 1,000 QPS	arXiv 2605.24217
Tokenizer different from the model’s (LLMPerf uses a universal LlamaTokenizer)	tok/s not comparable across different vocabularies	variable; up to 33 % of client overhead in the TPS denominator	NVIDIA blog (TPS = client overhead ÷ total duration)
ignore_eos missing or inconsistent OSL	underestimates throughput; artificially shortens the real OSL	up to premature termination; an unrealistic fixed-length benchmark	vLLM bench docs; NVIDIA fundamental concepts
Unrealistic ISL/OSL distribution (fixed length instead of a realistic distribution)	an optimistic knee that does not resemble production	the knee shifts; prefill understated with short prompts	AIPerf sequence-length-distributions docs
Missing warmup / warm prefix cache	overstates TTFT (artificially low on the first requests)	TTFT underestimated by an already populated KV cache	vLLM prefix caching docs; arXiv 2605.24217
`concurrency` vs `request-rate` as the load mode	concurrency saturates symmetrically; request-rate can build a queue	ITL overestimated if the queue grows without a ceiling with request-rate	NVIDIA fundamentals; AIPerf docs
Measurement window too short	does not capture the steady state; includes ramp-up	throughput overestimated; latency underestimated	meta-metrics arXiv 2508.10251
Thermal state and GPU clocks	thermal throttling reduces throughput without prior stabilisation	temperature > 75 °C triggers throttling; a difference of ~3 W of stable power	arXiv 2604.09048
Warm prefix cache between runs	overstates TTFT; unrealistic TTFT if the cache is pre-populated from the previous run	a severe effect in multi-prompt evaluations with shared prefixes	arXiv 2605.24217
Mean instead of percentiles	hides the latency tail	P99 can be 4–6× the median at high concurrency	NVIDIA fundamentals; Anyscale docs

The dominant bias: single-process client saturation

The largest bias is not in the engine but in the client that generates the load. The single-process tools (vLLM bench, SGLang bench, genai-perf before its replacement by AIPerf on 15 April 2026) use a single Python process with asyncio to manage the concurrent requests. The Python Global Interpreter Lock (GIL) prevents a single process from using more than one CPU core at a time, which introduces a bottleneck on the client side that becomes critical at high concurrency.

The mathematical effect: as the request rate rises, the client fails to dispatch requests at the configured pace, accumulates queueing time on the client side, and that time gets recorded as engine latency. The result is that TTFT and TPOT appear inflated and throughput appears depressed, without the engine having changed anything (arXiv 2605.24217).

The documented discrepancy at 1,000 QPS:

Client architecture	Tokens processed at 1,000 QPS	Ratio
Single-process (asyncio, Python GIL)	75,574	1× (baseline)
Multi-process (distributed load)	545,733	7.2×

Note: both clients point at the same endpoint; the difference is entirely attributable to the benchmark client, not to the engine.

Tokenizer bias: tok/s are not comparable across vocabularies

Tokens are not universal. Each model has its own tokenizer with its own vocabulary. Llama-3 uses a vocabulary of 128,256 tokens; Gemma uses 256,128; earlier models used 32,000–50,000. The same English text produces a different number of tokens depending on the model’s tokenizer.

The direct consequence for benchmarking: if the tool measures tok/s with a tokenizer different from that of the model being served, the token count, and therefore the throughput in tok/s and the cost per token, are biased. LLMPerf (archived in December 2025) used LlamaTokenizer universally for all models, which guaranteed internal consistency in the leaderboard but made the tok/s not comparable with measurements of other models with different vocabularies.

The TPS formula in LLMPerf additionally included the benchmark’s full denominator, the time spent generating prompts, preparing requests and storing responses, which NVIDIA estimated at up to 33 % of the total duration at concurrency 1. This makes LLMPerf’s TPS systematically lower than that of GenAI-Perf/AIPerf for the same system, without the engine being worse:

$$\text{TPS}_{\text{LLMPerf}} = \frac{\text{output tokens}}{T_{\text{end}} - T_{\text{start}}}$$ $$\text{TPS}_{\text{GenAI-Perf}} = \frac{\text{output tokens}}{T_y - T_x}$$

Where $T_{\text{start}}$ and $T_{\text{end}}$ include the client overheads, while $T_x$ and $T_y$ are the instant of the first request and that of the last token received, respectively (NVIDIA · Fundamental Concepts).

The operational rule: always count tokens with the tokenizer of the model being served, not with a proxy tokenizer.

The bias of ignore_eos and an inconsistent OSL

Most LLM models generate a special end-of-sequence token (EOS) when they consider the response complete. If the benchmark does not set ignore_eos=True, the real output length (OSL) varies from request to request according to the model’s natural length distribution, which produces:

Inconsistent OSL: two runs with the same seed can produce different OSL distributions if the model varies its natural output.
Spurious comparison: a “faster” model may be so simply because it generates shorter responses (it hits EOS sooner), not because it has more real throughput.
Underestimated throughput: if the benchmark expects OSL=256 but the model stops at OSL=80 on average, the throughput in tok/s appears higher but measures less work.

The ignore_eos parameter (or --ignore-eos in vLLM bench) instructs the engine to ignore the EOS token and continue until reaching max_tokens. It is mandatory for the OSL to be the configured one rather than the model’s natural one, and for two runs to be comparable (vLLM benchmark CLI docs).

The bias of the ISL/OSL distribution

The distribution of input (ISL) and output (OSL) lengths determines what proportion of the compute time goes to prefill (costly in TTFT) and how much to decode (costly in ITL). A benchmark with a fixed length, say ISL=128, OSL=128, produces results that do not resemble any real traffic.

Real use cases have very different distributions:

Use case	Typical ISL (tokens)	Typical OSL (tokens)	Dominated by
Translation	500–2,000	500–2,000	balanced
Generation (code, email)	~100	~1,000	decode (long OSL)
Summarisation / RAG	~1,000	~100	prefill (long ISL)
Reasoning (CoT)	~100	1,000–10,000	very long decode

A short-ISL benchmark with a model optimised for prefill will give an artificially low TTFT and an artificially high throughput. The knee of the sweep shifts: with ISL=64 the system admits more concurrency without breaking the TTFT SLO; with ISL=1,024 prefill saturates earlier and the knee appears earlier. Using the wrong distribution means sizing for traffic that does not exist.

AIPerf introduces sequence distributions with configurable per-component variance to reproduce realistic traffic mixes (AIPerf · Sequence Length Distributions):

--sequence-distribution "64|10,32|8:70;256|40,128|20:20;1024|100,512|50:10"

This creates 70 % of requests with $\text{ISL} \sim \mathcal{N}(64, 10)$ and $\text{OSL} \sim \mathcal{N}(32, 8)$, 20 % with medium ISL/OSL, and 10 % with long ISL/OSL, far more faithful to real chatbot traffic than a fixed length.

The bias of warmup and the prefix cache

Two related but distinct sources of bias:

Missing warmup. The first requests of a benchmark hit a “cold” engine: the GPU is in a low-clock state, the KV cache is empty, and the operating system may be paging memory. The TTFT of the first requests is structurally higher than that of the steady state. If the benchmark does not discard a warmup period, the mean TTFT includes these outliers and overestimates the real production latency. Some frameworks (GenAI-Perf/AIPerf) use a sliding window that excludes the ramp-up and ramp-down requests; others do not.

Warm prefix cache between runs. The prefix KV cache (prefix cache or prompt cache) stores the computed results of repeated prompt prefixes. If the benchmark runs multiple consecutive runs with the same prompts, the second and subsequent runs find the KV cache already populated and report an artificially low TTFT, that of the decode, not of the prefill. For a baseline benchmark, the prefix cache must be cold; for one that simulates production with repeated prompts, warm. The distinction must be made explicit (arXiv 2605.24217; vLLM · Automatic Prefix Caching).

The bias of `concurrency` vs `request-rate`

The two load control modes produce different latency distributions for the same system:

Mode	Semantics	When to use	Risk
concurrency N	keeps exactly N requests in flight; as soon as one finishes, it launches another	measuring the system under a fixed concurrency load	over-represents sustained load; does not simulate real arrivals
request-rate r (constant or Poisson)	launches one request every $1/r$ seconds (constant) or with interarrival $\sim \text{Exp}(1/r)$ (Poisson)	simulating real traffic (random arrivals)	if the engine cannot absorb r req/s, the queue grows without a ceiling

NVIDIA recommends the concurrency mode for most capacity benchmarks (NVIDIA · LLM Benchmarking Fundamental Concepts). The request-rate mode is more faithful for online traffic (a Poisson distribution of arrivals), but if the rate exceeds the engine’s capacity the queue grows indefinitely and the latency metrics include queueing time that can dominate the TTFT, mixing queue behaviour with engine behaviour.

A system measured at concurrency=16 and another at request-rate=16 req/s are not under the same load: fixed concurrency guarantees 16 simultaneous requests; the rate sets the arrival pace but not the instantaneous concurrency. Comparing their results without adjustment is incorrect.

The bias of the measurement window and the thermal state

Window too short. A 30-second benchmark may measure the system’s ramp-up, not the steady state. The recommendation of arXiv 2508.10251 is that the measurement window cover at least 3-5× the system’s ramp time under load, and that the metrics be computed only over the steady window, excluding warmup and cooldown.

Thermal state of the GPU. Datacenter GPUs (H100 SXM, A100) operate with thermal throttling above ~75 °C. If the GPU has not reached its steady-state temperature before the benchmark, the first measurements correspond to a higher-clock state than the sustainable one. Controlled experiments document that power must stabilise within a range of 3 W for at least 30 seconds before the measurements are representative (arXiv 2604.09048 · Watt Counts). The effect is especially severe in prefill benchmarks (TTFT), where the high clocks of the cold state produce an artificially low TTFT.

For a reproducible benchmark on 4×H100 SXM 80 GB: before measuring, run sustained load for at least 2–3 minutes until nvidia-smi reports a stable temperature and stable power (variation < 3 W over 30 s). Only then start the measurement window.

How the tools compute ITL differently

ITL (Inter-Token Latency) appears in every tool but its formula varies, and the differences are not small:

GenAI-Perf / AIPerf:

$$\text{ITL} = \frac{\text{e2e latency} - \text{TTFT}}{\text{output tokens} - 1}$$

TTFT is excluded from the numerator and the denominator discounts the first token. ITL is a pure decode metric, with no prefill contamination.

LLMPerf (archived Dec. 2025):

$$\text{ITL}_{\text{LLMPerf}} = \frac{\text{e2e latency}}{\text{output tokens}}$$

TTFT is included in the numerator. For short sequences (OSL < 50 tokens), TTFT can represent 50–80 % of the e2e_latency, so that LLMPerf’s ITL measures mainly prefill, not decode. Two systems with the same decode but different prefill will show different ITLs in LLMPerf even if they are identical in decode speed.

vLLM bench (benchmark_serving.py):

It computes ITL as the mean of the intervals between consecutive tokens of the output stream, including intra-request variance. It can reveal decode jitter that the others average away.

The practical consequence: never cross a GenAI-Perf ITL with an LLMPerf one as if they were the same metric. They are different formulas over the same signal.

Comparability across tools: why their numbers do not cross

The following table summarises the methodological differences that make one tool’s numbers structurally incompatible with another’s without an explicit adjustment:

Dimension	vLLM bench	LLMPerf (archived)	GenAI-Perf / AIPerf	GuideLLM
Client architecture	single-process asyncio	single-process	multi-process	multi-process
ITL includes TTFT	no	yes	no	no
TPS denominator	$T_y - T_x$	$T_{\text{end}} - T_{\text{start}}$ (overhead included)	$T_y - T_x$	per round
Tokenizer	that of the model served	universal LlamaTokenizer	that of the model served	that of the model served
Warmup / sliding window	not automatic	no	yes (sliding window)	per round
ignore_eos by default	no	no	explicitly recommended	configurable
ISL/OSL distribution	manual parameter	manual parameter	distributions with variance	configurable `--data`
Primary load mode	concurrency	concurrency (drained batches)	concurrency (recommended)	sweep + poisson
Structured output	basic JSON	JSON	JSON + CSV	JSON + HTML + CSV

An LLMPerf result and a GenAI-Perf result for the same endpoint can differ in TTFT, ITL and TPS simultaneously and in all of them for methodological reasons, not because of engine differences. The only way to cross them is to run both tools on the same system under the same conditions and compute the empirical conversion factor, which in practice amounts to re-measuring with a single tool.

The honest harness: a reproducibility checklist

A benchmark whose number can be defended in an audit has to come accompanied by all the metadata that allow it to be reproduced. The minimum to demand, organised by category:

Hardware

Metadata	Example 4×H100	Effect if not fixed
GPU (model, variant, count)	4× NVIDIA H100 SXM 80 GB	H100 PCIe vs SXM differ in memory BW and NVLink
GPU-GPU interconnect	NVLink 4.0	tensor parallelism depends on the interconnect BW
CPU, RAM, CPU-GPU bandwidth	2× Intel Xeon 8480+, 512 GB DDR5, PCIe 5.0	the tokenisation bottleneck can be on the CPU
Thermal state at the start	GPU temperature < 65 °C, stable power ± 3 W	throttling alters TPS and TTFT by up to ~15 %
GPU clocks	without `nvidia-smi -pm 1` they can vary	a difference of up to ~10 % in throughput

Software and model

Metadata	Example	Effect if not fixed
Engine + version + flags	vLLM 0.8.4, `--tensor-parallel-size 4 --gpu-memory-utilization 0.90`	each version changes the scheduler and the KV cache management
Model + precision	Llama-3-70B-Instruct, FP8	FP16 vs FP8 differ by ~1.4–1.8× in throughput
Tokenizer used for counting	tokenizer of the model served (HF tokenizer)	a universal LlamaTokenizer biases tok/s across vocabularies
Generation seed	`--seed 42`	without a fixed seed, the OSL varies run-to-run
`ignore_eos`	`True`	without it, the OSL varies with the prompt content
Sampling parameters	`temperature=1.0, top_p=0.95`	greedy vs sampling affect logit speed

Load

Metadata	Example	Effect if not fixed
Bench tool + version	AIPerf 0.2.1	each version changes formulas and warmup
ISL/OSL distribution	$\mathcal{N}(512, 64)$ ISL, $\mathcal{N}(128, 20)$ OSL	changing the distribution moves the knee
Load mode	concurrency, sweep 1–64	concurrency vs request-rate: different distributions
Concurrency levels	1, 2, 4, 8, 16, 32, 64 (full sweep, beyond the knee)	without passing the knee, the safe capacity is unknown
Duration per point	120 s minimum per level	short windows capture ramp-up, not the steady state
Warmup	30 s excluded from the metric computation	without warmup, the metrics include the cold state
Prefix cache treatment	cold (flush between runs) or warm (declared explicitly)	warm cache: unrealistic TTFT for a baseline

SLO and reported metrics

Metadata	Example	Effect if not declared
Declared SLO	TTFT P99 < 500 ms, ITL P95 < 50 ms	goodput depends on the SLO; without an SLO there is no goodput
Reported percentiles	P50, P95, P99 (not just the mean)	the mean hides the tail; irrelevant for an SLO
Throughput vs goodput	goodput under the SLO, not raw throughput	catalogue throughput can be 5–10× the goodput
Total number of requests	≥ 1,000 per concurrency level	small samples: high variance in percentiles

Methodological defence of the datum before an audit

A throughput number presented without the data sheet above is not a datum: it is an anecdote. The validation pattern for a technical audit:

1. Traceability of the tool. The tool and its version must be pinnable to a Git commit or a container image hash. AIPerf and GuideLLM export JSON with version metadata; vLLM bench omits them by default and they have to be captured manually.

2. Reproducibility of the environment. The engine deployment script (or the Kubernetes manifest) and the exact benchmark command must be enough for a third party to reproduce the number on the same hardware. GuideLLM exports the benchmark file as an authoritative record of the session: configuration, metadata, per-benchmark statistics and request entries with individual timings (GitHub vllm-project/guidellm).

3. Verification of the steady state. The metrics must come from the benchmark’s steady window (warmup excluded). The throughput vs concurrency curve must show the knee, the point where throughput saturates and latency blows up, and extend beyond it. Without a visible knee, the reported capacity may be the client’s limit, not the engine’s.

4. Goodput, not raw throughput. The auditable throughput is the goodput under the declared SLO, not the peak throughput. A system that reports 4,000 tok/s but whose goodput under TTFT P99 < 500 ms is 1,800 tok/s has a real capacity of 1,800 tok/s for interactive use cases.

5. Internal comparability. If two engines or two configurations are compared, the tool, the load distribution, the SLO, the hardware and the warmup treatment must be identical. Any difference in these dimensions contaminates the comparison.

The goodput formula as an auditable metric:

$$\text{goodput} = \text{throughput} \times \Pr[\text{TTFT} \leq \text{SLO}_{\text{TTFT}}] \times \Pr[\text{ITL} \leq \text{SLO}_{\text{ITL}}]$$

Where the probabilities are estimated over the empirical distribution of the run. An engine with a throughput of 4,000 tok/s and $\Pr[\text{TTFT} \leq 500\,\text{ms}] = 0.45$ has a goodput of $4{,}000 \times 0.45 = 1{,}800$ tok/s.

A numerical example: the same node, four different measurements

Reference hardware: 4×H100 SXM 80 GB, model Llama-3-70B-Instruct FP8, tensor parallel 4, SLO TTFT P99 < 500 ms.

Benchmark configuration	Reported throughput	Real goodput	Factor vs goodput
vLLM bench, concurrency=32, no ignore_eos, no warmup, fixed OSL=64	5,800 tok/s	~1,200 tok/s (short OSL inflates TPS)	4.8×
LLMPerf, concurrency=32, LlamaTokenizer, overhead included in the denominator	3,100 tok/s	~1,600 tok/s (ITL includes TTFT)	1.9×
AIPerf, concurrency=32, ignore_eos, sliding window, model’s tokenizer	3,400 tok/s	3,350 tok/s	1.01×
GuideLLM, sweep 1–64, poisson, realistic ISL/OSL distribution, declared SLO	3,330 tok/s goodput (knee at round 6)	3,330 tok/s	1.0× (honest baseline)

The range: from 1,200 tok/s to 5,800 tok/s for the same engine, the same hardware and the same model. The maximum factor is 4.8×. The cause is not the engine; it is the instrumentation decisions.

Decomposing the total bias

The total observable bias between two tools is the multiplicative composition of the individual biases. For a comparison of single-process vLLM bench vs multi-process GuideLLM on the same system:

$$\text{total factor} \approx \underbrace{f_{\text{client}}}_{\leq 7.2\times} \times \underbrace{f_{\text{tokenizer}}}_{1.0{-}1.3\times} \times \underbrace{f_{\text{ignore-eos}}}_{1.0{-}2.0\times} \times \underbrace{f_{\text{ISL/OSL}}}_{1.0{-}1.5\times} \times \underbrace{f_{\text{warmup}}}_{1.0{-}1.2\times}$$

The client saturation factor ($\leq 7.2\times$) dominates, but the other factors multiply. Under adverse conditions (single-process client + different tokenizer + no ignore_eos + unrealistic OSL + no warmup), the compound bias can exceed 15–20× for the same system.

State of the art 2026: what has changed

genai-perf → AIPerf migration (15 Apr 2026): NVIDIA retired genai-perf and replaced it with AIPerf, which is multi-process with automatic saturation detection, configurable ISL/OSL distributions and a built-in sliding window. The migration removes the single-process client bias from NVIDIA’s official tooling.
LLMPerf archived (Dec. 2025): the ray-project/llmperf repository is archived and in read-only mode. The historical results of the LLMPerf leaderboard are comparable only among themselves; they must not be crossed with modern measurements without adjustment.
GuideLLM 0.5.x (2025–2026): a complete architectural refactor, multimodal support, and authoritative JSON export with all the session metadata. It is the OSS standard for SLO-driven evaluation.
arXiv 2605.24217 (May 2026): the first formal characterisation of systematic measurement bias in production LLM benchmarks, with a mathematical demonstration of the GIL effect and a proposal for a multi-process harness.
arXiv 2508.10251: meta-metrics and good practice for system-level performance benchmarking; it establishes the minimum dimensions of the parameter space that must be declared for a result to be reproducible.
MLPerf Inference v5.1 (Sep. 2025): 27 participants, a participation record. The MLPerf rules require declaring the exact code, the dataset and the hardware, and admit public review; it is the only industry benchmark with a formal review process for reproducibility.

Cross-links from the benchmarking track

The biases described here apply to every tool in the catalogue. For the complete data sheet of each tool:

Full catalogue: LLM benchmarking tools, data sheet by data sheet
GuideLLM and SLO validation: GuideLLM in depth: validating the SLO under load
AIPerf (ex genai-perf) and saturation detection: NVIDIA GenAI-Perf in depth
State of the art of frameworks: LLM benchmarking: frameworks and state of the art
Engine decision on the Pareto axis: Serving engines compared: the Pareto frontier

Sources

arXiv 2605.24217 · Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks — https://arxiv.org/abs/2605.24217
arXiv 2508.10251 · Meta-Metrics and Best Practices for System-Level Inference Performance Benchmarking — https://arxiv.org/pdf/2508.10251
arXiv 2604.09048 · Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures — https://arxiv.org/html/2604.09048v1
NVIDIA Technical Blog · LLM Inference Benchmarking: Fundamental Concepts (ITL, TPS, ISL/OSL, ignore_eos, concurrency vs request-rate) — https://developer.nvidia.com/blog/llm-benchmarking-fundamental-concepts/
NVIDIA AIPerf Docs · Sequence Length Distributions for Advanced Benchmarking — https://docs.nvidia.com/aiperf/tutorials/datasets-inputs/sequence-length-distributions-for-advanced-benchmarking
NVIDIA AIPerf · Request Rate with Max Concurrency — https://docs.nvidia.com/aiperf/tutorials/load-patterns-scheduling/request-rate-with-max-concurrency
NVIDIA AIPerf · Comprehensive LLM Benchmarking Guide — https://lucaberton.com/blog/nvidia-aiperf-llm-inference-benchmarking-guide/
vLLM Documentation · Benchmark CLI (ignore_eos, métricas disponibles) — https://docs.vllm.ai/en/latest/benchmarking/cli/
vLLM Documentation · Automatic Prefix Caching — https://docs.vllm.ai/en/stable/design/prefix_caching/
GitHub · ray-project/llmperf (archivado dic. 2025) — https://github.com/ray-project/llmperf
GitHub · vllm-project/guidellm (exportación JSON autoritativa, metadatos de sesión) — https://github.com/vllm-project/guidellm
Red Hat Developer · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
MLCommons · MLPerf Inference v5.1 results (récord 27 participantes, reproducibilidad formal) — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
arXiv 2502.16721 · Speed and Conversational LLMs: Not All Is About Tokens per Second (tokenizer incompatibilidad y tok/s entre vocabularios) — https://arxiv.org/pdf/2502.16721
Medium · LLM Inference Benchmarking (genAI-perf y vLLM, discrepancia 7,2×) — https://kchandan.medium.com/llm-inference-benchmarking-genai-perf-and-vllm-5dd06b57428e

Reproducibilidad on lo0 — Tech Blog

The reproducible harness: measuring cost, performance and energy in a single auditable experiment

TL;DR

Why the three axes must be measured together

Architecture of the integrated bench

The Kubernetes Job: the full YAML

Performance layer: GuideLLM and AIPerf

GuideLLM — the SLO-driven sweep

AIPerf — NVIDIA’s concurrency profiler

Cost layer: OpenCost and LiteLLM

OpenCost — the denominator in euros per GPU-hour

LiteLLM — the per-request token counter

Energy layer: DCGM and Kepler

DCGM Exporter — GPU power and energy

Kepler — pod-level energy

The MLPerf Power reference

Unifying the metrics: the PromQL queries

The 3-axis scorecard: table and interpretation

The multi-objective Pareto frontier

Reproducible design: the twelve mandatory metadata

Idempotence and versioning of results

How it connects with the other posts in the series

Full flow of a benchmarking session

Cost of the experiment

What the harness does not measure (and why)

Sources

Measurement bias and reproducibility: why two benchmarks of the same model give figures that differ by up to 7×

TL;DR

The sources of bias: a catalogue with magnitudes

The dominant bias: single-process client saturation

Tokenizer bias: tok/s are not comparable across vocabularies

The bias of ignore_eos and an inconsistent OSL

The bias of the ISL/OSL distribution

The bias of warmup and the prefix cache

The bias of concurrency vs request-rate

The bias of the measurement window and the thermal state

How the tools compute ITL differently

Comparability across tools: why their numbers do not cross

The honest harness: a reproducibility checklist

Hardware

Software and model

Load

SLO and reported metrics

Methodological defence of the datum before an audit

A numerical example: the same node, four different measurements

Decomposing the total bias

State of the art 2026: what has changed

Cross-links from the benchmarking track

Sources

The bias of `concurrency` vs `request-rate`