The reproducible harness: measuring cost, performance and energy in a single auditable experiment

Notation: amounts in euros (N €), decimals with a point, thousands with a thin space. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

The “data” series has produced three independent measurement axes: cost per million tokens (OpenCost + LiteLLM), performance under an SLO (GuideLLM + AIPerf) and energy per token (DCGM + Kepler). The problem is that the three have been measured in different articles, with different loads and at different times: they are not comparable with each other. This closing article describes the integrated harness that runs the three axes in the same experiment, on the same node (4×H100 SXM, a generic reference), with all the metadata fixed, the output in versioned JSON/CSV and an idempotent Kubernetes Job. The result is the 3-axis scorecard (€/1M tok, Wh/token, TTFT/ITL P99) that allows configurations to be compared on a multi-objective Pareto frontier and any figure to be audited with the bench to reproduce it.

Why the three axes must be measured together

The series’ opening post (The three axes) established the identity:

Throughput is the common denominator. If it is measured in different experiments, different time, different load, different GPU temperature, then CPM and energy/token do not share a denominator: they are three anecdotes, not a scorecard. The harness captures them in the same time window, over the same load, with Prometheus windows aligned to the second. Only then is the scorecard row coherent by construction.

The second reason is reproducibility. The post Bias and reproducibility in benchmarking listed twelve biases that invalidate comparisons: an unpinned engine, an undeclared tokeniser, unfixed input/output lengths, missing warmup, a client outside the cluster. The harness removes all of them because the metadata are part of the Job, not of the documentation.

Architecture of the integrated bench

The harness has four layers. Each one is OSS, exports to Prometheus and lives in the same Kubernetes namespace:

MLPerf Power is used as an external comparability reference: its published results, with hardware documented in detail, allow the harness figures to be calibrated for plausibility. The in-house bench does not aim to be an MLPerf submission, but to be reproducible on your own cluster.

The Kubernetes Job: the full YAML

The experiment runs as a versioned Kubernetes Job. All the relevant metadata are environment variables declared in the manifest: not in ad-hoc scripts, not in external documentation. The Job is idempotent (same name = same run) and leaves traces in the pod log and in the output volume.

apiVersion: batch/v1
kind: Job
metadata:
  name: bench-llama3-70b-fp8-h100x4-20260616
  namespace: benchmark
  labels:
    bench/model: llama3-70b
    bench/precision: fp8
    bench/engine: vllm-0.9.1
    bench/hardware: h100x4-sxm
    bench/isl: "1024"
    bench/osl: "256"
    bench/concurrency: "32"
    bench/tokenizer: meta-llama-3-tokenizer-v3
    bench/run-id: "20260616T1200"
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      serviceAccountName: bench-runner
      volumes:
        - name: results
          persistentVolumeClaim:
            claimName: bench-results-pvc
      initContainers:
        # Warmup: 60 s of traffic before the experiment
        - name: warmup
          image: ghcr.io/vllm-project/guidellm:0.4.2
          command:
            - guidellm
            - benchmark
            - --target
            - http://vllm-svc.inference.svc.cluster.local:8000
            - --rate-type
            - concurrent
            - --rate
            - "4"
            - --max-seconds
            - "60"
            - --data
            - prompt_tokens=1024,output_tokens=256
          env:
            - name: GUIDELLM_ENV
              value: production
      containers:
        - name: guidellm-sweep
          image: ghcr.io/vllm-project/guidellm:0.4.2
          command:
            - guidellm
            - benchmark
            - --target
            - http://vllm-svc.inference.svc.cluster.local:8000
            - --rate-type
            - sweep
            - --max-seconds
            - "120"
            - --data
            - prompt_tokens=1024,output_tokens=256
            - --output-path
            - /results/guidellm-$(BENCH_RUN_ID).json
          env:
            - name: BENCH_RUN_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['bench/run-id']
          volumeMounts:
            - name: results
              mountPath: /results

        - name: aiperf-concurrent
          image: nvcr.io/nvidia/aiperf:0.2.0
          command:
            - aiperf
            - profile
            - --url
            - http://vllm-svc.inference.svc.cluster.local:8000/v1
            - --model
            - meta-llama/Meta-Llama-3-70B-Instruct
            - --concurrency
            - "4,8,16,32"
            - --input-tokens
            - "1024"
            - --output-tokens
            - "256"
            - --num-requests
            - "200"
            - --output
            - /results/aiperf-$(BENCH_RUN_ID).json
          volumeMounts:
            - name: results
              mountPath: /results

        - name: scorecard-exporter
          image: python:3.12-slim
          command:
            - python
            - /scripts/export_scorecard.py
            - --run-id
            - "$(BENCH_RUN_ID)"
            - --prometheus
            - http://prometheus.monitoring.svc.cluster.local:9090
            - --output
            - /results/scorecard-$(BENCH_RUN_ID).json
          env:
            - name: BENCH_RUN_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['bench/run-id']
          volumeMounts:
            - name: results
              mountPath: /results

The Job name (bench-llama3-70b-fp8-h100x4-20260616) is the unique identifier of the run. Changing it is enough to register a variant. The labels are the metadata that the scorecard-exporter reads to enrich the output JSON.

Performance layer: GuideLLM and AIPerf

GuideLLM — the SLO-driven sweep

GuideLLM (a vLLM project) generates realistic traffic patterns, synchronous, concurrent, poisson, throughput, sweep, and captures full TTFT and ITL distributions (Red Hat Developer). The sweep mode ramps from idle to saturation over 10 rounds and identifies the knee: the maximum load where goodput ≈ throughput under the declared SLO. The output is JSON/CSV with all the percentiles per round, ready to be versioned.

For the harness, the bench’s reference SLO is:

The Job command already appears in the YAML above. The JSON output includes, per round: rate (req/s), TTFT (P50/P95/P99), ITL (P50/P95/P99), throughput (tok/s) and goodput (tok/s). The goodput under the SLO at the knee is the value that enters the scorecard.

For the cross-verification layer of the data and comparability with published results, see the post GuideLLM in depth.

AIPerf — NVIDIA’s concurrency profiler

AIPerf (the successor to GenAI-Perf, repository ai-dynamo/aiperf) measures TTFT, ITL, throughput and latency distributions at fixed concurrencies (GitHub ai-dynamo/aiperf). Where GuideLLM gives the automatic sweep up to the knee, AIPerf gives the detailed profile at specific concurrencies (4, 8, 16, 32 in the example): it allows the throughput-latency curve to be characterised point by point.

The two are complementary: GuideLLM finds the knee automatically; AIPerf confirms it and characterises the behaviour in the neighbourhood of that knee. Both outputs go to the results volume with the same BENCH_RUN_ID. The deep analysis of AIPerf/GenAI-Perf is in GenAI-Perf in depth.

Metrics the harness extracts from AIPerf for the scorecard:

# TTFT P99 at concurrency 16 (the closest to the sweep's knee):
aiperf profile ... --concurrency 16 \
  | jq '.results[] | select(.concurrency==16) | .ttft_ms.p99'

# Goodput (tok/s) at concurrency 16:
aiperf profile ... --concurrency 16 \
  | jq '.results[] | select(.concurrency==16) | .output_token_throughput'

Cost layer: OpenCost and LiteLLM

OpenCost — the denominator in euros per GPU-hour

OpenCost (CNCF incubating, opencost.io) allocates Kubernetes cost to namespace, label, pod and container in real time (GitHub opencost/opencost). For the harness, OpenCost answers the question: how much does the vllm-svc pod cost in euros per hour during the experiment’s window?

The harness calls the OpenCost REST API when the experiment finishes:

# Cost of the inference namespace over the experiment window (1 hour)
curl -s "http://opencost.monitoring.svc.cluster.local:9003/allocation" \
  --data-urlencode 'window=2026-06-16T12:00:00Z,2026-06-16T13:00:00Z' \
  --data-urlencode 'aggregate=namespace' \
  --data-urlencode 'namespace=inference' \
  | jq '.data[0].inference.totalCost'

The returned cost (in EUR, configured with the node’s real prices) is divided by the throughput measured in that same window to obtain the CPM:

The post OpenCost: cost allocation in Kubernetes details the price configuration and the allocation by GPU label.

LiteLLM — the per-request token counter

LiteLLM (litellm.ai, GitHub BerriAI/litellm) acts as an OpenAI-compatible proxy with token accounting per request, model and team. In the harness, GuideLLM and AIPerf point at the LiteLLM endpoint (which in turn forwards to vLLM): every request is recorded with prompt_tokens, completion_tokens and cost (using the custom pricing configured for the on-prem node).

# litellm-config.yaml (fragment of custom on-prem pricing)
model_list:
  - model_name: llama3-70b-fp8
    litellm_params:
      model: openai/meta-llama/Meta-Llama-3-70B-Instruct
      api_base: http://vllm-svc.inference.svc.cluster.local:8000/v1
      api_key: sk-dummy
      input_cost_per_token: 0.00000109   # 1.09 EUR/1M tok on-prem
      output_cost_per_token: 0.00000109

The LiteLLM records are exported to Prometheus as litellm_request_total_tokens and litellm_spend_metric_total, which the scorecard exporter consumes to compute the real CPM per token type (input vs output).

Energy layer: DCGM and Kepler

DCGM Exporter — GPU power and energy

DCGM Exporter (GitHub NVIDIA/dcgm-exporter, docs) exposes GPU metrics on /metrics for Prometheus as a DaemonSet on the GPU nodes. The two energy metrics the harness uses:

To compute the energy consumed during the experiment’s window (without the idle baseline), the harness takes the difference of the counter before and after the sweep:

# Deployment as a DaemonSet (fragment of the official chart)
helm repo add gpu-helm-charts \
  https://nvidia.github.io/dcgm-exporter/helm-charts
helm install dcgm-exporter gpu-helm-charts/dcgm-exporter \
  --namespace monitoring \
  --set serviceMonitor.enabled=true \
  --set serviceMonitor.interval=15s

The DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION field (in mJ) allows the experiment’s energy to be computed with hardware-counter accuracy, not with an estimate:

Kepler — pod-level energy

Kepler (CNCF sandbox, GitHub sustainable-computing-io/kepler) uses eBPF to estimate energy consumption at container and pod level, exporting kepler_container_joules_total to Prometheus (Red Hat Emerging Technologies). It combines RAPL (CPU/DRAM), NVML (GPU) and regression models when no sensors are available.

In the harness, Kepler complements DCGM: DCGM gives the hardware measurement of the GPU (more precise for GPU-intensive loads such as LLM inference), while Kepler attributes the energy to the specific vLLM pod (including the node’s CPU contribution). The main metric the harness consumes:

# Energy of the vllm-svc pod during the experiment (J)
increase(
  kepler_container_joules_total{
    container_namespace="inference",
    container_name="vllm"
  }[${BENCH_DURATION}]
)

The comparative analysis of DCGM vs Kepler vs Zeus is in Energy tools: deployment, precision and overhead.

The MLPerf Power reference

MLPerf Power (MLCommons, IEEE HPCA 2025 paper) establishes the protocol for measuring the energy efficiency of ML systems with high-precision external measurement. The in-house bench is not an MLPerf submission (that requires external power meters and committee review), but its published results are the calibration reference: if the harness gives a result of the same order as the MLPerf submission for the same hardware with the same model, the measurement is plausible. If it differs by more than a factor of 2, there is a methodological problem. See the post MLPerf Power: energy efficiency for the current reference data.

Unifying the metrics: the PromQL queries

The scorecard-exporter is the Job container that, once GuideLLM and AIPerf have finished, collects all the metrics from Prometheus and builds the JSON of the scorecard row. The key queries:

# Mean GPU power during the sweep (W) — 4×H100 node
avg_over_time(
  sum(DCGM_FI_DEV_POWER_USAGE{Hostname=~"gpu-node-.*"})[${BENCH_DURATION}:15s]
)

# Total GPU energy during the sweep (mJ → convert to J)
(
  sum(DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION{Hostname=~"gpu-node-.*"}) offset 0
  -
  sum(DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION{Hostname=~"gpu-node-.*"}) offset ${BENCH_DURATION}
) * 0.001

# Tokens generated during the sweep (from LiteLLM)
increase(
  litellm_request_total_tokens{model="llama3-70b-fp8", token_type="completion"}[${BENCH_DURATION}]
)

# Cost of the inference namespace over the window (from the OpenCost API)
# → REST call at the start and at the end of the Job, difference of accrual

The ${BENCH_DURATION} variable is the real duration of the sweep (in Prometheus format, e.g. 22m), which the exporter computes as end_ts - start_ts and substitutes into every query.

The output JSON of each run has this structure:

{
  "run_id": "20260616T1200",
  "metadata": {
    "model": "meta-llama/Meta-Llama-3-70B-Instruct",
    "engine": "vllm-0.9.1",
    "precision": "fp8",
    "hardware": "4xH100-SXM-80GB",
    "isl_tokens": 1024,
    "osl_tokens": 256,
    "tokenizer": "meta-llama-3-tokenizer-v3",
    "slo_ttft_p99_ms": 500,
    "slo_itl_p95_ms": 50,
    "bench_tool_guidellm": "0.4.2",
    "bench_tool_aiperf": "0.2.0",
    "dcgm_exporter": "3.3.9",
    "kepler": "0.10.2"
  },
  "performance": {
    "goodput_tok_s": 3120,
    "ttft_p99_ms": 487,
    "itl_p95_ms": 42,
    "throughput_peak_tok_s": 3890,
    "elbow_concurrency": 28
  },
  "cost": {
    "cpm_eur_1m": 0.97,
    "gpu_cost_eur_h": 10.8,
    "cost_window_eur": 3.24
  },
  "energy": {
    "wh_per_token": 0.00044,
    "j_per_token": 1.58,
    "power_mean_w": 4924,
    "energy_total_kwh": 0.287,
    "pue": 1.4,
    "wh_per_token_pue_adjusted": 0.000616
  },
  "carbon": {
    "grid_intensity_gco2_kwh": 40,
    "co2_per_1m_tokens_g": 24.6
  }
}

All the fields are computed from the same time windows. The JSON is versioned in git alongside the harness code. Reproducing any scorecard row is: git checkout <run-id> && kubectl apply -f job.yaml.

The 3-axis scorecard: table and interpretation

The following table illustrates how five configurations of the same model (Llama 3 70B) on the same reference node (4×H100 SXM) compare with the harness. The figures are illustrative of the order of magnitude (the bench fills them in with real measurements):

How to read the table:

• B vs A: FP8 over FP16 cuts CPM by 41 % and energy per token by 41 % thanks to the same higher throughput. Both meet the SLO (TTFT P99 < 500 ms, ITL P95 < 50 ms). B dominates A on all three axes: it is Pareto-superior.
• C vs B: SGLang gives 9 % more goodput and 9 % lower CPM with FP8. Both meet the SLO. C is Pareto-superior to B (if the engine is indifferent for the stack).
• D vs B: a shorter ISL (512 tok) lowers latency (TTFT P99 294 ms vs 487 ms) but lowers goodput and raises CPM. If the use case demands TTFT < 300 ms, D is the candidate; if TTFT < 500 ms is enough, B is better on cost and energy.
• E vs C: max-batch 128 raises goodput (+5 %) and lowers CPM, but the TTFT P99 breaks the SLO (721 ms > 500 ms) and so does the ITL P95 (55 ms > 50 ms). E has the best raw throughput but is outside the SLO: it is not a valid candidate for the interactive chat use case.

The scorecard’s Pareto frontier, under the declared SLO, includes only A, B, C and D (E falls out for breaking the SLO). Of those four, C dominates B which dominates A. D only enters the frontier if the use case demands TTFT P99 < 300 ms. The decision is not a number; it is the whole row plus the SLO.

The multi-objective Pareto frontier

With three minimisation metrics, CPM (€/1M tok), Wh/tok and TTFT P99 (ms), the Pareto frontier is defined as the set of configurations where none dominates another on all three axes simultaneously. Formally, config \( i \) dominates config \( j \) if:

with at least one strict inequality. The scorecard allows it to be computed over the set of runs with a simple vector operation. In the table’s example, the frontier under the SLO (with E excluded) is \(\{C, D\}\): C dominates on cost/energy/goodput; D dominates on latency. Between C and D the choice depends on the TTFT SLO of the specific use case.

Reproducible design: the twelve mandatory metadata

The checklist that closes the bias dossier (Bias and reproducibility). For the harness JSON to be auditable, the twelve fields must be present in every run:

A scorecard missing any of these twelve fields is an anecdote. With all twelve, it is an auditable datum. The run-id in the Job name (bench-llama3-70b-fp8-h100x4-20260616) implicitly incorporates fields 1-3 and 7, forcing the Job name to change with every variant, which makes it impossible to overlap runs in the same namespace.

Idempotence and versioning of results

The Kubernetes Job is idempotent: if the same manifest is applied again (same name), the Job already exists and Kubernetes does not relaunch it (backoffLimit: 0 policy). This avoids accidental duplicate runs. To relaunch, the Job has to be deleted (kubectl delete job <name>) or the run-id changed.

Results are versioned with the following directory convention in the harness’s git repository:

results/
  20260616T1200/
    guidellm-20260616T1200.json
    aiperf-20260616T1200.json
    scorecard-20260616T1200.json
  20260617T0900/
    guidellm-20260617T0900.json
    aiperf-20260617T0900.json
    scorecard-20260617T0900.json
scorecard-aggregate.csv   # all rows, for analysis and charts

The scorecard-aggregate.csv is the scorecard table in flat format: each row is a run, each column a field of the JSON. It is generated with:

# Generates the aggregate CSV from all the JSON files in results/
python scripts/aggregate_scorecards.py results/ > scorecard-aggregate.csv

The CSV is what feeds the Pareto charts, the reports and the architecture decisions. It is versioned in git; its diff is the configuration change.

How it connects with the other posts in the series

The harness is not a standalone article: it is the synthesis of the twenty-eight articles. Each tool has its deep dive:

• Performance: GuideLLM in depth — sweep, SLO, goodput, knee; GenAI-Perf / AIPerf in depth — concurrency, rate sweep, distribution metrics.
• Cost: OpenCost: cost allocation in Kubernetes — allocation by namespace/label, custom on-prem prices.
• Energy: Energy tools: deployment, precision and overhead — DCGM vs Kepler vs Zeus, instrumentation overhead, relative precision; MLPerf Power: energy efficiency — the external calibration reference.
• Theoretical framework: The three axes: cost, performance and energy — the CPM/energy/throughput identity that justifies measuring them together.
• Bias: Bias and reproducibility in benchmarking — the twelve biases the harness removes.

The harness closes the series’ dossier because it makes the opening article’s promise possible: “when someone disputes a number in the proposal, the answer is not ‘a blog says so’, but ’this is the bench, this is the methodology, reproduce it’”. With the run-id and the git repo, reproducing is one command.

Full flow of a benchmarking session

The harness’s standard operating procedure, from start to finish:

# 1. Make sure the inference endpoint is up
kubectl get svc vllm-svc -n inference

# 2. Verify that DCGM and Kepler are scraping
curl -s http://prometheus.monitoring.svc.cluster.local:9090/api/v1/query \
  --data-urlencode 'query=DCGM_FI_DEV_POWER_USAGE' | jq '.data.result | length'

# 3. Record the energy counter BEFORE the experiment
PRE_ENERGY=$(kubectl exec -n monitoring dcgm-exporter-xxx -- \
  curl -s localhost:9400/metrics \
  | grep DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION | awk '{sum+=$2} END{print sum}')

# 4. Launch the Job (unique name per run)
kubectl apply -f jobs/bench-llama3-70b-fp8-h100x4-20260616.yaml

# 5. Wait for it to finish
kubectl wait --for=condition=complete job/bench-llama3-70b-fp8-h100x4-20260616 \
  -n benchmark --timeout=3600s

# 6. Record the energy counter AFTER
POST_ENERGY=$(kubectl exec -n monitoring dcgm-exporter-xxx -- \
  curl -s localhost:9400/metrics \
  | grep DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION | awk '{sum+=$2} END{print sum}')

# 7. Collect the results from the volume
kubectl cp benchmark/bench-run-pod:/results/ ./results/20260616T1200/

# 8. Compute the experiment's energy (mJ → kWh)
python scripts/calc_energy.py \
  --pre $PRE_ENERGY --post $POST_ENERGY \
  --run-id 20260616T1200

# 9. Add a row to the aggregate CSV
python scripts/aggregate_scorecards.py results/ > scorecard-aggregate.csv

# 10. Version it
git add results/20260616T1200/ scorecard-aggregate.csv
git commit -m "bench: llama3-70b fp8 4xH100 ISL1024 OSL256 (20260616T1200)"

Steps 6-9 are candidates for automation as a second post-processing Job launched when the main one completes (with ownerReferences or a clean-up CronJob). In the harness’s basic state, the manual steps are precisely the ones that force a review of the result before versioning it.

Cost of the experiment

A GuideLLM sweep of 10 rounds at 120 seconds per round occupies the node for ~22-26 minutes. The AIPerf profile at 4 concurrencies (200 requests each) adds ~8-12 minutes. Total per run: ~35-40 minutes of a 4×H100 node.

At the reference amortised cost (~10.8 €/h for 4×H100 on-prem):

At the reference European cloud price (4 × 2.73 = 10.92 €/h):

A continuous benchmarking programme (10 configurations × 3 models × 2 sweeps/week) adds up to ~415 EUR/week. That is why the harness includes short sweeps (5 rounds, ~15 minutes) for CI and full sweeps for releases.

What the harness does not measure (and why)

Sources

• GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
• GuideLLM · PyPI — https://pypi.org/project/guidellm/
• Red Hat Developer · desplegar y benchmarkear vLLM con GuideLLM en Kubernetes — https://developers.redhat.com/articles/2025/12/24/how-deploy-and-benchmark-vllm-guidellm-kubernetes
• Red Hat Developer · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
• AIPerf · GitHub (ai-dynamo/aiperf) — https://github.com/ai-dynamo/aiperf
• GenAI-Perf · documentación NVIDIA Triton — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/README.html
• NVIDIA · LLM performance benchmarking con GenAI-Perf — https://developer.nvidia.com/blog/llm-performance-benchmarking-measuring-nvidia-nim-performance-with-genai-perf
• NVIDIA NIM Benchmarking · métricas (TTFT, ITL, throughput) — https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html
• OpenCost · web oficial — https://opencost.io/
• OpenCost · GitHub (opencost/opencost) — https://github.com/opencost/opencost
• OpenCost · CNCF blog (sandbox → incubating) — https://www.cncf.io/blog/2022/12/06/opencost-a-new-cncf-sandbox-project-for-real-time-kubernetes-cost-monitoring/
• LiteLLM · web oficial — https://www.litellm.ai/
• LiteLLM · GitHub (BerriAI/litellm) — https://github.com/BerriAI/litellm
• LiteLLM · Spend Tracking docs — https://docs.litellm.ai/docs/proxy/cost_tracking
• NVIDIA DCGM Exporter · documentación oficial — https://docs.nvidia.com/datacenter/dcgm/latest/gpu-telemetry/dcgm-exporter.html
• NVIDIA DCGM Exporter · GitHub — https://github.com/NVIDIA/dcgm-exporter
• NVIDIA DCGM Exporter · métricas CSV — https://github.com/NVIDIA/dcgm-exporter/blob/main/etc/dcp-metrics-included.csv
• Kepler (Kubernetes-based Efficient Power Level Exporter) · GitHub — https://github.com/sustainable-computing-io/kepler
• Red Hat Emerging Technologies · Introducing Kepler — https://next.redhat.com/2023/08/22/introducing-kepler-efficient-power-monitoring-for-kubernetes/
• MLCommons · MLPerf Inference Datacenter benchmark — https://mlcommons.org/benchmarks/inference-datacenter/
• MLCommons · MLPerf Power presentado en IEEE HPCA 2025 — https://mlcommons.org/2025/03/ml-commons-power-hpca/
• IEEE Xplore · MLPerf Power paper (μWatts to MWatts) — https://ieeexplore.ieee.org/document/10946778/
• MLCommons · MLPerf Inference v5.1 results (septiembre 2025) — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
• Red Hat · Efficient and reproducible LLM inference: MLPerf Inference v5.1 — https://www.redhat.com/en/blog/efficient-and-reproducible-llm-inference-red-hat-mlperf-inference-v51-results
• BentoML · Beyond tokens-per-second: cost, speed and quality in LLM inference — https://www.bentoml.com/blog/beyond-tokens-per-second-how-to-balance-speed-cost-and-quality-in-llm-inference