Energy-Efficiency on lo0 — Tech Blog

LLM energy leaderboards: comparing models by Wh/token and choosing on efficiency

Tue, 16 Jun 2026 04:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter). Generic example hardware; no real infrastructure.

TL;DR

There are three OSS leaderboards with public data and documented methodology for comparing the energy efficiency of LLMs in inference: Hugging Face AI Energy Score (166 models, Wh/query on H100, 1–5 star scale, launched February 2025), ML.ENERGY Leaderboard v3 (University of Michigan, J/token per task, Zeus tooling, December 2025) and MLPerf Power (samples/joule certified with a physical Yokogawa WT310E power meter). The three measure different dimensions and are not directly interchangeable. The available data shows that reasoning models consume up to 700× more energy than their non-reasoning equivalents; that MoE models consume roughly 3× fewer J/token than a dense model with equivalent active parameters; and that INT4 quantisation cuts consumption by up to 79 % against FP16 under favourable conditions. The inference engine (vLLM vs Transformers) can move the result another 25–40 %. Without fixing hardware, engine, batch size and task, no comparison between leaderboards is valid.

Track context

This article is C5 of the energy pillar. The base context:

The fundamentals of quantisation are a prerequisite for the quantisation section of this article.

The three leaderboards: technical data sheet

1 · Hugging Face AI Energy Score

Field	Detail
URL	huggingface.co/AIEnergyScore · huggingface.co/spaces/AIEnergyScore/Leaderboard
Organisation	Hugging Face (Sasha Luccioni et al.), with Salesforce and Cohere as initial partners
Launch	February 2025 (AI Action Summit, Paris); v2 December 2025
Models indexed	166 (v1, Feb 2025); +39 new ones in v2 (Dec 2025)
Tasks measured	10 tasks: text generation, summarisation, classification, image generation, ASR, audio generation, translation, question answering, reasoning (added in v2)
Unit of measurement	Wh (watt-hour) per 1,000 queries of the task
Reference hardware	NVIDIA H100 exclusively (a single GPU for class A/B models; multiple for class C)
Measurement tooling	CodeCarbon (GPU energy) + the `ai-energy-benchmarks` package (OSS, PyPI)
Rating system	1–5 stars per task: quintiles of the energy range; ⭐⭐⭐⭐⭐ = the 20 % most efficient
Reference batch size	Batch size = 1 (does not reflect production with aggressive batching)
Access for proprietary models	Yes, via an audited Docker container
Update frequency	No fixed cadence; v1 Feb 2025, v2 Dec 2025
Project licence	Apache 2.0 (repository github.com/huggingface/AIEnergyScore)

Scope of the metric. The AI Energy Score measures GPU energy only (CodeCarbon); it does not capture CPU, DRAM or system overhead. The Wh/1k-queries unit covers the whole run time (prefill + decode + framework overhead), but at batch = 1. The results are therefore comparable between models under the same test conditions, but not extrapolable to a production environment with real concurrency without correction.

Model class (the project’s internal classification):

Class	Definition
A	Fits on a consumer GPU (≤ ~24 GB VRAM)
B	Requires a cloud GPU (≥ 40 GB VRAM)
C	Requires multiple GPUs

2 · ML.ENERGY Leaderboard

Field	Detail
URL	ml.energy/leaderboard
Organisation	Symbiotic Lab, University of Michigan (Mosharaf Chowdhury, Jae-Won Chung et al.)
Reference paper	arXiv 2505.06371 — “The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization” (NeurIPS 2025 D&B, Spotlight)
Current version	v3.0 (December 2025)
Measurement tooling	Zeus (github.com/ml-energy/zeus) via NVML + RAPL; measurement overhead in single-digit ms
Unit of measurement	J/token (energy per output token generated) and total energy per complete response
Reference hardware	NVIDIA A100 80 GB and H100 SXM (declared per submission; varies between models)
Tasks measured	6 tasks: chat (text conversation), reasoning, code generation, summarisation, visual question answering, video generation
Normalisation	Mean energy per complete response (prefill + decode). Output J/token is also reported. The task is stated explicitly because it determines the output length
Scope of the measurement	GPU via NVML + CPU/DRAM via RAPL; not a power meter at the wall
Models covered	~40 architectures in the NeurIPS 2025 version; the web leaderboard is updated with more
Licence	Apache 2.0 (zeus: github.com/ml-energy/zeus); MIT (benchmark: github.com/ml-energy/benchmark)
Update frequency	Continuous on the web leaderboard; the paper is a point-in-time snapshot

Zeus as tooling. Zeus is the measurement engine of the ML.ENERGY Leaderboard and also a standalone package (pip install zeus-ml). It supports NVIDIA GPU (NVML), AMD GPU (ROCm), CPU (RAPL), DRAM (RAPL), Apple Silicon and NVIDIA Jetson. The ZeusMonitor adds measurement overhead in single-digit milliseconds. Since May 2025 it has been a PyTorch ecosystem project. MIT licence.

3 · MLPerf Power

The full data sheet is in the C4 article. A summary of the points relevant for comparison with the previous two:

Field	Detail
URL	mlcommons.org/benchmarks/inference-datacenter/
Organisation	MLCommons Power Working Group (>20 orgs)
Unit of measurement	samples/joule (throughput/mean power) = the inverse of J/sample
Hardware	Complete node measured at the wall (AC); Yokogawa WT310E analyser (±0.1 % of reading)
LLM tasks	GPT-J 6B, Llama 2 70B, Mixtral 8×7B (from v5.0)
Granularity	Complete node (GPU + CPU + RAM + fans + PSU losses); no attribution to individual workloads
Node overhead over GPU	25–45 % of total consumption in submissions with a physical analyser
Licence of the corpus	Public results on GitHub (mlcommons/inference_results_vX.Y); PTDaemon requires MLCommons membership

Comparison of the three leaderboards

Dimension	HF AI Energy Score	ML.ENERGY Leaderboard	MLPerf Power
Unit	Wh/1k-queries	Output J/token	samples/J (complete node)
Fixed hardware	H100 (all models)	A100/H100 (varies)	Depends on the submitter
Measurement	CodeCarbon (GPU)	Zeus NVML+RAPL	Physical AC power meter (Yokogawa)
System coverage	GPU only	GPU + CPU + DRAM	Complete node including fans and PSU
Batch size	1	Varies by task	Per LoadGen scenario
Models covered	166+ (text, image, audio)	~40 generative LLMs	Few (GPT-J, Llama 2, Mixtral)
Proprietary models	Yes (audited Docker)	No (OSS only)	Yes (MLCommons members)
External certification	No	No	Yes (SPEC PTDaemon)
Frequency	Point-in-time (v1, v2)	Continuous	Half-yearly (MLPerf rounds)
Licence	Apache 2.0	Apache 2.0 / MIT	Public results; PTDaemon: membership

Incompatibility between leaderboards. The three measure different dimensions: Wh/query ≠ J/token ≠ node samples/J. A direct comparison requires converting units and assuming that the hardware, the engine and the task are equivalent, which is rarely the case across leaderboards.

How energy per token is measured and normalised

The base identity is developed in the C2 article:

$$E_{\text{token}} \,[\text{J/tok}] = \frac{\bar{P} \,[\text{W}]}{\text{throughput} \,[\text{tok/s}]}$$

To compare models against each other, every factor other than the model must be fixed:

Factor	Effect if it varies	How to fix it
Hardware	H100 vs A100 vs L40S changes the result by 2–4×	Declare the exact hardware; compare only within the same HW
Inference engine	vLLM vs Transformers: 25–40 % difference in J/token	Fix the engine and the version
Batch size / concurrency	Batch 1 vs batch 32: throughput rises but so does power; the ratio varies	Declare the batch size; compare within the same regime
Model precision	FP16 vs INT8 vs INT4: up to −79 % energy	Declare the precision; do not mix
Response length	A query with 50 tokens ≠ one with 500	Use a fixed dataset or normalise per token
Measurement window	Including warm-up or idle inflates the numerator	Align the power window with the token window (see C2)

Conversion formula Wh/query ↔ J/token:

$$E_{\text{J/tok}} = \frac{E_{\text{Wh/query}} \times 3600}{\bar{n}_{\text{tokens/query}}}$$

Example: if a model consumes 0.05 Wh/query (= 180 J/query) and generates an average of 200 tokens per query:

$$E_{\text{J/tok}} = \frac{0{.}05 \times 3600}{200} = \frac{180}{200} = 0{.}9 \,\text{J/tok}$$

AI Energy Score data: concrete examples

The v2 data (December 2025, H100 hardware, batch = 1, text generation task with reasoning on/off):

Model	Active params	Reasoning	GPU Wh/1k queries	Stars (text-gen)
DistilGPT-2	82 M	—	1.31	⭐⭐⭐⭐⭐
SmolLM3-3B	3 B	Off	18.35	⭐⭐⭐⭐
SmolLM3-3B	3 B	On	12,791.22	⭐
Phi-4-reasoning-plus	15 B	Off	18.42	⭐⭐⭐⭐
Phi-4-reasoning-plus	15 B	On	9,461.61	⭐
DeepSeek-R1-Distill-Llama-70B	70 B	Off	49.53	⭐⭐⭐
DeepSeek-R1-Distill-Llama-70B	70 B	On	7,626.53	⭐

Source: Hugging Face AI Energy Score v2 (Dec 2025).

The reasoning multiplier. The energy increase when reasoning is switched on ranges from ×154 (DeepSeek-R1-Distill-Llama-70B) to ×697 (SmolLM3-3B). The direct cause: reasoning models generate between 300 and 800 times more tokens than their non-reasoning equivalents (internal chains of thought). The mean across the v2 corpus is ×30 extra energy for reasoning.

Newer models are not always more efficient. Of the 14 comparable models (no reasoning, no MoE, similar size) between the Feb 2025 and Dec 2025 cohorts: 8 out of 14 had equal or higher energy. The range runs from 3 % of the reference model’s energy to almost 2×. Parameter scale is no longer enough to estimate efficiency.

ML.ENERGY Leaderboard data: J/token by family

The data from the arXiv 2505.06371 paper and from leaderboard v3 (A100/H100 hardware, vLLM as the reference engine):

Scale within one family (Llama 3):

Size	Params	Relative J/token (base = 1B)	Params/energy ratio
Llama 3 · 1B	1 B	1.0×	—
Llama 3 · 8B	8 B	~2.1×	8× params → 2.1× energy
Llama 3 · 70B	70 B	~7.3×	70× params → 7.3× energy

The sublinearity (70× params → 7.3× energy, not 70×) reflects that inference energy is dominated by memory bandwidth (memory-bandwidth bound), not by raw FLOPs.

Dense vs MoE:

Model	Type	Total params	Active params/token	Relative J/token
Llama 3 · 8B	Dense	8 B	8 B	1.0×
Mixtral 8×7B	MoE (top-2)	47 B	~13 B	~0.33×
Llama 3 · 70B	Dense	70 B	70 B	~3.5×

The MoE activates only 2 of 8 experts per token. Mixtral 8×7B consumes roughly ⅓ of the J/token of a dense model with 8B active, at a quality comparable to a dense model of larger scale. Routing overhead and holding all the experts in memory cancel out part of the theoretical gain.

Effect of the task (ML.ENERGY v3, same model):

Task	Energy multiplier per response (vs chat)
Chat (text conversation)	1× (reference)
Summarisation	~2–4×
Code generation	~3–6×
Reasoning	~25×
Image + text	1.1–5.2×
Video + text	1.3–15.0×

Reasoning uses ~10× more tokens per response, and the extra memory of the chain of thought reduces the effective batch size, raising energy per token through memory pressure.

Effect of quantisation on energy per token

Data on NVIDIA H100 hardware, Llama 3 family (arXiv 2508.16712 and arXiv 2504.03360):

Precision	Energy reduction vs FP16	Condition
FP16	reference (0 %)	—
BF16	~0 % (iso-energy)	Same hardware and engine
FP8	−25 to −35 %	H100/H200 with native hardware support
INT8	−23 to −44 % (median ~39 %)	Depends on batch size; more at low batches
INT4 (AWQ / GPTQ)	−50 to −79 %	Requires hardware with efficient low-precision support

Warning. On GPUs without native INT4 hardware support (or with suboptimal dequantisation kernels), quantisation can increase latency and energy per token rather than reduce them, because of run-time dequantisation overhead. The benefit of quantisation is real on H100/A100 with TensorRT-LLM or a well-configured llama.cpp, but it is not guaranteed with any engine.

Quantisation and throughput: the memory reduction per model frees VRAM, which allows larger batch sizes. At a larger batch, throughput rises more than power does, cutting J/token further still. The net effect can exceed the direct reduction in energy per operation.

Effect of the inference engine

The engine is a variable that model-level leaderboards tend to fix, but which in production is a decision of your own. Data from published comparisons (vLLM, TensorRT-LLM, naive Transformers, A100):

Engine	Relative J/token vs Transformers baseline
Transformers (naive, not optimised)	1.0× (reference)
vLLM (PagedAttention, continuous batching)	−25 to −35 %
TensorRT-LLM (optimised NVIDIA kernels, FP8)	−35 to −45 %
llama.cpp (hybrid CPU/GPU, INT4)	Variable; −30 to −60 % depending on hardware

Moving from naive Transformers to TensorRT-LLM can cut energy per token by more than moving from a 70B model to an 8B one of the same origin. The choice of engine is a first-order lever on energy efficiency.

Limits of energy leaderboards

Limit	Description
Hardware dependence	A ranking on H100 is not valid on A100 or L40S without correction. The hierarchy of models can change from one piece of hardware to another.
Engine dependence	The results are valid only for the engine they were measured with. A model that is ×2 more efficient on the leaderboard can fall behind if a slower engine is used.
Artificial batch size	AI Energy Score uses batch = 1. In production with aggressive batching, the efficiency relationship between large and small models changes: the large ones scale better with batch.
Training not captured	All the leaderboards measure inference only. The energy cost of training (which can exceed that of inference by 1,000× over the life of the model) is out of scope.
Incompatibility between leaderboards	Wh/query, J/token and samples/J measure different things. Converting between them requires knowing the mean output length, which varies by task and dataset.
Partial system coverage	AI Energy Score and ML.ENERGY measure GPU (+CPU/DRAM with Zeus); they do not capture the overhead of the complete system (PSU losses, fans, interconnect). MLPerf Power does, but covers few models.
Data latency	The leaderboards publish results months after the tests. New hardware (H200, B100, B200) may have no data available at the time of the decision.
No PUE	None of the three includes datacenter PUE. For real TCO, the leaderboard J/token must be multiplied by your own PUE.

Decision table: choosing a model on energy efficiency

The selection criteria in order, without recommendation prose:

Criterion	Question	Action
Task with reasoning	Does the task require step-by-step reasoning?	Yes → multiply the model’s base energy by ×30–700 before comparing. If there is a non-reasoning alternative with sufficient quality, prefer it.
Size vs minimum quality	What is the minimum acceptable quality for the task?	Consult quality benchmarks (see B7 when available). Pick the smallest model that clears the quality threshold; energy grows sublinearly with size.
Dense vs MoE	Does the hardware have enough memory for the full MoE?	If yes: the active-equivalent MoE consumes ~3× fewer J/token than the equivalent dense model. If not: paging or offload eats the gain.
Precision	Does the hardware have native FP8/INT4 support?	H100/H200: native FP8 (−30 %). With TensorRT-LLM: INT4 AWQ (−50 to −79 %). Without native support: stay on FP16 or BF16 until validated with your own benchmark.
Inference engine	Is the optimal engine for the hardware being used?	Measure with C3. If the engine is not optimised, changing engine can cut energy more than changing model.
Consult the leaderboard	Is the task covered by AI Energy Score or ML.ENERGY?	Filter by: same task, same hardware class, reasoning explicitly off/on. Do not compare models from different hardware classes or different engines.
Validate on your own hardware	Are the leaderboard results on the same HW as yours?	Always validate with Zeus or DCGM on your own hardware before taking the final decision. The leaderboard is a reference, not a prediction.

Quick signals table:

Signal	Effect on energy	Data source
Switching reasoning on	×30–700	AI Energy Score v2
Going from dense 8B to dense 70B	~×3.5	ML.ENERGY Leaderboard v3
Going from dense 8B to active-equivalent 8B MoE	~×0.33 (−67 %)	ML.ENERGY v3
FP16 → INT4 (compatible hardware)	−50 to −79 %	arXiv 2508.16712, 2504.03360
Naive Transformers → TensorRT-LLM FP8	−35 to −45 %	TokenPowerBench, ML.ENERGY
PUE 1.0 → PUE 1.5	+50 % on real datacenter energy	MLPerf Power (scope)

Reference data: energy in a generic node (4×H100 SXM)

Generic example hardware, to anchor the leaderboard values to a real node:

Parameter	Indicative value
TDP 4×H100 SXM 80 GB	4 × 700 W = 2,800 W (GPU only)
Complete-node system power (at the wall)	~3,500–5,000 W depending on load
Non-GPU overhead over GPU	25–45 %
J/token Llama 3 70B FP16, vLLM, batch 8	~1–3 J/tok (indicative, A100/H100)
J/token Llama 3 8B FP16, vLLM, batch 8	~0.3–0.7 J/tok (indicative)
J/token Mixtral 8×7B FP16, vLLM, batch 8	~0.4–0.8 J/tok (indicative)
Energy per 1M tokens (Llama 3 70B, PUE 1.4)	~0.5–1.2 kWh

The J/token values are indicative and depend heavily on batch size, prompt length, prefill/decode ratio and engine version. For certified values, consult the MLPerf Power submissions (mlcommons.org).

For the alternative reference node (4×A100 PCIe 80 GB, TDP ~300 W each):

Parameter	Indicative value
TDP 4×A100 PCIe	4 × 300 W = 1,200 W (GPU only)
Complete-node system power	~1,500–2,000 W
J/token Llama 3 70B FP16, vLLM	~2–5 J/tok (indicative; higher because of lower HBM bandwidth vs SXM)

How to use the leaderboards in practice

A decision flow based on the public data available:

Step	Action	Resource
1	Identify the dominant task of the workload	—
2	Consult AI Energy Score filtered by task and hardware class	huggingface.co/spaces/AIEnergyScore/Leaderboard
3	Note the models with ⭐⭐⭐⭐ or ⭐⭐⭐⭐⭐ on the task	Wh/1k-queries as a relative reference
4	Cross-check with ML.ENERGY for the J/token of each candidate	ml.energy/leaderboard
5	If any model is in MLPerf Power (Llama 2, GPT-J, Mixtral), consult the certified samples/J	mlcommons.org/benchmarks/inference-datacenter/
6	Select the 2–3 candidates with the best energy/quality ratio	—
7	Measure on your own hardware with Zeus or DCGM	github.com/ml-energy/zeus
8	Multiply the measured J/token by the datacenter PUE	J/token × PUE = effective J/token in the datacenter
9	Work out the electricity cost per token at the contracted price	See C2

Sources

Hugging Face · AI Energy Score · organización y leaderboard — https://huggingface.co/AIEnergyScore
Hugging Face · Announcing AI Energy Score Ratings (Luccioni et al., feb. 2025) — https://huggingface.co/blog/sasha/announcing-ai-energy-score
Hugging Face · AI Energy Score v2: Refreshed Leaderboard, now with Reasoning (Luccioni, Gamazaychikov, dic. 2025) — https://huggingface.co/blog/sasha/ai-energy-score-v2
Hugging Face · AIEnergyScore GitHub (Apache 2.0) — https://github.com/huggingface/AIEnergyScore
ML.ENERGY Initiative · Leaderboard — https://ml.energy/leaderboard
ML.ENERGY Initiative · Blog: Diagnosing Inference Energy Consumption with the ML.ENERGY Leaderboard v3.0 (dic. 2025) — https://ml.energy/blog/measurement/energy/diagnosing-inference-energy-consumption-with-the-mlenergy-leaderboard-v30/
arXiv 2505.06371 · The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization (Chung et al., NeurIPS 2025 D&B Spotlight) — https://arxiv.org/abs/2505.06371
ML.ENERGY Initiative · Zeus: Deep Learning Energy Measurement and Optimization — https://ml.energy/zeus/
GitHub ml-energy/zeus (MIT) — https://github.com/ml-energy/zeus
PyTorch Blog · Zeus: Deep Learning Energy Measurement and Optimization — https://pytorch.org/blog/zeus/
University of Michigan CSE · Power-hungry AI: Researchers evaluate energy consumption across models — https://cse.engin.umich.edu/stories/power-hungry-ai-researchers-evaluate-energy-consumption-across-models
arXiv 2512.03024 · TokenPowerBench: Benchmarking the Power Consumption of LLM Inference (dic. 2024) — https://arxiv.org/abs/2512.03024
arXiv 2508.16712 · Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective — https://arxiv.org/abs/2508.16712
arXiv 2504.03360 · Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency — https://arxiv.org/abs/2504.03360
Epoch AI · AI Energy Use: Data & Research — https://epoch.ai/topics/energy
MLCommons · MLPerf Inference Datacenter benchmark results — https://mlcommons.org/benchmarks/inference-datacenter/
arXiv 2410.12032 · MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems (Tschand et al., 2024) — https://arxiv.org/abs/2410.12032
Coalition for Sustainable AI · AI Energy Score as best practice in benchmarking — https://www.sustainableaicoalition.org/ai-energy-score-a-standardized-approach-to-evaluating-ai-model-energy-efficiency/

MLPerf Power: the standard energy-efficiency benchmark for on-premise ML systems

Mon, 15 Jun 2026 06:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

MLPerf Power is the only standardised benchmark for measuring the energy efficiency of complete ML systems. It is run by the MLCommons Power Working Group and backed by more than 20 organisations. It measures power at the wall (AC wall power) for the whole System Under Test, GPUs, CPUs, memory, interconnect and fans included, with a SPEC PTDaemon certified power analyser (Yokogawa WT310/WT5000) and the public tooling in mlcommons/power-dev. The central metric for datacenter inference is samples/joule (or tokens/joule for LLMs); for latency, energy per stream. The public corpus adds up to 1,841 reproducible measurements from 60 systems (590 datacenter, 792 edge, 447 tiny, 12 training). GPT-J and Llama 2 show improvements of more than 100× in samples/joule between the first and the latest available rounds. The direct comparison with the software-based measurement of post C3 (DCGM/NVML/RAPL/Kepler) shows that MLPerf Power offers maximum accuracy and reproducibility at the cost of dedicated hardware (~3,000 USD for the Yokogawa 310E) and of the requirement to be an MLCommons member to access PTDaemon; software measurement is continuous, needs no extra hardware, but carries a wider error bar and no external validation.

What MLPerf Power is and how it fits into the MLCommons ecosystem

MLCommons is the consortium of more than 100 organisations (NVIDIA, Google, Intel, Dell, AMD, Meta and others) that maintains the MLPerf benchmarks: Training, Inference (Datacenter and Edge), Tiny, HPC, Storage, Client and Automotive. The Power Working Group is the specific group that extends each of those benchmarks by adding the energy dimension (MLCommons Power Working Group).

MLPerf Power is not a standalone benchmark: it is an energy measurement layer overlaid on the existing performance benchmarks. For a submission to be valid with power, it must first satisfy the performance rules (MLPerf Inference, Training or Tiny) and then the additional power measurement rules.

Coupling with MLPerf Inference

The integration most relevant to on-premise inference is with MLPerf Inference Datacenter, which defines:

Scenarios: Server (latency, queries per second with a request queue) and Offline (pure throughput, batch with no latency constraint). Power is measured in both.
Benchmarks: ResNet-50, BERT, RNN-T, 3D-UNET, RetinaNet, DLRM, GPT-J 6B, Llama 2 70B (from v4.0), Mixtral 8×7B (from v5.0).
Divisions: closed (model and preprocessing fixed, only runtime optimisation allowed) and open (modifications to the model permitted).

Power is measured during the performance phase, not during accuracy or compliance. The same run that produces the performance log produces the power log: reporting the highest performance of three runs and the lowest power of another three is forbidden (MLPerf Inference Power Measurement Rules, §5.9).

Coupling with MLPerf Training and Tiny

MLPerf Training: large-scale power measurement (multi-node, 10K+ GPUs) does not use an external analyser, which would be impracticable. Node telemetry (IPMI/Redfish) plus an estimate for the interconnect network is used instead. The metric is energy to train (J or kWh). In the v4.0 submissions with power, the systems running Llama 2 70B Training range from single nodes to hundreds, revealing the non-linear scaling of energy (arXiv 2410.12032).
MLPerf Tiny: microcontroller systems (from 5.64 mW). Specialised micro-power instrumentation is used, with hardware pins to demarcate the start and end of the inference. The metric is energy per inference (J), not samples/joule.

Measurement methodology: certified wall power

The fundamental principle: the whole SUT, at the wall

Rule number one is absolute: power must be measured at system level, that is, including every component the benchmark activates: host processors, accelerators, memory, disks, fans, internal interconnect. Measuring the GPUs alone is not enough (MLPerf Inference Power Measurement Rules, §5.1):

“The power consumption must be measured at the system level, i.e. including all components that are sensitized by LoadGen e.g. the host processor on which LoadGen runs, accelerators, memory, fans, etc.”

Power is measured in AC (alternating current), at the SUT input, upstream of the PSUs. Any battery or energy storage between the mains socket and the system PSUs is forbidden.

Measurement hardware: SPEC-certified power analyser

MLPerf Power requires a power analyser certified by SPEC PTDaemon (official SPEC list). The most widespread in submissions is the Yokogawa WT310E (~3,000 USD), which connects to the director over USB (Linux) or Ethernet/serial (Windows). Voltage is measured in parallel and current in series with the SUT power line.

Relevant specifications of the Yokogawa WT310E:

Power accuracy: 0.1 % of reading + 0.1 % of range
Measurement range: µW to MW (the WT5000 reaches industrial installations)
Update rate: from 50 ms

For systems with more than one channel, or multi-PSU nodes, multi-analyser configurations are allowed. The ranging rule: first a ranging run is performed with the range in Auto mode to determine the maximum current and voltage values; the testing runs use fixed ranges based on those peaks, which maximises accuracy within the range. Auto mode is not permitted in the testing runs.

SPEC PTDaemon: the daemon that orchestrates the measurement

PTDaemon (Power Thermal Daemon) is the SPEC tool that handles communication with the analyser. MLCommons is licensed to use it within the MLPerf Power flow. Access requires MLCommons membership and signing the corresponding EULA (MLPerf Power FAQ).

The measurement flow is as follows:

The process step by step (mlcommons/power-dev):

NTP sync between Director and SUT to align timestamps.
Ranging run: power with ranges in Auto; the analyser determines the current and voltage peaks.
Testing run: fixed ranges; LoadGen runs the benchmark; the Director records power with timestamps; the SUT records the performance log with start/end timestamps of the execution phase.
Post-processing: the result summariser cross-references power log and performance log by timestamp to compute the mean power over the execution window.
A minimum of 60 seconds of valid power data. If the workload finishes before 60 s, it is run in a loop until that threshold is reached.

What is and is not inside the SUT

The SUT includes everything the benchmark activates:

Inside the SUT	Outside the SUT
GPUs / accelerators	PDU shared with other systems
Host CPU and RAM	Datacenter cooling infrastructure (PUE)
Disks / NVMe if the benchmark uses them	Management network (BMC, out of band)
Node fans and cooling systems	Datacenter network switches (not the node’s)
Internal interconnect (NVLink, PCIe)
Node PSUs

The paper makes it explicit: PUE is out of scope for MLPerf Power by design (arXiv 2410.12032, §III-C). MLPerf measures the efficiency of the ML system, not that of the datacenter. Including PUE would obscure the differences between systems by mixing hardware efficiency with building efficiency.

Metrics: formulas and definitions

Throughput benchmarks (Datacenter / Offline / Edge)

For throughput benchmarks, Offline and Server in datacenter plus some edge scenarios, the energy efficiency metric is:

$$\eta = \frac{\text{throughput (samples/s)}}{\text{mean power (W)}} \quad \Rightarrow \quad \left[\frac{\text{samples}}{\text{J}}\right]$$

where mean power is computed over the window of the execution phase of the performance run. The reciprocal gives energy per sample:

$$E_{\text{sample}} = \frac{\text{mean power (W)}}{\text{throughput (samples/s)}} \quad \left[\text{J/sample}\right]$$

For LLM benchmarks (GPT-J, Llama 2), where the output varies in length, a “sample” is one complete query (prompt + response). Energy per output token is obtained by dividing by the number of tokens generated, which varies per query and must be reported or estimated.

Latency benchmarks (Single Stream / Tiny)

For latency benchmarks, Single Stream in edge and tiny, where the processing time is fixed, the metric is the inverse of energy per inference:

$$\eta_{\text{latency}} = \frac{1}{E_{\text{inference}}} \quad \left[\frac{1}{\text{J}}\right]$$

The paper treats both metrics (samples/J and 1/J) as comparable within their category, although they are not interchangeable between categories.

System power and run energy

The three quantities that appear in every power submission:

Quantity	Definition	Unit
System power	mean of the AC power samples over the execution window	W
Run energy	mean power × window duration	J
Energy efficiency	throughput / system power = samples/J	samples/J

The run energy formula:

$$E_{\text{run}} = \bar{P} \times \Delta t = \frac{\sum_{i} P_i \cdot \Delta t_i}{\Delta t_{\text{total}}} \times \Delta t_{\text{total}} \quad [\text{J}]$$

where $ P_i $ are the analyser samples and $ \Delta t_i $ the intervals between samples, both inside the window demarcated by the LoadGen timestamps.

How to read an MLPerf Power submission

Divisions and availability categories

Submissions inherit the divisions of MLPerf Inference:

Division	Constraints	Comparability
Closed	Fixed model, fixed preprocessing, only the runtime and the hardware are optimised	High: submissions directly comparable with each other
Open	Modifications to the model allowed (aggressive quantisation, distillation, pruning)	Low: each system may use a different model

Within closed, there are system availability categories:

Category	Definition
Available	System commercially available at the closing date
Preview	Announced but not commercially available
RDI (Research, Development, Internal)	Systems for internal or experimental use

Only Available systems in closed are comparable without caveats. A Preview system with better energy than an Available one from another company is not a valid comparison for purchasing decisions.

What comparability MLPerf Power gives and what it does not

It gives:

An objective comparison of the energy efficiency of complete nodes (not just the GPU) under a standardised ML workload.
Verified reproducibility: the logs are published on GitHub and anyone can review them.
Trends over time between rounds: how much the efficiency of each hardware family improves version to version.
A basis for comparing heterogeneous systems running the same workload under the same rules.

It does not give:

An isolated GPU-to-GPU comparison (it measures the complete node, not the card alone).
Broad hardware coverage: the number of submissions with power is small. In v4.0 only four companies (Dell, Fujitsu, NVIDIA, Qualcomm) delivered power numbers for datacenter. In v5.1 there were two power submissions (Lenovo datacenter + GATEOverflow edge).
Representativeness of your own workloads: the benchmarks are fixed and may not match the distribution of your real load (batch size, prompt length, prefill/decode ratio).
Datacenter cooling data: PUE, liquid vs air, are out of scope.
Real-time comparison: results are published months after the closing date.

Structure of a submission

Every power submission publishes the following in the MLCommons results repository:

<division>/<submitter>/measurements/<system>/
├── analyzer_table.md # Analyser configuration (model, ranges)
├── power_settings.md # Power management configuration of the SUT
results/<system>/<benchmark>/<scenario>/
├── mlperf_log_summary.txt # Performance results (LoadGen)
├── spl.txt # Power log (power, current, voltage, timestamps)
└── ...

The spl.txt files contain the raw analyser readings with timestamps, which makes it possible to verify alignment with the LoadGen window.

Data: scale, trends and reference hardware

The corpus: 1,841 measurements from 60 systems

The MLPerf Power submissions database covers (arXiv 2410.12032):

Category	Submissions	Power range
Datacenter (Inference)	590	~200 W – ~10 kW per node
Edge (Inference)	792	~10 W – several kW
Tiny (Inference)	447	5.64 mW – hundreds of mW
Training / HPC	12	up to 500 kW (measured); ~10 MW estimated in HPC

The total range runs from microwatts to megawatts, 9 orders of magnitude, which means no single methodology applies across every segment.

Energy efficiency improvements: LLMs lead

The evolution of the samples/joule metric between rounds, normalised to the first submission of each model (arXiv 2410.12032, §V-A):

Workload	Category	Cumulative improvement (samples/J)
GPT-J 6B	Datacenter	>100×
Llama 2 70B	Datacenter	>100×
RetinaNet	Datacenter	Largest among the classic models
BERT-99.0	Edge	~4×
RNN-T	Edge	~4×
ResNet-50	Edge	~1.5×
ResNet-50	Tiny	>1,000×
Other Tiny	Tiny	79× – 596×

The >100× improvements in datacenter LLMs reflect the massive industry attention on software optimisation (FP8 kernels, FlashAttention, speculation) and hardware (tensor core architectures, HBM3). Tiny models show even larger gains in relative terms thanks to the low starting point and the progress in specialised chips.

On-premise reference hardware

For a node of 4×H100 SXM 80 GB (the track’s generic hardware), the typical values in MLPerf Power:

Parameter	Indicative value
Declared TDP 4×H100 SXM	4 × 700 W = 2,800 W (GPU only)
System power (complete node) measured at the wall	~3,500 – 5,000 W depending on load
CPU + RAM + fans overhead over GPU	25 – 45 % of the total
Efficiency: Llama 2 70B offline (closed)	a function of the runtime; order of magnitude ~0.1 – 1 tokens/J

The key point: the “node overhead”, the difference between what the analyser measures at the wall and what NVML reports for the GPUs, is 25–45 % of total consumption. A GPU-only measurement underestimates the real system consumption by that margin.

For the A100 PCIe 80 GB (4× per node), system consumption is lower (GPU TDP ~300 W each, node ~1,500–2,000 W), with better efficiency per watt but lower absolute throughput. The L40S (4× per node) has an intermediate profile: TDP ~350 W each, good FP8 inference performance, node consumption ~1,800–2,500 W.

Contrast with the software measurement of post C3

Post C3 of this track covers the DCGM/NVML/RAPL/Kepler stack in production. Here is the data-driven comparison against MLPerf Power:

Dimension	Software (DCGM/NVML/RAPL/Kepler)	MLPerf Power (analyser + PTDaemon)
What it measures	On-device sensors (GPU, CPU, DRAM)	AC wall power (the whole node)
Coverage	GPU + CPU + DRAM (≠ the wall)	Complete node including fans, PSU overhead
Node overhead	Not captured by default	Included in the measurement
Accuracy	±5–15 % vs power meter (Kepler estimate); NVML ~3–5 % on the GPU	±0.1 % reading + ±0.1 % range (Yokogawa WT310E)
Frequency / continuity	Continuous in production (sub-second)	Only during the benchmark run
Extra hardware	None (uses system sensors)	SPEC-certified analyser (~3,000 USD)
Attribution per workload	Per pod/MIG with Kepler	No: it measures the system, not the individual workload
External certification	No	Yes (SPEC PTDaemon + MLCommons review)
Data access	Real time in Prometheus	Public logs on GitHub after the round
Recommended use	Continuous monitoring in production	Certified comparative benchmarking

The operational conclusion: the two approaches are complementary, not competing. The DCGM/Kepler stack gives the continuous per-pod view for production; MLPerf Power gives the certified ground truth for hardware comparisons with reproducibility guarantees.

The myth of measuring only the GPU: data from the paper

The arXiv 2410.12032 paper devotes section §III-C to dismantling the myths of power measurement in ML systems. The one most relevant to on-premise inference is Myth 1:

“A common misconception is that measuring the power consumption of specific ML components, such as accelerators or GPUs, is adequate to assess system efficiency. In reality, overall system power consumption is crucial. Different components are active at various stages of ML workloads with varying duty cycles.”

The data from submissions with a wall analyser versus NVML readings confirms that the overhead of the node’s non-GPU components (CPU, memory, disks, fans, PSU losses) ranges between 25 and 45 % of total consumption depending on the system. An efficiency comparison based on NVML readings alone systematically underestimates consumption and can distort the ranking between systems with different CPU/GPU proportions.

Myth 2 (TDP and PSU as power proxies) is also relevant: the TDP of an H100 SXM is 700 W per card, but real consumption in inference with Llama 2 70B depends on the load, the prompt length and the batch size. The MLPerf Power data shows that systems routinely run well below TDP in online inference scenarios (server scenario), where latency limits throughput and the GPU is not at 100 % utilisation.

Limitations of the current submissions

The limitations are not of the method but of the coverage:

Few submissions with power in datacenter. In v4.0 only 4 companies delivered power numbers for datacenter; in v5.1, one (Lenovo). Performance benchmarking has dozens of submitters; energy benchmarking has few. This limits the representativeness of the corpus for comparing on-premise hardware.
Heterogeneous SUTs. Each submitter defines its own SUT: number of GPUs, server, interconnect, software stack. A Dell system with 4×H100 is not directly comparable to a Supermicro one with 8×H100, even if both use the same model. Platform differences (NVLink vs PCIe, server vs rack) are captured but not separated out.
Fixed workloads. The MLPerf benchmarks are representative but they are not your load. The distribution of prompt lengths, the prefill/decode ratio and your users’ batch size may differ significantly from the MLPerf synthetic datasets. The benchmark’s samples/J result is a reference, not a prediction of your workload.
Publication latency. Results are published months after the submission deadline. New hardware (H200, B100, B200) may have no power results available at the moment of the purchasing decision.
No PUE. MLPerf Power does not include datacenter efficiency. Two identical systems in two datacenters with PUE 1.1 and 1.5 give the same MLPerf Power result but a very different electricity cost. For TCO, the MLPerf J/sample must be multiplied by your own datacenter PUE.

Usage flow for an on-premise platform

For anyone who wants to use MLPerf Power as a purchasing reference or as their own benchmark, the practical flow:

Step	Action	Resource
1	Consult the publicly available submissions	mlcommons.org/benchmarks/inference-datacenter
2	Filter by model (Llama 2 70B, GPT-J), scenario (Offline/Server) and division (Closed/Available)	GitHub mlcommons/inference_results_vX.Y
3	Compare samples/J between systems with similar hardware	power log + performance summary
4	Adjust for your own PUE	samples/J × (1/PUE) for real energy in the datacenter
5	If you want to measure your own hardware: buy a Yokogawa WT310E, join MLCommons, get access to PTDaemon	docs.mlcommons.org/inference/power
6	Calibrate the software stack (DCGM/Kepler) against the analyser once	see post C3

Cross-links of the energy track

This article is C4 of the energy pillar. The related articles:

C1 — State of the art: energy benchmarking of LLM frameworks: inventory of tools and overview of the field.
C2 — Energy per token: methodology and the Spanish electricity market: the J/token identity and how the electricity price multiplies it.
C3 — Measurement tooling in production: Kepler, DCGM and a practical stack: the continuous stack for production that MLPerf Power complements.
Benchmarking track — state of the art of frameworks: context of the performance benchmarking that MLPerf Power extends into energy.

Sources

MLCommons Power Working Group — https://mlcommons.org/working-groups/benchmarks/power/
arXiv 2410.12032 · MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI (Tschand et al., 2024) — https://arxiv.org/abs/2410.12032
MLPerf Inference Power Measurement Rules v2.0 (power_measurement.adoc) — https://github.com/mlcommons/inference_policies/blob/master/power_measurement.adoc
MLCommons power-dev · repositorio público de herramientas de medición — https://github.com/mlcommons/power-dev
MLPerf Inference Power Measurement Documentation (MLCFlow) — https://docs.mlcommons.org/inference/power/
SPEC PTDaemon · lista de dispositivos certificados — https://open.spec.org/power/docs/specpower-device_list/
MLCommons · MLPerf Inference v1.0 con las primeras mediciones de potencia (abril 2021) — https://mlcommons.org/2021/04/mlperf-inference-v1-0-results-with-first-power-measurements/
MLCommons · MLPerf Inference v4.1 results (agosto 2024) — https://mlcommons.org/2024/08/mlperf-inference-v4-1-results/
MLCommons · MLPerf Inference v5.1 results (septiembre 2025) — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
MLCommons · MLPerf Power benchmark presentado en IEEE HPCA 2025 — https://mlcommons.org/2025/03/ml-commons-power-hpca/
SPEC Updates PTDaemon Interface (GlobeNewswire, febrero 2024) — https://www.globenewswire.com/news-release/2024/02/22/2833367/0/en/SPEC-Updates-PTDaemon-Interface-to-Meet-Evolving-Industry-Requirements.html
Yokogawa WT310E Power Analyzer — https://tmi.yokogawa.com/us/solutions/products/power-analyzers/

Energy-Efficiency on lo0 — Tech Blog

LLM energy leaderboards: comparing models by Wh/token and choosing on efficiency

TL;DR

Track context

The three leaderboards: technical data sheet

1 · Hugging Face AI Energy Score

2 · ML.ENERGY Leaderboard

3 · MLPerf Power

Comparison of the three leaderboards

How energy per token is measured and normalised

AI Energy Score data: concrete examples

ML.ENERGY Leaderboard data: J/token by family

Effect of quantisation on energy per token

Effect of the inference engine

Limits of energy leaderboards

Decision table: choosing a model on energy efficiency

Reference data: energy in a generic node (4×H100 SXM)

How to use the leaderboards in practice

See also

Sources

MLPerf Power: the standard energy-efficiency benchmark for on-premise ML systems

TL;DR

What MLPerf Power is and how it fits into the MLCommons ecosystem

Coupling with MLPerf Inference

Coupling with MLPerf Training and Tiny

Measurement methodology: certified wall power

The fundamental principle: the whole SUT, at the wall

Measurement hardware: SPEC-certified power analyser

SPEC PTDaemon: the daemon that orchestrates the measurement

What is and is not inside the SUT

Metrics: formulas and definitions

Throughput benchmarks (Datacenter / Offline / Edge)

Latency benchmarks (Single Stream / Tiny)

System power and run energy

How to read an MLPerf Power submission

Divisions and availability categories

What comparability MLPerf Power gives and what it does not

Structure of a submission

Data: scale, trends and reference hardware

The corpus: 1,841 measurements from 60 systems

Energy efficiency improvements: LLMs lead

On-premise reference hardware

Contrast with the software measurement of post C3

The myth of measuring only the GPU: data from the paper

Limitations of the current submissions

Usage flow for an on-premise platform

Cross-links of the energy track

See also

Sources