Mlperf-Power on lo0 — Tech Blog

MLPerf Power: the standard energy-efficiency benchmark for on-premise ML systems

Mon, 15 Jun 2026 06:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

MLPerf Power is the only standardised benchmark for measuring the energy efficiency of complete ML systems. It is run by the MLCommons Power Working Group and backed by more than 20 organisations. It measures power at the wall (AC wall power) for the whole System Under Test, GPUs, CPUs, memory, interconnect and fans included, with a SPEC PTDaemon certified power analyser (Yokogawa WT310/WT5000) and the public tooling in mlcommons/power-dev. The central metric for datacenter inference is samples/joule (or tokens/joule for LLMs); for latency, energy per stream. The public corpus adds up to 1,841 reproducible measurements from 60 systems (590 datacenter, 792 edge, 447 tiny, 12 training). GPT-J and Llama 2 show improvements of more than 100× in samples/joule between the first and the latest available rounds. The direct comparison with the software-based measurement of post C3 (DCGM/NVML/RAPL/Kepler) shows that MLPerf Power offers maximum accuracy and reproducibility at the cost of dedicated hardware (~3,000 USD for the Yokogawa 310E) and of the requirement to be an MLCommons member to access PTDaemon; software measurement is continuous, needs no extra hardware, but carries a wider error bar and no external validation.

What MLPerf Power is and how it fits into the MLCommons ecosystem

MLCommons is the consortium of more than 100 organisations (NVIDIA, Google, Intel, Dell, AMD, Meta and others) that maintains the MLPerf benchmarks: Training, Inference (Datacenter and Edge), Tiny, HPC, Storage, Client and Automotive. The Power Working Group is the specific group that extends each of those benchmarks by adding the energy dimension (MLCommons Power Working Group).

MLPerf Power is not a standalone benchmark: it is an energy measurement layer overlaid on the existing performance benchmarks. For a submission to be valid with power, it must first satisfy the performance rules (MLPerf Inference, Training or Tiny) and then the additional power measurement rules.

Coupling with MLPerf Inference

The integration most relevant to on-premise inference is with MLPerf Inference Datacenter, which defines:

Scenarios: Server (latency, queries per second with a request queue) and Offline (pure throughput, batch with no latency constraint). Power is measured in both.
Benchmarks: ResNet-50, BERT, RNN-T, 3D-UNET, RetinaNet, DLRM, GPT-J 6B, Llama 2 70B (from v4.0), Mixtral 8×7B (from v5.0).
Divisions: closed (model and preprocessing fixed, only runtime optimisation allowed) and open (modifications to the model permitted).

Power is measured during the performance phase, not during accuracy or compliance. The same run that produces the performance log produces the power log: reporting the highest performance of three runs and the lowest power of another three is forbidden (MLPerf Inference Power Measurement Rules, §5.9).

Coupling with MLPerf Training and Tiny

MLPerf Training: large-scale power measurement (multi-node, 10K+ GPUs) does not use an external analyser, which would be impracticable. Node telemetry (IPMI/Redfish) plus an estimate for the interconnect network is used instead. The metric is energy to train (J or kWh). In the v4.0 submissions with power, the systems running Llama 2 70B Training range from single nodes to hundreds, revealing the non-linear scaling of energy (arXiv 2410.12032).
MLPerf Tiny: microcontroller systems (from 5.64 mW). Specialised micro-power instrumentation is used, with hardware pins to demarcate the start and end of the inference. The metric is energy per inference (J), not samples/joule.

Measurement methodology: certified wall power

The fundamental principle: the whole SUT, at the wall

Rule number one is absolute: power must be measured at system level, that is, including every component the benchmark activates: host processors, accelerators, memory, disks, fans, internal interconnect. Measuring the GPUs alone is not enough (MLPerf Inference Power Measurement Rules, §5.1):

“The power consumption must be measured at the system level, i.e. including all components that are sensitized by LoadGen e.g. the host processor on which LoadGen runs, accelerators, memory, fans, etc.”

Power is measured in AC (alternating current), at the SUT input, upstream of the PSUs. Any battery or energy storage between the mains socket and the system PSUs is forbidden.

Measurement hardware: SPEC-certified power analyser

MLPerf Power requires a power analyser certified by SPEC PTDaemon (official SPEC list). The most widespread in submissions is the Yokogawa WT310E (~3,000 USD), which connects to the director over USB (Linux) or Ethernet/serial (Windows). Voltage is measured in parallel and current in series with the SUT power line.

Relevant specifications of the Yokogawa WT310E:

Power accuracy: 0.1 % of reading + 0.1 % of range
Measurement range: µW to MW (the WT5000 reaches industrial installations)
Update rate: from 50 ms

For systems with more than one channel, or multi-PSU nodes, multi-analyser configurations are allowed. The ranging rule: first a ranging run is performed with the range in Auto mode to determine the maximum current and voltage values; the testing runs use fixed ranges based on those peaks, which maximises accuracy within the range. Auto mode is not permitted in the testing runs.

SPEC PTDaemon: the daemon that orchestrates the measurement

PTDaemon (Power Thermal Daemon) is the SPEC tool that handles communication with the analyser. MLCommons is licensed to use it within the MLPerf Power flow. Access requires MLCommons membership and signing the corresponding EULA (MLPerf Power FAQ).

The measurement flow is as follows:

The process step by step (mlcommons/power-dev):

NTP sync between Director and SUT to align timestamps.
Ranging run: power with ranges in Auto; the analyser determines the current and voltage peaks.
Testing run: fixed ranges; LoadGen runs the benchmark; the Director records power with timestamps; the SUT records the performance log with start/end timestamps of the execution phase.
Post-processing: the result summariser cross-references power log and performance log by timestamp to compute the mean power over the execution window.
A minimum of 60 seconds of valid power data. If the workload finishes before 60 s, it is run in a loop until that threshold is reached.

What is and is not inside the SUT

The SUT includes everything the benchmark activates:

Inside the SUT	Outside the SUT
GPUs / accelerators	PDU shared with other systems
Host CPU and RAM	Datacenter cooling infrastructure (PUE)
Disks / NVMe if the benchmark uses them	Management network (BMC, out of band)
Node fans and cooling systems	Datacenter network switches (not the node’s)
Internal interconnect (NVLink, PCIe)
Node PSUs

The paper makes it explicit: PUE is out of scope for MLPerf Power by design (arXiv 2410.12032, §III-C). MLPerf measures the efficiency of the ML system, not that of the datacenter. Including PUE would obscure the differences between systems by mixing hardware efficiency with building efficiency.

Metrics: formulas and definitions

Throughput benchmarks (Datacenter / Offline / Edge)

For throughput benchmarks, Offline and Server in datacenter plus some edge scenarios, the energy efficiency metric is:

$$\eta = \frac{\text{throughput (samples/s)}}{\text{mean power (W)}} \quad \Rightarrow \quad \left[\frac{\text{samples}}{\text{J}}\right]$$

where mean power is computed over the window of the execution phase of the performance run. The reciprocal gives energy per sample:

$$E_{\text{sample}} = \frac{\text{mean power (W)}}{\text{throughput (samples/s)}} \quad \left[\text{J/sample}\right]$$

For LLM benchmarks (GPT-J, Llama 2), where the output varies in length, a “sample” is one complete query (prompt + response). Energy per output token is obtained by dividing by the number of tokens generated, which varies per query and must be reported or estimated.

Latency benchmarks (Single Stream / Tiny)

For latency benchmarks, Single Stream in edge and tiny, where the processing time is fixed, the metric is the inverse of energy per inference:

$$\eta_{\text{latency}} = \frac{1}{E_{\text{inference}}} \quad \left[\frac{1}{\text{J}}\right]$$

The paper treats both metrics (samples/J and 1/J) as comparable within their category, although they are not interchangeable between categories.

System power and run energy

The three quantities that appear in every power submission:

Quantity	Definition	Unit
System power	mean of the AC power samples over the execution window	W
Run energy	mean power × window duration	J
Energy efficiency	throughput / system power = samples/J	samples/J

The run energy formula:

$$E_{\text{run}} = \bar{P} \times \Delta t = \frac{\sum_{i} P_i \cdot \Delta t_i}{\Delta t_{\text{total}}} \times \Delta t_{\text{total}} \quad [\text{J}]$$

where $ P_i $ are the analyser samples and $ \Delta t_i $ the intervals between samples, both inside the window demarcated by the LoadGen timestamps.

How to read an MLPerf Power submission

Divisions and availability categories

Submissions inherit the divisions of MLPerf Inference:

Division	Constraints	Comparability
Closed	Fixed model, fixed preprocessing, only the runtime and the hardware are optimised	High: submissions directly comparable with each other
Open	Modifications to the model allowed (aggressive quantisation, distillation, pruning)	Low: each system may use a different model

Within closed, there are system availability categories:

Category	Definition
Available	System commercially available at the closing date
Preview	Announced but not commercially available
RDI (Research, Development, Internal)	Systems for internal or experimental use

Only Available systems in closed are comparable without caveats. A Preview system with better energy than an Available one from another company is not a valid comparison for purchasing decisions.

What comparability MLPerf Power gives and what it does not

It gives:

An objective comparison of the energy efficiency of complete nodes (not just the GPU) under a standardised ML workload.
Verified reproducibility: the logs are published on GitHub and anyone can review them.
Trends over time between rounds: how much the efficiency of each hardware family improves version to version.
A basis for comparing heterogeneous systems running the same workload under the same rules.

It does not give:

An isolated GPU-to-GPU comparison (it measures the complete node, not the card alone).
Broad hardware coverage: the number of submissions with power is small. In v4.0 only four companies (Dell, Fujitsu, NVIDIA, Qualcomm) delivered power numbers for datacenter. In v5.1 there were two power submissions (Lenovo datacenter + GATEOverflow edge).
Representativeness of your own workloads: the benchmarks are fixed and may not match the distribution of your real load (batch size, prompt length, prefill/decode ratio).
Datacenter cooling data: PUE, liquid vs air, are out of scope.
Real-time comparison: results are published months after the closing date.

Structure of a submission

Every power submission publishes the following in the MLCommons results repository:

<division>/<submitter>/measurements/<system>/
├── analyzer_table.md # Analyser configuration (model, ranges)
├── power_settings.md # Power management configuration of the SUT
results/<system>/<benchmark>/<scenario>/
├── mlperf_log_summary.txt # Performance results (LoadGen)
├── spl.txt # Power log (power, current, voltage, timestamps)
└── ...

The spl.txt files contain the raw analyser readings with timestamps, which makes it possible to verify alignment with the LoadGen window.

Data: scale, trends and reference hardware

The corpus: 1,841 measurements from 60 systems

The MLPerf Power submissions database covers (arXiv 2410.12032):

Category	Submissions	Power range
Datacenter (Inference)	590	~200 W – ~10 kW per node
Edge (Inference)	792	~10 W – several kW
Tiny (Inference)	447	5.64 mW – hundreds of mW
Training / HPC	12	up to 500 kW (measured); ~10 MW estimated in HPC

The total range runs from microwatts to megawatts, 9 orders of magnitude, which means no single methodology applies across every segment.

Energy efficiency improvements: LLMs lead

The evolution of the samples/joule metric between rounds, normalised to the first submission of each model (arXiv 2410.12032, §V-A):

Workload	Category	Cumulative improvement (samples/J)
GPT-J 6B	Datacenter	>100×
Llama 2 70B	Datacenter	>100×
RetinaNet	Datacenter	Largest among the classic models
BERT-99.0	Edge	~4×
RNN-T	Edge	~4×
ResNet-50	Edge	~1.5×
ResNet-50	Tiny	>1,000×
Other Tiny	Tiny	79× – 596×

The >100× improvements in datacenter LLMs reflect the massive industry attention on software optimisation (FP8 kernels, FlashAttention, speculation) and hardware (tensor core architectures, HBM3). Tiny models show even larger gains in relative terms thanks to the low starting point and the progress in specialised chips.

On-premise reference hardware

For a node of 4×H100 SXM 80 GB (the track’s generic hardware), the typical values in MLPerf Power:

Parameter	Indicative value
Declared TDP 4×H100 SXM	4 × 700 W = 2,800 W (GPU only)
System power (complete node) measured at the wall	~3,500 – 5,000 W depending on load
CPU + RAM + fans overhead over GPU	25 – 45 % of the total
Efficiency: Llama 2 70B offline (closed)	a function of the runtime; order of magnitude ~0.1 – 1 tokens/J

The key point: the “node overhead”, the difference between what the analyser measures at the wall and what NVML reports for the GPUs, is 25–45 % of total consumption. A GPU-only measurement underestimates the real system consumption by that margin.

For the A100 PCIe 80 GB (4× per node), system consumption is lower (GPU TDP ~300 W each, node ~1,500–2,000 W), with better efficiency per watt but lower absolute throughput. The L40S (4× per node) has an intermediate profile: TDP ~350 W each, good FP8 inference performance, node consumption ~1,800–2,500 W.

Contrast with the software measurement of post C3

Post C3 of this track covers the DCGM/NVML/RAPL/Kepler stack in production. Here is the data-driven comparison against MLPerf Power:

Dimension	Software (DCGM/NVML/RAPL/Kepler)	MLPerf Power (analyser + PTDaemon)
What it measures	On-device sensors (GPU, CPU, DRAM)	AC wall power (the whole node)
Coverage	GPU + CPU + DRAM (≠ the wall)	Complete node including fans, PSU overhead
Node overhead	Not captured by default	Included in the measurement
Accuracy	±5–15 % vs power meter (Kepler estimate); NVML ~3–5 % on the GPU	±0.1 % reading + ±0.1 % range (Yokogawa WT310E)
Frequency / continuity	Continuous in production (sub-second)	Only during the benchmark run
Extra hardware	None (uses system sensors)	SPEC-certified analyser (~3,000 USD)
Attribution per workload	Per pod/MIG with Kepler	No: it measures the system, not the individual workload
External certification	No	Yes (SPEC PTDaemon + MLCommons review)
Data access	Real time in Prometheus	Public logs on GitHub after the round
Recommended use	Continuous monitoring in production	Certified comparative benchmarking

The operational conclusion: the two approaches are complementary, not competing. The DCGM/Kepler stack gives the continuous per-pod view for production; MLPerf Power gives the certified ground truth for hardware comparisons with reproducibility guarantees.

The myth of measuring only the GPU: data from the paper

The arXiv 2410.12032 paper devotes section §III-C to dismantling the myths of power measurement in ML systems. The one most relevant to on-premise inference is Myth 1:

“A common misconception is that measuring the power consumption of specific ML components, such as accelerators or GPUs, is adequate to assess system efficiency. In reality, overall system power consumption is crucial. Different components are active at various stages of ML workloads with varying duty cycles.”

The data from submissions with a wall analyser versus NVML readings confirms that the overhead of the node’s non-GPU components (CPU, memory, disks, fans, PSU losses) ranges between 25 and 45 % of total consumption depending on the system. An efficiency comparison based on NVML readings alone systematically underestimates consumption and can distort the ranking between systems with different CPU/GPU proportions.

Myth 2 (TDP and PSU as power proxies) is also relevant: the TDP of an H100 SXM is 700 W per card, but real consumption in inference with Llama 2 70B depends on the load, the prompt length and the batch size. The MLPerf Power data shows that systems routinely run well below TDP in online inference scenarios (server scenario), where latency limits throughput and the GPU is not at 100 % utilisation.

Limitations of the current submissions

The limitations are not of the method but of the coverage:

Few submissions with power in datacenter. In v4.0 only 4 companies delivered power numbers for datacenter; in v5.1, one (Lenovo). Performance benchmarking has dozens of submitters; energy benchmarking has few. This limits the representativeness of the corpus for comparing on-premise hardware.
Heterogeneous SUTs. Each submitter defines its own SUT: number of GPUs, server, interconnect, software stack. A Dell system with 4×H100 is not directly comparable to a Supermicro one with 8×H100, even if both use the same model. Platform differences (NVLink vs PCIe, server vs rack) are captured but not separated out.
Fixed workloads. The MLPerf benchmarks are representative but they are not your load. The distribution of prompt lengths, the prefill/decode ratio and your users’ batch size may differ significantly from the MLPerf synthetic datasets. The benchmark’s samples/J result is a reference, not a prediction of your workload.
Publication latency. Results are published months after the submission deadline. New hardware (H200, B100, B200) may have no power results available at the moment of the purchasing decision.
No PUE. MLPerf Power does not include datacenter efficiency. Two identical systems in two datacenters with PUE 1.1 and 1.5 give the same MLPerf Power result but a very different electricity cost. For TCO, the MLPerf J/sample must be multiplied by your own datacenter PUE.

Usage flow for an on-premise platform

For anyone who wants to use MLPerf Power as a purchasing reference or as their own benchmark, the practical flow:

Step	Action	Resource
1	Consult the publicly available submissions	mlcommons.org/benchmarks/inference-datacenter
2	Filter by model (Llama 2 70B, GPT-J), scenario (Offline/Server) and division (Closed/Available)	GitHub mlcommons/inference_results_vX.Y
3	Compare samples/J between systems with similar hardware	power log + performance summary
4	Adjust for your own PUE	samples/J × (1/PUE) for real energy in the datacenter
5	If you want to measure your own hardware: buy a Yokogawa WT310E, join MLCommons, get access to PTDaemon	docs.mlcommons.org/inference/power
6	Calibrate the software stack (DCGM/Kepler) against the analyser once	see post C3

Cross-links of the energy track

This article is C4 of the energy pillar. The related articles:

C1 — State of the art: energy benchmarking of LLM frameworks: inventory of tools and overview of the field.
C2 — Energy per token: methodology and the Spanish electricity market: the J/token identity and how the electricity price multiplies it.
C3 — Measurement tooling in production: Kepler, DCGM and a practical stack: the continuous stack for production that MLPerf Power complements.
Benchmarking track — state of the art of frameworks: context of the performance benchmarking that MLPerf Power extends into energy.

Sources

MLCommons Power Working Group — https://mlcommons.org/working-groups/benchmarks/power/
arXiv 2410.12032 · MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI (Tschand et al., 2024) — https://arxiv.org/abs/2410.12032
MLPerf Inference Power Measurement Rules v2.0 (power_measurement.adoc) — https://github.com/mlcommons/inference_policies/blob/master/power_measurement.adoc
MLCommons power-dev · repositorio público de herramientas de medición — https://github.com/mlcommons/power-dev
MLPerf Inference Power Measurement Documentation (MLCFlow) — https://docs.mlcommons.org/inference/power/
SPEC PTDaemon · lista de dispositivos certificados — https://open.spec.org/power/docs/specpower-device_list/
MLCommons · MLPerf Inference v1.0 con las primeras mediciones de potencia (abril 2021) — https://mlcommons.org/2021/04/mlperf-inference-v1-0-results-with-first-power-measurements/
MLCommons · MLPerf Inference v4.1 results (agosto 2024) — https://mlcommons.org/2024/08/mlperf-inference-v4-1-results/
MLCommons · MLPerf Inference v5.1 results (septiembre 2025) — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
MLCommons · MLPerf Power benchmark presentado en IEEE HPCA 2025 — https://mlcommons.org/2025/03/ml-commons-power-hpca/
SPEC Updates PTDaemon Interface (GlobeNewswire, febrero 2024) — https://www.globenewswire.com/news-release/2024/02/22/2833367/0/en/SPEC-Updates-PTDaemon-Interface-to-Meet-Evolving-Industry-Requirements.html
Yokogawa WT310E Power Analyzer — https://tmi.yokogawa.com/us/solutions/products/power-analyzers/

Energy benchmarking in AI: frameworks, metrics and state of the art (tool by tool)

Sat, 13 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The energy and carbon references are European (France, Germany, Spain), since this is a sovereign proposal. The dollar sign is not used (on this site it is a formula delimiter).

What this introduction covers

Fourth article in the data series and deep dive on the energy axis. Measuring the watts of an AI workload looks like a sustainability detail, but it is at once a cost lever (electricity is 30–50 % of TCO), an axis of sovereignty (carbon per token depends on the country) and, increasingly, a regulatory obligation (emissions reporting). This article inventories the metrics, where each figure physically comes from, the data sheet for each measurement framework, and how to go from watt to carbon with European figures. No recommendations; only data and methodology, because in energy, even more than in performance, the measurement method changes the result.

Why measuring energy matters (with numbers)

Three quantified reasons, connecting with the opening article:

Reason	Figure	Implication
Cost	electricity = 30–50 % of TCO	measuring energy is measuring almost half the cost
Sovereignty/carbon	France ~9× fewer gCO₂/token than Germany	the country of the cluster is a lever
Scale/regulation	datacenters ~460 TWh in 2025 (≈1.8 % worldwide), >800 TWh projected for 2028	emissions reporting (CSRD) mandatory

At 700 W per H100, an 8-card node draws of the order of 5.6 kW from the boards alone, ~49,000 kWh a year. Applying PUE 1.4 that becomes ~68,700 kWh/year, and there the country decides the bill: at the French wholesale price (~0.058 €/kWh) that is ~3,980 €/year of electricity per node; in Germany (~0.091 €/kWh), ~6,250 €/year, 57 % more for the same iron and the same work. Multiplied across a fleet, it is a first-order cost line, and a carbon footprint that, in Europe, has to be measured and reported. Energy stopped being an engineering detail and became a business and compliance variable.

The energy metrics

Metric	Definition	Unit
Mean power	average instantaneous consumption	W
Energy per token	energy consumed per generated token	J/token
Energy per inference	energy per request	µJ – Wh
Total energy	integral of power over time	Wh, kWh
PUE (Power Usage Effectiveness)	datacenter overhead (cooling, losses)	ratio ≥ 1
Carbon	energy × grid intensity	gCO₂eq

The two base identities of the whole axis:

$$\text{energy per token (J)} = \frac{\text{mean power (W)}}{\text{throughput (tok/s)}}$$ $$\text{carbon} = \text{energy (kWh)} \times \text{PUE} \times \text{grid intensity (gCO}_2\text{/kWh)}$$

The first has the same shape as cost per token: it shares the throughput denominator, so raising tokens/s lowers energy per token. The second brings in the two multipliers external to the compute, the datacenter PUE and the country’s grid intensity, which can change carbon per token by an order of magnitude without touching the stack.

Worked example: energy per token of an 8×H100 node

To anchor the identities with numbers, the end-to-end calculation on the example node (8×H100, Llama 3.1 70B FP16, vLLM at 2,800 tok/s):

Step	Calculation	Result
Board power	8 × 700 W	5,600 W
Energy per token (boards)	5,600 ÷ 2,800	2.0 J/token
Energy per token (with PUE 1.4)	2.0 × 1.4	2.8 J/token
Energy per 1M tokens (boards)	2 × 10⁶ J	0.56 kWh
Energy per 1M tokens (with PUE)	0.56 × 1.4	0.78 kWh

From there come the electricity cost and the carbon, by country:

Country	Electricity cost / 1M tok	Carbon / 1M tok
France (0.058 €/kWh; ~40 gCO₂/kWh)	~0.045 €	~31 gCO₂
Spain (0.054 €/kWh; ~160 gCO₂/kWh)	~0.042 €	~125 gCO₂
Germany (0.091 €/kWh; ~363 gCO₂/kWh)	~0.071 €	~283 gCO₂

And the link with performance: if an optimisation (FP8, say) raised throughput to 4,200 tok/s, energy per token would drop to ~1.33 J/token (boards) and everything below it in the same proportion. Electricity cost and carbon per token fall with throughput, by the identity. Note: these are board + PUE figures; the complete node (CPU, NICs, PSUs) consumes somewhat more, and energy measured with a power meter can differ from energy estimated with nvidia-smi.

Where the figure comes from: the measurement layers

Not every energy figure is equally reliable. The figure can come from four layers, with increasing accuracy and decreasing context:

The GPU is the dominant source in LLM inference, and its power is read from a hardware counter (NVML/nvidia-smi, aggregated by DCGM, the same basis as GPU observability). RAPL gives CPU and DRAM. The eBPF + model layer (Kepler) splits the total watt between containers and pods without instrumenting the application. And the physical power meter is the ground truth against which estimates are calibrated: no software estimate is better than the meter, only more convenient.

How it is instrumented (and where the error creeps in)

Four instrumentation decisions explain most of the discrepancies between energy measurements:

Sampling rate. nvidia-smi/DCGM read power at intervals; sampling that is too coarse misses the peaks and underestimates the energy. It must be fine enough to capture the power variation between prefill (high) and decode (lower).
Board vs node. GPU (board) power does not include CPU, NICs, fans or PSU losses (typically an extra 10–20 %). Comparing a board energy figure with a complete-node one is comparing different things.
Idle baseline. A powered GPU with no work already draws power. Is that idle attributed to the workload or discounted? The decision changes energy per token, and idle matters: it is real energy that someone pays for.
Time window. Energy is the integral of power over time; the measurement window has to line up exactly with the workload (without counting warm-up or shutdown), or the number does not correspond to the work measured.

These decisions, as in performance benchmarking, mean that the measurement method matters as much as the system measured. A J/token without a stated sampling rate, board/node boundary and idle treatment is not comparable.

Frameworks, tool by tool

Zeus (ml.energy, University of Michigan)

What it does: measure and optimise the energy of deep learning workloads. Method: NVML counters. Scope: NVIDIA and AMD GPUs, CPU, DRAM, Apple Silicon and NVIDIA Jetson. Licence: OSS; academic backing (NSDI'23 paper). Differentiator: it does not only measure, it also optimises (power capping, frequency selection) to cut energy without significant loss of performance (Zeus Project, NSDI'23). It is the reference tool when the goal is to lower J/token, not just to measure them.

CodeCarbon

What it does: estimates the energy and the CO₂eq of compute processes. Method: reads the GPU with nvidia-smi and estimates CPU/RAM. Scope: CPU + GPU + RAM. Licence: OSS (Python). Differentiator: it integrates into ML flows and CI pipelines, and it is the one that comes closest to the power meter among the Python tools (comparison). It brings carbon context out of the box. Limit: a high-level estimate, less exact than direct hardware measurement.

CarbonTracker

What it does: tracks energy and carbon at GPU run time. Scope: GPU only, it measures neither CPU nor memory. Licence: OSS. Differentiator: lightweight, easy to add to a training/inference script. Limit: partial coverage (no CPU/DRAM), so it undercounts the total system energy.

Scaphandre

What it does: a power metrology agent written in Rust. Method: RAPL (reads /proc, splits joules by CPU ticks). Scope: process or VM, with the raw accuracy of the hardware layer. Licence: OSS. Differentiator: the highest direct accuracy among the general-purpose tools, at the cost of bringing no carbon context (it is pure metrology) (Scaphandre · GitHub). Limit: focused on CPU/RAPL; for GPU it has to be combined with NVML.

Kepler (CNCF)

What it does: a Prometheus exporter that measures energy at container, pod and node level in Kubernetes. Method: eBPF + ML models to estimate consumption per workload. Language: Go + C. Licence: Apache 2.0 (the sustainable-computing-io project) (Kepler · GitHub). Differentiator: it is the cloud-native option for attributing energy per pod without instrumenting the app, just as OpenCost attributes cost. Limit: attribution via eBPF + model is estimation, not direct measurement; its accuracy depends on the model.

MLPerf Power (MLCommons)

What it does: a standardised benchmark of energy efficiency, from µW to MW. Maintainer: MLCommons. Differentiator: cross-vendor comparability of efficiency, with a dataset spanning several years and workload versions (MLPerf Power, arXiv 2410.12032). It is to energy what MLPerf Inference is to performance: the comparison standard between vendors.

Comparison table

Tool	Method	Scope	Licence	Accuracy / note
Zeus	NVML / counters	NV+AMD GPU, CPU, DRAM, Apple, Jetson	OSS	measures and optimises DL energy
CodeCarbon	nvidia-smi + estimation	CPU+GPU+RAM, CO₂eq	OSS (Python)	the closest to the power meter among Python tools
CarbonTracker	GPU runtime	GPU only	OSS	lightweight; no CPU/mem
Scaphandre	RAPL (/proc)	process/VM	OSS (Rust)	raw accuracy; no carbon context
Kepler	eBPF + model	container/pod/node (K8s)	Apache 2.0 (CNCF)	cloud-native; model-based estimation
MLPerf Power	standardised measurement	µW to MW	MLCommons	cross-vendor comparability

Accuracy: who to believe

The difference between tools is not a nuance: Scaphandre offers the raw accuracy of the hardware layer (RAPL), while the Python ones such as CodeCarbon give higher-level, carbon-aware estimates, with more context but less direct exactness; among the Python tools, CodeCarbon is the one that comes closest to the power meter, followed by CarbonTracker, with variability across infrastructures (comparison). The rule: for sizing and design, a software estimate will do; to defend a figure before a committee or an auditor, calibrate against a power meter at least once, and use the software tool as a continuous proxy. The measured figure beats the estimated one; the rest is convenience.

Energy leaderboards and benchmarks

To compare efficiency between models without building your own test bench:

Resource	What it provides
MLPerf Power	standardised efficiency measurement, from µW to MW
HF AI Energy Score	comparable per-model energy efficiency ratings (HF)
ML.ENERGY Benchmark	automated measurement and optimisation of inference energy
awesome-green-ai	curated index of Green AI resources and tools (repo)

These resources give the order of magnitude of the energy per token of each model, useful for choosing a model on efficiency at the design stage. But they do not replace measurement on your hardware and your workload: a leaderboard measures on one specific bench, not on your node.

From watt to carbon: France, Germany, Spain

Here the energy axis turns into a sovereignty argument. Carbon per token comes from multiplying the energy by the country’s grid intensity, which in Europe varies by almost two orders of magnitude:

Country	Wholesale (€/MWh, Mar 2026)	Grid carbon (gCO₂/kWh)	Profile
France	~58	~20–60	nuclear: cheap and very clean
Spain	~54	~150–170	renewables + gas: cheap, medium carbon
Germany	~91	~363 (2024 average)	coal/gas + renewables: expensive and dirty

Sources: wholesale prices (TradingEconomics), Germany’s carbon ~363 gCO₂/kWh (2024 average, Wikipedia), nuclear ~17–35 gCO₂e/kWh (EEA), Spain ~167 gCO₂/kWh (2020). Hourly intensity comes from ElectricityMaps.

The calculation, on the example node (0.78 kWh per million tokens, with PUE): in France (~40 gCO₂/kWh) it is ~31 gCO₂ per million tokens; in Germany (~363 gCO₂/kWh), ~283 gCO₂, a factor of ~9×, without touching a line of the stack. Placing the cluster in France or Spain is at once a cost lever, a carbon lever and a compliance lever (EU AI Act, controls ENS × 42001 × EU AI Act).

Energy efficiency levers

Once it is measured, it can be brought down. The levers, ordered by the identity J/token = W ÷ throughput (lower the numerator or raise the denominator):

Lever	Mechanism	Effect
Quantisation (FP8/INT4)	more tokens per GPU-hour, less VRAM/KV	raises throughput → lowers J/token
Batching / continuous batching	amortises the fixed power cost	raises throughput → lowers J/token
Power capping (Zeus)	limits GPU power	lowers W with little loss of throughput
Efficiency-aware scheduling	consolidate workloads, power down idle GPUs	eliminates idle consumption
Location (country)	cleaner grid	lowers carbon per token
Avoiding over-reasoning	fewer reasoning tokens that add nothing	less compute → less energy

Quantisation (see quantisation for inference) is the highest-return lever because it moves all three axes at once: it raises throughput, lowers cost and lowers energy per token. Zeus’s power capping is energy-specific: it trims peak power with little performance loss, lowering the W in the numerator. And idle, the GPU powered but not working, is pure wasted energy that scheduling recovers.

From design to production: where energy is measured

As with the other axes, energy is measured at three moments, with different tools and increasing accuracy:

Moment	Tool	What it gives
Design	TDP × hours (datasheet)	ceiling estimate for sizing
Benchmark	Zeus, MLPerf Power, power meter	energy per token measured on a controlled bench
Production	Kepler (eBPF), DCGM	continuous per-pod consumption in the cluster

The design figure (TDP) serves to size the utility feed and the cooling, but it overestimates (the GPU is rarely at TDP 100 % of the time). The benchmark figure is the one used for the proposal’s cost and carbon per token, because it is measured and reproducible. And the production figure (Kepler/DCGM) closes the loop: it checks that real consumption matches the budget and feeds the footprint report. The three have to be traceable to each other, which is what the harness in article S4 does.

Energy in TCO and European regulation

The closing of the axis: energy is not only sustainability, it is cost and compliance.

Cost: being 30–50 % of TCO, every J/token improvement translates directly into euros. In the example, moving from Germany (~0.091 €/kWh) to France (~0.058 €/kWh) lowers the electricity share by ~37 % on top of the carbon.
Regulation (CSRD): European companies must report their carbon footprint (scope 2). A low-carbon grid (France) directly improves that reported figure, and measuring energy per token stops being optional and becomes a reporting requirement.
EU AI Act: efficiency and energy consumption form part of the documentation expected of AI systems, which makes energy measurement part of the technical file.

One reporting nuance worth knowing: the electricity footprint is scope 2 (indirect emissions from purchased energy), and it can be accounted for by location (the real intensity of the country’s grid) or by market (according to the guarantees of origin contracted). Location-based accounting is the one that genuinely rewards putting the cluster on a clean grid such as the French one; market-based accounting allows green energy to be “bought” through certificates. Knowing which one your CSRD report requires changes which lever (physical location vs contract) lowers the figure, and both start from measuring energy per token, which is what this axis provides.

In other words: measuring energy covers at once a first-order cost line, the carbon and sovereignty argument, and a regulatory obligation. For a sovereign European proposal, the energy axis is not the “nice sustainability bit”: it is one of the three columns with a number attached.

Checklist for a reproducible energy measurement

For an energy or carbon figure to be defensible before a committee or an auditor, it has to come with its context. The minimum to fix and publish:

What to fix	Why
Tool + version	each layer (RAPL, NVML, eBPF) and tool measures differently
Sampling rate	coarse sampling misses peaks
Board vs node boundary	±10–20 % depending on whether CPU/PSUs are included
PUE used	it multiplies all the compute energy
Source and time of the grid intensity	carbon varies ~9× by country and by hour
Idle treatment	attributed or discounted changes the J/token
Model, precision and workload	energy depends on what is served and how
Calibration vs power meter	at least once, to validate the estimate

The rule: if you cannot hand over this table alongside the gCO₂/token figure, the figure is not auditable. The reproducible harness of article S4 records all these parameters so that the energy figure is as traceable as the cost and performance ones, and so that the reported footprint (CSRD) holds up under audit.

State of the art 2026

LLM-specific resolution is missing: general-purpose tools measure well at machine or process level, but they lack LLM-specific resolution (energy per token, per prefill/decode phase), and the simulators lack emissions modelling, an open gap in the field.
eBPF as a trend (Kepler) for attributing energy per pod without instrumenting the app, in parallel to how OpenCost attributes cost.
Standardisation of comparison via MLPerf Power and HF AI Energy Score.
Optimisation, not only measurement (Zeus): the field is moving from “how much do I spend” to “how do I spend less without losing performance”.

Limits and traps (data-driven)

Estimation vs measurement. An nvidia-smi figure is not a power meter. Calibrate against hardware at least once before defending a number.
Partial coverage. CarbonTracker measures GPU only; Scaphandre, mostly CPU/RAPL. Add the layers or you undercount the total energy.
Forgetting PUE. Board energy does not include cooling or losses; without PUE you underestimate the real consumption and carbon.
Carbon without a country. A gCO₂/token without a stated grid (and hour) means nothing: it varies ~9× between France and Germany.
Confusing energy with sustainability. Here it is cost and compliance, with a number; treating it as a “green” extra is losing one of the three columns of the proposal.

That closes the run of introductions: cost, performance and energy, each with its frameworks and its methodology. The rest of the series goes tool by tool through to the dashboard that underpins the sovereign architecture decision, with European figures, measured and reproducible.

Closing

Energy is the axis easiest to dismiss as “the green one” and the one that hides the most: it is almost half the cost, it defines the footprint Europe requires to be reported, and its carbon per token changes ~9× depending on whether the cluster sits in France or in Germany. Measuring it properly, with the right layer, the right sampling, PUE included and the country’s grid, turns a soft sustainability argument into a hard column of the proposal, with a number and a source. And the sovereign conclusion stands on its own with the data in this article: an inference platform in France or Spain is at once cheaper, cleaner and compliant with European jurisdiction than the same platform on a US hyperscaler, and that is demonstrated with measured J/token, gCO₂/kWh by country and euros, not with intentions. The energy axis does not decorate the proposal: it closes it.

Sources

Zeus Project (ml.energy, UMich) — https://ml.energy/zeus/
Zeus · USENIX NSDI'23 — https://www.usenix.org/system/files/nsdi23-you.pdf
Kepler (CNCF, eBPF) · GitHub — https://github.com/sustainable-computing-io/kepler
Scaphandre · GitHub — https://github.com/hubblo-org/scaphandre
CarbonTracker · GitHub — https://github.com/saintslab/carbontracker
MLPerf Power (arXiv 2410.12032) — https://arxiv.org/html/2410.12032v2
HF AI Energy Score — https://huggingface.github.io/AIEnergyScore/
awesome-green-ai — https://github.com/samuelrince/awesome-green-ai
TradingEconomics · precio electricidad Francia/Alemania/España — https://tradingeconomics.com/france/electricity-price
EEA · intensidad de emisiones de la electricidad en Europa — https://www.eea.europa.eu/en/analysis/indicators/greenhouse-gas-emission-intensity-of-1
Electricity sector in Germany (carbono ~363 gCO₂/kWh) — https://en.wikipedia.org/wiki/Electricity_sector_in_Germany

Mlperf-Power on lo0 — Tech Blog

MLPerf Power: the standard energy-efficiency benchmark for on-premise ML systems

TL;DR

What MLPerf Power is and how it fits into the MLCommons ecosystem

Coupling with MLPerf Inference

Coupling with MLPerf Training and Tiny

Measurement methodology: certified wall power

The fundamental principle: the whole SUT, at the wall

Measurement hardware: SPEC-certified power analyser

SPEC PTDaemon: the daemon that orchestrates the measurement

What is and is not inside the SUT

Metrics: formulas and definitions

Throughput benchmarks (Datacenter / Offline / Edge)

Latency benchmarks (Single Stream / Tiny)

System power and run energy

How to read an MLPerf Power submission

Divisions and availability categories

What comparability MLPerf Power gives and what it does not

Structure of a submission

Data: scale, trends and reference hardware

The corpus: 1,841 measurements from 60 systems

Energy efficiency improvements: LLMs lead

On-premise reference hardware

Contrast with the software measurement of post C3

The myth of measuring only the GPU: data from the paper

Limitations of the current submissions

Usage flow for an on-premise platform

Cross-links of the energy track

See also

Sources

Energy benchmarking in AI: frameworks, metrics and state of the art (tool by tool)

What this introduction covers

Why measuring energy matters (with numbers)

The energy metrics

Worked example: energy per token of an 8×H100 node

Where the figure comes from: the measurement layers

How it is instrumented (and where the error creeps in)

Frameworks, tool by tool

Zeus (ml.energy, University of Michigan)

CodeCarbon

CarbonTracker

Scaphandre

Kepler (CNCF)

MLPerf Power (MLCommons)

Comparison table

Accuracy: who to believe

Energy leaderboards and benchmarks

From watt to carbon: France, Germany, Spain

Energy efficiency levers

From design to production: where energy is measured

Energy in TCO and European regulation

Checklist for a reproducible energy measurement

State of the art 2026

Limits and traps (data-driven)

Closing

See also

Sources