MLPerf Power: the standard energy-efficiency benchmark for on-premise ML systems

· By José Manuel Román
Contents

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

MLPerf Power is the only standardised benchmark for measuring the energy efficiency of complete ML systems. It is run by the MLCommons Power Working Group and backed by more than 20 organisations. It measures power at the wall (AC wall power) for the whole System Under Test, GPUs, CPUs, memory, interconnect and fans included, with a SPEC PTDaemon certified power analyser (Yokogawa WT310/WT5000) and the public tooling in mlcommons/power-dev. The central metric for datacenter inference is samples/joule (or tokens/joule for LLMs); for latency, energy per stream. The public corpus adds up to 1,841 reproducible measurements from 60 systems (590 datacenter, 792 edge, 447 tiny, 12 training). GPT-J and Llama 2 show improvements of more than 100× in samples/joule between the first and the latest available rounds. The direct comparison with the software-based measurement of post C3 (DCGM/NVML/RAPL/Kepler) shows that MLPerf Power offers maximum accuracy and reproducibility at the cost of dedicated hardware (~3,000 USD for the Yokogawa 310E) and of the requirement to be an MLCommons member to access PTDaemon; software measurement is continuous, needs no extra hardware, but carries a wider error bar and no external validation.

What MLPerf Power is and how it fits into the MLCommons ecosystem

MLCommons is the consortium of more than 100 organisations (NVIDIA, Google, Intel, Dell, AMD, Meta and others) that maintains the MLPerf benchmarks: Training, Inference (Datacenter and Edge), Tiny, HPC, Storage, Client and Automotive. The Power Working Group is the specific group that extends each of those benchmarks by adding the energy dimension (MLCommons Power Working Group).

MLPerf Power is not a standalone benchmark: it is an energy measurement layer overlaid on the existing performance benchmarks. For a submission to be valid with power, it must first satisfy the performance rules (MLPerf Inference, Training or Tiny) and then the additional power measurement rules.

Coupling with MLPerf Inference

The integration most relevant to on-premise inference is with MLPerf Inference Datacenter, which defines:

  • Scenarios: Server (latency, queries per second with a request queue) and Offline (pure throughput, batch with no latency constraint). Power is measured in both.
  • Benchmarks: ResNet-50, BERT, RNN-T, 3D-UNET, RetinaNet, DLRM, GPT-J 6B, Llama 2 70B (from v4.0), Mixtral 8×7B (from v5.0).
  • Divisions: closed (model and preprocessing fixed, only runtime optimisation allowed) and open (modifications to the model permitted).

Power is measured during the performance phase, not during accuracy or compliance. The same run that produces the performance log produces the power log: reporting the highest performance of three runs and the lowest power of another three is forbidden (MLPerf Inference Power Measurement Rules, §5.9).

Coupling with MLPerf Training and Tiny

  • MLPerf Training: large-scale power measurement (multi-node, 10K+ GPUs) does not use an external analyser, which would be impracticable. Node telemetry (IPMI/Redfish) plus an estimate for the interconnect network is used instead. The metric is energy to train (J or kWh). In the v4.0 submissions with power, the systems running Llama 2 70B Training range from single nodes to hundreds, revealing the non-linear scaling of energy (arXiv 2410.12032).
  • MLPerf Tiny: microcontroller systems (from 5.64 mW). Specialised micro-power instrumentation is used, with hardware pins to demarcate the start and end of the inference. The metric is energy per inference (J), not samples/joule.

Measurement methodology: certified wall power

The fundamental principle: the whole SUT, at the wall

Rule number one is absolute: power must be measured at system level, that is, including every component the benchmark activates: host processors, accelerators, memory, disks, fans, internal interconnect. Measuring the GPUs alone is not enough (MLPerf Inference Power Measurement Rules, §5.1):

“The power consumption must be measured at the system level, i.e. including all components that are sensitized by LoadGen e.g. the host processor on which LoadGen runs, accelerators, memory, fans, etc.”

Power is measured in AC (alternating current), at the SUT input, upstream of the PSUs. Any battery or energy storage between the mains socket and the system PSUs is forbidden.

Measurement hardware: SPEC-certified power analyser

MLPerf Power requires a power analyser certified by SPEC PTDaemon (official SPEC list). The most widespread in submissions is the Yokogawa WT310E (~3,000 USD), which connects to the director over USB (Linux) or Ethernet/serial (Windows). Voltage is measured in parallel and current in series with the SUT power line.

Relevant specifications of the Yokogawa WT310E:

  • Power accuracy: 0.1 % of reading + 0.1 % of range
  • Measurement range: µW to MW (the WT5000 reaches industrial installations)
  • Update rate: from 50 ms

For systems with more than one channel, or multi-PSU nodes, multi-analyser configurations are allowed. The ranging rule: first a ranging run is performed with the range in Auto mode to determine the maximum current and voltage values; the testing runs use fixed ranges based on those peaks, which maximises accuracy within the range. Auto mode is not permitted in the testing runs.

SPEC PTDaemon: the daemon that orchestrates the measurement

PTDaemon (Power Thermal Daemon) is the SPEC tool that handles communication with the analyser. MLCommons is licensed to use it within the MLPerf Power flow. Access requires MLCommons membership and signing the corresponding EULA (MLPerf Power FAQ).

The measurement flow is as follows:

Director (PC)server.py + PTDaemonNTP sync → CSV logsAnalyserYokogawa WT310EAC wall powerSUT (System Under Test)client.py + LoadGen4×H100 SXM + CPU + RAMfans + interconnect→ performance log + timestampscurrent in seriesvoltage in parallelThe analyser measures AC at the SUT input. Director and SUT sync over NTP. Power log and performance log are aligned by timestamp to compute the final metric.

The process step by step (mlcommons/power-dev):

  1. NTP sync between Director and SUT to align timestamps.
  2. Ranging run: power with ranges in Auto; the analyser determines the current and voltage peaks.
  3. Testing run: fixed ranges; LoadGen runs the benchmark; the Director records power with timestamps; the SUT records the performance log with start/end timestamps of the execution phase.
  4. Post-processing: the result summariser cross-references power log and performance log by timestamp to compute the mean power over the execution window.
  5. A minimum of 60 seconds of valid power data. If the workload finishes before 60 s, it is run in a loop until that threshold is reached.

What is and is not inside the SUT

The SUT includes everything the benchmark activates:

Inside the SUTOutside the SUT
GPUs / acceleratorsPDU shared with other systems
Host CPU and RAMDatacenter cooling infrastructure (PUE)
Disks / NVMe if the benchmark uses themManagement network (BMC, out of band)
Node fans and cooling systemsDatacenter network switches (not the node’s)
Internal interconnect (NVLink, PCIe)
Node PSUs

The paper makes it explicit: PUE is out of scope for MLPerf Power by design (arXiv 2410.12032, §III-C). MLPerf measures the efficiency of the ML system, not that of the datacenter. Including PUE would obscure the differences between systems by mixing hardware efficiency with building efficiency.

Metrics: formulas and definitions

Throughput benchmarks (Datacenter / Offline / Edge)

For throughput benchmarks, Offline and Server in datacenter plus some edge scenarios, the energy efficiency metric is:

$$\eta = \frac{\text{throughput (samples/s)}}{\text{mean power (W)}} \quad \Rightarrow \quad \left[\frac{\text{samples}}{\text{J}}\right]$$

where mean power is computed over the window of the execution phase of the performance run. The reciprocal gives energy per sample:

$$E_{\text{sample}} = \frac{\text{mean power (W)}}{\text{throughput (samples/s)}} \quad \left[\text{J/sample}\right]$$

For LLM benchmarks (GPT-J, Llama 2), where the output varies in length, a “sample” is one complete query (prompt + response). Energy per output token is obtained by dividing by the number of tokens generated, which varies per query and must be reported or estimated.

Latency benchmarks (Single Stream / Tiny)

For latency benchmarks, Single Stream in edge and tiny, where the processing time is fixed, the metric is the inverse of energy per inference:

$$\eta_{\text{latency}} = \frac{1}{E_{\text{inference}}} \quad \left[\frac{1}{\text{J}}\right]$$

The paper treats both metrics (samples/J and 1/J) as comparable within their category, although they are not interchangeable between categories.

System power and run energy

The three quantities that appear in every power submission:

QuantityDefinitionUnit
System powermean of the AC power samples over the execution windowW
Run energymean power × window durationJ
Energy efficiencythroughput / system power = samples/Jsamples/J

The run energy formula:

$$E_{\text{run}} = \bar{P} \times \Delta t = \frac{\sum_{i} P_i \cdot \Delta t_i}{\Delta t_{\text{total}}} \times \Delta t_{\text{total}} \quad [\text{J}]$$

where \( P_i \) are the analyser samples and \( \Delta t_i \) the intervals between samples, both inside the window demarcated by the LoadGen timestamps.

How to read an MLPerf Power submission

Divisions and availability categories

Submissions inherit the divisions of MLPerf Inference:

DivisionConstraintsComparability
ClosedFixed model, fixed preprocessing, only the runtime and the hardware are optimisedHigh: submissions directly comparable with each other
OpenModifications to the model allowed (aggressive quantisation, distillation, pruning)Low: each system may use a different model

Within closed, there are system availability categories:

CategoryDefinition
AvailableSystem commercially available at the closing date
PreviewAnnounced but not commercially available
RDI (Research, Development, Internal)Systems for internal or experimental use

Only Available systems in closed are comparable without caveats. A Preview system with better energy than an Available one from another company is not a valid comparison for purchasing decisions.

What comparability MLPerf Power gives and what it does not

It gives:

  • An objective comparison of the energy efficiency of complete nodes (not just the GPU) under a standardised ML workload.
  • Verified reproducibility: the logs are published on GitHub and anyone can review them.
  • Trends over time between rounds: how much the efficiency of each hardware family improves version to version.
  • A basis for comparing heterogeneous systems running the same workload under the same rules.

It does not give:

  • An isolated GPU-to-GPU comparison (it measures the complete node, not the card alone).
  • Broad hardware coverage: the number of submissions with power is small. In v4.0 only four companies (Dell, Fujitsu, NVIDIA, Qualcomm) delivered power numbers for datacenter. In v5.1 there were two power submissions (Lenovo datacenter + GATEOverflow edge).
  • Representativeness of your own workloads: the benchmarks are fixed and may not match the distribution of your real load (batch size, prompt length, prefill/decode ratio).
  • Datacenter cooling data: PUE, liquid vs air, are out of scope.
  • Real-time comparison: results are published months after the closing date.

Structure of a submission

Every power submission publishes the following in the MLCommons results repository:

<division>/<submitter>/measurements/<system>/
├── analyzer_table.md        # Analyser configuration (model, ranges)
├── power_settings.md        # Power management configuration of the SUT
results/<system>/<benchmark>/<scenario>/
├── mlperf_log_summary.txt   # Performance results (LoadGen)
├── spl.txt                  # Power log (power, current, voltage, timestamps)
└── ...

The spl.txt files contain the raw analyser readings with timestamps, which makes it possible to verify alignment with the LoadGen window.

The corpus: 1,841 measurements from 60 systems

The MLPerf Power submissions database covers (arXiv 2410.12032):

CategorySubmissionsPower range
Datacenter (Inference)590~200 W – ~10 kW per node
Edge (Inference)792~10 W – several kW
Tiny (Inference)4475.64 mW – hundreds of mW
Training / HPC12up to 500 kW (measured); ~10 MW estimated in HPC

The total range runs from microwatts to megawatts, 9 orders of magnitude, which means no single methodology applies across every segment.

Energy efficiency improvements: LLMs lead

The evolution of the samples/joule metric between rounds, normalised to the first submission of each model (arXiv 2410.12032, §V-A):

WorkloadCategoryCumulative improvement (samples/J)
GPT-J 6BDatacenter>100×
Llama 2 70BDatacenter>100×
RetinaNetDatacenterLargest among the classic models
BERT-99.0Edge~4×
RNN-TEdge~4×
ResNet-50Edge~1.5×
ResNet-50Tiny>1,000×
Other TinyTiny79× – 596×

The >100× improvements in datacenter LLMs reflect the massive industry attention on software optimisation (FP8 kernels, FlashAttention, speculation) and hardware (tensor core architectures, HBM3). Tiny models show even larger gains in relative terms thanks to the low starting point and the progress in specialised chips.

On-premise reference hardware

For a node of 4×H100 SXM 80 GB (the track’s generic hardware), the typical values in MLPerf Power:

ParameterIndicative value
Declared TDP 4×H100 SXM4 × 700 W = 2,800 W (GPU only)
System power (complete node) measured at the wall~3,500 – 5,000 W depending on load
CPU + RAM + fans overhead over GPU25 – 45 % of the total
Efficiency: Llama 2 70B offline (closed)a function of the runtime; order of magnitude ~0.1 – 1 tokens/J

The key point: the “node overhead”, the difference between what the analyser measures at the wall and what NVML reports for the GPUs, is 25–45 % of total consumption. A GPU-only measurement underestimates the real system consumption by that margin.

For the A100 PCIe 80 GB (4× per node), system consumption is lower (GPU TDP ~300 W each, node ~1,500–2,000 W), with better efficiency per watt but lower absolute throughput. The L40S (4× per node) has an intermediate profile: TDP ~350 W each, good FP8 inference performance, node consumption ~1,800–2,500 W.

Contrast with the software measurement of post C3

Post C3 of this track covers the DCGM/NVML/RAPL/Kepler stack in production. Here is the data-driven comparison against MLPerf Power:

Software measurement (C3)DCGM/NVML → GPU power (on-device)RAPL → CPU + DRAM (on-device)Kepler eBPF → per pod/MIGContinuous in production, no extra hardwareError bar: ±5–15 % vs power meterNot externally certifiedMLPerf Power (C4)Yokogawa WT310E analyser (AC, at the wall)SPEC PTDaemon (SPEC certified)Measures the whole SUT as a systemOnly during the benchmark (not continuous)Accuracy: ±0.1 % reading + ±0.1 % rangeExternally certified and reproducible
DimensionSoftware (DCGM/NVML/RAPL/Kepler)MLPerf Power (analyser + PTDaemon)
What it measuresOn-device sensors (GPU, CPU, DRAM)AC wall power (the whole node)
CoverageGPU + CPU + DRAM (≠ the wall)Complete node including fans, PSU overhead
Node overheadNot captured by defaultIncluded in the measurement
Accuracy±5–15 % vs power meter (Kepler estimate); NVML ~3–5 % on the GPU±0.1 % reading + ±0.1 % range (Yokogawa WT310E)
Frequency / continuityContinuous in production (sub-second)Only during the benchmark run
Extra hardwareNone (uses system sensors)SPEC-certified analyser (~3,000 USD)
Attribution per workloadPer pod/MIG with KeplerNo: it measures the system, not the individual workload
External certificationNoYes (SPEC PTDaemon + MLCommons review)
Data accessReal time in PrometheusPublic logs on GitHub after the round
Recommended useContinuous monitoring in productionCertified comparative benchmarking

The operational conclusion: the two approaches are complementary, not competing. The DCGM/Kepler stack gives the continuous per-pod view for production; MLPerf Power gives the certified ground truth for hardware comparisons with reproducibility guarantees.

The myth of measuring only the GPU: data from the paper

The arXiv 2410.12032 paper devotes section §III-C to dismantling the myths of power measurement in ML systems. The one most relevant to on-premise inference is Myth 1:

“A common misconception is that measuring the power consumption of specific ML components, such as accelerators or GPUs, is adequate to assess system efficiency. In reality, overall system power consumption is crucial. Different components are active at various stages of ML workloads with varying duty cycles.”

The data from submissions with a wall analyser versus NVML readings confirms that the overhead of the node’s non-GPU components (CPU, memory, disks, fans, PSU losses) ranges between 25 and 45 % of total consumption depending on the system. An efficiency comparison based on NVML readings alone systematically underestimates consumption and can distort the ranking between systems with different CPU/GPU proportions.

Myth 2 (TDP and PSU as power proxies) is also relevant: the TDP of an H100 SXM is 700 W per card, but real consumption in inference with Llama 2 70B depends on the load, the prompt length and the batch size. The MLPerf Power data shows that systems routinely run well below TDP in online inference scenarios (server scenario), where latency limits throughput and the GPU is not at 100 % utilisation.

Limitations of the current submissions

The limitations are not of the method but of the coverage:

  1. Few submissions with power in datacenter. In v4.0 only 4 companies delivered power numbers for datacenter; in v5.1, one (Lenovo). Performance benchmarking has dozens of submitters; energy benchmarking has few. This limits the representativeness of the corpus for comparing on-premise hardware.

  2. Heterogeneous SUTs. Each submitter defines its own SUT: number of GPUs, server, interconnect, software stack. A Dell system with 4×H100 is not directly comparable to a Supermicro one with 8×H100, even if both use the same model. Platform differences (NVLink vs PCIe, server vs rack) are captured but not separated out.

  3. Fixed workloads. The MLPerf benchmarks are representative but they are not your load. The distribution of prompt lengths, the prefill/decode ratio and your users’ batch size may differ significantly from the MLPerf synthetic datasets. The benchmark’s samples/J result is a reference, not a prediction of your workload.

  4. Publication latency. Results are published months after the submission deadline. New hardware (H200, B100, B200) may have no power results available at the moment of the purchasing decision.

  5. No PUE. MLPerf Power does not include datacenter efficiency. Two identical systems in two datacenters with PUE 1.1 and 1.5 give the same MLPerf Power result but a very different electricity cost. For TCO, the MLPerf J/sample must be multiplied by your own datacenter PUE.

Usage flow for an on-premise platform

For anyone who wants to use MLPerf Power as a purchasing reference or as their own benchmark, the practical flow:

StepActionResource
1Consult the publicly available submissionsmlcommons.org/benchmarks/inference-datacenter
2Filter by model (Llama 2 70B, GPT-J), scenario (Offline/Server) and division (Closed/Available)GitHub mlcommons/inference_results_vX.Y
3Compare samples/J between systems with similar hardwarepower log + performance summary
4Adjust for your own PUEsamples/J × (1/PUE) for real energy in the datacenter
5If you want to measure your own hardware: buy a Yokogawa WT310E, join MLCommons, get access to PTDaemondocs.mlcommons.org/inference/power
6Calibrate the software stack (DCGM/Kepler) against the analyser oncesee post C3

This article is C4 of the energy pillar. The related articles:

See also

Sources