Wh-Por-Token on lo0 — Tech Blog

Measuring the power of a GPU: NVML, DCGM and the sampling errors that invalidate your watts

Sat, 29 Aug 2026 08:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

TL;DR

A GPU power sensor does not return what almost everyone believes. The NVML documentation is explicit: on Ampere except GA100, and on later architectures including H100, nvmlDeviceGetPowerUsage returns power averaged over a one-second interval, not instantaneous power. The University of Oxford study presented at SC24, covering more than 70 GPUs across 25 models checked against an external 1 mΩ meter, adds the fact that breaks most published measurements: on the A100 and H100 the sensor only samples 25 % of the time (a 25 ms window within a 101 ms period), so during the remaining 75 % the GPU may be consuming something radically different. Integrating power naively over nine real benchmarks, the mean error was 39.27 %; applying good practice brought it down to 4.89 %. The operational consequence is short: if the hardware is Volta or later, use the accumulated energy counter (nvmlDeviceGetTotalEnergyConsumption, in millijoules) and subtract, rather than integrating samples. And the GPU counter is not the bill: on measured 8× H100 nodes, the node reaches 8.4 kW against 5.6 kW of summed GPU TDP, node idle is 1.8 kW, and with an average PUE of 1.54 the NVML watts are of the order of 43 % of the watts the utility bills.

What the hardware exposes

NVML: power

nvmlDeviceGetPowerUsage(device, unsigned int* power) returns milliwatts for the GPU and its associated circuitry, for example the memory. It has been supported since Fermi. The note in the official documentation is the part almost nobody quotes (NVML Device Queries):

On Fermi and Kepler the reading is accurate to ±5 % of current consumption. On Ampere (except GA100) or later, the API returns power averaged over a 1 s interval. On GA100 and earlier architectures instantaneous power is returned.

In other words, on an H100 or an L40S the usual call already delivers a one-second moving average. Anyone sampling it at 1 Hz and integrating is applying a second filter on top of an already filtered signal, without declaring it. To disambiguate, NVML exposes two separate fields accessible via nvmlDeviceGetFieldValues: NVML_FI_DEV_POWER_AVERAGE and NVML_FI_DEV_POWER_INSTANT, with command-line equivalents (nvidia-smi --query-gpu=power.draw.average and power.draw.instant).

NVML: the energy counter

nvmlDeviceGetTotalEnergyConsumption returns accumulated energy in millijoules since the last driver reload, supported on Volta or later. It is a 64-bit integer, so overflow is not a practical problem: an H100 at 700 W consumes of the order of 6.1·10¹⁰ J per year against a counter range of 1.8·10¹⁶ J. The only event that resets it is a driver reload or a system restart, and that does need watching: a modprobe -r nvidia halfway through a campaign produces a negative delta.

NVIDIA does not document the internal time resolution of the counter, its accumulation mechanism or its accuracy, and I have found no peer-reviewed work validating it against an external meter. The recommendation to use it, which the rest of this sheet rests on, is a structural argument (it removes quadrature error and aliasing error), not a published empirical validation.

NVML: power limits

Function	Unit	Semantics
`nvmlDeviceGetPowerManagementLimit`	mW	Currently configured limit
`nvmlDeviceGetPowerManagementLimitConstraints`	mW	Legal minimum and maximum range
`nvmlDeviceGetPowerManagementDefaultLimit`	mW	Limit the card boots with
`nvmlDeviceGetEnforcedPowerLimit`	mW	Effective limit after considering all limiters, including the out-of-band interface

For an auditable measurement you have to record EnforcedPowerLimit, not PowerManagementLimit: the BMC may be imposing an out-of-band ceiling that the latter does not reflect, and two apparently identical nodes may be operating at different ceilings.

DCGM: the equivalent fields

Field	ID	Unit	Type in the exporter
`DCGM_FI_DEV_POWER_USAGE`	155	watts in floating point	gauge
`DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION`	156	accumulated millijoules	counter
`DCGM_FI_DEV_POWER_MGMT_LIMIT`	160	mW	gauge
`DCGM_FI_DEV_ENFORCED_POWER_LIMIT`	164	mW	gauge

Both energy fields are active in the default counter file of dcgm-exporter. The audit conclusion is direct: a standard Prometheus panel integrating DCGM_FI_DEV_POWER_USAGE is integrating a gauge that is already a one-second average of the sensor, sampled at the exporter’s collect-interval. For energy per token the correct metric is counter 156 with increase().

The real accuracy of the sensor

The reference is the work of Yang, Adámek and Armour (University of Oxford), published at SC24 and available as arXiv:2312.02741: more than 70 GPUs across 25 models, including 10 H100s and 10 A100s, checked against an external meter with a 1 mΩ shunt, a 12-bit ADC and internal sampling at 34 kHz.

Update period and averaging window

GPU or architecture	Update period	Averaging window
Kepler and Maxwell	10-50 Hz	logarithmic growth, ~200 ms
Volta (V100)	20 ms	10 ms
Turing	100 ms	100 ms
A100 (GA100)	101 ms	25 ms
Non-GA100 Ampere and Ada, driver earlier than March 2023	100 ms	1 s
Non-GA100 Ampere and Ada, driver 530	100 ms	100 ms
H100	100 ms	25 ms with `.instant`; 1 s with `.average` and with the default field
GH200	100 ms	20 ms on the GPU, 10 ms on the Grace CPU

Two readings of this table. The first: the driver is an experimental variable, not an infrastructure detail; the averaging window changed between drivers earlier than 530, driver 530, and the later ones. Any measurement meant to be compared over time has to record the driver version.

The second is the central finding of the paper. On the A100 and H100 the 25 ms window sits inside a 101 ms period, which gives a sensor duty cycle close to 25 %. The sensor does not average the full interval: it averages a quarter of it and the system presents the result as if it described the whole interval. In the words of the work itself, during the other 75 % of the time the GPU may be consuming radically different power and nvidia-smi never notices.

The steady-state error

The paper also quantifies the claim in the nvidia-smi documentation, which spoke of an accuracy of ±5 watts: the real error is proportional, ±5 %, not absolute. On a GPU capable of consuming 700 W that is ±35 W of over- or underestimation. The errors measured against the external meter, after correcting the time offset, were −4.70 % and −4.53 % on the RTX 3090 and −5.43 % on the A100, and the residue is attributed to the tolerance of the card’s own physical shunt, which no software can correct.

The response to transients

Four distinct behaviours were observed in the 10 % to 90 % rise time: an almost instantaneous real rise with the reading following on the next tick and a delay of 0 to 100 ms; a real rise of several hundred milliseconds with the reading still updating on the next tick; linear growth over a second, which corresponds to power.draw.average; and logarithmic growth over 200 ms, only on Kepler and Maxwell. Out of this comes the most uncomfortable warning in the work: when running a short program, the measured power probably corresponds to the activity before the program.

The two ways of obtaining energy

The first is to integrate samples, which is what any poller does:

$$E \approx \sum_i P(t_i) \cdot \Delta t$$

It accumulates five sources of error: the ±5 % of the sensor, quadrature error from the finite step, aliasing if the load has spectral content above half the sampling frequency, the fact that each sample is already a moving average, and the jitter of 0 to 100 ms between the host clock and the sensor tick.

The second is to subtract the counter:

$$E = \text{counter}(t_1) - \text{counter}(t_0)$$

Two calls and a subtraction. It removes quadrature, aliasing and time misalignment; the ±5 % of the sensor remains.

Why 1 Hz destroys a 200 ms inference

With sampling at 1 Hz, the sampling theorem only allows components below 0.5 Hz to be reconstructed, while a 200 ms pulse has content in the region of 5 Hz. The expected number of samples inside the event is 0.2, so with probability 0.8 none falls inside it: the result is not noisy, it is undefined.

Even sampling at the maximum useful rate, which on the A100 and H100 is about 10 Hz given the 101 ms period, a 200 ms inference produces one or two distinct sensor updates, and each has observed only 25 ms of real activity: effective coverage is around 25 % of the event. And because of the fourth transient behaviour in the previous section, the reading obtained during those 200 ms may describe what was happening before the kernel was launched.

Aliasing with periodic loads

The paper observes the phenomenon directly on the A100: with a square wave of period slightly different from 100 ms, the reading fluctuates between high and low values with a clear beat. An inference server with periodic arrivals, health checks, fixed-size batches or a scheduler tick is exactly that pathological case.

When only instantaneous power is available, the measured good practice from the Oxford work consists of running 32 consecutive iterations or a minimum of 5 s, inserting 8 evenly spaced controlled delays if the window is smaller than the period, repeating over 4 separate trials with a random delay between them, and shifting the series in post-processing to synchronise it with the real activity. Over nine real benchmarks that took the mean error from 39.27 % to 4.89 %.

What the GPU counter does not measure

The scope declared by NVML is the GPU and its associated circuitry: the SXM module or the PCIe card, with its HBM and its regulators. Left out are the CPU, host DRAM, NVSwitch, network cards, storage, fans and power supply losses.

The measured node figures come from the empirical calibration work on 8× H100 SXM5 nodes (arXiv:2506.14551):

Quantity	Value
Nominal node TDP declared by the manufacturer	10.2 kW
Node power at idle	1.8 kW
Maximum measured under saturating load	8.4 kW
Peak on real production loads	never above 76 % of TDP

Two conversion factors come out of that, both my own estimates from those figures: 8 × 700 W is 5.6 kW of GPU against 8.4 kW of node, that is, the GPU is of the order of 67 % of node power and the GPU-to-node factor is around 1.50×. Chaining that with an industry average PUE of 1.54 (Uptime Institute, 2025), the total factor from the NVML counter to the utility feed is around 2.3×; with a PUE of 1.1 it drops to 1.65×. Put in the form that matters to whoever signs the bill: the watts NVML reports are roughly 43 % of the watts the utility bills in an average datacenter.

As a complementary reference, an 80 PLUS Titanium supply at 230 V is 96 % efficient at 50 % load and 91 % at 100 %, so supply losses add between 4 % and 9 % that is already included if you measure at the socket.

The tax of keeping a context open

A GPU serving a loaded model with no traffic does not consume its nominal idle. Measurements over 335,267 production samples from 14 H100s across 18 days, plus controlled experiments, give this table:

GPU	Idle without a CUDA context	Idle with a CUDA context	Extra cost	Percentage of TDP
H100	71.8 W	121.7 W	+49.9 W	7.1 %
A100	53.7 W	80.0 W	+26.3 W	8.8 %
L40S	35.6 W	102.1 W	+66.4 W	19.0 %

The result that changes the mental model: more than 98 % of that extra cost is produced by the open CUDA context, independently of the memory occupied. Varying allocated VRAM between 0 and 72 GB moves power by less than 1 W. The size of the loaded model does not cost watts; keeping the context open does.

That cost is not marginal on a real platform. Over 11,791 long-running jobs, the measured split was 24 % of the time and 7 % of the energy in deep idle, 15 % of the time and 10 % of the energy idle with an active context, and 61 % of the time with 83 % of the energy in execution. On serving loads, GPUs can spend 48 % of their energy in low-activity periods.

CPU and DRAM: RAPL

To close the node you have to measure what is not GPU, and there the interface is RAPL through the kernel powercap framework, at /sys/devices/virtual/powercap/intel-rapl/. The hierarchy exposes intel-rapl:N as the socket and intel-rapl:N:M as subzones (core, uncore, dram), plus the psys domain for the full SoC since Skylake.

File	Content
`energy_uj`	Energy counter in microjoules; writing “0” resets it if the counter supports it
`max_energy_range_uj`	Counter range, that is, the overflow point
`power_uw`	Current power in microwatts
`name`	Zone name

The figures that condition its use come from the characterisation published in ACM TOMPECS: an update roughly every 1 ms, with jitter; an energy quantum of 61 µJ on Haswell and Skylake, 15.3 µJ on Sandy Bridge; and counter overflow in 52 minutes on a Haswell at 84 W, which forces polling at a far shorter period and detecting the wrap with max_energy_range_uj. There is also a thermal drift that is rarely declared: package power for the same load grows between 10 % and 12 % between 37 °C and 74 °C on Haswell. Warming up beforehand is not optional.

Access, on the other hand, is no longer free. The PLATYPUS attack (IEEE S&P 2021) demonstrated a purely software power side channel through RAPL, with CVE-2020-8694 and CVE-2020-8695 attached. The Linux mitigation arrived in commit 949dd0104c49, included in 5.10.0-rc4, which changed the default permissions so that only root can read energy_uj. The microcode mitigation associated with SGX goes further and introduces random noise into the reported energy and changes the reporting frequency: on machines with that mitigation active, RAPL readings are deliberately degraded. Measuring CPU and DRAM from a container therefore requires root, a udev rule relaxing the permissions, or a privileged daemon.

The software stack

Tool	Source of the GPU figure	Counter or power	Default interval	Overhead
Zeus	NVML	Counter if the architecture is Volta or later; polling otherwise	Delimited by windows	Less than 10 ms per call
dcgm-exporter	NVML via DCGM	Both fields available	`collect-interval` = 30,000 ms	5-10 W extra on the server’s IPMI reading
CodeCarbon	NVML	Counter	`measure_power_secs` = 15 s	5.38 % to 46.75 % of time at 1 kHz
Scaphandre	No GPU support; RAPL	`energy_uj` counter	Not documented	3.81 % to 28.38 % at 1 kHz
Kepler	RAPL and IPMI; GPU only as an experimental option	Hardware counters	—	—

Four details that change the choice:

Zeus decides in code with nvmlDeviceGetArchitecture and uses the accumulated counter on Volta or later, with a nestable window API (begin_window / end_window). It is the reference implementation of what this sheet recommends. Its NSDI'23 paper also documents energy reductions of 23.8 % to 75.7 % from choosing batch size and power limit well, and of 3.0 % to 31.5 % from touching only the limit, with five seconds of profiling per point enough for stable results.
Kepler has removed eBPF. The CNCF announcement of June 2026 describes the redesign towards reading /proc and /sys, motivated by the CAP_BPF and CAP_SYSADMIN permissions that blocked deployments, and by the loss of short-lived processes that underestimated the footprint. The new node metric closely follows the IPMI pattern and removes the spurious multi-kilowatt spikes of the previous version. For GPUs it remains an experimental option: today it is not the tool for measuring Wh/token on an accelerator.
CodeCarbon uses the NVML counter, which is correct, but its RAM estimate is a heuristic of 5 W per module derived from the total gigabytes, and the code itself acknowledges it.
DCGM delivers profiling metrics at 1 Hz by default, and warns that collecting at higher frequencies returns zeros because it groups metrics internally.

On the cost of measuring, the empirical overhead study of RAPL-based tools, all forced to 1 kHz, leaves a clear ordering:

Tool	Time overhead
CodeCarbon	5.38 % to 46.75 %
Scaphandre	3.81 % to 28.38 %
Turbostat	2.73 % to 14.37 %
PowerJoular	1.67 % to 8.88 %
perf	−1.00 % to 4.26 %
RAPL read in user space	−0.70 % to 2.93 %
RAPL read in kernel space	−0.17 % to 0.99 %

The underlying difference is that a system call costs of the order of 1.36·10⁻³ ms and an rdmsr instruction between 2.3·10⁻⁴ and 5.6·10⁻⁴ ms, an order of magnitude less. The authors recommend matching the granularity to the phenomenon being measured instead of raising the default frequency, and limiting polling to the necessary domains.

Procedure for making a Wh/token auditable

Fix and record the electrical state. Note EnforcedPowerLimit, the default limit and the driver version; lock clocks or the power ceiling if the experiment requires it.
Warm up to a stable temperature. RAPL thermal drift is 10 % to 12 % between 37 °C and 74 °C, and the power.draw.average transient takes up to a second to settle.
Measure the idle baseline twice: without a CUDA context and with the model loaded. They are two different values and both are needed.
Use windows of at least 5 s or 32 consecutive iterations.
Repeat over 4 separate trials with a random delay, and add 8 evenly spaced delays if you are integrating instantaneous power with a window smaller than the period.
Prefer the accumulated counter whenever the architecture is Volta or later, watching for driver reloads.
Report dispersion. The Oxford paper itself publishes the mean and standard deviation of the error; without an interval, a Wh/token figure is not auditable.
Declare the measurement boundary: GPU, node or utility feed. Without that label, two figures from the same facility differ by a factor of 2.3 with nobody knowing why.

Subtracting idle, or not

There is no consensus, and the honest approach is to treat it as a declared decision. In production the GPU is parked with the model loaded, and that consumption shows up on the bill: subtracting it erases a real cost. To compare the marginal efficiency of two models or two kernels, by contrast, subtracting it isolates the variable. The workable way out is to report both figures with labels: gross Wh/token, which includes idle and corresponds to the bill, and marginal Wh/token, comparable between models. What does not work is mixing them in the same table without distinguishing them.

One estimation trap remains alive in many spreadsheets: multiplying TDP by time overestimates real energy by up to 4.1 times, according to the ML.ENERGY Benchmark.

Power capping: what it costs and what it saves

Configuration	Energy saving	Performance loss
V100 at 200 W against 300 W (67 % of TDP), BERT training	~15 %	barely any degradation
V100 at 200 W, set of workloads	10-20 %	less than 5 %
V100 at 100 W (33 % of TDP)	40-60 %	30-40 %
A100 at 175 W against 400 W (44 % of TDP), LLaMA 65B inference	22-24 %	5-8 %

The asymmetry between phases explains the result. The Microsoft measurement at ASPLOS'24 documents that the prompt phase is short and reaches or exceeds TDP, while the token generation phase is longer and consumes less: the power ceiling penalises TTFT far more than TPOT. In their production cluster, inference uses 79 % of peak power against 97 % for training, and their measurements show that up to 20 % of power can be recovered with less than a 7 % performance loss.

The published sweep from 200 W to 700 W on the H100 and H200 adds the nuance of where the knee is: between 500 W and 700 W each 100 W step only contributes around 10 % of performance on compute-bound loads, while the H100 holds peak memory bandwidth even at 200 W. The most stable operating zones identified are 400 W on the H100 and 500 W on the H200. That the memory subsystem is protected against the trim is the physical reason why the decode phase of an LLM, which is memory-bound, loses so little performance under a power ceiling.

Two warnings about those figures: the H100/H200 sweep was measured with nvidia-smi polling every 10 s, exactly the problem described above, although on long steady loads the bias averages out; and there is no publication isolating the effect of memory clock locking on Wh/token in LLM inference, so any concrete figure on that point has to be your own measurement.

Reference figures

GPU	TDP
H100 SXM	up to 700 W, configurable
H100 NVL	350-400 W, configurable
A100 80GB SXM	400 W
A100 80GB PCIe	300 W
RTX 5090	575 W, with a recommended 1000 W supply
HGX node of 8× H100 SXM	10.2 kW nominal

Average power measured in inference: on an HPC cluster with vLLM on A100, the aggregate power of the GPU set ranged between 999 W and 2,983 W depending on model and parallelism, with Nemotron 70B on 8× A100 at 2,983 W, about 373 W per GPU, around 93 % of TDP.

Energy per token, with the warning that published figures measure different boundaries and cannot be mixed without normalising by GPU count and output tokens:

Source	Figure
TokenPowerBench, H100 94 GB	40 J per token on a standard load, more than 60 J per token in high-throughput configurations
TokenPowerBench, batch effect	−25 % of energy per token between batch 32 and 256 on a 70B model
TokenPowerBench, context effect	from 2K to 10K tokens of context on Llama 3 70B: energy ×3
TokenPowerBench, quantisation effect	Llama 3 405B in FP8 against FP16: −30 % of energy per token
TokenPowerBench, engine effect	TensorRT-LLM and vLLM against the Transformers engine: −25 % to −40 %
Per-prompt measurement in batch	0.0074-0.0289 Wh on 7B to 14B models; 0.0835-0.6912 Wh on 70B to 405B; single unbatched requests, between 10 and 100 times more expensive

Cross-links from the energy track

C2 — Energy per token: methodology and the electricity market: what to do with the joule once it is measured properly, and how the electricity price multiplies it.
C3 — Measuring energy in production: Kepler, DCGM and the practical stack: the continuous deployment of the stack whose metrology this sheet documents.
C3 — Energy benchmarking: frameworks, metrics and the state of the art: the full catalogue of tools and their metrics.
C4 — MLPerf Power: the certified alternative, measuring at the wall with an external analyser instead of the card’s sensor.
C6 — From watt to carbon: PUE, grid and grams of CO₂: the chain that turns the measured watt into emissions and into euros.
C7 — Energy efficiency levers: the quantified catalogue where the power ceiling of this sheet is just one more lever.

Sources

NVIDIA, NVML API Reference, Device Queries — https://docs.nvidia.com/deploy/nvml-api/group__nvmlDeviceQueries.html
NVIDIA, DCGM Library API, Field Identifiers — https://docs.nvidia.com/datacenter/dcgm/3.1/dcgm-api/dcgm-api-field-ids.html
NVIDIA, DCGM Feature Overview — https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html
NVIDIA, dcgm-exporter — https://github.com/NVIDIA/dcgm-exporter
NVIDIA, H100 Tensor Core GPU, especificaciones — https://www.nvidia.com/en-us/data-center/h100/
NVIDIA, A100 Tensor Core GPU, especificaciones — https://www.nvidia.com/en-us/data-center/a100/
NVIDIA, GeForce RTX 5090, especificaciones — https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/
Yang, Adámek y Armour (Oxford), Part-time Power Measurements: nvidia-smi’s Lack of Attention / SC24 — https://arxiv.org/html/2312.02741v3
Yang, GPU_Power_Benchmark, microbenchmark del trabajo anterior — https://github.com/JimZeyuYang/GPU_Power_Benchmark
ML.ENERGY, Measuring GPU Energy: Best Practices — https://ml.energy/blog/energy/measurement/measuring-gpu-energy-best-practices/
You et al., Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training, NSDI'23 — https://www.usenix.org/system/files/nsdi23-you.pdf
Zeus, Measuring Energy — https://ml.energy/zeus/measure/
ML.ENERGY, The ML.ENERGY Benchmark — https://arxiv.org/html/2505.06371v1
CNCF, Kepler, re-architected: improved power accuracy (junio 2026) — https://www.cncf.io/blog/2026/06/30/kepler-re-architected-improved-power-accuracy-and-a-community-call-to-action/
CodeCarbon, How Power Estimation Works — https://docs.codecarbon.io/latest/explanation/power-estimation/
Hubblo, Scaphandre: cálculo del consumo por proceso — https://github.com/hubblo-org/scaphandre/blob/main/docs_src/explanations/how-scaph-computes-per-process-power-consumption.md
Linux Kernel, Power Capping Framework (powercap sysfs) — https://docs.kernel.org/power/powercap/powercap.html
Intel, Running Average Power Limit Energy Reporting, INTEL-SA-00389 — https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/advisory-guidance/running-average-power-limit-energy-reporting.html
Lipp et al., PLATYPUS: With Great Power comes Great Leakage, IEEE S&P 2021 — https://platypusattack.com/
Khan et al., RAPL in Action: Experiences in Using RAPL for Power Measurements, ACM TOMPECS — https://www.devsustainability.com/p/paper-notes-rapl-in-action
What Is the Cost of Energy Monitoring? An Empirical Study on the Overhead of RAPL-Based Tools — https://arxiv.org/html/2604.26815v1
Patel et al. (Microsoft), Characterizing Power Management Opportunities for LLMs in the Cloud, ASPLOS'24 — https://www.microsoft.com/en-us/research/wp-content/uploads/2024/03/GPU_Power_ASPLOS_24.pdf
MIT Lincoln Laboratory, Sustainable Supercomputing for AI: GPU Power Capping at HPC Scale — https://arxiv.org/html/2402.18593
Architectural Trade-offs in the Energy-Efficient Era: power-capping NVIDIA H100 and H200 — https://arxiv.org/html/2604.11391v1
Empirically-Calibrated H100 Node Power Models for Datacenter Energy Analysis — https://arxiv.org/pdf/2506.14551
The Model Parking Tax: Quantifying the Hidden Energy Cost of Always-On GPU Model Deployment — https://arxiv.org/html/2605.23918
The Energy Cost of Execution-Idle in GPU Clusters — https://arxiv.org/html/2604.04745
TokenPowerBench: Benchmarking the Power Consumption of LLM Inference — https://arxiv.org/pdf/2512.03024
From Prompts to Power: Measuring the Energy Footprint of LLM Inference — https://arxiv.org/html/2511.05597
Serving LLMs in HPC Clusters — https://arxiv.org/abs/2507.00418
Uptime Institute, Global Data Center Survey 2025 — https://datacenter.uptimeinstitute.com/rs/711-RIA-145/images/2025.Annual.Survey.Report.pdf

LLM energy leaderboards: comparing models by Wh/token and choosing on efficiency

Tue, 16 Jun 2026 04:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter). Generic example hardware; no real infrastructure.

TL;DR

There are three OSS leaderboards with public data and documented methodology for comparing the energy efficiency of LLMs in inference: Hugging Face AI Energy Score (166 models, Wh/query on H100, 1–5 star scale, launched February 2025), ML.ENERGY Leaderboard v3 (University of Michigan, J/token per task, Zeus tooling, December 2025) and MLPerf Power (samples/joule certified with a physical Yokogawa WT310E power meter). The three measure different dimensions and are not directly interchangeable. The available data shows that reasoning models consume up to 700× more energy than their non-reasoning equivalents; that MoE models consume roughly 3× fewer J/token than a dense model with equivalent active parameters; and that INT4 quantisation cuts consumption by up to 79 % against FP16 under favourable conditions. The inference engine (vLLM vs Transformers) can move the result another 25–40 %. Without fixing hardware, engine, batch size and task, no comparison between leaderboards is valid.

Track context

This article is C5 of the energy pillar. The base context:

The fundamentals of quantisation are a prerequisite for the quantisation section of this article.

The three leaderboards: technical data sheet

1 · Hugging Face AI Energy Score

Field	Detail
URL	huggingface.co/AIEnergyScore · huggingface.co/spaces/AIEnergyScore/Leaderboard
Organisation	Hugging Face (Sasha Luccioni et al.), with Salesforce and Cohere as initial partners
Launch	February 2025 (AI Action Summit, Paris); v2 December 2025
Models indexed	166 (v1, Feb 2025); +39 new ones in v2 (Dec 2025)
Tasks measured	10 tasks: text generation, summarisation, classification, image generation, ASR, audio generation, translation, question answering, reasoning (added in v2)
Unit of measurement	Wh (watt-hour) per 1,000 queries of the task
Reference hardware	NVIDIA H100 exclusively (a single GPU for class A/B models; multiple for class C)
Measurement tooling	CodeCarbon (GPU energy) + the `ai-energy-benchmarks` package (OSS, PyPI)
Rating system	1–5 stars per task: quintiles of the energy range; ⭐⭐⭐⭐⭐ = the 20 % most efficient
Reference batch size	Batch size = 1 (does not reflect production with aggressive batching)
Access for proprietary models	Yes, via an audited Docker container
Update frequency	No fixed cadence; v1 Feb 2025, v2 Dec 2025
Project licence	Apache 2.0 (repository github.com/huggingface/AIEnergyScore)

Scope of the metric. The AI Energy Score measures GPU energy only (CodeCarbon); it does not capture CPU, DRAM or system overhead. The Wh/1k-queries unit covers the whole run time (prefill + decode + framework overhead), but at batch = 1. The results are therefore comparable between models under the same test conditions, but not extrapolable to a production environment with real concurrency without correction.

Model class (the project’s internal classification):

Class	Definition
A	Fits on a consumer GPU (≤ ~24 GB VRAM)
B	Requires a cloud GPU (≥ 40 GB VRAM)
C	Requires multiple GPUs

2 · ML.ENERGY Leaderboard

Field	Detail
URL	ml.energy/leaderboard
Organisation	Symbiotic Lab, University of Michigan (Mosharaf Chowdhury, Jae-Won Chung et al.)
Reference paper	arXiv 2505.06371 — “The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization” (NeurIPS 2025 D&B, Spotlight)
Current version	v3.0 (December 2025)
Measurement tooling	Zeus (github.com/ml-energy/zeus) via NVML + RAPL; measurement overhead in single-digit ms
Unit of measurement	J/token (energy per output token generated) and total energy per complete response
Reference hardware	NVIDIA A100 80 GB and H100 SXM (declared per submission; varies between models)
Tasks measured	6 tasks: chat (text conversation), reasoning, code generation, summarisation, visual question answering, video generation
Normalisation	Mean energy per complete response (prefill + decode). Output J/token is also reported. The task is stated explicitly because it determines the output length
Scope of the measurement	GPU via NVML + CPU/DRAM via RAPL; not a power meter at the wall
Models covered	~40 architectures in the NeurIPS 2025 version; the web leaderboard is updated with more
Licence	Apache 2.0 (zeus: github.com/ml-energy/zeus); MIT (benchmark: github.com/ml-energy/benchmark)
Update frequency	Continuous on the web leaderboard; the paper is a point-in-time snapshot

Zeus as tooling. Zeus is the measurement engine of the ML.ENERGY Leaderboard and also a standalone package (pip install zeus-ml). It supports NVIDIA GPU (NVML), AMD GPU (ROCm), CPU (RAPL), DRAM (RAPL), Apple Silicon and NVIDIA Jetson. The ZeusMonitor adds measurement overhead in single-digit milliseconds. Since May 2025 it has been a PyTorch ecosystem project. MIT licence.

3 · MLPerf Power

The full data sheet is in the C4 article. A summary of the points relevant for comparison with the previous two:

Field	Detail
URL	mlcommons.org/benchmarks/inference-datacenter/
Organisation	MLCommons Power Working Group (>20 orgs)
Unit of measurement	samples/joule (throughput/mean power) = the inverse of J/sample
Hardware	Complete node measured at the wall (AC); Yokogawa WT310E analyser (±0.1 % of reading)
LLM tasks	GPT-J 6B, Llama 2 70B, Mixtral 8×7B (from v5.0)
Granularity	Complete node (GPU + CPU + RAM + fans + PSU losses); no attribution to individual workloads
Node overhead over GPU	25–45 % of total consumption in submissions with a physical analyser
Licence of the corpus	Public results on GitHub (mlcommons/inference_results_vX.Y); PTDaemon requires MLCommons membership

Comparison of the three leaderboards

Dimension	HF AI Energy Score	ML.ENERGY Leaderboard	MLPerf Power
Unit	Wh/1k-queries	Output J/token	samples/J (complete node)
Fixed hardware	H100 (all models)	A100/H100 (varies)	Depends on the submitter
Measurement	CodeCarbon (GPU)	Zeus NVML+RAPL	Physical AC power meter (Yokogawa)
System coverage	GPU only	GPU + CPU + DRAM	Complete node including fans and PSU
Batch size	1	Varies by task	Per LoadGen scenario
Models covered	166+ (text, image, audio)	~40 generative LLMs	Few (GPT-J, Llama 2, Mixtral)
Proprietary models	Yes (audited Docker)	No (OSS only)	Yes (MLCommons members)
External certification	No	No	Yes (SPEC PTDaemon)
Frequency	Point-in-time (v1, v2)	Continuous	Half-yearly (MLPerf rounds)
Licence	Apache 2.0	Apache 2.0 / MIT	Public results; PTDaemon: membership

Incompatibility between leaderboards. The three measure different dimensions: Wh/query ≠ J/token ≠ node samples/J. A direct comparison requires converting units and assuming that the hardware, the engine and the task are equivalent, which is rarely the case across leaderboards.

How energy per token is measured and normalised

The base identity is developed in the C2 article:

$$E_{\text{token}} \,[\text{J/tok}] = \frac{\bar{P} \,[\text{W}]}{\text{throughput} \,[\text{tok/s}]}$$

To compare models against each other, every factor other than the model must be fixed:

Factor	Effect if it varies	How to fix it
Hardware	H100 vs A100 vs L40S changes the result by 2–4×	Declare the exact hardware; compare only within the same HW
Inference engine	vLLM vs Transformers: 25–40 % difference in J/token	Fix the engine and the version
Batch size / concurrency	Batch 1 vs batch 32: throughput rises but so does power; the ratio varies	Declare the batch size; compare within the same regime
Model precision	FP16 vs INT8 vs INT4: up to −79 % energy	Declare the precision; do not mix
Response length	A query with 50 tokens ≠ one with 500	Use a fixed dataset or normalise per token
Measurement window	Including warm-up or idle inflates the numerator	Align the power window with the token window (see C2)

Conversion formula Wh/query ↔ J/token:

$$E_{\text{J/tok}} = \frac{E_{\text{Wh/query}} \times 3600}{\bar{n}_{\text{tokens/query}}}$$

Example: if a model consumes 0.05 Wh/query (= 180 J/query) and generates an average of 200 tokens per query:

$$E_{\text{J/tok}} = \frac{0{.}05 \times 3600}{200} = \frac{180}{200} = 0{.}9 \,\text{J/tok}$$

AI Energy Score data: concrete examples

The v2 data (December 2025, H100 hardware, batch = 1, text generation task with reasoning on/off):

Model	Active params	Reasoning	GPU Wh/1k queries	Stars (text-gen)
DistilGPT-2	82 M	—	1.31	⭐⭐⭐⭐⭐
SmolLM3-3B	3 B	Off	18.35	⭐⭐⭐⭐
SmolLM3-3B	3 B	On	12,791.22	⭐
Phi-4-reasoning-plus	15 B	Off	18.42	⭐⭐⭐⭐
Phi-4-reasoning-plus	15 B	On	9,461.61	⭐
DeepSeek-R1-Distill-Llama-70B	70 B	Off	49.53	⭐⭐⭐
DeepSeek-R1-Distill-Llama-70B	70 B	On	7,626.53	⭐

Source: Hugging Face AI Energy Score v2 (Dec 2025).

The reasoning multiplier. The energy increase when reasoning is switched on ranges from ×154 (DeepSeek-R1-Distill-Llama-70B) to ×697 (SmolLM3-3B). The direct cause: reasoning models generate between 300 and 800 times more tokens than their non-reasoning equivalents (internal chains of thought). The mean across the v2 corpus is ×30 extra energy for reasoning.

Newer models are not always more efficient. Of the 14 comparable models (no reasoning, no MoE, similar size) between the Feb 2025 and Dec 2025 cohorts: 8 out of 14 had equal or higher energy. The range runs from 3 % of the reference model’s energy to almost 2×. Parameter scale is no longer enough to estimate efficiency.

ML.ENERGY Leaderboard data: J/token by family

The data from the arXiv 2505.06371 paper and from leaderboard v3 (A100/H100 hardware, vLLM as the reference engine):

Scale within one family (Llama 3):

Size	Params	Relative J/token (base = 1B)	Params/energy ratio
Llama 3 · 1B	1 B	1.0×	—
Llama 3 · 8B	8 B	~2.1×	8× params → 2.1× energy
Llama 3 · 70B	70 B	~7.3×	70× params → 7.3× energy

The sublinearity (70× params → 7.3× energy, not 70×) reflects that inference energy is dominated by memory bandwidth (memory-bandwidth bound), not by raw FLOPs.

Dense vs MoE:

Model	Type	Total params	Active params/token	Relative J/token
Llama 3 · 8B	Dense	8 B	8 B	1.0×
Mixtral 8×7B	MoE (top-2)	47 B	~13 B	~0.33×
Llama 3 · 70B	Dense	70 B	70 B	~3.5×

The MoE activates only 2 of 8 experts per token. Mixtral 8×7B consumes roughly ⅓ of the J/token of a dense model with 8B active, at a quality comparable to a dense model of larger scale. Routing overhead and holding all the experts in memory cancel out part of the theoretical gain.

Effect of the task (ML.ENERGY v3, same model):

Task	Energy multiplier per response (vs chat)
Chat (text conversation)	1× (reference)
Summarisation	~2–4×
Code generation	~3–6×
Reasoning	~25×
Image + text	1.1–5.2×
Video + text	1.3–15.0×

Reasoning uses ~10× more tokens per response, and the extra memory of the chain of thought reduces the effective batch size, raising energy per token through memory pressure.

Effect of quantisation on energy per token

Data on NVIDIA H100 hardware, Llama 3 family (arXiv 2508.16712 and arXiv 2504.03360):

Precision	Energy reduction vs FP16	Condition
FP16	reference (0 %)	—
BF16	~0 % (iso-energy)	Same hardware and engine
FP8	−25 to −35 %	H100/H200 with native hardware support
INT8	−23 to −44 % (median ~39 %)	Depends on batch size; more at low batches
INT4 (AWQ / GPTQ)	−50 to −79 %	Requires hardware with efficient low-precision support

Warning. On GPUs without native INT4 hardware support (or with suboptimal dequantisation kernels), quantisation can increase latency and energy per token rather than reduce them, because of run-time dequantisation overhead. The benefit of quantisation is real on H100/A100 with TensorRT-LLM or a well-configured llama.cpp, but it is not guaranteed with any engine.

Quantisation and throughput: the memory reduction per model frees VRAM, which allows larger batch sizes. At a larger batch, throughput rises more than power does, cutting J/token further still. The net effect can exceed the direct reduction in energy per operation.

Effect of the inference engine

The engine is a variable that model-level leaderboards tend to fix, but which in production is a decision of your own. Data from published comparisons (vLLM, TensorRT-LLM, naive Transformers, A100):

Engine	Relative J/token vs Transformers baseline
Transformers (naive, not optimised)	1.0× (reference)
vLLM (PagedAttention, continuous batching)	−25 to −35 %
TensorRT-LLM (optimised NVIDIA kernels, FP8)	−35 to −45 %
llama.cpp (hybrid CPU/GPU, INT4)	Variable; −30 to −60 % depending on hardware

Moving from naive Transformers to TensorRT-LLM can cut energy per token by more than moving from a 70B model to an 8B one of the same origin. The choice of engine is a first-order lever on energy efficiency.

Limits of energy leaderboards

Limit	Description
Hardware dependence	A ranking on H100 is not valid on A100 or L40S without correction. The hierarchy of models can change from one piece of hardware to another.
Engine dependence	The results are valid only for the engine they were measured with. A model that is ×2 more efficient on the leaderboard can fall behind if a slower engine is used.
Artificial batch size	AI Energy Score uses batch = 1. In production with aggressive batching, the efficiency relationship between large and small models changes: the large ones scale better with batch.
Training not captured	All the leaderboards measure inference only. The energy cost of training (which can exceed that of inference by 1,000× over the life of the model) is out of scope.
Incompatibility between leaderboards	Wh/query, J/token and samples/J measure different things. Converting between them requires knowing the mean output length, which varies by task and dataset.
Partial system coverage	AI Energy Score and ML.ENERGY measure GPU (+CPU/DRAM with Zeus); they do not capture the overhead of the complete system (PSU losses, fans, interconnect). MLPerf Power does, but covers few models.
Data latency	The leaderboards publish results months after the tests. New hardware (H200, B100, B200) may have no data available at the time of the decision.
No PUE	None of the three includes datacenter PUE. For real TCO, the leaderboard J/token must be multiplied by your own PUE.

Decision table: choosing a model on energy efficiency

The selection criteria in order, without recommendation prose:

Criterion	Question	Action
Task with reasoning	Does the task require step-by-step reasoning?	Yes → multiply the model’s base energy by ×30–700 before comparing. If there is a non-reasoning alternative with sufficient quality, prefer it.
Size vs minimum quality	What is the minimum acceptable quality for the task?	Consult quality benchmarks (see B7 when available). Pick the smallest model that clears the quality threshold; energy grows sublinearly with size.
Dense vs MoE	Does the hardware have enough memory for the full MoE?	If yes: the active-equivalent MoE consumes ~3× fewer J/token than the equivalent dense model. If not: paging or offload eats the gain.
Precision	Does the hardware have native FP8/INT4 support?	H100/H200: native FP8 (−30 %). With TensorRT-LLM: INT4 AWQ (−50 to −79 %). Without native support: stay on FP16 or BF16 until validated with your own benchmark.
Inference engine	Is the optimal engine for the hardware being used?	Measure with C3. If the engine is not optimised, changing engine can cut energy more than changing model.
Consult the leaderboard	Is the task covered by AI Energy Score or ML.ENERGY?	Filter by: same task, same hardware class, reasoning explicitly off/on. Do not compare models from different hardware classes or different engines.
Validate on your own hardware	Are the leaderboard results on the same HW as yours?	Always validate with Zeus or DCGM on your own hardware before taking the final decision. The leaderboard is a reference, not a prediction.

Quick signals table:

Signal	Effect on energy	Data source
Switching reasoning on	×30–700	AI Energy Score v2
Going from dense 8B to dense 70B	~×3.5	ML.ENERGY Leaderboard v3
Going from dense 8B to active-equivalent 8B MoE	~×0.33 (−67 %)	ML.ENERGY v3
FP16 → INT4 (compatible hardware)	−50 to −79 %	arXiv 2508.16712, 2504.03360
Naive Transformers → TensorRT-LLM FP8	−35 to −45 %	TokenPowerBench, ML.ENERGY
PUE 1.0 → PUE 1.5	+50 % on real datacenter energy	MLPerf Power (scope)

Reference data: energy in a generic node (4×H100 SXM)

Generic example hardware, to anchor the leaderboard values to a real node:

Parameter	Indicative value
TDP 4×H100 SXM 80 GB	4 × 700 W = 2,800 W (GPU only)
Complete-node system power (at the wall)	~3,500–5,000 W depending on load
Non-GPU overhead over GPU	25–45 %
J/token Llama 3 70B FP16, vLLM, batch 8	~1–3 J/tok (indicative, A100/H100)
J/token Llama 3 8B FP16, vLLM, batch 8	~0.3–0.7 J/tok (indicative)
J/token Mixtral 8×7B FP16, vLLM, batch 8	~0.4–0.8 J/tok (indicative)
Energy per 1M tokens (Llama 3 70B, PUE 1.4)	~0.5–1.2 kWh

The J/token values are indicative and depend heavily on batch size, prompt length, prefill/decode ratio and engine version. For certified values, consult the MLPerf Power submissions (mlcommons.org).

For the alternative reference node (4×A100 PCIe 80 GB, TDP ~300 W each):

Parameter	Indicative value
TDP 4×A100 PCIe	4 × 300 W = 1,200 W (GPU only)
Complete-node system power	~1,500–2,000 W
J/token Llama 3 70B FP16, vLLM	~2–5 J/tok (indicative; higher because of lower HBM bandwidth vs SXM)

How to use the leaderboards in practice

A decision flow based on the public data available:

Step	Action	Resource
1	Identify the dominant task of the workload	—
2	Consult AI Energy Score filtered by task and hardware class	huggingface.co/spaces/AIEnergyScore/Leaderboard
3	Note the models with ⭐⭐⭐⭐ or ⭐⭐⭐⭐⭐ on the task	Wh/1k-queries as a relative reference
4	Cross-check with ML.ENERGY for the J/token of each candidate	ml.energy/leaderboard
5	If any model is in MLPerf Power (Llama 2, GPT-J, Mixtral), consult the certified samples/J	mlcommons.org/benchmarks/inference-datacenter/
6	Select the 2–3 candidates with the best energy/quality ratio	—
7	Measure on your own hardware with Zeus or DCGM	github.com/ml-energy/zeus
8	Multiply the measured J/token by the datacenter PUE	J/token × PUE = effective J/token in the datacenter
9	Work out the electricity cost per token at the contracted price	See C2

Sources

Hugging Face · AI Energy Score · organización y leaderboard — https://huggingface.co/AIEnergyScore
Hugging Face · Announcing AI Energy Score Ratings (Luccioni et al., feb. 2025) — https://huggingface.co/blog/sasha/announcing-ai-energy-score
Hugging Face · AI Energy Score v2: Refreshed Leaderboard, now with Reasoning (Luccioni, Gamazaychikov, dic. 2025) — https://huggingface.co/blog/sasha/ai-energy-score-v2
Hugging Face · AIEnergyScore GitHub (Apache 2.0) — https://github.com/huggingface/AIEnergyScore
ML.ENERGY Initiative · Leaderboard — https://ml.energy/leaderboard
ML.ENERGY Initiative · Blog: Diagnosing Inference Energy Consumption with the ML.ENERGY Leaderboard v3.0 (dic. 2025) — https://ml.energy/blog/measurement/energy/diagnosing-inference-energy-consumption-with-the-mlenergy-leaderboard-v30/
arXiv 2505.06371 · The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization (Chung et al., NeurIPS 2025 D&B Spotlight) — https://arxiv.org/abs/2505.06371
ML.ENERGY Initiative · Zeus: Deep Learning Energy Measurement and Optimization — https://ml.energy/zeus/
GitHub ml-energy/zeus (MIT) — https://github.com/ml-energy/zeus
PyTorch Blog · Zeus: Deep Learning Energy Measurement and Optimization — https://pytorch.org/blog/zeus/
University of Michigan CSE · Power-hungry AI: Researchers evaluate energy consumption across models — https://cse.engin.umich.edu/stories/power-hungry-ai-researchers-evaluate-energy-consumption-across-models
arXiv 2512.03024 · TokenPowerBench: Benchmarking the Power Consumption of LLM Inference (dic. 2024) — https://arxiv.org/abs/2512.03024
arXiv 2508.16712 · Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective — https://arxiv.org/abs/2508.16712
arXiv 2504.03360 · Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency — https://arxiv.org/abs/2504.03360
Epoch AI · AI Energy Use: Data & Research — https://epoch.ai/topics/energy
MLCommons · MLPerf Inference Datacenter benchmark results — https://mlcommons.org/benchmarks/inference-datacenter/
arXiv 2410.12032 · MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems (Tschand et al., 2024) — https://arxiv.org/abs/2410.12032
Coalition for Sustainable AI · AI Energy Score as best practice in benchmarking — https://www.sustainableaicoalition.org/ai-energy-score-a-standardized-approach-to-evaluating-ai-model-energy-efficiency/

Wh-Por-Token on lo0 — Tech Blog

Measuring the power of a GPU: NVML, DCGM and the sampling errors that invalidate your watts

TL;DR

What the hardware exposes

NVML: power

NVML: the energy counter

NVML: power limits

DCGM: the equivalent fields

The real accuracy of the sensor

Update period and averaging window

The steady-state error

The response to transients

The two ways of obtaining energy

Why 1 Hz destroys a 200 ms inference

Aliasing with periodic loads

What the GPU counter does not measure

The tax of keeping a context open

CPU and DRAM: RAPL

The software stack

Procedure for making a Wh/token auditable

Subtracting idle, or not

Power capping: what it costs and what it saves

Reference figures

Cross-links from the energy track

See also

Sources

LLM energy leaderboards: comparing models by Wh/token and choosing on efficiency

TL;DR

Track context

The three leaderboards: technical data sheet

1 · Hugging Face AI Energy Score

2 · ML.ENERGY Leaderboard

3 · MLPerf Power

Comparison of the three leaderboards

How energy per token is measured and normalised

AI Energy Score data: concrete examples

ML.ENERGY Leaderboard data: J/token by family

Effect of quantisation on energy per token

Effect of the inference engine

Limits of energy leaderboards

Decision table: choosing a model on energy efficiency

Reference data: energy in a generic node (4×H100 SXM)

How to use the leaderboards in practice

See also

Sources