Zeus on lo0 — Tech Blog

Measuring the power of a GPU: NVML, DCGM and the sampling errors that invalidate your watts

Sat, 29 Aug 2026 08:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

TL;DR

A GPU power sensor does not return what almost everyone believes. The NVML documentation is explicit: on Ampere except GA100, and on later architectures including H100, nvmlDeviceGetPowerUsage returns power averaged over a one-second interval, not instantaneous power. The University of Oxford study presented at SC24, covering more than 70 GPUs across 25 models checked against an external 1 mΩ meter, adds the fact that breaks most published measurements: on the A100 and H100 the sensor only samples 25 % of the time (a 25 ms window within a 101 ms period), so during the remaining 75 % the GPU may be consuming something radically different. Integrating power naively over nine real benchmarks, the mean error was 39.27 %; applying good practice brought it down to 4.89 %. The operational consequence is short: if the hardware is Volta or later, use the accumulated energy counter (nvmlDeviceGetTotalEnergyConsumption, in millijoules) and subtract, rather than integrating samples. And the GPU counter is not the bill: on measured 8× H100 nodes, the node reaches 8.4 kW against 5.6 kW of summed GPU TDP, node idle is 1.8 kW, and with an average PUE of 1.54 the NVML watts are of the order of 43 % of the watts the utility bills.

What the hardware exposes

NVML: power

nvmlDeviceGetPowerUsage(device, unsigned int* power) returns milliwatts for the GPU and its associated circuitry, for example the memory. It has been supported since Fermi. The note in the official documentation is the part almost nobody quotes (NVML Device Queries):

On Fermi and Kepler the reading is accurate to ±5 % of current consumption. On Ampere (except GA100) or later, the API returns power averaged over a 1 s interval. On GA100 and earlier architectures instantaneous power is returned.

In other words, on an H100 or an L40S the usual call already delivers a one-second moving average. Anyone sampling it at 1 Hz and integrating is applying a second filter on top of an already filtered signal, without declaring it. To disambiguate, NVML exposes two separate fields accessible via nvmlDeviceGetFieldValues: NVML_FI_DEV_POWER_AVERAGE and NVML_FI_DEV_POWER_INSTANT, with command-line equivalents (nvidia-smi --query-gpu=power.draw.average and power.draw.instant).

NVML: the energy counter

nvmlDeviceGetTotalEnergyConsumption returns accumulated energy in millijoules since the last driver reload, supported on Volta or later. It is a 64-bit integer, so overflow is not a practical problem: an H100 at 700 W consumes of the order of 6.1·10¹⁰ J per year against a counter range of 1.8·10¹⁶ J. The only event that resets it is a driver reload or a system restart, and that does need watching: a modprobe -r nvidia halfway through a campaign produces a negative delta.

NVIDIA does not document the internal time resolution of the counter, its accumulation mechanism or its accuracy, and I have found no peer-reviewed work validating it against an external meter. The recommendation to use it, which the rest of this sheet rests on, is a structural argument (it removes quadrature error and aliasing error), not a published empirical validation.

NVML: power limits

Function	Unit	Semantics
`nvmlDeviceGetPowerManagementLimit`	mW	Currently configured limit
`nvmlDeviceGetPowerManagementLimitConstraints`	mW	Legal minimum and maximum range
`nvmlDeviceGetPowerManagementDefaultLimit`	mW	Limit the card boots with
`nvmlDeviceGetEnforcedPowerLimit`	mW	Effective limit after considering all limiters, including the out-of-band interface

For an auditable measurement you have to record EnforcedPowerLimit, not PowerManagementLimit: the BMC may be imposing an out-of-band ceiling that the latter does not reflect, and two apparently identical nodes may be operating at different ceilings.

DCGM: the equivalent fields

Field	ID	Unit	Type in the exporter
`DCGM_FI_DEV_POWER_USAGE`	155	watts in floating point	gauge
`DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION`	156	accumulated millijoules	counter
`DCGM_FI_DEV_POWER_MGMT_LIMIT`	160	mW	gauge
`DCGM_FI_DEV_ENFORCED_POWER_LIMIT`	164	mW	gauge

Both energy fields are active in the default counter file of dcgm-exporter. The audit conclusion is direct: a standard Prometheus panel integrating DCGM_FI_DEV_POWER_USAGE is integrating a gauge that is already a one-second average of the sensor, sampled at the exporter’s collect-interval. For energy per token the correct metric is counter 156 with increase().

The real accuracy of the sensor

The reference is the work of Yang, Adámek and Armour (University of Oxford), published at SC24 and available as arXiv:2312.02741: more than 70 GPUs across 25 models, including 10 H100s and 10 A100s, checked against an external meter with a 1 mΩ shunt, a 12-bit ADC and internal sampling at 34 kHz.

Update period and averaging window

GPU or architecture	Update period	Averaging window
Kepler and Maxwell	10-50 Hz	logarithmic growth, ~200 ms
Volta (V100)	20 ms	10 ms
Turing	100 ms	100 ms
A100 (GA100)	101 ms	25 ms
Non-GA100 Ampere and Ada, driver earlier than March 2023	100 ms	1 s
Non-GA100 Ampere and Ada, driver 530	100 ms	100 ms
H100	100 ms	25 ms with `.instant`; 1 s with `.average` and with the default field
GH200	100 ms	20 ms on the GPU, 10 ms on the Grace CPU

Two readings of this table. The first: the driver is an experimental variable, not an infrastructure detail; the averaging window changed between drivers earlier than 530, driver 530, and the later ones. Any measurement meant to be compared over time has to record the driver version.

The second is the central finding of the paper. On the A100 and H100 the 25 ms window sits inside a 101 ms period, which gives a sensor duty cycle close to 25 %. The sensor does not average the full interval: it averages a quarter of it and the system presents the result as if it described the whole interval. In the words of the work itself, during the other 75 % of the time the GPU may be consuming radically different power and nvidia-smi never notices.

The steady-state error

The paper also quantifies the claim in the nvidia-smi documentation, which spoke of an accuracy of ±5 watts: the real error is proportional, ±5 %, not absolute. On a GPU capable of consuming 700 W that is ±35 W of over- or underestimation. The errors measured against the external meter, after correcting the time offset, were −4.70 % and −4.53 % on the RTX 3090 and −5.43 % on the A100, and the residue is attributed to the tolerance of the card’s own physical shunt, which no software can correct.

The response to transients

Four distinct behaviours were observed in the 10 % to 90 % rise time: an almost instantaneous real rise with the reading following on the next tick and a delay of 0 to 100 ms; a real rise of several hundred milliseconds with the reading still updating on the next tick; linear growth over a second, which corresponds to power.draw.average; and logarithmic growth over 200 ms, only on Kepler and Maxwell. Out of this comes the most uncomfortable warning in the work: when running a short program, the measured power probably corresponds to the activity before the program.

The two ways of obtaining energy

The first is to integrate samples, which is what any poller does:

$$E \approx \sum_i P(t_i) \cdot \Delta t$$

It accumulates five sources of error: the ±5 % of the sensor, quadrature error from the finite step, aliasing if the load has spectral content above half the sampling frequency, the fact that each sample is already a moving average, and the jitter of 0 to 100 ms between the host clock and the sensor tick.

The second is to subtract the counter:

$$E = \text{counter}(t_1) - \text{counter}(t_0)$$

Two calls and a subtraction. It removes quadrature, aliasing and time misalignment; the ±5 % of the sensor remains.

Why 1 Hz destroys a 200 ms inference

With sampling at 1 Hz, the sampling theorem only allows components below 0.5 Hz to be reconstructed, while a 200 ms pulse has content in the region of 5 Hz. The expected number of samples inside the event is 0.2, so with probability 0.8 none falls inside it: the result is not noisy, it is undefined.

Even sampling at the maximum useful rate, which on the A100 and H100 is about 10 Hz given the 101 ms period, a 200 ms inference produces one or two distinct sensor updates, and each has observed only 25 ms of real activity: effective coverage is around 25 % of the event. And because of the fourth transient behaviour in the previous section, the reading obtained during those 200 ms may describe what was happening before the kernel was launched.

Aliasing with periodic loads

The paper observes the phenomenon directly on the A100: with a square wave of period slightly different from 100 ms, the reading fluctuates between high and low values with a clear beat. An inference server with periodic arrivals, health checks, fixed-size batches or a scheduler tick is exactly that pathological case.

When only instantaneous power is available, the measured good practice from the Oxford work consists of running 32 consecutive iterations or a minimum of 5 s, inserting 8 evenly spaced controlled delays if the window is smaller than the period, repeating over 4 separate trials with a random delay between them, and shifting the series in post-processing to synchronise it with the real activity. Over nine real benchmarks that took the mean error from 39.27 % to 4.89 %.

What the GPU counter does not measure

The scope declared by NVML is the GPU and its associated circuitry: the SXM module or the PCIe card, with its HBM and its regulators. Left out are the CPU, host DRAM, NVSwitch, network cards, storage, fans and power supply losses.

The measured node figures come from the empirical calibration work on 8× H100 SXM5 nodes (arXiv:2506.14551):

Quantity	Value
Nominal node TDP declared by the manufacturer	10.2 kW
Node power at idle	1.8 kW
Maximum measured under saturating load	8.4 kW
Peak on real production loads	never above 76 % of TDP

Two conversion factors come out of that, both my own estimates from those figures: 8 × 700 W is 5.6 kW of GPU against 8.4 kW of node, that is, the GPU is of the order of 67 % of node power and the GPU-to-node factor is around 1.50×. Chaining that with an industry average PUE of 1.54 (Uptime Institute, 2025), the total factor from the NVML counter to the utility feed is around 2.3×; with a PUE of 1.1 it drops to 1.65×. Put in the form that matters to whoever signs the bill: the watts NVML reports are roughly 43 % of the watts the utility bills in an average datacenter.

As a complementary reference, an 80 PLUS Titanium supply at 230 V is 96 % efficient at 50 % load and 91 % at 100 %, so supply losses add between 4 % and 9 % that is already included if you measure at the socket.

The tax of keeping a context open

A GPU serving a loaded model with no traffic does not consume its nominal idle. Measurements over 335,267 production samples from 14 H100s across 18 days, plus controlled experiments, give this table:

GPU	Idle without a CUDA context	Idle with a CUDA context	Extra cost	Percentage of TDP
H100	71.8 W	121.7 W	+49.9 W	7.1 %
A100	53.7 W	80.0 W	+26.3 W	8.8 %
L40S	35.6 W	102.1 W	+66.4 W	19.0 %

The result that changes the mental model: more than 98 % of that extra cost is produced by the open CUDA context, independently of the memory occupied. Varying allocated VRAM between 0 and 72 GB moves power by less than 1 W. The size of the loaded model does not cost watts; keeping the context open does.

That cost is not marginal on a real platform. Over 11,791 long-running jobs, the measured split was 24 % of the time and 7 % of the energy in deep idle, 15 % of the time and 10 % of the energy idle with an active context, and 61 % of the time with 83 % of the energy in execution. On serving loads, GPUs can spend 48 % of their energy in low-activity periods.

CPU and DRAM: RAPL

To close the node you have to measure what is not GPU, and there the interface is RAPL through the kernel powercap framework, at /sys/devices/virtual/powercap/intel-rapl/. The hierarchy exposes intel-rapl:N as the socket and intel-rapl:N:M as subzones (core, uncore, dram), plus the psys domain for the full SoC since Skylake.

File	Content
`energy_uj`	Energy counter in microjoules; writing “0” resets it if the counter supports it
`max_energy_range_uj`	Counter range, that is, the overflow point
`power_uw`	Current power in microwatts
`name`	Zone name

The figures that condition its use come from the characterisation published in ACM TOMPECS: an update roughly every 1 ms, with jitter; an energy quantum of 61 µJ on Haswell and Skylake, 15.3 µJ on Sandy Bridge; and counter overflow in 52 minutes on a Haswell at 84 W, which forces polling at a far shorter period and detecting the wrap with max_energy_range_uj. There is also a thermal drift that is rarely declared: package power for the same load grows between 10 % and 12 % between 37 °C and 74 °C on Haswell. Warming up beforehand is not optional.

Access, on the other hand, is no longer free. The PLATYPUS attack (IEEE S&P 2021) demonstrated a purely software power side channel through RAPL, with CVE-2020-8694 and CVE-2020-8695 attached. The Linux mitigation arrived in commit 949dd0104c49, included in 5.10.0-rc4, which changed the default permissions so that only root can read energy_uj. The microcode mitigation associated with SGX goes further and introduces random noise into the reported energy and changes the reporting frequency: on machines with that mitigation active, RAPL readings are deliberately degraded. Measuring CPU and DRAM from a container therefore requires root, a udev rule relaxing the permissions, or a privileged daemon.

The software stack

Tool	Source of the GPU figure	Counter or power	Default interval	Overhead
Zeus	NVML	Counter if the architecture is Volta or later; polling otherwise	Delimited by windows	Less than 10 ms per call
dcgm-exporter	NVML via DCGM	Both fields available	`collect-interval` = 30,000 ms	5-10 W extra on the server’s IPMI reading
CodeCarbon	NVML	Counter	`measure_power_secs` = 15 s	5.38 % to 46.75 % of time at 1 kHz
Scaphandre	No GPU support; RAPL	`energy_uj` counter	Not documented	3.81 % to 28.38 % at 1 kHz
Kepler	RAPL and IPMI; GPU only as an experimental option	Hardware counters	—	—

Four details that change the choice:

Zeus decides in code with nvmlDeviceGetArchitecture and uses the accumulated counter on Volta or later, with a nestable window API (begin_window / end_window). It is the reference implementation of what this sheet recommends. Its NSDI'23 paper also documents energy reductions of 23.8 % to 75.7 % from choosing batch size and power limit well, and of 3.0 % to 31.5 % from touching only the limit, with five seconds of profiling per point enough for stable results.
Kepler has removed eBPF. The CNCF announcement of June 2026 describes the redesign towards reading /proc and /sys, motivated by the CAP_BPF and CAP_SYSADMIN permissions that blocked deployments, and by the loss of short-lived processes that underestimated the footprint. The new node metric closely follows the IPMI pattern and removes the spurious multi-kilowatt spikes of the previous version. For GPUs it remains an experimental option: today it is not the tool for measuring Wh/token on an accelerator.
CodeCarbon uses the NVML counter, which is correct, but its RAM estimate is a heuristic of 5 W per module derived from the total gigabytes, and the code itself acknowledges it.
DCGM delivers profiling metrics at 1 Hz by default, and warns that collecting at higher frequencies returns zeros because it groups metrics internally.

On the cost of measuring, the empirical overhead study of RAPL-based tools, all forced to 1 kHz, leaves a clear ordering:

Tool	Time overhead
CodeCarbon	5.38 % to 46.75 %
Scaphandre	3.81 % to 28.38 %
Turbostat	2.73 % to 14.37 %
PowerJoular	1.67 % to 8.88 %
perf	−1.00 % to 4.26 %
RAPL read in user space	−0.70 % to 2.93 %
RAPL read in kernel space	−0.17 % to 0.99 %

The underlying difference is that a system call costs of the order of 1.36·10⁻³ ms and an rdmsr instruction between 2.3·10⁻⁴ and 5.6·10⁻⁴ ms, an order of magnitude less. The authors recommend matching the granularity to the phenomenon being measured instead of raising the default frequency, and limiting polling to the necessary domains.

Procedure for making a Wh/token auditable

Fix and record the electrical state. Note EnforcedPowerLimit, the default limit and the driver version; lock clocks or the power ceiling if the experiment requires it.
Warm up to a stable temperature. RAPL thermal drift is 10 % to 12 % between 37 °C and 74 °C, and the power.draw.average transient takes up to a second to settle.
Measure the idle baseline twice: without a CUDA context and with the model loaded. They are two different values and both are needed.
Use windows of at least 5 s or 32 consecutive iterations.
Repeat over 4 separate trials with a random delay, and add 8 evenly spaced delays if you are integrating instantaneous power with a window smaller than the period.
Prefer the accumulated counter whenever the architecture is Volta or later, watching for driver reloads.
Report dispersion. The Oxford paper itself publishes the mean and standard deviation of the error; without an interval, a Wh/token figure is not auditable.
Declare the measurement boundary: GPU, node or utility feed. Without that label, two figures from the same facility differ by a factor of 2.3 with nobody knowing why.

Subtracting idle, or not

There is no consensus, and the honest approach is to treat it as a declared decision. In production the GPU is parked with the model loaded, and that consumption shows up on the bill: subtracting it erases a real cost. To compare the marginal efficiency of two models or two kernels, by contrast, subtracting it isolates the variable. The workable way out is to report both figures with labels: gross Wh/token, which includes idle and corresponds to the bill, and marginal Wh/token, comparable between models. What does not work is mixing them in the same table without distinguishing them.

One estimation trap remains alive in many spreadsheets: multiplying TDP by time overestimates real energy by up to 4.1 times, according to the ML.ENERGY Benchmark.

Power capping: what it costs and what it saves

Configuration	Energy saving	Performance loss
V100 at 200 W against 300 W (67 % of TDP), BERT training	~15 %	barely any degradation
V100 at 200 W, set of workloads	10-20 %	less than 5 %
V100 at 100 W (33 % of TDP)	40-60 %	30-40 %
A100 at 175 W against 400 W (44 % of TDP), LLaMA 65B inference	22-24 %	5-8 %

The asymmetry between phases explains the result. The Microsoft measurement at ASPLOS'24 documents that the prompt phase is short and reaches or exceeds TDP, while the token generation phase is longer and consumes less: the power ceiling penalises TTFT far more than TPOT. In their production cluster, inference uses 79 % of peak power against 97 % for training, and their measurements show that up to 20 % of power can be recovered with less than a 7 % performance loss.

The published sweep from 200 W to 700 W on the H100 and H200 adds the nuance of where the knee is: between 500 W and 700 W each 100 W step only contributes around 10 % of performance on compute-bound loads, while the H100 holds peak memory bandwidth even at 200 W. The most stable operating zones identified are 400 W on the H100 and 500 W on the H200. That the memory subsystem is protected against the trim is the physical reason why the decode phase of an LLM, which is memory-bound, loses so little performance under a power ceiling.

Two warnings about those figures: the H100/H200 sweep was measured with nvidia-smi polling every 10 s, exactly the problem described above, although on long steady loads the bias averages out; and there is no publication isolating the effect of memory clock locking on Wh/token in LLM inference, so any concrete figure on that point has to be your own measurement.

Reference figures

GPU	TDP
H100 SXM	up to 700 W, configurable
H100 NVL	350-400 W, configurable
A100 80GB SXM	400 W
A100 80GB PCIe	300 W
RTX 5090	575 W, with a recommended 1000 W supply
HGX node of 8× H100 SXM	10.2 kW nominal

Average power measured in inference: on an HPC cluster with vLLM on A100, the aggregate power of the GPU set ranged between 999 W and 2,983 W depending on model and parallelism, with Nemotron 70B on 8× A100 at 2,983 W, about 373 W per GPU, around 93 % of TDP.

Energy per token, with the warning that published figures measure different boundaries and cannot be mixed without normalising by GPU count and output tokens:

Source	Figure
TokenPowerBench, H100 94 GB	40 J per token on a standard load, more than 60 J per token in high-throughput configurations
TokenPowerBench, batch effect	−25 % of energy per token between batch 32 and 256 on a 70B model
TokenPowerBench, context effect	from 2K to 10K tokens of context on Llama 3 70B: energy ×3
TokenPowerBench, quantisation effect	Llama 3 405B in FP8 against FP16: −30 % of energy per token
TokenPowerBench, engine effect	TensorRT-LLM and vLLM against the Transformers engine: −25 % to −40 %
Per-prompt measurement in batch	0.0074-0.0289 Wh on 7B to 14B models; 0.0835-0.6912 Wh on 70B to 405B; single unbatched requests, between 10 and 100 times more expensive

Cross-links from the energy track

C2 — Energy per token: methodology and the electricity market: what to do with the joule once it is measured properly, and how the electricity price multiplies it.
C3 — Measuring energy in production: Kepler, DCGM and the practical stack: the continuous deployment of the stack whose metrology this sheet documents.
C3 — Energy benchmarking: frameworks, metrics and the state of the art: the full catalogue of tools and their metrics.
C4 — MLPerf Power: the certified alternative, measuring at the wall with an external analyser instead of the card’s sensor.
C6 — From watt to carbon: PUE, grid and grams of CO₂: the chain that turns the measured watt into emissions and into euros.
C7 — Energy efficiency levers: the quantified catalogue where the power ceiling of this sheet is just one more lever.

Sources

NVIDIA, NVML API Reference, Device Queries — https://docs.nvidia.com/deploy/nvml-api/group__nvmlDeviceQueries.html
NVIDIA, DCGM Library API, Field Identifiers — https://docs.nvidia.com/datacenter/dcgm/3.1/dcgm-api/dcgm-api-field-ids.html
NVIDIA, DCGM Feature Overview — https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html
NVIDIA, dcgm-exporter — https://github.com/NVIDIA/dcgm-exporter
NVIDIA, H100 Tensor Core GPU, especificaciones — https://www.nvidia.com/en-us/data-center/h100/
NVIDIA, A100 Tensor Core GPU, especificaciones — https://www.nvidia.com/en-us/data-center/a100/
NVIDIA, GeForce RTX 5090, especificaciones — https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/
Yang, Adámek y Armour (Oxford), Part-time Power Measurements: nvidia-smi’s Lack of Attention / SC24 — https://arxiv.org/html/2312.02741v3
Yang, GPU_Power_Benchmark, microbenchmark del trabajo anterior — https://github.com/JimZeyuYang/GPU_Power_Benchmark
ML.ENERGY, Measuring GPU Energy: Best Practices — https://ml.energy/blog/energy/measurement/measuring-gpu-energy-best-practices/
You et al., Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training, NSDI'23 — https://www.usenix.org/system/files/nsdi23-you.pdf
Zeus, Measuring Energy — https://ml.energy/zeus/measure/
ML.ENERGY, The ML.ENERGY Benchmark — https://arxiv.org/html/2505.06371v1
CNCF, Kepler, re-architected: improved power accuracy (junio 2026) — https://www.cncf.io/blog/2026/06/30/kepler-re-architected-improved-power-accuracy-and-a-community-call-to-action/
CodeCarbon, How Power Estimation Works — https://docs.codecarbon.io/latest/explanation/power-estimation/
Hubblo, Scaphandre: cálculo del consumo por proceso — https://github.com/hubblo-org/scaphandre/blob/main/docs_src/explanations/how-scaph-computes-per-process-power-consumption.md
Linux Kernel, Power Capping Framework (powercap sysfs) — https://docs.kernel.org/power/powercap/powercap.html
Intel, Running Average Power Limit Energy Reporting, INTEL-SA-00389 — https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/advisory-guidance/running-average-power-limit-energy-reporting.html
Lipp et al., PLATYPUS: With Great Power comes Great Leakage, IEEE S&P 2021 — https://platypusattack.com/
Khan et al., RAPL in Action: Experiences in Using RAPL for Power Measurements, ACM TOMPECS — https://www.devsustainability.com/p/paper-notes-rapl-in-action
What Is the Cost of Energy Monitoring? An Empirical Study on the Overhead of RAPL-Based Tools — https://arxiv.org/html/2604.26815v1
Patel et al. (Microsoft), Characterizing Power Management Opportunities for LLMs in the Cloud, ASPLOS'24 — https://www.microsoft.com/en-us/research/wp-content/uploads/2024/03/GPU_Power_ASPLOS_24.pdf
MIT Lincoln Laboratory, Sustainable Supercomputing for AI: GPU Power Capping at HPC Scale — https://arxiv.org/html/2402.18593
Architectural Trade-offs in the Energy-Efficient Era: power-capping NVIDIA H100 and H200 — https://arxiv.org/html/2604.11391v1
Empirically-Calibrated H100 Node Power Models for Datacenter Energy Analysis — https://arxiv.org/pdf/2506.14551
The Model Parking Tax: Quantifying the Hidden Energy Cost of Always-On GPU Model Deployment — https://arxiv.org/html/2605.23918
The Energy Cost of Execution-Idle in GPU Clusters — https://arxiv.org/html/2604.04745
TokenPowerBench: Benchmarking the Power Consumption of LLM Inference — https://arxiv.org/pdf/2512.03024
From Prompts to Power: Measuring the Energy Footprint of LLM Inference — https://arxiv.org/html/2511.05597
Serving LLMs in HPC Clusters — https://arxiv.org/abs/2507.00418
Uptime Institute, Global Data Center Survey 2025 — https://datacenter.uptimeinstitute.com/rs/711-RIA-145/images/2025.Annual.Survey.Report.pdf

Levers for spending fewer watts per token: a quantified catalogue for LLM inference

Tue, 16 Jun 2026 09:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

TL;DR

Seven independent levers act on the numerator or the denominator of the identity $\text{J/token} = \text{power (W)} / \text{throughput (tok/s)}$. Applied in order of cost-benefit: quantisation (FP16→FP8) cuts J/token by 30–50% without changing hardware; power capping to 70% of TDP reduces power by 30% with less than a 10% drop in throughput on memory-bound loads; DVFS in the decode phase saves a further 22–45%; continuous batching lowers J/token by 25–40% against unoptimised stacks; speculative decoding trims up to 29% off J/token at low batch; prefix caching eliminates the prefill recompute for repeated prompts; and scheduling to clean hours shifts the carbon footprint without changing physical consumption. Starting reference: a vLLM FP16 stack on an H100 SXM (700 W TDP) with Llama-3 70B produces ~0.39 J/token under realistic batch loads; an unoptimised stack (PyTorch baseline) starts 2–6× higher. The levers stack; the combined effect can reach a 73% reduction against the unoptimised baseline (arXiv 2504.17674).

The reference identity

$$\text{J/token} = \frac{\overline{P}\ [\text{W}]}{\text{throughput}\ [\text{tok/s}]}$$

where $\overline{P}$ is the average power over the measurement window (see the methodology in energy per token). The levers act on one of the two factors:

They reduce $\overline{P}$: power capping, DVFS, scheduling to low-intensity hours.
They raise throughput (the denominator): quantisation, continuous batching, speculative decoding, prefix caching, chunked prefill, model selection.
Both at once: quantisation (smaller weight footprint → a larger batch fits in VRAM → higher throughput at similar or lower power).

The expanded formula including infrastructure overhead:

$$\text{J/token}_{\text{effective}} = \frac{\overline{P}_{\text{GPU}} + \overline{P}_{\text{rest of node}}}{\text{throughput}_{\text{tok/s}}} \times \text{PUE}$$

For the lever calculations the work is done on $\overline{P}_{\text{GPU}}$ for clarity; the PUE factor (see energy per token) amplifies or attenuates the effect of each lever proportionally.

Master table: levers × impact × cost × tool

Lever	J/token reduction (%)	Effect on latency	Effect on quality	Main OSS tool	Source
Quantisation FP16→FP8	30–50%	−8–33% TPOT (improvement)	<1% degradation on benchmarks	vLLM, TRT-LLM, llm-compressor	arXiv 2504.17674, baseten.co
Quantisation FP16→INT4 (AWQ/GPTQ)	45–60%	latency improves (lower BW)	1–3% degradation depending on model	AutoAWQ, llm-compressor, vLLM	arXiv 2504.17674
Continuous vs static batching	25–40% vs PyTorch	+5× throughput (improvement)	no impact	vLLM, TGI, TRT-LLM	arXiv 2504.17674, TokenPowerBench
Power capping 70% TDP	20–30% power	<10% throughput (decode)	no impact	nvidia-smi, DCGM, Zeus	arXiv 2604.11391, NERSC docs
DVFS decode-phase	22–45% in decode	1–6% total latency	no impact	nvidia-smi, GreenLLM, DVFS-GPT	arXiv 2501.08219, HAL 05190051
Speculative decoding	up to 29% (batch≤16)	TTFT improves	no demonstrable impact	vLLM, TRT-LLM	arXiv 2602.09113
Prefix caching	50–90% on reusable prefill	TTFT −50–80% (repeated prompts)	no impact	vLLM, SGLang	vLLM docs
Chunked prefill	~10–15% on mixed loads	more stable TTFT	no impact	vLLM (SARATHI)	arXiv 2308.16369
SLM vs dense LLM selection	60–80% (7B vs 70B model)	low latency	degradation varies by task	Ollama, vLLM, llama.cpp	arXiv 2504.17674
Scheduling to clean hours	0% J/token, up to −70% CO₂	no impact on the load	no impact	Kueue, Volcano, cron	ACM SLIT, arXiv 2507.09942

Note: the percentages are against the effective J/token of the load, not against idle board power. The effects are always measured under conditions of equal output quality (same model, same serving stack, same prompt distribution).

Lever 1: quantisation

Mechanics

Quantisation reduces the numerical precision of the weights (and optionally the activations), which shortens the time spent reading weights from HBM per generated token. Decode is memory-bandwidth-bound: the bottleneck is reading the weights, not computing. Reducing bit width frees bandwidth for higher throughput or allows a larger batch in VRAM.

Format	Bits/weight	BW reduction vs FP32	J/token vs FP16	Tool
FP32	32	1× (reference)	+100% (worse)	—
BF16/FP16	16	2×	reference	vLLM native
FP8 (W8A8)	8	~4×	−30–50%	vLLM, TRT-LLM, llm-compressor
INT4 AWQ (W4A16)	4	~8×	−45–60%	AutoAWQ, llm-compressor
INT4 GPTQ (W4A16)	4	~8×	−40–55%	AutoGPTQ, vLLM

Concrete reference figures:

Llama-3 405B on H100, FP16→FP8: −30% J/token measured; TPOT improves by 33% (Baseten, baseten.co/blog/33-faster-llm-inference-with-fp8-quantization).
The vLLM+CUDA graphs stack vs the PyTorch baseline: 2–6× more efficient in J/token (arXiv 2504.17674).
Llama-65B on V100 FP16: 3–4 J/token. Llama-3 70B on H100 FP8: 0.39 J/token. Combined hardware+quantisation reduction: ~90% (MLCommons, arXiv 2504.17674).

Quality cost

FP8 quantisation with calibration produces <1% degradation on standard benchmarks (MMLU, HumanEval). INT4 AWQ can produce 1–3% degradation depending on the model and the task; INT4 GPTQ without fine calibration, up to 2–5%. Validate with LLM energy leaderboards and your own quality benchmarks.

OSS tooling

# vLLM with FP8 (load-time quantisation, H100/A100)
vllm serve meta-llama/Llama-3-70B-Instruct \
 --quantization fp8 \
 --dtype auto

# llm-compressor: offline quantisation to W4A16 (AWQ)
python -m llmcompressor.transformers.compression.helpers \
 --model meta-llama/Llama-3-70B-Instruct \
 --recipe w4a16_awq.yaml \
 --output-dir llama-70b-w4a16/

Lever 2: continuous batching

Mechanics

Static batching keeps the batch fixed: if one request finishes early, the slot stays empty until the end of the batch. Continuous batching inserts new requests as soon as a slot frees up, keeping the GPU always at the highest possible utilisation.

The energy effect: GPU power is almost constant (the GPU is always active); the throughput grows. The result is a larger denominator at similar power:

$$\text{J/token}_{\text{continuous batch}} = \frac{\overline{P}}{\text{throughput}_{\uparrow}}$$

Measured figures (arXiv 2504.17674, TokenPowerBench arXiv 2512.03024):

vLLM with PagedAttention vs unoptimised Transformers: −25–40% J/token on high batch loads.
Online continuous batching vs static offline: a further −5% of J/token (small but consistent).
Batch 32→256 tokens: −25% J/token on Llama-3 70B (kernel economies of scale).
Throughput: ×5 against static batching stacks at equivalent load.

The efficiency knee on the H100 sits at around 256–512 tokens of batch; beyond that, J/token flattens or rises slightly.

OSS tooling

vLLM, TGI (HuggingFace Text Generation Inference), TRT-LLM. The relevant flag in vLLM: --max-num-seqs (maximum concurrent sequences); tuning it to the available VRAM maximises the effective batch.

Lever 3: power capping

Mechanics

The power cap sets a ceiling on instantaneous power (W). The GPU respects it by cutting the clock (internal DVFS). The efficiency trick: LLM decode is memory-bandwidth-bound; the GPU is already not at usefully full clock, and lowering the TDP reduces power without degrading throughput proportionally.

The power cap vs throughput curve (H100 SXM)

Figures from (arXiv 2604.11391) and NERSC (docs.nersc.gov/jobs/power-capping):

Power cap (W)	% of TDP (700 W)	Relative throughput (memory-bound)	Efficiency (tok/W)
700 W	100%	100%	1.00×
560 W	80%	~95%	~1.19×
490 W	70%	~90–95%	~1.27× (sweet spot)
420 W	60%	~80–85%	~1.33× (latency rises)
350 W	50%	~75% (the H100 reaches maximum BW at 350 W memory-bound)	~1.50×
200 W	29%	~50%	frequency saturation

Key finding from arXiv 2604.11391: the H100 reaches its maximum effective memory bandwidth with only ~350 W on memory-bound loads (STriad benchmark), against a TDP of 700 W. Going from 700 W to 490 W (−30% power) produces less than a 10% drop in decode throughput: J/token improves by ~20–27%.

For compute-bound loads (heavy prefill), the sweet spot shifts to 400–500 W, where performance scales almost linearly with the cap; from 500 W to 700 W, the gain is only ~10% extra throughput for each additional 100 W.

J/token formula with a power cap

$$\text{J/token}(P_{\text{cap}}) = \frac{P_{\text{cap}} \times \eta(P_{\text{cap}})}{\text{throughput}_{\text{base}} \times f(P_{\text{cap}})}$$

where $\eta(P_{\text{cap}})$ is the fraction of power actually consumed under the cap (on memory-bound loads, the GPU no longer consumes the whole cap), and $f(P_{\text{cap}})$ is the resulting fraction of throughput. In the H100’s 490–560 W zone, $\eta \approx 0{.}9$ and $f \approx 0{.}92\text{–}0{.}95$: the numerator falls more than the denominator, so J/token improves.

Commands

# Set the power cap to 490 W on GPU 0 (H100 SXM, 700 W TDP)
sudo nvidia-smi -i 0 -pl 490

# Check the applied limit and the power draw in real time
nvidia-smi --query-gpu=power.limit,power.draw --format=csv -l 1

# With DCGM: read DCGM_FI_DEV_POWER_USAGE in Prometheus
# and DCGM_FI_DEV_POWER_LIMIT to confirm the cap is active

# Apply the cap to every GPU on the node (systemd/cron script)
for i in $(seq 0 3); do
 sudo nvidia-smi -i $i -pl 490
done

With DCGM in production, the cap can be set via dcgmi policy --set powercap:490 or through the NVIDIA GPU Operator (the powerLimit field in the DevicePlugin). Zeus (ml.energy/zeus) can sweep caps automatically and measure the resulting J/token per configuration.

Lever 4: DVFS (Dynamic Voltage and Frequency Scaling)

Mechanics

DVFS is finer-grained than a power cap: it allows the SM frequency to be set directly (nvidia-smi -lgc to lock the GPU clock) or left for the driver to adjust dynamically according to available power and load. In LLM inference, the asymmetry between phases is the key fact:

Prefill: compute-bound. Benefits from maximum clock. Sensitive to frequency reduction.
Decode: memory-bandwidth-bound. The SM clock barely matters; what rules is HBM bandwidth.

This asymmetry is what LLM-specific DVFS systems exploit:

System	Energy saving	Latency penalty	Hardware	Source
DVFS-GPT (HAL)	up to 32% at iso-latency	~1–2%	A100, RTX 4090	HAL 05190051
EcoInfer	up to 25.4% (average 21.5%)	<5%	A100	arXiv MDPI Electronics
DualScale (disaggregated prefill/decode)	22–45% with SLO	1–6%	H100	arXiv 2602.18755
GreenLLM (SLO-aware DVFS)	18–30%	within the SLO	—	arXiv 2508.16449

The paper arXiv 2501.08219 measures that reducing the GPU frequency from 2,842 MHz to 180 MHz achieves an average saving of 42% with only a 1–6% latency increase in the decode phase, because decode is memory-bound and the SM frequency is not the bottleneck.

Commands (manual DVFS)

# Lock the SM clock at 1,200 MHz (H100 boost: 1,980 MHz) for decode
sudo nvidia-smi -i 0 -lgc 1200,1200

# Restore the dynamic clock
sudo nvidia-smi -i 0 -rgc

# Persistent mode (needed for the lock to survive between jobs)
sudo nvidia-smi -pm 1

For automatic DVFS tuned to the SLO (target TPOT), systems such as GreenLLM or DVFS-GPT measure the time of each iteration and adjust the frequency in real time, without overrunning the latency budget.

Lever 5: model selection (SLM, MoE vs dense)

The effect of scale

J/token scales with model size, but sublinearly. For Llama-3 (arXiv 2504.17674):

Model	Parameters	Relative J/token	Factor vs 1B
Llama-3 1B	1B	1.0×	reference
Llama-3 8B	8B	~2.3×	×2.3 (not ×8)
Llama-3 70B	70B	~7.3×	×7.3 (not ×70)
Llama-3 405B	405B	~40×	×40 (not ×405)

J/token grows ~7.3× from 1B to 70B, while parameters grow ×70: hardware efficiency (caches, tensor parallelism, kernels) cushions the scaling. Even so, moving from 70B to 8B cuts J/token by ~68% if the quality is sufficient for the task.

MoE vs dense

MoE models activate only a subset of experts per token (typically 2 of 8–64), which reduces active compute. But there is an implementation trap:

OLMoE-1B-7B (MoE, 1B active) vs OLMo-1B (dense): the MoE consumes 54.24% more energy per token than the dense model of the same active size (arXiv 2504.17674), because the fused expert kernels are ~19–63% slower than an equivalent dense GEMM.
OLMoE-1B-7B vs OLMo-7B (dense): here the MoE is indeed more efficient (7B active → only 1B active, a ~7× reduction in active FLOPs per token).

The rule: MoE is efficient when the comparison is equal total parameters, fewer active (for example Mixtral 8×7B vs dense Llama-3 70B). It is not efficient when the comparison is equal active parameters per token.

Model selection table by use case

Use case	Recommended model	Efficiency rationale
Simple chat, entity extraction	SLM 7–8B FP8	−68% J/token vs 70B
RAG with long contexts	70B FP8 or MoE 8×7B	optimal quality/J at K>2K tokens
Complex code, reasoning	70B FP8 or 405B INT4	quality is non-negotiable
Classification, embeddings	dedicated model (e.g. E5, BGE)	orders of magnitude less J
Deferrable offline batch	large MoE (DeepSeek, Mixtral)	high quality, batch throughput

Lever 6: speculative decoding

Mechanics

Speculative decoding uses a small draft model to propose several tokens, which the target model verifies in parallel in a single forward pass. If the tokens are accepted, K tokens are generated per step instead of 1: potentially K× throughput.

The energy efficiency condition

The J/token benefit depends on batch size (arXiv 2602.09113):

Batch size	Effect on J/token	Reason
1–16 (low batch)	−up to 29%	the draft model fills the waiting time; verification is cheap
32–64 (medium batch)	~0% (neutral)	verification overhead cancels out the gain
128+ (high batch)	+25% (worse)	GPU saturated; the draft model adds overhead without saving

Conclusion: speculative decoding is a J/token lever only in a low-latency regime (batch≤16); in high-throughput serving it is counterproductive in J/token terms (although it reduces TPOT latency).

OSS tooling

vLLM supports speculative decoding with --speculative-model and --num-speculative-tokens. TRT-LLM does too. The draft model must come from the same family (for example Llama-3 8B as draft for Llama-3 70B).

Lever 7: prefix caching and chunked prefill

Prefix caching

The prefill KV-cache of a shared prompt (system prompt, few-shot examples, document context) is identical across requests. Storing and reusing it eliminates the prefill recompute for that fraction of the prompt.

Energy impact: prefill is compute-bound and draws power spikes. Removing the repeated prefill removes those spikes. In scenarios with a fixed 1,000-token system prompt and 200-token user prompts, the saving is ~1,000/(1,000+200) = ~83% of prefill energy. In total J/token, the effect depends on the length of the shared prompt against the generation length.

Enabling it in vLLM:

vllm serve meta-llama/Llama-3-70B-Instruct \
 --enable-prefix-caching \
 --max-num-seqs 256

Chunked prefill (SARATHI)

Chunked prefill splits the prefill into fixed-size chunks and processes each chunk together with decode tokens from other requests. The benefit: the prefill of a long prompt no longer blocks the decode of other requests (reducing TPOT spikes), which allows higher GPU utilisation over time.

In terms of total J/token, chunked prefill reduces the node’s peak power (fewer prefill spikes) and improves average utilisation, which translates into ~10–15% less J/token on mixed loads with long prompts (arXiv 2308.16369).

# Enable chunked prefill in vLLM (chunk size: 512 tokens)
vllm serve meta-llama/Llama-3-70B-Instruct \
 --enable-chunked-prefill \
 --max-num-batched-tokens 512

Lever 8: scheduling to clean hours

This lever does not reduce physical J/token, but it does reduce the gCO₂/token of deferrable load. The mechanics: moving training, document ingestion, batch re-ranking and fine-tuning to the hours of lowest grid carbon intensity.

Impact figures for temporal shifting (ACM SLIT, arXiv 2507.09942):

CO₂ reduction from temporal shifting: up to 34.7% for flexible loads.
With carbon-aware reinforcement learning (Eco-Orchestrator): up to a 70% offset in integrated deployments.
In Spain, the hourly variation in intensity is ~80–250 gCO₂/kWh: a factor of ~3× within the same day (see from watt to carbon).

The practical implementation uses Kueue or Volcano on Kubernetes, with a grid intensity metric (the ElectricityMaps API or esios) as the scheduling signal. Loads are marked with a carbon-deadline annotation (the maximum tolerable wait) and the scheduler places them in the minimum-intensity hours within that window.

# Kueue annotation for carbon-aware scheduling
metadata:
 annotations:
 kueue.x-k8s.io/carbon-aware: "true"
 kueue.x-k8s.io/carbon-deadline: "8h"

The power cap vs throughput curve: the sweet spot

Reading the chart: the power curve (solid line) falls faster than the throughput curve (dashed line) in the 490–560 W zone. Above 560 W, each additional 70 W gives only ~5% extra throughput; below 420 W, throughput starts falling faster than power. The efficiency sweet spot (minimum J/token) sits at 490–560 W for decode-bound loads.

For compute-bound loads (pure prefill), the sweet spot shifts to 500–600 W: prefill saturates SM compute and needs more power to sustain throughput.

How to measure the effect and validate quality

Measurement stack

The full stack for measuring the effect of each lever:

What to measure	Tool	Metric
Real-time GPU power	DCGM (`DCGM_FI_DEV_POWER_USAGE`)	W per GPU, 100ms
J/token in production	Kepler + vLLM tokens	`rate(kepler_joules) / rate(vllm_tokens)`
J/token on the bench	Zeus (`ZeusMonitor`)	J measured during the benchmark
Throughput	vLLM metrics (`vllm:generation_tokens_total`)	tok/s
TPOT latency	vLLM (`vllm:e2e_request_latency`)	ms/tok
Output quality	lm-evaluation-harness	MMLU, HumanEval, etc.

See the full stack in energy tooling.

Measurement protocol per lever

# Real-time J/token (Kepler + vLLM in Prometheus)
sum(rate(kepler_container_joules_total{container="vllm"}[5m]))
 /
sum(rate(vllm_generation_tokens_total[5m]))

# Average power per GPU (DCGM)
avg by (gpu) (DCGM_FI_DEV_POWER_USAGE)

# Check that the power cap is active
avg by (gpu) (DCGM_FI_DEV_POWER_LIMIT)

With Zeus, the configuration sweep is automatic:

from zeus.monitor import ZeusMonitor

monitor = ZeusMonitor(gpu_indices=[0, 1, 2, 3])

with monitor.begin_window("fp8_inference"):
 # run the benchmark with vLLM FP8
 run_benchmark(model="llama-3-70b-fp8", num_tokens=10_000)

result = monitor.end_window("fp8_inference")
print(f"J/token: {result.total_energy / 10_000:.4f}")

Quality validation

Validating that the lever does not degrade quality is done with lm-evaluation-harness (github.com/EleutherAI/lm-evaluation-harness):

# Compare MMLU between FP16 and FP8
lm_eval --model vllm \
 --model_args pretrained=meta-llama/Llama-3-70B-Instruct,dtype=float16 \
 --tasks mmlu --num_fewshot 5 --output_path ./results/fp16/

lm_eval --model vllm \
 --model_args pretrained=meta-llama/Llama-3-70B-Instruct,quantization=fp8 \
 --tasks mmlu --num_fewshot 5 --output_path ./results/fp8/

The MMLU score difference between FP16 and FP8 with calibration is typically <1%. Without calibration it can be 2–4%.

Pareto decision table (energy / latency / quality)

Lever	J/token	TPOT (latency)	Quality	Implementation cost	Reversibility
FP8 quantisation	↓↓↓	↓ (improves)	~ (−<1%)	Low (1 flag)	High
INT4 AWQ quantisation	↓↓↓↓	↓ (improves)	↓ (−1–3%)	Medium (recalibrate)	High
Continuous batching	↓↓↓	~ (neutral on TPOT)	~	Low (new engine)	High
Power cap 70% TDP	↓↓	↑ <10%	~	Very low (1 command)	Immediate
DVFS decode	↓↓↓	↑ <6%	~	Medium (script + tuning)	High
Speculative decoding	↓ (low batch)	↓ (improves TTFT)	~	Medium (draft model)	High
Prefix caching	↓↓ (prefill)	↓↓ (TTFT improves)	~	Low (1 flag)	High
Chunked prefill	↓	↑ stable TPOT	~	Low (1 flag)	High
SLM (7B vs 70B)	↓↓↓↓	↓↓ (improves a lot)	↓↓ (−variable)	High (quality re-eval)	Medium
MoE (vs dense, same active size)	↑ (+54%)	↑	~	—	—
Scheduling to clean hours	~ J/token, ↓↓ CO₂	~ (for deferrable)	~	Medium (scheduler)	High

Key: ↓↓↓↓ strong improvement · ↓↓↓ significant improvement · ↓↓ moderate improvement · ↓ slight improvement · ~ neutral · ↑ worsens · ↑↑ worsens significantly.

Stacking levers: the combined effect

The levers are mostly orthogonal and they stack. The maximum combined impact documented in (arXiv 2504.17674):

“Correct application of the relevant inference optimisations can reduce total energy consumption by up to 73% against unoptimised baselines.”

Recommended sequence by cost-benefit:

Optimised engine (vLLM, TRT-LLM) with continuous batching: −25–40% immediately.
FP8 quantisation (if the hardware supports it: H100, A100 with limited support): a further −30–50% on the remaining J/token.
Power cap at 70% TDP: −20–30% power with <10% of throughput. Low cost.
Prefix caching: turn it on if there are system prompts or repeated contexts. No cost.
Chunked prefill: if the load has long variable prompts.
DVFS in decode: a further −22–45% if the SLO allows it.
Speculative decoding: only if the batch is low (<16) and TTFT latency matters.
Model selection: move to an SLM if the quality is sufficient (evaluate with lm-evaluation-harness on your specific task).

Quantified example on 4×H100 SXM (2,800 W node TDP), Llama-3 70B, chat load:

Configuration	Power (W)	Throughput (tok/s)	J/token
PyTorch FP16, static batching	~2,100	~400	5.25
vLLM FP16, continuous batching	~2,200	~1,800	1.22
vLLM FP8, continuous batching	~2,100	~2,400	0.875
+ power cap 490 W/GPU (1,960 W node)	~1,960	~2,160	0.907
+ prefix caching (40% hit rate)	~1,960	~2,600	0.754
+ DVFS decode 1,200 MHz	~1,500	~2,400	0.625

The values are illustrative estimates based on the cited papers applied to a 4×H100 node. Real J/token depends on the prompt distribution, the effective batch and the temperature of the models. Measuring it with Zeus or DCGM+Kepler is mandatory.

Reference hardware: TDP and power cap ranges

GPU	TDP (W)	Supported cap range	Decode sweet spot
H100 SXM 80GB	700	200–700 W	~490–560 W
H100 PCIe 80GB	350	100–350 W	~245–280 W
A100 SXM 80GB	400	100–400 W	~280–320 W
A100 PCIe 80GB	300	100–300 W	~210–240 W
L40S 48GB	350	100–350 W	~245–280 W
RTX 5090 (reference, PCIe)	575	~100–575 W	~400–460 W

For the exact ranges: nvidia-smi -q -d POWER | grep -E "Min|Max|Current".

Executive summary by role

Role	First action	Second action	Key tool
Cluster operator	`nvidia-smi -pl [70% TDP]` on every node	Enable persistent mode	nvidia-smi, DCGM
Serving engineer	Migrate to vLLM FP8 with continuous batching	Enable prefix caching	vLLM, llm-compressor
Platform architect	Select the smallest model that meets the quality SLO	Design a carbon-based scheduling policy	lm-evaluation-harness, Kueue
MLOps team	Set up a power cap sweep with Zeus	Instrument J/token in Prometheus	Zeus, DCGM exporter

See the full measurement stack in energy tooling and the J/token methodology in energy per token. For the context of reference energy leaderboards and benchmarks, see LLM energy leaderboards. The connection with full TCO is in GPU utilisation as FinOps.

Sources

arXiv 2504.17674 · Energy Considerations of LLM Inference and Efficiency Optimizations (CMU/Hugging Face, 2025) — https://arxiv.org/pdf/2504.17674
arXiv 2604.11391 · Architectural Trade-offs: power-capping NVIDIA H100 and H200 (FAU Erlangen, 2026) — https://arxiv.org/html/2604.11391v1
arXiv 2512.03024 · TokenPowerBench: Benchmarking the Power Consumption of LLM Inference — https://arxiv.org/pdf/2512.03024
arXiv 2602.09113 · Benchmarking the Energy Savings with Speculative Decoding Strategies — https://arxiv.org/pdf/2602.09113
arXiv 2501.08219 · Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling — https://arxiv.org/abs/2501.08219
arXiv 2602.18755 · DualScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFS — https://arxiv.org/pdf/2602.18755
arXiv 2508.16449 · GreenLLM: SLO-Aware Dynamic Frequency Scaling for Energy-Efficient LLM Serving — https://arxiv.org/pdf/2508.16449
HAL 05190051 · DVFS-GPT: Dynamic Voltage and Frequency Scaling for Energy-Efficient LLMs — https://hal.science/hal-05190051
arXiv 2308.16369 · SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — https://arxiv.org/pdf/2308.16369
arXiv 2507.09942 · Green-LLM: Optimal Workload Allocation for Environmentally-Aware Distributed Inference — https://arxiv.org/pdf/2507.09942
arXiv 2505.06371 · The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization — https://arxiv.org/html/2505.06371v1
Zeus Project (ml.energy) — https://ml.energy/zeus/
Zeus · PyTorch blog: Deep Learning Energy Measurement and Optimization — https://pytorch.org/blog/zeus/
Baseten · 33% faster LLM inference with FP8 quantization — https://www.baseten.co/blog/33-faster-llm-inference-with-fp8-quantization/
NVIDIA · Managing Power Capping (DGX H100/H200 User Guide) — https://docs.nvidia.com/dgx/dgxh100-user-guide/power-capping.html
NERSC · GPU Power Capping documentation — https://docs.nersc.gov/jobs/power-capping/
MLCommons · MLPerf Inference v5.1 results — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
vLLM · Disaggregated Prefill documentation — https://docs.vllm.ai/en/latest/features/disagg_prefill/
Microsoft Research · Characterizing Power Management Opportunities for LLMs in the Cloud (ASPLOS 2024) — https://www.microsoft.com/en-us/research/wp-content/uploads/2024/03/GPU_Power_ASPLOS_24.pdf
EcoInfer (MDPI Electronics) · Optimizing Energy Efficiency with Latency Guarantees Through Iteration-Level GPU Frequency Control — https://doi.org/10.3390/electronics15102139

Measuring energy in production: Kepler, DCGM and the practical stack (accuracy and overhead)

Sun, 14 Jun 2026 05:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Third article of the energy track (C3). C1 made the conceptual inventory of the tools; here we get practical: how the measurement stack is deployed on Kubernetes, where each piece takes its figures from, how accurate it is against a power meter, how much overhead it adds, and how energy is attributed per pod and per MIG instance. It is the equivalent of B2/B3 for energy: to stop talking about what each tool measures and start building the bench that measures it. With commands; no recommendations, only the mechanics.

The measurement stack on Kubernetes

To measure energy per workload on a GPU cluster, the practical stack has four pieces, and the good news is that three of them you already have if you do observability:

The new piece compared with a normal observability stack is Kepler, which adds energy attribution per pod. DCGM already gives GPU power; Prometheus stores it; Grafana draws it. It is reused infrastructure, not a new system, in line with GPU observability via DCGM.

DCGM exporter: the base of GPU power

GPU power, the dominant source in LLM inference, comes from NVIDIA’s dcgm-exporter (part of the GPU Operator), which exposes DCGM metrics to Prometheus. The key metric is DCGM_FI_DEV_POWER_USAGE (instantaneous power in W), alongside utilisation (DCGM_FI_DEV_GPU_UTIL) and memory. With that, energy is the integral of power over time across the workload’s window. DCGM gives power at card level; to split it per pod or per MIG instance you need a layer on top, which is Kepler.

Kepler in depth: eBPF and the sources

Kepler (Kubernetes-based Efficient Power Level Exporter) is a Prometheus exporter that measures energy at container, pod and node level, using eBPF to keep overhead low and scientific methods to improve accuracy (Kepler · GitHub). Its strength is that it combines several power sources:

Source	What it covers
RAPL	CPU and DRAM
NVML	NVIDIA GPU power
ACPI	platform power management
Redfish / IPMI	platform power (BMC)
Regression models	estimation when there is no real metric

For the GPU, the dcgmEndpoint parameter points at the dcgm-exporter metrics URL, which enables power attribution per MIG instance (Kepler · configuration). That is: DCGM gives the card’s power, Kepler splits it between the pods (and MIG partitions) that use it, via eBPF and models. The result is per-pod energy metrics in Prometheus, ready for Grafana, usable for scheduling or scaling, consumption reporting and visualisation (Kepler · GitHub).

Deployment: a DaemonSet on top of Prometheus

The deployment is a DaemonSet (one Kepler per node), pointing at the existing Prometheus and at the dcgm-exporter:

# Helm, pointing at the dcgm-exporter for GPU attribution
helm install kepler kepler/kepler \
 --namespace kepler --create-namespace \
 --set dcgmEndpoint="http://dcgm-exporter:9400/metrics"

Prerequisites: a kernel with eBPF, the dcgm-exporter running (from the GPU Operator) and Prometheus scraping. Once deployed, Kepler exposes metrics such as kepler_container_joules_total per pod, queried with PromQL and drawn in Grafana. Energy per workload stops being a manual estimate and becomes a continuous time series, just like cost with OpenCost.

PromQL queries: energy per pod

With Kepler exporting, these are the queries that build the panels. The exact names vary by version; check your /metrics:

# Cumulative energy per pod (joules)
kepler_container_joules_total

# Mean power per namespace over the last hour (W)
sum by (namespace) (rate(kepler_container_joules_total[1h]))

# GPU power per node (from DCGM, W)
sum by (node) (DCGM_FI_DEV_POWER_USAGE)

# Energy per token: power ÷ throughput (with a gateway/engine metric)
sum(rate(kepler_container_joules_total[5m])) / sum(rate(vllm_generation_tokens_total[5m]))

The last one closes the circle with the cost track: it crosses Kepler’s energy with the engine/gateway tokens to give J/token in real time, per workload. With that in Grafana you have energy per pod, per namespace and per token, continuously, the energy equivalent of the OpenCost cost report.

A calibration procedure

Since Kepler estimates, it is worth calibrating once per node type. A simple procedure:

Power meter at the node input (or a PDU/BMC reading), measuring the node’s real power.
Controlled load: a stable sweep at a known power (the engine at 80 % GPU, say).
Compare: the node’s sum of kepler_container_joules_total plus idle against the meter reading over the same window.
Adjust Kepler’s factor or model if the deviation is noticeable, and document the residual error.
Repeat for each node type (calibration is not transferable between different hardware).

With that residual error documented, Kepler’s continuous figure has a known error bar, which is exactly what a CSRD auditor or a technical committee will ask for. Measuring without calibrating gives a number with no credibility; calibrating once turns the continuous estimate into a defensible figure.

Accuracy: estimation vs power meter

The data honesty of the energy track: Kepler estimates (eBPF + models + counters), it does not measure with a power meter. Accuracy depends on the source: with NVML/DCGM and RAPL available, the figure is fairly faithful (it reads hardware counters); when it falls back on regression models (with no real metric), it is an estimate with more error. Good practice: calibrate against a physical power meter at least once per node type, adjust, and use Kepler as a continuous proxy. The meter is the point-in-time ground truth; Kepler, the continuous series calibrated against it. For a CSRD report that an auditor may challenge, that initial calibration is what gives the continuous figure credibility.

Overhead: the cost of measuring

Measuring energy has its own compute cost, worth budgeting for. In a Kubernetes environment, the overhead of the monitoring system is split between the Prometheus server and the targets: the Prometheus server consumes ~150 % of CPU (1.5 cores) and ~15 GB of RAM, while the targets (node exporter and dcgm-exporter) together use ~35 % of CPU and ~300 MB of RAM (arXiv · Container-level Energy Observability). Kepler uses eBPF precisely to keep its own overhead low. The reading: the bulk of the cost of measuring is in Prometheus (which you probably already have for observability), not in the energy exporters. Adding Kepler to an existing observability stack is marginal; standing up Prometheus just for this would be expensive. That is why measuring energy is nearly free if you already observe, and a project if you do not.

Zeus: for the bench and for optimisation, not for continuous production

Zeus (ml.energy) plays a different role: it is not a continuous production exporter, it is the bench tool for measuring and optimising the energy of one specific workload (a benchmark, a training run), with power capping and frequency selection. It fits into the benchmarking flow (alongside GuideLLM/AIPerf): during a sweep, Zeus measures the energy per configuration and lets you test the effect of power capping on performance. The practical division: Kepler/DCGM in production (continuous per-pod attribution), Zeus on the bench (fine measurement + optimisation of one workload), CodeCarbon if you want carbon context integrated into an ML pipeline.

CodeCarbon and Scaphandre in practice

The other two pieces, with their specific role:

CodeCarbon integrates inside the code of an ML pipeline (a decorator or a context manager wrapping the training or the inference), and returns the energy and the CO₂eq of that block, using nvidia-smi for the GPU and estimating CPU/RAM. It fits when you want the carbon figure per experiment or per ML job, integrated into the data flow, not at infrastructure level. It is the closest to the power meter among the Python tools.
Scaphandre is a metrology agent in Rust that reads RAPL and gives process/VM power with raw hardware accuracy. For CPU it is very accurate; for GPU it has to be combined with NVML. It fits when you need the most faithful CPU metrology, or a lightweight agent independent of Kubernetes.

The division with Kepler: Kepler attributes energy per pod in the cluster (the infrastructure view); CodeCarbon gives carbon per ML job (the pipeline view); Scaphandre gives raw CPU metrology. They do not compete; they cover different layers, and a mature platform uses whichever matches the question.

Choosing the stack by objective

Objective	Tool	Why
Continuous per-pod attribution (production)	Kepler + DCGM	eBPF, per pod/MIG, into Prometheus
Fine measurement + optimisation (bench)	Zeus	measures and optimises (power capping)
Carbon in an ML pipeline	CodeCarbon	integrated CO₂eq, close to the power meter
Raw CPU accuracy (RAPL)	Scaphandre	hardware metrology
Calibration / ground truth	power meter	the standard everything is calibrated against

It is not “one tool”, it is a stack by objective: continuous production (Kepler/DCGM), bench (Zeus), reporting (CodeCarbon), calibration (power meter). Mixing them according to the phase is what gives a complete and credible energy measurement.

From figure to action

Measuring is the start; the value is in acting. With energy per pod in Prometheus:

Reporting (CSRD): the carbon footprint per workload, calibrated, ready for regulatory reporting.
Energy/carbon-aware scheduling: move deferrable load to hours or nodes of lower intensity (connects with the hourly carbon of the Spanish grid).
Power capping: with Zeus, trim peak power with little performance loss, lowering the W in the numerator of J/token.
Optimisation: identify pods with poor energy efficiency (a lot of energy per token) and attack them with quantisation or batching.

The measurement stack is not a panel to look at: it is the input to scheduling, optimisation and reporting decisions. Just as OpenCost closes the cost loop, Kepler+DCGM closes the energy one.

From energy to carbon: the final layer

The energy figure (kWh) becomes carbon by multiplying by the grid intensity, and here the practical stack connects with the hourly source. In Spain, the intensity is published by Red Eléctrica (esios) and aggregated by ElectricityMaps, hour by hour. The pattern:

# Carbon per workload: energy (kWh) × grid intensity (gCO2/kWh)
(sum by (namespace) (increase(kepler_container_joules_total[1h])) / 3.6e6)
 * on() group_left() grid_carbon_intensity_gco2_kwh

where grid_carbon_intensity_gco2_kwh is a metric you feed from esios/ElectricityMaps (an exporter or a periodic push). With that, the Grafana panel shows gCO₂ per namespace and per token, hour by hour, not an annual average. And since Spanish intensity varies a lot (from ~80 in solar hours to ~250 on gas-heavy nights), that hourly resolution is what enables carbon-aware scheduling: moving the deferrable work to the clean hours lowers the measured carbon. The measurement stack does not only report carbon; with the hourly figure, it allows it to be reduced.

Energy efficiency KPIs to watch

KPI	What it indicates	Where from
J/token per model	energy efficiency	Kepler ÷ engine tokens
Mean power per pod	consumption per workload	`rate(kepler_container_joules_total)`
gCO₂/token (hourly)	real footprint	energy × grid intensity
Idle energy	waste	power of pods with no traffic
Effective PUE	DC overhead	total energy ÷ compute energy

These five turn the energy panel into a decision tool: which model is more efficient, which pod wastes, when it pays to schedule. Without them the stack is a pretty panel; with them, it is the input to optimisation and reporting.

Example: energy attribution on a multi-tenant cluster

With Kepler splitting per pod, an energy report per team on the example node (8×H100, 5.6 kW of boards, ~7.84 kW with PUE), over one month:

Team	Energy/month (kWh)	gCO₂ (ES grid ~160)	Mean J/token
A · chat-prod	~3,100	~496 kg	~2.2 J/tok
B · batch	~1,400	~224 kg	~2.8 J/tok
C · experimentation	~900	~144 kg	~6.5 J/tok
idle	~250	~40 kg	—

What it reveals: A is efficient (FP8, high utilisation → 2.2 J/token); C is inefficient (6.5 J/token, underused GPU), just as it dragged the idle cost in the FinOps track. The carbon column takes the figure into the per-team CSRD report. And the idle row (~250 kWh, ~40 kg CO₂) is pure wasted energy. Without Kepler, that split does not exist: you would only see the node’s total consumption, with no idea whose each watt is. With it, energy is attributable and actionable, exactly like cost with OpenCost. In fact, the two panels together (OpenCost cost + Kepler energy) give the complete picture of efficiency per team.

Energy and carbon-aware scheduling

The highest-return action the stack enables: scheduling by energy and carbon. With energy per pod and hourly grid intensity in Prometheus, two policies:

By carbon: move the deferrable load (training, ingestion, overnight batch) to the hours of lowest grid intensity. In Spain, the hours of peak solar (~80 gCO₂/kWh) against gas-heavy nights (~250). The same workload emits ~3× less when scheduled at the right hour.
By efficiency: consolidate workloads to eliminate idle (a powered GPU with no work is wasted energy), and direct each workload to the hardware most efficient for it.

Both use the same data (energy per pod + hourly intensity) that the stack already exposes, and they connect with cluster scheduling (Kueue/Volcano). It is the difference between measuring carbon (and reporting it) and reducing it (by scheduling). The C3 stack enables both.

Deployment checklist

Step	Action	Verification
1	dcgm-exporter (GPU Operator)	`DCGM_FI_DEV_POWER_USAGE` in Prometheus
2	Kepler DaemonSet, kernel with eBPF	`kepler_container_joules_total` per pod
3	`dcgmEndpoint` pointing at the exporter	GPU/MIG attribution works
4	Grid intensity metric (esios)	`grid_carbon_intensity` in Prometheus
5	Calibration vs power meter (once per node type)	error bar documented
6	Grafana panels (J/token, carbon, idle)	KPIs visible

Step 5 (calibration) is the one most often skipped and the one that gives the figure credibility; step 3 (dcgmEndpoint) is the one most often forgotten, and without it the GPU is not attributed.

Energy by phase: prefill vs decode

A level of detail the stack allows and that guides optimisation: power is not constant during a request. Prefill (processing the prompt) is compute-bound and draws a lot of power in a burst; decode (generating tokens) is memory-bound and draws a lower peak but for longer. With fine DCGM sampling that bimodal profile is visible, and it has consequences:

A workload with long prompts (a lot of prefill) has a different power profile from one with long generation (a lot of decode), even if they produce the same tokens.
Power capping affects each phase differently: trimming peak power mostly touches the prefill.

For most cases the mean J/token is enough; to optimise energy efficiency thoroughly, separating prefill and decode consumption (aligning the DCGM window with each phase) reveals where the spend is. It is the energy equivalent of separating TTFT and TPOT in benchmarking, the same principle, measuring the two phases separately because they behave differently.

State of the art: the LLM resolution gap

An honest note on the limits of current tooling. The general-purpose tools (Kepler, CodeCarbon, Scaphandre) measure well at machine, process and pod level, but they lack LLM-specific resolution: out of the box they do not distinguish prefill from decode energy, nor energy per token by phase, nor the energy cost of a reasoning block against a normal response. It is an open gap in the field: per-pod attribution exists, but attribution per token and per inference phase has to be built by crossing power (DCGM/Kepler) with engine metrics (vLLM exposes tokens per phase). The 2026 state of the art is a mature stack for infrastructure and an emerging one for LLM-specific resolution, exactly the crossing that the reproducible harness (S4) has to build by hand. Acknowledging it avoids promising a per-token accuracy that the general-purpose tools do not deliver on their own.

Limits and traps (data-driven)

Uncalibrated estimation. Kepler estimates; without calibrating against a power meter at least once, the figure can carry a noticeable error, especially when it falls back on regression models.
Forgetting the dcgmEndpoint. Without pointing Kepler at the dcgm-exporter, GPU (and MIG) attribution does not work; it is the key parameter for GPU.
Accounting for overhead badly. The bulk of the cost of measuring is in Prometheus; if you stand it up just for energy, budget for it (1.5 cores, 15 GB).
Confusing bench and production. Zeus measures one workload finely; Kepler attributes continuously in production. Using one for the other gives the wrong figure.
Misaligned windows. For energy per token, the power window and the token window must coincide; if they do not, the J/token does not correspond.

With the measurement stack in place, the energy track has its instrumentation; the following articles (C6 carbon, C8 TCO and regulation) build on this data. The foundation is this: a continuous per-pod measurement, calibrated, on top of the observability infrastructure you already have.

Closing

Measuring the energy of an AI cluster in production does not require a new system: it requires adding Kepler, per-pod attribution via eBPF, to the DCGM, Prometheus and Grafana stack you already have for observability. That is the central idea of C3: the cost of measuring is almost entirely in the Prometheus you already pay for, and Kepler turns it from “card power” into “energy per pod, per team and per token”. With two cautions that separate a credible figure from a decorative panel: calibrate once against a power meter (to have an error bar that survives an audit) and point the dcgmEndpoint (without which the GPU is not attributed). Done that way, energy stops being the soft axis of sustainability and becomes an actionable time series: the J/token per model that exposes inefficiency, the hourly carbon that enables clean scheduling, the idle that gets recovered, and the CSRD report that holds up. For the sovereign platform, the Kepler+DCGM stack closes the energy loop just as OpenCost closes the cost one, and together, with GuideLLM’s goodput, they give the three faces of the dashboard measured on the same iron, in euros, joules and grams of CO₂.

Sources

Kepler · GitHub (CNCF, eBPF, por pod/nodo) — https://github.com/sustainable-computing-io/kepler
Kepler · configuración (fuentes, dcgmEndpoint para MIG) — https://github.com/sustainable-computing-io/kepler/blob/main/docs/user/configuration.md
CNCF · explorando Kepler (consumo de aplicaciones cloud) — https://www.cncf.io/blog/2023/10/11/exploring-keplers-potentials-unveiling-cloud-application-power-consumption/
Red Hat · Kepler: monitorización eficiente de energía para Kubernetes — https://next.redhat.com/2023/08/22/introducing-kepler-efficient-power-monitoring-for-kubernetes/
arXiv · Container-level Energy Observability in Kubernetes (overhead) — https://arxiv.org/pdf/2504.10702
NVIDIA · dcgm-exporter — https://github.com/NVIDIA/dcgm-exporter
Zeus Project (ml.energy) — https://ml.energy/zeus/

Energy benchmarking in AI: frameworks, metrics and state of the art (tool by tool)

Sat, 13 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The energy and carbon references are European (France, Germany, Spain), since this is a sovereign proposal. The dollar sign is not used (on this site it is a formula delimiter).

What this introduction covers

Fourth article in the data series and deep dive on the energy axis. Measuring the watts of an AI workload looks like a sustainability detail, but it is at once a cost lever (electricity is 30–50 % of TCO), an axis of sovereignty (carbon per token depends on the country) and, increasingly, a regulatory obligation (emissions reporting). This article inventories the metrics, where each figure physically comes from, the data sheet for each measurement framework, and how to go from watt to carbon with European figures. No recommendations; only data and methodology, because in energy, even more than in performance, the measurement method changes the result.

Why measuring energy matters (with numbers)

Three quantified reasons, connecting with the opening article:

Reason	Figure	Implication
Cost	electricity = 30–50 % of TCO	measuring energy is measuring almost half the cost
Sovereignty/carbon	France ~9× fewer gCO₂/token than Germany	the country of the cluster is a lever
Scale/regulation	datacenters ~460 TWh in 2025 (≈1.8 % worldwide), >800 TWh projected for 2028	emissions reporting (CSRD) mandatory

At 700 W per H100, an 8-card node draws of the order of 5.6 kW from the boards alone, ~49,000 kWh a year. Applying PUE 1.4 that becomes ~68,700 kWh/year, and there the country decides the bill: at the French wholesale price (~0.058 €/kWh) that is ~3,980 €/year of electricity per node; in Germany (~0.091 €/kWh), ~6,250 €/year, 57 % more for the same iron and the same work. Multiplied across a fleet, it is a first-order cost line, and a carbon footprint that, in Europe, has to be measured and reported. Energy stopped being an engineering detail and became a business and compliance variable.

The energy metrics

Metric	Definition	Unit
Mean power	average instantaneous consumption	W
Energy per token	energy consumed per generated token	J/token
Energy per inference	energy per request	µJ – Wh
Total energy	integral of power over time	Wh, kWh
PUE (Power Usage Effectiveness)	datacenter overhead (cooling, losses)	ratio ≥ 1
Carbon	energy × grid intensity	gCO₂eq

The two base identities of the whole axis:

$$\text{energy per token (J)} = \frac{\text{mean power (W)}}{\text{throughput (tok/s)}}$$ $$\text{carbon} = \text{energy (kWh)} \times \text{PUE} \times \text{grid intensity (gCO}_2\text{/kWh)}$$

The first has the same shape as cost per token: it shares the throughput denominator, so raising tokens/s lowers energy per token. The second brings in the two multipliers external to the compute, the datacenter PUE and the country’s grid intensity, which can change carbon per token by an order of magnitude without touching the stack.

Worked example: energy per token of an 8×H100 node

To anchor the identities with numbers, the end-to-end calculation on the example node (8×H100, Llama 3.1 70B FP16, vLLM at 2,800 tok/s):

Step	Calculation	Result
Board power	8 × 700 W	5,600 W
Energy per token (boards)	5,600 ÷ 2,800	2.0 J/token
Energy per token (with PUE 1.4)	2.0 × 1.4	2.8 J/token
Energy per 1M tokens (boards)	2 × 10⁶ J	0.56 kWh
Energy per 1M tokens (with PUE)	0.56 × 1.4	0.78 kWh

From there come the electricity cost and the carbon, by country:

Country	Electricity cost / 1M tok	Carbon / 1M tok
France (0.058 €/kWh; ~40 gCO₂/kWh)	~0.045 €	~31 gCO₂
Spain (0.054 €/kWh; ~160 gCO₂/kWh)	~0.042 €	~125 gCO₂
Germany (0.091 €/kWh; ~363 gCO₂/kWh)	~0.071 €	~283 gCO₂

And the link with performance: if an optimisation (FP8, say) raised throughput to 4,200 tok/s, energy per token would drop to ~1.33 J/token (boards) and everything below it in the same proportion. Electricity cost and carbon per token fall with throughput, by the identity. Note: these are board + PUE figures; the complete node (CPU, NICs, PSUs) consumes somewhat more, and energy measured with a power meter can differ from energy estimated with nvidia-smi.

Where the figure comes from: the measurement layers

Not every energy figure is equally reliable. The figure can come from four layers, with increasing accuracy and decreasing context:

The GPU is the dominant source in LLM inference, and its power is read from a hardware counter (NVML/nvidia-smi, aggregated by DCGM, the same basis as GPU observability). RAPL gives CPU and DRAM. The eBPF + model layer (Kepler) splits the total watt between containers and pods without instrumenting the application. And the physical power meter is the ground truth against which estimates are calibrated: no software estimate is better than the meter, only more convenient.

How it is instrumented (and where the error creeps in)

Four instrumentation decisions explain most of the discrepancies between energy measurements:

Sampling rate. nvidia-smi/DCGM read power at intervals; sampling that is too coarse misses the peaks and underestimates the energy. It must be fine enough to capture the power variation between prefill (high) and decode (lower).
Board vs node. GPU (board) power does not include CPU, NICs, fans or PSU losses (typically an extra 10–20 %). Comparing a board energy figure with a complete-node one is comparing different things.
Idle baseline. A powered GPU with no work already draws power. Is that idle attributed to the workload or discounted? The decision changes energy per token, and idle matters: it is real energy that someone pays for.
Time window. Energy is the integral of power over time; the measurement window has to line up exactly with the workload (without counting warm-up or shutdown), or the number does not correspond to the work measured.

These decisions, as in performance benchmarking, mean that the measurement method matters as much as the system measured. A J/token without a stated sampling rate, board/node boundary and idle treatment is not comparable.

Frameworks, tool by tool

Zeus (ml.energy, University of Michigan)

What it does: measure and optimise the energy of deep learning workloads. Method: NVML counters. Scope: NVIDIA and AMD GPUs, CPU, DRAM, Apple Silicon and NVIDIA Jetson. Licence: OSS; academic backing (NSDI'23 paper). Differentiator: it does not only measure, it also optimises (power capping, frequency selection) to cut energy without significant loss of performance (Zeus Project, NSDI'23). It is the reference tool when the goal is to lower J/token, not just to measure them.

CodeCarbon

What it does: estimates the energy and the CO₂eq of compute processes. Method: reads the GPU with nvidia-smi and estimates CPU/RAM. Scope: CPU + GPU + RAM. Licence: OSS (Python). Differentiator: it integrates into ML flows and CI pipelines, and it is the one that comes closest to the power meter among the Python tools (comparison). It brings carbon context out of the box. Limit: a high-level estimate, less exact than direct hardware measurement.

CarbonTracker

What it does: tracks energy and carbon at GPU run time. Scope: GPU only, it measures neither CPU nor memory. Licence: OSS. Differentiator: lightweight, easy to add to a training/inference script. Limit: partial coverage (no CPU/DRAM), so it undercounts the total system energy.

Scaphandre

What it does: a power metrology agent written in Rust. Method: RAPL (reads /proc, splits joules by CPU ticks). Scope: process or VM, with the raw accuracy of the hardware layer. Licence: OSS. Differentiator: the highest direct accuracy among the general-purpose tools, at the cost of bringing no carbon context (it is pure metrology) (Scaphandre · GitHub). Limit: focused on CPU/RAPL; for GPU it has to be combined with NVML.

Kepler (CNCF)

What it does: a Prometheus exporter that measures energy at container, pod and node level in Kubernetes. Method: eBPF + ML models to estimate consumption per workload. Language: Go + C. Licence: Apache 2.0 (the sustainable-computing-io project) (Kepler · GitHub). Differentiator: it is the cloud-native option for attributing energy per pod without instrumenting the app, just as OpenCost attributes cost. Limit: attribution via eBPF + model is estimation, not direct measurement; its accuracy depends on the model.

MLPerf Power (MLCommons)

What it does: a standardised benchmark of energy efficiency, from µW to MW. Maintainer: MLCommons. Differentiator: cross-vendor comparability of efficiency, with a dataset spanning several years and workload versions (MLPerf Power, arXiv 2410.12032). It is to energy what MLPerf Inference is to performance: the comparison standard between vendors.

Comparison table

Tool	Method	Scope	Licence	Accuracy / note
Zeus	NVML / counters	NV+AMD GPU, CPU, DRAM, Apple, Jetson	OSS	measures and optimises DL energy
CodeCarbon	nvidia-smi + estimation	CPU+GPU+RAM, CO₂eq	OSS (Python)	the closest to the power meter among Python tools
CarbonTracker	GPU runtime	GPU only	OSS	lightweight; no CPU/mem
Scaphandre	RAPL (/proc)	process/VM	OSS (Rust)	raw accuracy; no carbon context
Kepler	eBPF + model	container/pod/node (K8s)	Apache 2.0 (CNCF)	cloud-native; model-based estimation
MLPerf Power	standardised measurement	µW to MW	MLCommons	cross-vendor comparability

Accuracy: who to believe

The difference between tools is not a nuance: Scaphandre offers the raw accuracy of the hardware layer (RAPL), while the Python ones such as CodeCarbon give higher-level, carbon-aware estimates, with more context but less direct exactness; among the Python tools, CodeCarbon is the one that comes closest to the power meter, followed by CarbonTracker, with variability across infrastructures (comparison). The rule: for sizing and design, a software estimate will do; to defend a figure before a committee or an auditor, calibrate against a power meter at least once, and use the software tool as a continuous proxy. The measured figure beats the estimated one; the rest is convenience.

Energy leaderboards and benchmarks

To compare efficiency between models without building your own test bench:

Resource	What it provides
MLPerf Power	standardised efficiency measurement, from µW to MW
HF AI Energy Score	comparable per-model energy efficiency ratings (HF)
ML.ENERGY Benchmark	automated measurement and optimisation of inference energy
awesome-green-ai	curated index of Green AI resources and tools (repo)

These resources give the order of magnitude of the energy per token of each model, useful for choosing a model on efficiency at the design stage. But they do not replace measurement on your hardware and your workload: a leaderboard measures on one specific bench, not on your node.

From watt to carbon: France, Germany, Spain

Here the energy axis turns into a sovereignty argument. Carbon per token comes from multiplying the energy by the country’s grid intensity, which in Europe varies by almost two orders of magnitude:

Country	Wholesale (€/MWh, Mar 2026)	Grid carbon (gCO₂/kWh)	Profile
France	~58	~20–60	nuclear: cheap and very clean
Spain	~54	~150–170	renewables + gas: cheap, medium carbon
Germany	~91	~363 (2024 average)	coal/gas + renewables: expensive and dirty

Sources: wholesale prices (TradingEconomics), Germany’s carbon ~363 gCO₂/kWh (2024 average, Wikipedia), nuclear ~17–35 gCO₂e/kWh (EEA), Spain ~167 gCO₂/kWh (2020). Hourly intensity comes from ElectricityMaps.

The calculation, on the example node (0.78 kWh per million tokens, with PUE): in France (~40 gCO₂/kWh) it is ~31 gCO₂ per million tokens; in Germany (~363 gCO₂/kWh), ~283 gCO₂, a factor of ~9×, without touching a line of the stack. Placing the cluster in France or Spain is at once a cost lever, a carbon lever and a compliance lever (EU AI Act, controls ENS × 42001 × EU AI Act).

Energy efficiency levers

Once it is measured, it can be brought down. The levers, ordered by the identity J/token = W ÷ throughput (lower the numerator or raise the denominator):

Lever	Mechanism	Effect
Quantisation (FP8/INT4)	more tokens per GPU-hour, less VRAM/KV	raises throughput → lowers J/token
Batching / continuous batching	amortises the fixed power cost	raises throughput → lowers J/token
Power capping (Zeus)	limits GPU power	lowers W with little loss of throughput
Efficiency-aware scheduling	consolidate workloads, power down idle GPUs	eliminates idle consumption
Location (country)	cleaner grid	lowers carbon per token
Avoiding over-reasoning	fewer reasoning tokens that add nothing	less compute → less energy

Quantisation (see quantisation for inference) is the highest-return lever because it moves all three axes at once: it raises throughput, lowers cost and lowers energy per token. Zeus’s power capping is energy-specific: it trims peak power with little performance loss, lowering the W in the numerator. And idle, the GPU powered but not working, is pure wasted energy that scheduling recovers.

From design to production: where energy is measured

As with the other axes, energy is measured at three moments, with different tools and increasing accuracy:

Moment	Tool	What it gives
Design	TDP × hours (datasheet)	ceiling estimate for sizing
Benchmark	Zeus, MLPerf Power, power meter	energy per token measured on a controlled bench
Production	Kepler (eBPF), DCGM	continuous per-pod consumption in the cluster

The design figure (TDP) serves to size the utility feed and the cooling, but it overestimates (the GPU is rarely at TDP 100 % of the time). The benchmark figure is the one used for the proposal’s cost and carbon per token, because it is measured and reproducible. And the production figure (Kepler/DCGM) closes the loop: it checks that real consumption matches the budget and feeds the footprint report. The three have to be traceable to each other, which is what the harness in article S4 does.

Energy in TCO and European regulation

The closing of the axis: energy is not only sustainability, it is cost and compliance.

Cost: being 30–50 % of TCO, every J/token improvement translates directly into euros. In the example, moving from Germany (~0.091 €/kWh) to France (~0.058 €/kWh) lowers the electricity share by ~37 % on top of the carbon.
Regulation (CSRD): European companies must report their carbon footprint (scope 2). A low-carbon grid (France) directly improves that reported figure, and measuring energy per token stops being optional and becomes a reporting requirement.
EU AI Act: efficiency and energy consumption form part of the documentation expected of AI systems, which makes energy measurement part of the technical file.

One reporting nuance worth knowing: the electricity footprint is scope 2 (indirect emissions from purchased energy), and it can be accounted for by location (the real intensity of the country’s grid) or by market (according to the guarantees of origin contracted). Location-based accounting is the one that genuinely rewards putting the cluster on a clean grid such as the French one; market-based accounting allows green energy to be “bought” through certificates. Knowing which one your CSRD report requires changes which lever (physical location vs contract) lowers the figure, and both start from measuring energy per token, which is what this axis provides.

In other words: measuring energy covers at once a first-order cost line, the carbon and sovereignty argument, and a regulatory obligation. For a sovereign European proposal, the energy axis is not the “nice sustainability bit”: it is one of the three columns with a number attached.

Checklist for a reproducible energy measurement

For an energy or carbon figure to be defensible before a committee or an auditor, it has to come with its context. The minimum to fix and publish:

What to fix	Why
Tool + version	each layer (RAPL, NVML, eBPF) and tool measures differently
Sampling rate	coarse sampling misses peaks
Board vs node boundary	±10–20 % depending on whether CPU/PSUs are included
PUE used	it multiplies all the compute energy
Source and time of the grid intensity	carbon varies ~9× by country and by hour
Idle treatment	attributed or discounted changes the J/token
Model, precision and workload	energy depends on what is served and how
Calibration vs power meter	at least once, to validate the estimate

The rule: if you cannot hand over this table alongside the gCO₂/token figure, the figure is not auditable. The reproducible harness of article S4 records all these parameters so that the energy figure is as traceable as the cost and performance ones, and so that the reported footprint (CSRD) holds up under audit.

State of the art 2026

LLM-specific resolution is missing: general-purpose tools measure well at machine or process level, but they lack LLM-specific resolution (energy per token, per prefill/decode phase), and the simulators lack emissions modelling, an open gap in the field.
eBPF as a trend (Kepler) for attributing energy per pod without instrumenting the app, in parallel to how OpenCost attributes cost.
Standardisation of comparison via MLPerf Power and HF AI Energy Score.
Optimisation, not only measurement (Zeus): the field is moving from “how much do I spend” to “how do I spend less without losing performance”.

Limits and traps (data-driven)

Estimation vs measurement. An nvidia-smi figure is not a power meter. Calibrate against hardware at least once before defending a number.
Partial coverage. CarbonTracker measures GPU only; Scaphandre, mostly CPU/RAPL. Add the layers or you undercount the total energy.
Forgetting PUE. Board energy does not include cooling or losses; without PUE you underestimate the real consumption and carbon.
Carbon without a country. A gCO₂/token without a stated grid (and hour) means nothing: it varies ~9× between France and Germany.
Confusing energy with sustainability. Here it is cost and compliance, with a number; treating it as a “green” extra is losing one of the three columns of the proposal.

That closes the run of introductions: cost, performance and energy, each with its frameworks and its methodology. The rest of the series goes tool by tool through to the dashboard that underpins the sovereign architecture decision, with European figures, measured and reproducible.

Closing

Energy is the axis easiest to dismiss as “the green one” and the one that hides the most: it is almost half the cost, it defines the footprint Europe requires to be reported, and its carbon per token changes ~9× depending on whether the cluster sits in France or in Germany. Measuring it properly, with the right layer, the right sampling, PUE included and the country’s grid, turns a soft sustainability argument into a hard column of the proposal, with a number and a source. And the sovereign conclusion stands on its own with the data in this article: an inference platform in France or Spain is at once cheaper, cleaner and compliant with European jurisdiction than the same platform on a US hyperscaler, and that is demonstrated with measured J/token, gCO₂/kWh by country and euros, not with intentions. The energy axis does not decorate the proposal: it closes it.

Sources

Zeus Project (ml.energy, UMich) — https://ml.energy/zeus/
Zeus · USENIX NSDI'23 — https://www.usenix.org/system/files/nsdi23-you.pdf
Kepler (CNCF, eBPF) · GitHub — https://github.com/sustainable-computing-io/kepler
Scaphandre · GitHub — https://github.com/hubblo-org/scaphandre
CarbonTracker · GitHub — https://github.com/saintslab/carbontracker
MLPerf Power (arXiv 2410.12032) — https://arxiv.org/html/2410.12032v2
HF AI Energy Score — https://huggingface.github.io/AIEnergyScore/
awesome-green-ai — https://github.com/samuelrince/awesome-green-ai
TradingEconomics · precio electricidad Francia/Alemania/España — https://tradingeconomics.com/france/electricity-price
EEA · intensidad de emisiones de la electricidad en Europa — https://www.eea.europa.eu/en/analysis/indicators/greenhouse-gas-emission-intensity-of-1
Electricity sector in Germany (carbono ~363 gCO₂/kWh) — https://en.wikipedia.org/wiki/Electricity_sector_in_Germany

Zeus on lo0 — Tech Blog

Measuring the power of a GPU: NVML, DCGM and the sampling errors that invalidate your watts

TL;DR

What the hardware exposes

NVML: power

NVML: the energy counter

NVML: power limits

DCGM: the equivalent fields

The real accuracy of the sensor

Update period and averaging window

The steady-state error

The response to transients

The two ways of obtaining energy

Why 1 Hz destroys a 200 ms inference

Aliasing with periodic loads

What the GPU counter does not measure

The tax of keeping a context open

CPU and DRAM: RAPL

The software stack

Procedure for making a Wh/token auditable

Subtracting idle, or not

Power capping: what it costs and what it saves

Reference figures

Cross-links from the energy track

See also

Sources

Levers for spending fewer watts per token: a quantified catalogue for LLM inference

TL;DR

The reference identity

Master table: levers × impact × cost × tool

Lever 1: quantisation

Mechanics

Quality cost

OSS tooling

Lever 2: continuous batching

Mechanics

OSS tooling

Lever 3: power capping

Mechanics

The power cap vs throughput curve (H100 SXM)

J/token formula with a power cap

Commands

Lever 4: DVFS (Dynamic Voltage and Frequency Scaling)

Mechanics

Commands (manual DVFS)

Lever 5: model selection (SLM, MoE vs dense)

The effect of scale

MoE vs dense

Model selection table by use case

Lever 6: speculative decoding

Mechanics

The energy efficiency condition

OSS tooling

Lever 7: prefix caching and chunked prefill

Prefix caching

Chunked prefill (SARATHI)

Lever 8: scheduling to clean hours

The power cap vs throughput curve: the sweet spot

How to measure the effect and validate quality

Measurement stack

Measurement protocol per lever

Quality validation

Pareto decision table (energy / latency / quality)

Stacking levers: the combined effect

Reference hardware: TDP and power cap ranges

Executive summary by role

Sources

Measuring energy in production: Kepler, DCGM and the practical stack (accuracy and overhead)

What this article covers

The measurement stack on Kubernetes

DCGM exporter: the base of GPU power

Kepler in depth: eBPF and the sources

Deployment: a DaemonSet on top of Prometheus

PromQL queries: energy per pod

A calibration procedure

Accuracy: estimation vs power meter

Overhead: the cost of measuring

Zeus: for the bench and for optimisation, not for continuous production

CodeCarbon and Scaphandre in practice

Choosing the stack by objective