Energy on lo0 — Tech Blog

Measuring the power of a GPU: NVML, DCGM and the sampling errors that invalidate your watts

Sat, 29 Aug 2026 08:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

TL;DR

A GPU power sensor does not return what almost everyone believes. The NVML documentation is explicit: on Ampere except GA100, and on later architectures including H100, nvmlDeviceGetPowerUsage returns power averaged over a one-second interval, not instantaneous power. The University of Oxford study presented at SC24, covering more than 70 GPUs across 25 models checked against an external 1 mΩ meter, adds the fact that breaks most published measurements: on the A100 and H100 the sensor only samples 25 % of the time (a 25 ms window within a 101 ms period), so during the remaining 75 % the GPU may be consuming something radically different. Integrating power naively over nine real benchmarks, the mean error was 39.27 %; applying good practice brought it down to 4.89 %. The operational consequence is short: if the hardware is Volta or later, use the accumulated energy counter (nvmlDeviceGetTotalEnergyConsumption, in millijoules) and subtract, rather than integrating samples. And the GPU counter is not the bill: on measured 8× H100 nodes, the node reaches 8.4 kW against 5.6 kW of summed GPU TDP, node idle is 1.8 kW, and with an average PUE of 1.54 the NVML watts are of the order of 43 % of the watts the utility bills.

What the hardware exposes

NVML: power

nvmlDeviceGetPowerUsage(device, unsigned int* power) returns milliwatts for the GPU and its associated circuitry, for example the memory. It has been supported since Fermi. The note in the official documentation is the part almost nobody quotes (NVML Device Queries):

On Fermi and Kepler the reading is accurate to ±5 % of current consumption. On Ampere (except GA100) or later, the API returns power averaged over a 1 s interval. On GA100 and earlier architectures instantaneous power is returned.

In other words, on an H100 or an L40S the usual call already delivers a one-second moving average. Anyone sampling it at 1 Hz and integrating is applying a second filter on top of an already filtered signal, without declaring it. To disambiguate, NVML exposes two separate fields accessible via nvmlDeviceGetFieldValues: NVML_FI_DEV_POWER_AVERAGE and NVML_FI_DEV_POWER_INSTANT, with command-line equivalents (nvidia-smi --query-gpu=power.draw.average and power.draw.instant).

NVML: the energy counter

nvmlDeviceGetTotalEnergyConsumption returns accumulated energy in millijoules since the last driver reload, supported on Volta or later. It is a 64-bit integer, so overflow is not a practical problem: an H100 at 700 W consumes of the order of 6.1·10¹⁰ J per year against a counter range of 1.8·10¹⁶ J. The only event that resets it is a driver reload or a system restart, and that does need watching: a modprobe -r nvidia halfway through a campaign produces a negative delta.

NVIDIA does not document the internal time resolution of the counter, its accumulation mechanism or its accuracy, and I have found no peer-reviewed work validating it against an external meter. The recommendation to use it, which the rest of this sheet rests on, is a structural argument (it removes quadrature error and aliasing error), not a published empirical validation.

NVML: power limits

Function	Unit	Semantics
`nvmlDeviceGetPowerManagementLimit`	mW	Currently configured limit
`nvmlDeviceGetPowerManagementLimitConstraints`	mW	Legal minimum and maximum range
`nvmlDeviceGetPowerManagementDefaultLimit`	mW	Limit the card boots with
`nvmlDeviceGetEnforcedPowerLimit`	mW	Effective limit after considering all limiters, including the out-of-band interface

For an auditable measurement you have to record EnforcedPowerLimit, not PowerManagementLimit: the BMC may be imposing an out-of-band ceiling that the latter does not reflect, and two apparently identical nodes may be operating at different ceilings.

DCGM: the equivalent fields

Field	ID	Unit	Type in the exporter
`DCGM_FI_DEV_POWER_USAGE`	155	watts in floating point	gauge
`DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION`	156	accumulated millijoules	counter
`DCGM_FI_DEV_POWER_MGMT_LIMIT`	160	mW	gauge
`DCGM_FI_DEV_ENFORCED_POWER_LIMIT`	164	mW	gauge

Both energy fields are active in the default counter file of dcgm-exporter. The audit conclusion is direct: a standard Prometheus panel integrating DCGM_FI_DEV_POWER_USAGE is integrating a gauge that is already a one-second average of the sensor, sampled at the exporter’s collect-interval. For energy per token the correct metric is counter 156 with increase().

The real accuracy of the sensor

The reference is the work of Yang, Adámek and Armour (University of Oxford), published at SC24 and available as arXiv:2312.02741: more than 70 GPUs across 25 models, including 10 H100s and 10 A100s, checked against an external meter with a 1 mΩ shunt, a 12-bit ADC and internal sampling at 34 kHz.

Update period and averaging window

GPU or architecture	Update period	Averaging window
Kepler and Maxwell	10-50 Hz	logarithmic growth, ~200 ms
Volta (V100)	20 ms	10 ms
Turing	100 ms	100 ms
A100 (GA100)	101 ms	25 ms
Non-GA100 Ampere and Ada, driver earlier than March 2023	100 ms	1 s
Non-GA100 Ampere and Ada, driver 530	100 ms	100 ms
H100	100 ms	25 ms with `.instant`; 1 s with `.average` and with the default field
GH200	100 ms	20 ms on the GPU, 10 ms on the Grace CPU

Two readings of this table. The first: the driver is an experimental variable, not an infrastructure detail; the averaging window changed between drivers earlier than 530, driver 530, and the later ones. Any measurement meant to be compared over time has to record the driver version.

The second is the central finding of the paper. On the A100 and H100 the 25 ms window sits inside a 101 ms period, which gives a sensor duty cycle close to 25 %. The sensor does not average the full interval: it averages a quarter of it and the system presents the result as if it described the whole interval. In the words of the work itself, during the other 75 % of the time the GPU may be consuming radically different power and nvidia-smi never notices.

The steady-state error

The paper also quantifies the claim in the nvidia-smi documentation, which spoke of an accuracy of ±5 watts: the real error is proportional, ±5 %, not absolute. On a GPU capable of consuming 700 W that is ±35 W of over- or underestimation. The errors measured against the external meter, after correcting the time offset, were −4.70 % and −4.53 % on the RTX 3090 and −5.43 % on the A100, and the residue is attributed to the tolerance of the card’s own physical shunt, which no software can correct.

The response to transients

Four distinct behaviours were observed in the 10 % to 90 % rise time: an almost instantaneous real rise with the reading following on the next tick and a delay of 0 to 100 ms; a real rise of several hundred milliseconds with the reading still updating on the next tick; linear growth over a second, which corresponds to power.draw.average; and logarithmic growth over 200 ms, only on Kepler and Maxwell. Out of this comes the most uncomfortable warning in the work: when running a short program, the measured power probably corresponds to the activity before the program.

The two ways of obtaining energy

The first is to integrate samples, which is what any poller does:

$$E \approx \sum_i P(t_i) \cdot \Delta t$$

It accumulates five sources of error: the ±5 % of the sensor, quadrature error from the finite step, aliasing if the load has spectral content above half the sampling frequency, the fact that each sample is already a moving average, and the jitter of 0 to 100 ms between the host clock and the sensor tick.

The second is to subtract the counter:

$$E = \text{counter}(t_1) - \text{counter}(t_0)$$

Two calls and a subtraction. It removes quadrature, aliasing and time misalignment; the ±5 % of the sensor remains.

Why 1 Hz destroys a 200 ms inference

With sampling at 1 Hz, the sampling theorem only allows components below 0.5 Hz to be reconstructed, while a 200 ms pulse has content in the region of 5 Hz. The expected number of samples inside the event is 0.2, so with probability 0.8 none falls inside it: the result is not noisy, it is undefined.

Even sampling at the maximum useful rate, which on the A100 and H100 is about 10 Hz given the 101 ms period, a 200 ms inference produces one or two distinct sensor updates, and each has observed only 25 ms of real activity: effective coverage is around 25 % of the event. And because of the fourth transient behaviour in the previous section, the reading obtained during those 200 ms may describe what was happening before the kernel was launched.

Aliasing with periodic loads

The paper observes the phenomenon directly on the A100: with a square wave of period slightly different from 100 ms, the reading fluctuates between high and low values with a clear beat. An inference server with periodic arrivals, health checks, fixed-size batches or a scheduler tick is exactly that pathological case.

When only instantaneous power is available, the measured good practice from the Oxford work consists of running 32 consecutive iterations or a minimum of 5 s, inserting 8 evenly spaced controlled delays if the window is smaller than the period, repeating over 4 separate trials with a random delay between them, and shifting the series in post-processing to synchronise it with the real activity. Over nine real benchmarks that took the mean error from 39.27 % to 4.89 %.

What the GPU counter does not measure

The scope declared by NVML is the GPU and its associated circuitry: the SXM module or the PCIe card, with its HBM and its regulators. Left out are the CPU, host DRAM, NVSwitch, network cards, storage, fans and power supply losses.

The measured node figures come from the empirical calibration work on 8× H100 SXM5 nodes (arXiv:2506.14551):

Quantity	Value
Nominal node TDP declared by the manufacturer	10.2 kW
Node power at idle	1.8 kW
Maximum measured under saturating load	8.4 kW
Peak on real production loads	never above 76 % of TDP

Two conversion factors come out of that, both my own estimates from those figures: 8 × 700 W is 5.6 kW of GPU against 8.4 kW of node, that is, the GPU is of the order of 67 % of node power and the GPU-to-node factor is around 1.50×. Chaining that with an industry average PUE of 1.54 (Uptime Institute, 2025), the total factor from the NVML counter to the utility feed is around 2.3×; with a PUE of 1.1 it drops to 1.65×. Put in the form that matters to whoever signs the bill: the watts NVML reports are roughly 43 % of the watts the utility bills in an average datacenter.

As a complementary reference, an 80 PLUS Titanium supply at 230 V is 96 % efficient at 50 % load and 91 % at 100 %, so supply losses add between 4 % and 9 % that is already included if you measure at the socket.

The tax of keeping a context open

A GPU serving a loaded model with no traffic does not consume its nominal idle. Measurements over 335,267 production samples from 14 H100s across 18 days, plus controlled experiments, give this table:

GPU	Idle without a CUDA context	Idle with a CUDA context	Extra cost	Percentage of TDP
H100	71.8 W	121.7 W	+49.9 W	7.1 %
A100	53.7 W	80.0 W	+26.3 W	8.8 %
L40S	35.6 W	102.1 W	+66.4 W	19.0 %

The result that changes the mental model: more than 98 % of that extra cost is produced by the open CUDA context, independently of the memory occupied. Varying allocated VRAM between 0 and 72 GB moves power by less than 1 W. The size of the loaded model does not cost watts; keeping the context open does.

That cost is not marginal on a real platform. Over 11,791 long-running jobs, the measured split was 24 % of the time and 7 % of the energy in deep idle, 15 % of the time and 10 % of the energy idle with an active context, and 61 % of the time with 83 % of the energy in execution. On serving loads, GPUs can spend 48 % of their energy in low-activity periods.

CPU and DRAM: RAPL

To close the node you have to measure what is not GPU, and there the interface is RAPL through the kernel powercap framework, at /sys/devices/virtual/powercap/intel-rapl/. The hierarchy exposes intel-rapl:N as the socket and intel-rapl:N:M as subzones (core, uncore, dram), plus the psys domain for the full SoC since Skylake.

File	Content
`energy_uj`	Energy counter in microjoules; writing “0” resets it if the counter supports it
`max_energy_range_uj`	Counter range, that is, the overflow point
`power_uw`	Current power in microwatts
`name`	Zone name

The figures that condition its use come from the characterisation published in ACM TOMPECS: an update roughly every 1 ms, with jitter; an energy quantum of 61 µJ on Haswell and Skylake, 15.3 µJ on Sandy Bridge; and counter overflow in 52 minutes on a Haswell at 84 W, which forces polling at a far shorter period and detecting the wrap with max_energy_range_uj. There is also a thermal drift that is rarely declared: package power for the same load grows between 10 % and 12 % between 37 °C and 74 °C on Haswell. Warming up beforehand is not optional.

Access, on the other hand, is no longer free. The PLATYPUS attack (IEEE S&P 2021) demonstrated a purely software power side channel through RAPL, with CVE-2020-8694 and CVE-2020-8695 attached. The Linux mitigation arrived in commit 949dd0104c49, included in 5.10.0-rc4, which changed the default permissions so that only root can read energy_uj. The microcode mitigation associated with SGX goes further and introduces random noise into the reported energy and changes the reporting frequency: on machines with that mitigation active, RAPL readings are deliberately degraded. Measuring CPU and DRAM from a container therefore requires root, a udev rule relaxing the permissions, or a privileged daemon.

The software stack

Tool	Source of the GPU figure	Counter or power	Default interval	Overhead
Zeus	NVML	Counter if the architecture is Volta or later; polling otherwise	Delimited by windows	Less than 10 ms per call
dcgm-exporter	NVML via DCGM	Both fields available	`collect-interval` = 30,000 ms	5-10 W extra on the server’s IPMI reading
CodeCarbon	NVML	Counter	`measure_power_secs` = 15 s	5.38 % to 46.75 % of time at 1 kHz
Scaphandre	No GPU support; RAPL	`energy_uj` counter	Not documented	3.81 % to 28.38 % at 1 kHz
Kepler	RAPL and IPMI; GPU only as an experimental option	Hardware counters	—	—

Four details that change the choice:

Zeus decides in code with nvmlDeviceGetArchitecture and uses the accumulated counter on Volta or later, with a nestable window API (begin_window / end_window). It is the reference implementation of what this sheet recommends. Its NSDI'23 paper also documents energy reductions of 23.8 % to 75.7 % from choosing batch size and power limit well, and of 3.0 % to 31.5 % from touching only the limit, with five seconds of profiling per point enough for stable results.
Kepler has removed eBPF. The CNCF announcement of June 2026 describes the redesign towards reading /proc and /sys, motivated by the CAP_BPF and CAP_SYSADMIN permissions that blocked deployments, and by the loss of short-lived processes that underestimated the footprint. The new node metric closely follows the IPMI pattern and removes the spurious multi-kilowatt spikes of the previous version. For GPUs it remains an experimental option: today it is not the tool for measuring Wh/token on an accelerator.
CodeCarbon uses the NVML counter, which is correct, but its RAM estimate is a heuristic of 5 W per module derived from the total gigabytes, and the code itself acknowledges it.
DCGM delivers profiling metrics at 1 Hz by default, and warns that collecting at higher frequencies returns zeros because it groups metrics internally.

On the cost of measuring, the empirical overhead study of RAPL-based tools, all forced to 1 kHz, leaves a clear ordering:

Tool	Time overhead
CodeCarbon	5.38 % to 46.75 %
Scaphandre	3.81 % to 28.38 %
Turbostat	2.73 % to 14.37 %
PowerJoular	1.67 % to 8.88 %
perf	−1.00 % to 4.26 %
RAPL read in user space	−0.70 % to 2.93 %
RAPL read in kernel space	−0.17 % to 0.99 %

The underlying difference is that a system call costs of the order of 1.36·10⁻³ ms and an rdmsr instruction between 2.3·10⁻⁴ and 5.6·10⁻⁴ ms, an order of magnitude less. The authors recommend matching the granularity to the phenomenon being measured instead of raising the default frequency, and limiting polling to the necessary domains.

Procedure for making a Wh/token auditable

Fix and record the electrical state. Note EnforcedPowerLimit, the default limit and the driver version; lock clocks or the power ceiling if the experiment requires it.
Warm up to a stable temperature. RAPL thermal drift is 10 % to 12 % between 37 °C and 74 °C, and the power.draw.average transient takes up to a second to settle.
Measure the idle baseline twice: without a CUDA context and with the model loaded. They are two different values and both are needed.
Use windows of at least 5 s or 32 consecutive iterations.
Repeat over 4 separate trials with a random delay, and add 8 evenly spaced delays if you are integrating instantaneous power with a window smaller than the period.
Prefer the accumulated counter whenever the architecture is Volta or later, watching for driver reloads.
Report dispersion. The Oxford paper itself publishes the mean and standard deviation of the error; without an interval, a Wh/token figure is not auditable.
Declare the measurement boundary: GPU, node or utility feed. Without that label, two figures from the same facility differ by a factor of 2.3 with nobody knowing why.

Subtracting idle, or not

There is no consensus, and the honest approach is to treat it as a declared decision. In production the GPU is parked with the model loaded, and that consumption shows up on the bill: subtracting it erases a real cost. To compare the marginal efficiency of two models or two kernels, by contrast, subtracting it isolates the variable. The workable way out is to report both figures with labels: gross Wh/token, which includes idle and corresponds to the bill, and marginal Wh/token, comparable between models. What does not work is mixing them in the same table without distinguishing them.

One estimation trap remains alive in many spreadsheets: multiplying TDP by time overestimates real energy by up to 4.1 times, according to the ML.ENERGY Benchmark.

Power capping: what it costs and what it saves

Configuration	Energy saving	Performance loss
V100 at 200 W against 300 W (67 % of TDP), BERT training	~15 %	barely any degradation
V100 at 200 W, set of workloads	10-20 %	less than 5 %
V100 at 100 W (33 % of TDP)	40-60 %	30-40 %
A100 at 175 W against 400 W (44 % of TDP), LLaMA 65B inference	22-24 %	5-8 %

The asymmetry between phases explains the result. The Microsoft measurement at ASPLOS'24 documents that the prompt phase is short and reaches or exceeds TDP, while the token generation phase is longer and consumes less: the power ceiling penalises TTFT far more than TPOT. In their production cluster, inference uses 79 % of peak power against 97 % for training, and their measurements show that up to 20 % of power can be recovered with less than a 7 % performance loss.

The published sweep from 200 W to 700 W on the H100 and H200 adds the nuance of where the knee is: between 500 W and 700 W each 100 W step only contributes around 10 % of performance on compute-bound loads, while the H100 holds peak memory bandwidth even at 200 W. The most stable operating zones identified are 400 W on the H100 and 500 W on the H200. That the memory subsystem is protected against the trim is the physical reason why the decode phase of an LLM, which is memory-bound, loses so little performance under a power ceiling.

Two warnings about those figures: the H100/H200 sweep was measured with nvidia-smi polling every 10 s, exactly the problem described above, although on long steady loads the bias averages out; and there is no publication isolating the effect of memory clock locking on Wh/token in LLM inference, so any concrete figure on that point has to be your own measurement.

Reference figures

GPU	TDP
H100 SXM	up to 700 W, configurable
H100 NVL	350-400 W, configurable
A100 80GB SXM	400 W
A100 80GB PCIe	300 W
RTX 5090	575 W, with a recommended 1000 W supply
HGX node of 8× H100 SXM	10.2 kW nominal

Average power measured in inference: on an HPC cluster with vLLM on A100, the aggregate power of the GPU set ranged between 999 W and 2,983 W depending on model and parallelism, with Nemotron 70B on 8× A100 at 2,983 W, about 373 W per GPU, around 93 % of TDP.

Energy per token, with the warning that published figures measure different boundaries and cannot be mixed without normalising by GPU count and output tokens:

Source	Figure
TokenPowerBench, H100 94 GB	40 J per token on a standard load, more than 60 J per token in high-throughput configurations
TokenPowerBench, batch effect	−25 % of energy per token between batch 32 and 256 on a 70B model
TokenPowerBench, context effect	from 2K to 10K tokens of context on Llama 3 70B: energy ×3
TokenPowerBench, quantisation effect	Llama 3 405B in FP8 against FP16: −30 % of energy per token
TokenPowerBench, engine effect	TensorRT-LLM and vLLM against the Transformers engine: −25 % to −40 %
Per-prompt measurement in batch	0.0074-0.0289 Wh on 7B to 14B models; 0.0835-0.6912 Wh on 70B to 405B; single unbatched requests, between 10 and 100 times more expensive

Cross-links from the energy track

C2 — Energy per token: methodology and the electricity market: what to do with the joule once it is measured properly, and how the electricity price multiplies it.
C3 — Measuring energy in production: Kepler, DCGM and the practical stack: the continuous deployment of the stack whose metrology this sheet documents.
C3 — Energy benchmarking: frameworks, metrics and the state of the art: the full catalogue of tools and their metrics.
C4 — MLPerf Power: the certified alternative, measuring at the wall with an external analyser instead of the card’s sensor.
C6 — From watt to carbon: PUE, grid and grams of CO₂: the chain that turns the measured watt into emissions and into euros.
C7 — Energy efficiency levers: the quantified catalogue where the power ceiling of this sheet is just one more lever.

Sources

NVIDIA, NVML API Reference, Device Queries — https://docs.nvidia.com/deploy/nvml-api/group__nvmlDeviceQueries.html
NVIDIA, DCGM Library API, Field Identifiers — https://docs.nvidia.com/datacenter/dcgm/3.1/dcgm-api/dcgm-api-field-ids.html
NVIDIA, DCGM Feature Overview — https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html
NVIDIA, dcgm-exporter — https://github.com/NVIDIA/dcgm-exporter
NVIDIA, H100 Tensor Core GPU, especificaciones — https://www.nvidia.com/en-us/data-center/h100/
NVIDIA, A100 Tensor Core GPU, especificaciones — https://www.nvidia.com/en-us/data-center/a100/
NVIDIA, GeForce RTX 5090, especificaciones — https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/
Yang, Adámek y Armour (Oxford), Part-time Power Measurements: nvidia-smi’s Lack of Attention / SC24 — https://arxiv.org/html/2312.02741v3
Yang, GPU_Power_Benchmark, microbenchmark del trabajo anterior — https://github.com/JimZeyuYang/GPU_Power_Benchmark
ML.ENERGY, Measuring GPU Energy: Best Practices — https://ml.energy/blog/energy/measurement/measuring-gpu-energy-best-practices/
You et al., Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training, NSDI'23 — https://www.usenix.org/system/files/nsdi23-you.pdf
Zeus, Measuring Energy — https://ml.energy/zeus/measure/
ML.ENERGY, The ML.ENERGY Benchmark — https://arxiv.org/html/2505.06371v1
CNCF, Kepler, re-architected: improved power accuracy (junio 2026) — https://www.cncf.io/blog/2026/06/30/kepler-re-architected-improved-power-accuracy-and-a-community-call-to-action/
CodeCarbon, How Power Estimation Works — https://docs.codecarbon.io/latest/explanation/power-estimation/
Hubblo, Scaphandre: cálculo del consumo por proceso — https://github.com/hubblo-org/scaphandre/blob/main/docs_src/explanations/how-scaph-computes-per-process-power-consumption.md
Linux Kernel, Power Capping Framework (powercap sysfs) — https://docs.kernel.org/power/powercap/powercap.html
Intel, Running Average Power Limit Energy Reporting, INTEL-SA-00389 — https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/advisory-guidance/running-average-power-limit-energy-reporting.html
Lipp et al., PLATYPUS: With Great Power comes Great Leakage, IEEE S&P 2021 — https://platypusattack.com/
Khan et al., RAPL in Action: Experiences in Using RAPL for Power Measurements, ACM TOMPECS — https://www.devsustainability.com/p/paper-notes-rapl-in-action
What Is the Cost of Energy Monitoring? An Empirical Study on the Overhead of RAPL-Based Tools — https://arxiv.org/html/2604.26815v1
Patel et al. (Microsoft), Characterizing Power Management Opportunities for LLMs in the Cloud, ASPLOS'24 — https://www.microsoft.com/en-us/research/wp-content/uploads/2024/03/GPU_Power_ASPLOS_24.pdf
MIT Lincoln Laboratory, Sustainable Supercomputing for AI: GPU Power Capping at HPC Scale — https://arxiv.org/html/2402.18593
Architectural Trade-offs in the Energy-Efficient Era: power-capping NVIDIA H100 and H200 — https://arxiv.org/html/2604.11391v1
Empirically-Calibrated H100 Node Power Models for Datacenter Energy Analysis — https://arxiv.org/pdf/2506.14551
The Model Parking Tax: Quantifying the Hidden Energy Cost of Always-On GPU Model Deployment — https://arxiv.org/html/2605.23918
The Energy Cost of Execution-Idle in GPU Clusters — https://arxiv.org/html/2604.04745
TokenPowerBench: Benchmarking the Power Consumption of LLM Inference — https://arxiv.org/pdf/2512.03024
From Prompts to Power: Measuring the Energy Footprint of LLM Inference — https://arxiv.org/html/2511.05597
Serving LLMs in HPC Clusters — https://arxiv.org/abs/2507.00418
Uptime Institute, Global Data Center Survey 2025 — https://datacenter.uptimeinstitute.com/rs/711-RIA-145/images/2025.Annual.Survey.Report.pdf

Datacenter physical infrastructure (1/4): the power chain

Mon, 22 Jun 2026 20:00:00 +0200

This blog has devoted more than a hundred articles to the software layer of inference: serving engines, KV-cache, observability, FinOps. But all of that rests on something far more prosaic and far less forgiving when it fails: the current coming in through the wall. With this series we open a new vertical, the physical infrastructure of the datacenter, the cross-cutting topics an AI architect tends to take for granted until the day a power cut takes down an entire cluster. We start at the very beginning: the power chain.

The approach is deliberately mixed. The concepts (electrical chain, redundancy, Tier levels, power factor) are universal and apply to any datacenter. But in each section we will point out why the AI era, with its racks of 120 kW and more, stresses that infrastructure to extremes the classical design never anticipated.

Why power is the foundation of everything

In a datacenter, power is not an ancillary service: it is the primary constraint. You can survive a slow disk or a congested network, but not the loss of supply to critical equipment. And unlike software, electrical infrastructure is not hot-patched or scaled with a kubectl scale: it is designed, built, and lived with for fifteen or twenty years. Getting the sizing or the redundancy wrong is not a bug, it is a construction project.

In 2025-2026 this has gone from being a facilities concern to being the bottleneck of the entire sector. Power availability has become the factor that limits where and when an AI datacenter can be built: grid constraints stretch connection lead times to between 24 and 72 months, made worse by the global shortage of transformers, switchgear and even gas turbines. The challenge used to be finding GPUs; now, increasingly, it is finding megawatts.

The chain, link by link

Power travels a long road from the utility feed to the chips. Each link transforms, protects or distributes, and each one is a point where reliability is won or lost. The canonical sequence is this:

Grid (medium voltage) → MV switchgear → MV/LV transformer → low-voltage boards → UPS → floor power distribution units (PDU) → rack PDU or busway → server power supplies.

The utility feed arrives at medium voltage (MV) for facilities whose IT load exceeds a megawatt, which is already most of the relevant ones. Typical input voltages range from 11 to 33 kV. The MV switchgear is the entry and protection point; vacuum circuit breakers are today the indoor standard. From there, the transformers (usually 2.5 or 3 MVA) step the MV down to low voltage (LV), typically 400-480 V, which is what the hall handles.

Downstream of the transformer, the low-voltage boards distribute power towards the heart of the protection system: the UPS (uninterruptible power supply). The UPS performs two functions worth keeping separate: it conditions the power (filtering disturbances, voltage sags, harmonics) and, above all, it bridges the seconds the generator takes to start and pick up load when the grid fails. The dominant topology in serious installations is double-conversion online, which isolates the load completely from the incoming supply. We will return to the UPS in the fourth article of the series, because its relationship with the generator is the heart of resilience.

After the UPS, the floor PDUs (Power Distribution Units) feed the rows, and the rack PDUs (or an overhead busway) deliver power to each cabinet. Modern PDUs are not dumb power strips: metered units measure consumption per outlet and intelligent ones integrate with DCIM, add environmental sensors and allow outlets to be switched remotely. Finally, the server power supplies convert that LV into the DC voltages the components consume.

The A/B dual path

There is a design principle that runs through the whole chain in high-availability installations: the dual A and B path. Instead of a single route from the transformer to the rack, two electrically independent paths are built, with separate circuits, boards, UPS, PDUs and breakers, and no common points of failure. Every critical server carries two power supplies (dual-corded) and connects simultaneously to path A and path B. If one path goes down entirely, whether through a fault or planned maintenance, the server keeps running on the other without flinching. This architecture is the physical basis of 2N redundancy, and it is what allows half the installation to be maintained while the other half keeps serving.

Redundancy: the language of the “N"s

The whole conversation about electrical reliability is encoded in a compact notation worth mastering, because it defines the cost and the availability of the facility:

N is the strict minimum capacity to support the load, with no reserve at all. A failure of any component brings the system down.
N+1 adds one backup component to the set. It is the most efficient option in cost and energy, and the most common in cooling and, often, in generation.
N+2 adds two backups, to tolerate a second failure or a failure during the maintenance of another component.
2N duplicates the entire system: two complete, independent sets (the A and B paths). It tolerates the total failure of either path with no impact, and allows one half to be maintained while the other operates. It is the basis of fault tolerance.
2N+1 is the 2N duplication plus an extra component, to survive a failure and a maintenance activity at the same time.
Distributed redundancy (block redundant, sometimes “3N/2”): instead of backups dedicated to each block, it shares the reserve among several blocks, seeking a middle ground between the cost of N+1 and the robustness of 2N.

Two operational concepts sum up the intent of all this. A system is concurrently maintainable when any component can be taken out of service in a planned way without affecting the IT load; and it is fault tolerant when it also survives any single, unforeseen failure. The difference between the two is, essentially, the difference between being able to do maintenance without an outage and being able to absorb an unexpected event during that maintenance as well.

The economic trade-off is the axis of the whole decision. Going from N to N+1 adds a single component and is relatively cheap; going to 2N literally duplicates half the installation (two UPS, two paths, often two electrical rooms) and drives up capex, floor space and the no-load consumption of the redundancy alike. Distributed or block redundancy exists precisely to capture much of the robustness of 2N at a cost closer to N+1, sharing the reserve among several blocks instead of dedicating it. There is no universal answer: the right architecture is the one that matches the cost of unavailability for your business against the cost of avoiding it. For a critical workload under SLA, the usual order of magnitude is a concurrently maintainable distribution with a dual A/B path; for workloads that tolerate maintenance windows, N+1 can be perfectly reasonable and much cheaper.

The Uptime Institute Tier levels

The most widespread way of classifying this robustness is the Uptime Institute Tier system, with four levels in Roman numerals:

Tier I (basic capacity): single components, a single distribution path, no redundancy. Any maintenance or failure means an outage.
Tier II (redundant capacity): redundant components (N+1 in capacity) but a single, non-redundant distribution path.
Tier III (concurrently maintainable): multiple independent distribution paths; any component can be taken out for maintenance with no impact. It usually combines 2N in power with N+1 in cooling.
Tier IV (fault tolerant): 2N in every system; it survives any single infrastructure failure, as well as being concurrently maintainable.

Here an important warning is needed that many guides leave out. It is very common to see tables associating each Tier with an availability percentage (99.671 % for Tier I, 99.741 % for Tier II, 99.982 % for Tier III and 99.995 % for Tier IV) translated into minutes of downtime per year. Those percentages are not guarantees or official Uptime Institute definitions. In fact, Uptime removed the references to “expected annual downtime” from its standard in 2009, and maintains that its definitions are based on topology and behaviour (redundant components, concurrent maintainability, fault tolerance), not on a magic number of nines. A badly operated Tier IV can go down, and a well-operated Tier III can have a spotless service record. Quote those percentages, if at all, as popular reference figures, never as a promise.

Two more nuances for an architect. First, design is not the same as built: Uptime distinguishes the Tier Certification of Design Documents, which validates only the drawings and expires after two years, from the Constructed Facility certification, which verifies the real building with live demonstrations. A “designed for Tier III” without a constructed-facility certification is an intention, not a fact. Second, the European alternative: the EN 50600 standard, now folded into ISO/IEC 22237, classifies with Availability Classes 1 to 4 analogous to the Tiers, but with a more holistic approach covering power, cooling, telecommunications, security and efficiency, and it can be certified by independent bodies. In Europe it pays to speak both in Tiers and in EN 50600 classes.

The metrics you really have to understand

Beyond the nomenclature, there are three pairs of concepts an architect must handle in order not to get the sizing wrong.

kW versus kVA and the power factor

This is the most common sizing error. kW (active or real power) is the power that does useful work: what the servers actually consume. kVA (apparent power) is the power the electrical infrastructure (the UPS, the cables, the generator) has to be capable of delivering. The relationship between the two is the power factor (PF):

$$\text{PF} = \frac{P_{\text{kW}}}{S_{\text{kVA}}}$$

From which you get what really matters when sizing: the apparent power the equipment must supply,

$$S_{\text{kVA}} = \frac{P_{\text{kW}}}{\text{PF}}$$

If you size a UPS or a generator by the kW of the load alone, ignoring the power factor, you underestimate the capacity needed. Modern servers with active power factor correction work close to unity (PF from 0.9 to 0.99), but generators are usually specified at a PF of 0.8: a 1000 kW set is catalogued as 1200 kVA. And a 100 kVA UPS at PF 0.9 delivers only 90 real kW. A load of 150 kW at PF 0.8 demands 187.5 kVA of equipment, not 150. Confusing these two quantities has left more than one hall with no headroom the day it filled up.

PUE: the efficiency of the whole facility

PUE (Power Usage Effectiveness) measures how much total energy the facility consumes for every unit that actually reaches the IT equipment:

$$\text{PUE} = \frac{E_{\text{total}}}{E_{\text{IT}}}$$

A PUE of 1.0 would be the theoretical ideal (all the energy goes to IT, nothing is lost in cooling, conversion or lighting). The sector’s declared average is around 1.5-1.56, the big hyperscalers come down to 1.09-1.20 (Google reports around 1.09) and regulation is tightening: the German energy efficiency act requires a PUE of 1.2 or lower for new datacenters from 2026. PUE is, at bottom, a measure of how much energy “tax” the facility pays above what it computes.

Availability and the nines

Finally, availability is expressed in “nines”, and it pays to have internalised its translation into annual downtime: 99.9 % is almost 9 hours a year; 99.99 %, about 53 minutes; 99.999 % (the famous five nines), about 5 minutes. The economic key is that the cost of each additional nine is not linear: going from four to five nines does not cost a little more, it costs a multiple of the investment in redundancy, automation and, often, multiple sites. Chasing nines the business does not need is one of the most expensive forms of over-engineering.

A sizing example

Let us see how these quantities fit together in a concrete case, because the devil is in the detail. Suppose a row of eight GB200-class inference racks at 130 kW each: 1040 kW of real IT load. To size the UPS, which works on apparent power, we apply the load’s power factor (say 0.95):

$$S_{\text{kVA}} = \frac{1040}{0{.}95} \approx 1095 \text{ kVA}$$

that is, we need a UPS of around 1100 kVA for this row alone, and that without counting growth headroom or redundancy. To size the generator, on the other hand, the IT load is not enough: you have to add everything hanging off the emergency side, above all the cooling, which at these densities is liquid and consumes its share. If we apply a PUE of 1.3, the total facility load for this row comes to around 1350 kW, and since sets are rated at PF 0.8, the generator must deliver on the order of 1690 kVA. You would choose a 2 MVA set, and an N+1 architecture would need two of them. The jump is revealing: a row that “consumes 1 MW” demands, via power factor and cooling, more than 4 MVA of installed generation if you want redundancy. Anyone sizing in their head, multiplying racks by chip watts, falls short at every link.

The AI earthquake: rack density

This is where the AI era breaks the assumptions. For decades, rack density sat between 5 and 15 kW, and the sector average has barely risen: the Uptime Institute’s 2025 survey puts it at 7.5 kW per rack, with the bulk still between 5 and 9 kW. The whole classical electrical chain, and air cooling with it, was designed around those figures.

A current AI rack pulverises them. The NVIDIA GB200 NVL72 is specified at 120 kW per rack, and real deployments have measured 130-132 kW at full load: eight to ten times the density of a traditional rack, concentrated in the same cabinet. At that density air stops being viable (rear-door units of 30-40 kW fall short) and direct-to-chip liquid cooling goes from option to requirement. And the electrical chain has to carry those 130 kW to a single rack: dedicated three-phase circuits, PDUs of 200 A or more, a lot more copper.

And this has only just begun. NVIDIA’s roadmap points to some 190-230 kW per rack with the Vera Rubin generation, and to a staggering 600 kW per rack with Rubin Ultra “Kyber” in 2027, which NVIDIA proposes to feed directly at 800 V DC to reduce conversion stages and copper. The sector is already talking openly about 1 MW-class racks. These figures should be taken for what they are, vendor roadmap and industry debate rather than deployed installations, but the direction is unmistakable: density is rising faster than traditional electrical infrastructure knows how to absorb.

The aggregate impact is macroeconomic. Datacenters consumed around 415 TWh in 2024 (about 1.5 % of the world’s electricity), and the International Energy Agency projects that the global figure could approach 945 TWh in 2030, pushed by AI. In 2025 datacenter electricity demand grew by 17 %, and that of AI-specific datacenters by 50 %. That is why grid connection, and not silicon, has become the real limiting factor for many projects.

High-density distribution and the jump to 800 V DC

Taking 130 kW, and tomorrow 600 kW, to a single rack forces a rethink of distribution. By Joule’s law, carrying more power at the same voltage means more current, and more current means more copper, more losses and more heat in the distribution itself. The industry is responding along two lines. One is raising the internal distribution voltage: NVIDIA has proposed for its next-generation racks a direct supply at 800 V DC, which reduces AC-DC conversion stages, cuts the copper needed and improves efficiency against the classical scheme of 480 V AC converted to DC in several steps inside the rack. It is an emerging transition and still under debate, but it marks the direction. The other line is the high-current overhead busway instead of traditional PDUs, which makes it easier to reconfigure and scale power per row without rewiring. For an architect, the lesson is that distribution topology (voltage, busway, number of phases) stops being an installation detail and becomes a design decision with a direct impact on cost and on the maximum density achievable.

The grid as the limit: siting and on-site generation

There is a link upstream of the whole chain we have described, and in 2025-2026 it has become the most critical of all: the connection to the utility grid. Getting the grid to deliver tens or hundreds of megawatts at a specific point has become the sector’s real bottleneck. Connection lead times have stretched to between two and six years in many geographies, weighed down by substation saturation and by the global shortage of transformers and medium-voltage switchgear, whose delivery times are now counted in years, not months.

This has pushed a trend worth knowing about even though it falls outside the classical chain: on-site generation (behind-the-meter). More and more AI projects, unable to wait for the grid, install their own generation (gas turbines, gas engines, and in some cases fuel cells) to start up before having a firm connection, or so as not to depend on one. It is an inversion of the traditional paradigm, in which the generator was only an emergency backup: now, sometimes, it is the primary source while the grid is awaited. This dynamic has cost, emissions and regulatory implications that an AI architect would do well to keep on the radar, because they determine where you can build and how fast. The siting of an AI datacenter is, increasingly, a decision about power availability before anything else.

The facility is only as reliable as its operation

It is worth closing with a warning the Tier system already hinted at: topology is a necessary but not a sufficient condition. A facility with impeccable 2N redundancy can go down all the same through a botched switching operation, maintenance without a procedure, a mislabelled breaker or a transfer test that was never rehearsed. A substantial share of real outages is born not of an equipment failure but of human operation on that equipment. That is why the Uptime Institute insists that a badly operated Tier IV guarantees nothing, and why constructed-facility certification includes live demonstrations: seeing the system transfer load for real, not reading it off a drawing.

The practical consequence is that electrical reliability is managed with the same disciplines as any critical system: observability and rehearsal. Observability is provided by DCIM (Data Center Infrastructure Management), which instruments the whole chain (consumption per outlet at the PDUs, temperature, status of the UPS and the generators, fuel levels, switch positions) and allows the system to be seen in real time and problems to be anticipated. Rehearsal is provided by periodic testing: starting the generators under real load (with load banks if necessary), running full transfers to the emergency source, and verifying that the UPS holds the bridge. A continuity plan that has not been rehearsed at realistic scale is a hypothesis, not a guarantee; and the day of the power cut is not the moment to discover that the generator will not pick up load or that the UPS cannot hold the necessary seconds. The physical layer, like the logical one, rewards whoever measures and rehearses, and punishes whoever trusts the paperwork.

For an inference factory

What does someone building or running an inference factory take away from all this? Three ideas. First, that the electrical design must start from the real density of AI racks, not from historical averages: planning an inference hall at 10 kW per rack is planning for a facility that does not exist. Second, that liquid cooling and high-density distribution stop being exotic and become the base case, with everything that implies for sizing copper, PDUs and boards. And third, that redundancy has a cost that has to be matched to the business: an inference factory under SLA probably justifies a concurrently maintainable architecture (Tier III order, dual A/B path), but chasing full fault tolerance at every link can be a luxury the use case does not pay for. The availability of compute, as we saw in the storage series, depends as much on the physical layer as on the logical one.

This article has drawn the complete map of the chain. The next three go into the links where resilience against a power cut is decided: the generators that sustain the facility when the utility goes away, the switching (ATS and STS) that transfers the load between sources, and the UPS that bridges those critical seconds. Because a perfectly sized chain is worth nothing if, the day the grid blinks, there is nobody to take over in time.

Sources

Giga Energy, Data center electrical distribution system: Key elements — https://www.gigaenergy.com/blog/data-center-electrical-distribution-system
GBC Engineers, The Basics of Electrical Data Center Design in 2025 — https://gbc-engineers.com/news/electrical-data-center-design
CoreSite, What is Data Center Redundancy? N, N+1, 2N, 2N+1 — https://www.coresite.com/blog/data-center-redundancy-n-1-vs-2n-1
Dgtl Infra, Data Center Redundancy: N, N+1, 2N, and 2N+1 Explained — https://dgtlinfra.com/data-center-redundancy/
Uptime Institute, Tier Classification System — https://uptimeinstitute.com/tiers
Uptime Institute, Tier Classification Myths and Misconceptions — https://uptimeinstitute.com/myths
Uptime Institute Journal, Myths and Misconceptions Regarding the Tier Certification System — https://journal.uptimeinstitute.com/myths-and-misconceptions-regarding-the-uptime-institutes-tier-certification-system/
Techerati, Explaining the new family of ISO Data Centre Standards (EN 50600 / ISO 22237) — https://www.techerati.com/features-hub/explaining-the-new-family-of-iso-data-centre-standards/
Fuji Electric, The Importance of Power Factor in Data Center UPS Selection — https://americas.fujielectric.com/the-importance-of-power-factor-in-data-center-ups-selection/
Server Room Environments, How To Size UPS Systems In kVA and kW — https://www.serverroomenvironments.co.uk/blog/how-to-size-ups-systems-in-kva-and-kw
Statista, Data center average annual PUE worldwide 2025 — https://www.statista.com/statistics/1229367/data-center-average-annual-pue-worldwide/
Google Data Centers, Power usage effectiveness — https://datacenters.google/efficiency/
Uptime Institute, Global Data Center Survey 2025 — https://intelligence.uptimeinstitute.com/resource/uptime-institute-global-data-center-survey-2025
NVIDIA, GB200 NVL72 — https://www.nvidia.com/en-us/data-center/gb200-nvl72/
ToneCooling, GB200 NVL72 Cooling Requirements — https://tonecooling.com/nvidia-gb200-nvl72-cooling-requirements/
Tom’s Hardware, Nvidia shows off Rubin Ultra with 600,000-Watt Kyber racks (2027) — https://www.tomshardware.com/pc-components/gpus/nvidia-shows-off-rubin-ultra-with-600-000-watt-kyber-racks-and-infrastructure-coming-in-2027
SemiAnalysis, Inside the 800VDC Revolution – Part 1 — https://newsletter.semianalysis.com/p/inside-the-800vdc-revolution-part
IEA, Energy demand from AI — https://www.iea.org/reports/energy-and-ai/energy-demand-from-ai

Datacenter physical infrastructure (2/4): the generators

Mon, 22 Jun 2026 19:00:00 +0200

In the first article we mapped the complete power chain, from the utility feed to the rack, and left pending the links that decide resilience against a power cut. We start with the one that carries the weight of the hours: the generator set. The UPS, which we will see at the end of the series, is a sprinter that covers seconds; the generator is a long-distance runner, capable of sustaining the entire facility for hours or days while the utility fails to come back. And in the AI era it has stopped being a mere insurance policy to become, at times, the main source of power.

The generator’s role: long-duration backup

A datacenter’s line of defence against a grid failure is clear: the UPS takes over instantly, with no perceptible interruption, and sustains the load during the few seconds the generator needs to start, stabilise and accept load. From there on, the generator takes over the work and the UPS returns to its role as conditioner and reserve. The UPS saves the gap; the generator saves the outage. For a datacenter, units of 1 to 3 MW are typically used, sized to support 100 % of the load for the whole duration of the event, not to give a brief push.

What has changed in 2025-2026 is the purpose. The AI boom, combined with the impossibility of getting a grid connection within reasonable timescales, has pushed many projects to use on-site generation as the primary source (behind-the-meter), not as an emergency supply. Some 92 % of announced behind-the-meter projects (around 82 GW) date from the start of 2025 onwards. The extreme case is the off-grid Oracle and OpenAI datacenter in Texas: a 700 MW natural gas microgrid with almost two hundred Jenbacher engines dedicated to prime power and a dozen more for backup, to feed up to 1.4 GW of compute. The generator, which for decades was the actor waiting in the shadows, has walked onto the main stage. This change of role has cascading consequences, in the rating that has to be specified, in the applicable emissions regime, in fuel logistics and in maintenance, which run through this entire article. It is worth keeping in mind: many of the design decisions depend on whether the generator is an insurance policy that starts once a year or a power station that runs for thousands of hours.

Starting in 10 seconds

The requirement that defines the design of a mission-critical generator is the starting time. NFPA 110, which governs emergency power systems, classifies installations by level and by type. Datacenters usually use Level 1, Type 10: the system must deliver acceptable power to the loads within 10 seconds of the grid failure, and at Level 1 all loads are transferred within that time, regardless of the size of the facility. Ten seconds is, precisely, the gap the UPS has to cover with its batteries.

But starting on time is only half the challenge. The other half is accepting the load without destabilising. When a newly started generator receives a large block of load all at once, the rectifier bank of a UPS for example, the alternator voltage and the engine frequency dip momentarily (voltage dip, frequency dip) before recovering. If those dips are excessive, the server power supplies can trip and reboot in a loop even though the generator is running: the power cut has been dodged, but the system goes down anyway through a badly managed transient. This is the difference between a generator that “starts” and one that “sustains”.

Here is one of the technical reasons why diesel still dominates over gas. Diesel injects the fuel directly into the cylinder and can accept load blocks close to 100 % with tolerable dips. The gas engine, by contrast, opens an intake valve and the air-fuel mixture must travel through the manifold before burning, which introduces a delay: a turbocharged gas engine can accept steps as low as 10-15 %, against 60-80 % for diesel. For a sensitive load that appears all at once, that difference is decisive.

The framework that quantifies all this is the ISO 8528-5 standard, which defines performance classes G1 to G4 with increasingly strict limits on voltage and frequency deviation. For sensitive electronic loads (UPS, drives) the reference class is G3, which requires voltage regulation around ±1 % and frequency regulation of ±0.25 % in steady state, with fast recovery from transients. Class G4, stricter still, is agreed between manufacturer and customer. It is worth remembering that UPS units are especially sensitive to the rate of change of frequency: variations above half a hertz per second can make them reject the source.

The ratings: why “standby” is not enough

Not all generators of the same nominal power are equivalent; what distinguishes them is their rating, which defines how many hours and at what load factor they can work. ISO 8528-1 defines four:

ESP (Emergency Standby Power): variable load, up to about 200 hours a year, with an average load factor not exceeding 70 % over 24 hours. It is the classic “emergency backup”.
PRP (Prime Power): variable load, unlimited hours, average load factor around 70 %.
LTP (Limited-Time Power): up to about 500 hours a year.
COP (Continuous Operating Power): unlimited hours at constant load, normally at 100 % indefinitely.

To those four the industry has added a commercial rating specific to our world: DCC (Data Center Continuous). It is not part of ISO 8528-1, being manufacturer nomenclature (Cummins, Caterpillar/MTU), but it has been reviewed by the Uptime Institute and is considered suitable for Tier III and IV. Its distinguishing feature is that it allows unlimited hours with no average load factor restriction, neither variable nor constant. Why does this matter? Because a datacenter operating behind-the-meter, doing peak shaving, or simply wanting Uptime certification without the hour and load limitations of the Standby rating cannot settle for an ESP. The choice of rating is not a catalogue detail: it determines whether the generator is legally and technically fit for the use it is going to be put to.

Sizing it right: the devil in the transients

As we saw in the first article, generators are sized in apparent power (kVA), applying the power factor, with 0.8 the industry standard for mixed loads:

$$S_{\text{kVA}} = \frac{P_{\text{kW}}}{\text{PF}}$$

But the real sizing of a generator is rarely set by the steady-state load; it is set by the transients and the problem loads. Three factors force oversizing:

First, motor starting. Induction motors demand five to seven times their rated current on direct starting, with a very low power factor (0.2-0.35), which can translate into a peak of 2.5 to 3.5 times the motor’s rated kVA. Cooling pumps and compressors starting at the same time can sink the frequency of a badly sized generator. Soft starters and drives reduce that blow.

Second, non-linear loads and harmonics. UPS units and drives inject harmonic distortion (THD) that heats the alternator and degrades the waveform. The rule of thumb is to reduce the available capacity by around 0.8 % for each point of THD above 5 %, which in practice can require oversizing by between 15 and 25 %, or specifying an alternator with 2/3 pitch winding and permanent magnet excitation (PMG), which is more tolerant of distortion.

Third, derating for ambient conditions: above 40 °C of temperature or 1000 m of altitude, the generator loses capacity and this must be corrected. Adding it all up, the manufacturers and NFPA 110 itself recommend a safety margin of the order of 20-25 %. Sizing a generator “exactly” for the nominal load is the recipe for the cooling not starting on the day of the outage.

It is worth seeing the combined effect in a case. Returning to the example from the first article, a row that added up to the order of 1350 kW of total load with its cooling, in steady state that called for some 1690 kVA of generation. But if the cooling of that row includes several pumps and compressors starting at the same time after the outage, the transient starting peak can momentarily demand considerably more than those 1690 kVA, and the alternator must hold the frequency during that instant without the electronic loads dropping out. It is common that, after adding the margin for motor starting, the harmonic tolerance of the UPS and the ambient derating, the set selected for a row “of 1 MW of IT” ends up at 2 to 2.5 MVA, and that two are installed in N+1. A generator is not sized by what the load consumes, but by the worst instant it has to survive.

Diesel, gas, hydrogen and batteries

The choice of generation technology is one of the great debates of 2025-2026, and it pays to understand why, despite all the alternatives, diesel keeps winning.

The figure for the “diesel boom” is eloquent: installed diesel generation capacity in United States datacenters went from some 20 GW in 2018 to 55 GW in 2024, nearly tripling; in Virginia alone more than 10,500 units, 27 GW, had been permitted by the end of 2025. The reason is not nostalgia but fit. An AI datacenter is commissioned in 18-24 months, while a new gas plant depends on turbines with delivery times of 2 to 4 years and on grid connection queues that reach 7 years. Diesel is the only option that matches the speed, the risk tolerance and the regulatory framework of AI deployment, and it also accepts large load blocks and does not depend on a pipeline. Cummins has delivered more than 39 GW of equipment to datacenters and doubled its production capacity in 2025.

The alternatives are advancing, but slowly. Natural gas, with its greater autonomy, is the basis of the large behind-the-meter prime power projects, at the cost of a slower transient response. Bi-fuel systems combine diesel and gas to extend autonomy, reverting to diesel if the gas fails. Fuel cells are gaining traction: Bloom Energy (solid oxide) has signed agreements with hyperscalers and is moving from backup to prime power, and Microsoft tested with Caterpillar a 3 MW hydrogen fuel cell capable of feeding a facility for 48 hours, with the goal of eliminating diesel by 2030. Batteries (BESS), for their part, are being integrated into medium-voltage architectures to cover short events and reduce generator running hours, with the generator only starting on extended disturbances. But diesel remains the reference standard, and the alternatives coexist with it more than they replace it.

The table sums up the trade-offs each technology settles:

Technology	Transient response	Autonomy	Emissions in use	Maturity 2026
Diesel	Excellent (blocks ~100 %)	Limited by tank (24-72 h)	High; requires Tier 4 / SCR+DPF	Dominant
Natural gas	Slow (low steps)	High (depends on pipeline)	Lower than diesel	Growing (prime / behind-the-meter)
Bi-fuel	Good (reverts to diesel)	High	Intermediate	Niche
Fuel cell (H₂/SOFC)	Variable	Depends on H₂/gas supply	Very low or none	Emerging
BESS (batteries)	Instantaneous	Short (minutes to a few hours)	None in use	Complementary

The reading for an architect is that there is no “best” technology, but rather a combination appropriate to each case: diesel provides the response and the supply independence that critical starting demands, gas and fuel cells provide autonomy and low emissions for prolonged operation, and batteries cut engine hours. The most advanced architectures of 2025-2026 are hybrid for precisely that reason.

Emissions: the tension with regulation

Burning diesel at gigawatt scale has an environmental cost that regulation watches closely, and here there is a tension every AI architect must know about. The United States EPA Tier 4 Final standard for diesel engines requires drastic reductions in nitrogen oxides and particulates, of the order of 93-94 % against the first generations, which forces full aftertreatment with a particulate filter (DPF) and selective catalytic reduction (SCR) with urea (DEF). Meeting Tier 4 Final is expensive and bulky.

The critical nuance is that strictly emergency engines are exempt from the most severe requirements: they can run up to about 100 hours a year for testing and maintenance (of which a maximum of 50 for non-strictly-emergency uses), with no time limit during a real emergency. That framework fits perfectly with the traditional role of the generator as backup. But it collides head-on with behind-the-meter use as a primary source, which means thousands of hours a year and requires far more demanding environmental permits.

This tension is one of the knots of AI deployment. A generator classified as “emergency” is cheap to permit and to maintain, but it cannot be the main source without breaking the framework that grants it that exemption; a prime power generator is legally fit to run for thousands of hours, but it carries the cost and the bulk of full Tier 4 aftertreatment and environmental permits that are much slower to obtain. Where the rush to deploy compute collides with local air quality, projects fall back on “minor source” permits and on spreading the load across many small units, an exercise in regulatory as much as electrical engineering. And social pressure is rising: communities near large datacenters look ever more closely at the megawatts of diesel installed beside them. The industry navigates that tension with “minor source” permits and with architectures that spread the load, but it is regulatory ground in full evolution. In Europe, the equivalent standard is Stage V, which also requires aftertreatment for large engines. And a decarbonisation route with no retrofit is emerging: HVO (renewable diesel of vegetable origin), a drop-in substitute that reduces CO₂ by up to 90 %, is stored for up to ten years with low microbial risk, and which the main manufacturers already certify for their sets.

The fuel: autonomy and degradation

A generator is worth no more than its fuel supply. The prevailing design in large datacenters combines a belly tank integrated under each set with refuelling by tanker truck. The recommended autonomy goes from 12-24 hours per generator in normal facilities to 48 or 72 hours in mission-critical ones, and more still in seismic or remote areas.

The autonomy calculation is direct: the time the generator holds out is the fuel volume divided by the consumption,

$$t_{\text{autonomy}} = \frac{V_{\text{tank}}}{\dot{m}_{\text{consumption}}}$$

and consumption follows a useful rule of thumb: about 90 gallons per hour per MW at full load, equivalent to approximately 0.1 gallons of diesel per kWh generated. A 2 MW set can consume of the order of 140 gallons per hour at full load, so 72 hours at 75 % load requires of the order of 7,500 gallons in store. Multiply by an N+1 fleet of several sets and you understand why fuel logistics is a project in itself.

And there is a trap that gets forgotten: diesel degrades. It starts to oxidise at 6-12 months, accumulates water through condensation and encourages microbial growth that generates sludge and acids capable of clogging filters and injectors. The solution is polishing, filtering the stored fuel without emptying the tank, every 3-6 months, and not storing diesel for more than a year without treatment. A generator that will not start on the day of the outage because its fuel has spoiled is as absolute a failure as not having one at all.

The fuel architecture usually has two levels: a main tank (bulk, or the belly tank under the set) with the total autonomy, and a smaller day tank next to the engine that refills automatically and guarantees immediate, stable supply. Redundancy of supply, with refuelling contracts for tankers and priority in the face of a regional disaster affecting many customers at once, is part of the resilience design, not a logistical patch. And here the transition to HVO changes the equation for the better: besides cutting CO₂ by up to 90 % with no engine modification, it is far more stable in storage, holding up for as much as ten years with low microbial risk, which relieves precisely the degradation problem of fossil diesel. It is no coincidence that the major manufacturers already certify their sets for HVO and that several European operators have adopted it.

Paralleling: several sets as one

A serious facility rarely depends on a single generator. N+1 redundancy requires several sets working in parallel, and coordinating them is the job of the paralleling switchgear. Putting two generators in parallel is not trivial: before closing the breaker that joins them, three parameters must match, namely voltage, frequency and, above all, phase angle, because closing out of synchronism can cause severe mechanical damage. Digital controls (Cummins PowerCommand, Caterpillar EMCP, Woodward) decide the safe instant to close.

Once in parallel, the system shares the load proportionally among the sets (load sharing), both active and reactive power. In island operation, the case of a power cut, isochronous control keeps the frequency constant; when paralleling with the grid, droop control is used. And then the magic of redundancy: when a running set fails, the controls start, synchronise and bring in the standby one, rebalancing the load in seconds. The cost of all this capability is not small: a turnkey diesel system at large scale moves in market ranges of the order of 650,000 to more than 1,000,000 USD per MW, and the paralleling switchgear for a hyperscaler can add up to several million.

The choice between N+1 and more robust configurations reappears here with a nuance specific to generation. An N+1 generator plant, four sets for a load that needs three, for example, tolerates the failure of one set without losing the facility, and it is the usual configuration for its cost-reliability balance. But it pays to look at where the single point of failure is: if all the sets feed a single bus through one paralleling switchgear lineup, that board becomes a critical link that must be designed with its own redundancy or segmentation. Replicating generators without replicating the bus that joins them is solving half the problem. In fault-tolerant facilities, the logic of the two A/B paths we saw in the first article extends to generation as well: two independent plants, each feeding its own path, with no common point where a single failure brings everything down.

Maintenance and testing: the enemy is low load

It ends where the first article ended: reliability is earned by rehearsing. And with diesel generators there is a specific and counter-intuitive enemy: low load. Running a diesel for a long time below its optimal load causes wet stacking, the accumulation of unburnt fuel in the exhaust, which reduces efficiency and can damage the engine. That is why testing does not consist merely of starting the set, but of loading it for real.

The instrument is the load bank, which applies real load to the generator to burn off those residues and, along the way, reveal latent faults in cooling, injectors or voltage regulation that a no-load start would never show. NFPA 110 sets the rhythm: a monthly test of the diesel at a minimum of 30 % of its rated power, and an annual load bank protocol applying 50 % for half an hour followed by 75 % for an hour. Skipping these tests is building paper resilience: the generator appears to be there, but nobody has checked that it will sustain the load when it really matters.

There is a paradox in all this worth bearing in mind. The generator is probably the most tested and least used machine in a well-run datacenter: it spends 99.9 % of its life switched off, waiting for an event that one hopes never arrives, and yet it is rehearsed religiously every month. That discipline is not bureaucracy: it is the only way to be certain that, in the seconds when the grid blinks and the UPS drains its batteries, the handover happens. A generator that is never tested under load is not a backup, it is an expensive assumption. The reliability of the electrical layer, like that of any critical system, is built with redundancy and confirmed by rehearsal.

Key takeaways

The generator is the long-distance runner of the power chain: it starts in ten seconds, accepts load blocks without destabilising, and sustains the facility for hours or days with a rating, ideally Continuous or DCC, matched to its real use. Sizing it right means thinking about transients and harmonics, not just the nominal load; feeding it right requires fuel logistics with autonomy and polishing; and relying on it requires redundant paralleling and load bank testing. In the AI era, moreover, it has crossed a boundary: from emergency insurance to, increasingly, primary source of power, with all that implies for emissions and regulation.

But the generator does not act alone. Between the grid, the generator and the load there is a referee that decides at every instant where the power comes from: the switching. The third article of the series, which we will publish next, will go into ATS and STS units, and into how the load is transferred between sources without the compute noticing.

Sources

Powerlink, Understanding Data Center Backup Power — https://powerlinkenergy.us/blogs/understanding-data-center-backup-power/
Cleanview, Bypassing the Grid: Behind-the-Meter Data Centers — https://cleanview.co/reports/behind-the-meter-data-centers
Construction Owners, Oracle and OpenAI to Power New Texas Data Center Off the Grid — https://www.constructionowners.com/news/oracle-and-openai-to-power-new-texas-data-center-off-the-grid
Cummins, Understanding ISO 8528-1 Generator Set Ratings — https://www.cummins.com/sites/default/files/2018-08/201707%20PowerHour_Understanding%20ISO%208528%20GeneratorSetRatings.pdf
Cummins, Data Center Continuous (DCC) Ratings — https://mart.cummins.com/imagelibrary/data/assetfiles/0059642.pdf
Cummins, NFPA 110 Type 10 Requirements — https://www.cummins.com/sites/default/files/2019-03/PowerHour_NFPA110.pdf
Caterpillar, Transient Performance Specifications for Diesel Generator Sets — https://www.cat.com/en_US/by-industry/electric-power/Articles/White-papers/transient-performance-specifications-for-diesel-generator-sets.html
Kohler, ISO 8528-5 and Generator Transient Performance — https://techcomm.kohler.com/techcomm/pdf/ISO%208528-5%20and%20Generator%20Transient%20Performance_WP.pdf
Latitude Media, The data center boom is a diesel generator boom — https://www.latitudemedia.com/news/the-data-center-boom-is-a-diesel-generator-boom/
Bloom Energy, AI Data Centers — https://www.bloomenergy.com/blog/ai-data-centers/
Microsoft Source, Hydrogen fuel cells for datacenters — https://news.microsoft.com/source/features/sustainability/hydrogen-datacenters/
PowerGen Enterprises, Tier 4 Generator Requirements 2026 — https://powergenenterprises.com/tier-4-generator-requirements-compliance-guide-2026/
EPA, Specifics about Provisions Related to Stationary Engines — https://www.epa.gov/stationary-engines/fact-sheet-specifics-about-provisions-related
Earthsafe, The New Logistics of Data Center Fuel Supply — https://www.earthsafe.com/the-new-logistics-of-data-center-fuel-supply
Generator Source, Fuel Consumption Charts — https://generatorsource.com/tools-info/fuel-consumption-charts/
Caterpillar, Renewable Liquid Fuels (HVO) — https://www.cat.com/en_US/by-industry/electric-power/electric-power-industries/renewable-liquid-fuels.html
Generator Source, Paralleling Switchgear Explained — https://generatorsource.com/industries-served/data-centers/paralleling-switchgear-explained-how-we-power-hyperscale-data-center-growth/
MGI EPSS, Understanding NFPA 110 Testing Requirements — https://www.mgiepss.com/blog/understanding-nfpa-110-generator-testing-requirements
SecondWatt, Data Center Generators 2026: Capacity, Cost, Speed-to-Power — https://secondwatt.com/resources/data-center-generators-2026-capacity-cost-speed-to-power

Datacenter physical infrastructure (3/4): the switching

Mon, 22 Jun 2026 18:00:00 +0200

We already have the power chain mapped out and the generators that sustain it when the grid goes away. What is missing is the referee: the component that, at every instant, decides which source (the grid, the generator, path A or path B) feeds each load, and that executes the change from one to the other without the compute noticing. That is the switching, and it is where the battle of the milliseconds is fought.

Because the problem of transferring load between sources is not deciding whether to do it, which is easy, but doing it fast enough that the server power supplies never notice the gap. And there the choice of technology is everything.

The referee of the chain

When a source fails or degrades, someone has to move the load to the alternative source without human intervention. That someone is a transfer switch, and there are two very different families of them that live in different places in the hierarchy.

At the building level, where the choice is between grid and generator, the electromechanical switch reigns: the ATS (Automatic Transfer Switch), or, in large installations, the transfer logic built into the generator paralleling switchgear itself. It is a transfer between sources that are hundreds of metres apart and that may take seconds to become available (the generator has to start), so here switching speed is measured in tens or hundreds of milliseconds.

Close to the critical load, by contrast, downstream of the UPS, between paths A and B, the solid-state switch governs: the STS (Static Transfer Switch). Here both sources are alive and available, and the aim is to transfer so fast that the load does not flinch. The boundary between “utility capacity” and “IT capacity” sits, precisely, around the PDUs and the rack distribution panels, and that is where the STS does its work.

The window that decides everything

To understand why there are two technologies, you have to know the figure that rules: how long a server power supply can go without power before shutting down. The answer, according to the ITIC/CBEMA curves and the IEC 62040-3 standard, is that most enterprise power supplies tolerate a total loss of between 10 and 20 milliseconds before dropping out. That is the window. The whole design of the switching consists of transferring within it, or of having something to cover the gap when you cannot.

That tolerance is not a manufacturer’s whim: the ITIC and CBEMA curves, which define the “envelope” of voltage disturbances an IT device must withstand without failing, capture the hold-up time provided by the capacitors inside the power supply. For a few milliseconds, those capacitors keep delivering energy even though the input has disappeared; past that margin, the internal voltage collapses and the server reboots. All the engineering of switching revolves around not exhausting that small cushion.

A grid cycle at 50 Hz lasts 20 ms; at 60 Hz, about 16.7 ms. A switch that takes a whole cycle is already at the limit. That is why the time of an STS is measured in fractions of a cycle: a quarter of a cycle is

$$t = \frac{1}{4f}$$

that is, about 4.17 ms at 60 Hz. Inside the window, with margin. An electromechanical ATS, by contrast, takes much longer, and there lies the fundamental difference between the two.

The ATS: robust and slow

The ATS transfers by physically moving the connection. It uses electromechanical contactors or a motorised mechanism that opens one set of contacts while closing the other, like an electrically driven double-throw switch. That mechanism has a consequence: it is slow. A standard open-transition ATS switches in 60 to 200 milliseconds; a fast one, with a stored-energy mechanism, comes down to 50-100 ms. Eaton documents times of the order of 150 ms, some nine cycles. Any of those figures far exceeds the server’s tolerance window, and that is why an ATS, on its own, cannot feed critical IT load without a UPS behind it to cover the jump.

In exchange, the ATS is robust and cheap: it costs of the order of 60-70 % less than an equivalent STS, and its contactors with arc chutes withstand higher fault currents than the semiconductors of an STS of the same size. That is why it is the king of grid↔generator transfer at building level and of non-critical loads (lighting, HVAC, office IT), where a few hundred milliseconds of gap harm nobody. The reference standard for its construction and testing is UL 1008, which defines, among other things, the short-circuit withstand and closing ratings (WCR) according to the upstream protective device.

The ATS also supports several operating modes designed for different cases: standard open transition, delayed transition (with a programmed delay of half a second to three seconds, intended for motor loads, where reconnecting too fast and out of phase would damage the equipment) and closed transition, which we will see shortly. For an IT load, however, delayed transition is ruled out: that delay is an eternity for a server.

An important variant is the bypass-isolation ATS (maintenance bypass), which allows the switch to be isolated and maintained or tested while the load keeps being fed by an alternative route. In concurrently maintainable facilities, Tier III and IV, this capability is not optional: without it, maintaining the ATS would force an outage.

The STS: fast and electronic

The STS moves nothing. It uses thyristors (SCR), power semiconductors, as switching elements, and transfers the current from one source to another in a quarter of a cycle, 2 to 4 milliseconds, with no moving parts. Its conduction voltage drop is barely 1 or 2 volts and its efficiency is around 99.5-99.9 %. If the two sources are synchronised, it switches in make-before-break mode; if they are not, it does break-before-make in a brief interruption that, depending on the phase angle difference, stays around 4-10 ms, still inside the server’s window.

Its logic is that of a prudent referee: it continuously monitors the magnitude, frequency and phase angle of both sources, and transfers when the active one goes outside the configured thresholds (typically ±10 % of voltage or ±3 Hz), but only if the alternative is healthy. If both sources are out of tolerance, it stays where it is: transferring to a bad source is worse than holding on a degraded one. And after transferring, it waits a configurable delay (from 30 seconds to several minutes) before going back, so as not to enter a back-and-forth cycle.

The STS is listed under UL 1008S, which distinguishes it from the UL 1008 of the ATS, and it exists at several voltage levels: single-phase at 208 V for rack level, three-phase at 208 V for PDU level and three-phase at 480 V for the bus, as well as medium-voltage versions. Its flagship application is giving source redundancy to single-corded equipment: a server or a network device with only one power inlet can, thanks to an STS, receive power from both paths A and B without needing a second power supply. It is the way to put single-corded equipment into a 2N world.

Beyond transfer time, there are specifications an architect must look at when choosing an STS: the configurable retransfer delay (from 30 seconds to several minutes) to avoid cycling between sources; the fault capacity, which withstands between 10 and 50 kA for half a cycle or a cycle; and the integration with the management system via Modbus TCP, SNMP and dry contacts, so that switching is observable, something that, as we have repeated throughout this series, is not a luxury but part of reliability. The downside of the STS against the ATS is precisely its lower fault capacity: semiconductors tolerate less short-circuit current than contactors with arc chutes, which in sites with high fault current conditions the choice.

The high-end STS units of 2025-2026 have also refined details that matter in real installations: switching synchronised to the voltage zero crossing to achieve sub-cycle transfers (under 4 ms), true neutral switching in four-pole units, needed to transfer unbalanced loads safely, which are common in a datacenter, and redundant SCR controllers and drivers so that the STS itself does not introduce a single point of failure. The reliability of a static switch lies not only in the speed of its thyristors, but in the redundancy of the intelligence that decides when to fire them.

The table sums up the two worlds:

	ATS (electromechanical)	STS (solid state)
Mechanism	Contactors / motorised	Thyristors (SCR)
Transfer time	60-200 ms	2-4 ms (¼ cycle)
Inside the server window?	No (requires UPS)	Yes
Typical location	Grid↔generator, building level	A/B paths, PDU/rack level
Fault capacity	High	Lower (limited by SCR)
Relative cost	Baseline	~60-70 % more expensive
Standard	UL 1008	UL 1008S

Open or closed transition

There is a second dimension, orthogonal to the previous one, that defines how the change is made: whether the old connection is broken before or after the new one is made.

Open transition (break-before-make) disconnects from source one, pauses, and connects to source two. It is the natural and safe way when the two sources are not synchronised, because they must not touch each other, but it introduces the 60-200 ms gap of the ATS we already know. Hence it requires UPS backup for critical load.

Closed transition (make-before-break) does the opposite: it connects the second source before releasing the first, so that the load is never without power. The price is that, for an instant, the two sources are in parallel, and that is only admissible if they are synchronised and if the utility authorises it. The standards bound it: the United States electrical code (NEC 700.5(B)) limits that overlap to less than 100 milliseconds, and requires strict synchronisation conditions to close, with a voltage difference below 5 %, a frequency difference below 0.2 Hz and a phase angle difference of a few degrees. Closed transition is what allows you to test or return the load when the grid comes back without a single blink, a very valuable capability in mission-critical work. And its relative, soft loading or extended parallel transition, holds the parallel for a few seconds to ramp the load from one source to the other smoothly, ideal for the transfer tests that, as we insisted in the previous articles, have to be done without an outage.

The sequence of a power cut, step by step

It is worth stringing the pieces together in the order in which they act when, on an ordinary day, the grid goes away. At instant zero, the utility disappears. The IT load does not notice, because the UPS, which we will see in the last article, was already feeding it through its inverter (in double conversion) or transfers to its batteries in a few milliseconds: that is the ride-through, the bridge. Meanwhile, the control logic detects the grid failure and orders the generators to start. These take up to ten seconds to start and stabilise (NFPA 110, Type 10). When they are ready and synchronised, the paralleling switchgear, or the building ATS, transfers the load from the dead grid to the live generator; since that transfer happens upstream of the UPS, its gap of tens or hundreds of milliseconds is absorbed by the UPS itself without the load noticing. From there on, the generator sustains the facility.

When the grid comes back, the sequence reverses, and here closed transition shines: if the generator can synchronise with the grid, the system returns the load to the utility without a single blink, ramping it smoothly (soft loading) before shutting down the sets. Throughout this whole dance, the STS units of paths A and B have been doing their work in parallel, at load level, guaranteeing that every single-corded device always had a healthy source. Four components (UPS, generator, transfer switchgear and STS) choreographed so that the compute never knows that, out there, the lights went out.

The regulatory framework

Switching is heavily regulated because lives depend on it, not just bytes. The United States electrical code classifies systems by criticality: article 700 covers emergency systems (where an outage puts life at risk), 701 the legally required standby systems (which must come online in 60 seconds or less), 702 the optional ones, and 708 critical operations power systems. Each category imposes different requirements on the switch and on its listing. On that basis, UL 1008 governs transfer switches and UL 1008S the static ones, and closed transition is conditional on the interconnection agreement with the utility, because it means, even if for less than 100 ms, connecting your own generation in parallel with the grid. An architect who designs the switching without bearing this regulatory layer in mind is exposed to expensive surprises at the permitting stage.

Synchronising in order to close

Both closed transition and the generator paralleling we saw in the previous article depend on the same thing: synchronising before closing the breaker. The peaks and troughs of the waveforms of both sources must line up precisely, matching in voltage, frequency and, above all, phase angle, before they are joined, because closing out of synchronism causes currents and mechanical stresses capable of destroying the equipment. The guardian of that operation is the synchro-check relay (ANSI 25), a permissive device that only authorises the breaker to close when the three parameters are within limits. In large facilities, all this intelligence (synchronisation, load sharing, start sequence) lives in the paralleling switchgear governed by a master PLC, which is at once the brain of generation and that of large-scale switching.

The trend in these controls points to digitalisation and to grid-interactive functions: digital synchronisers with remote monitoring, predictive maintenance and grid-interactive capabilities (frequency regulation, reactive power control, black-start) that turn the datacenter’s generation and switching plant into an actor capable of talking to the electrical grid, not just of disconnecting from it. It is the same logic we saw emerging in the generators article with behind-the-meter generation: the boundary between the datacenter and the grid is blurring, and the switching plane is the point where that boundary is managed.

A/B architecture and the single point of failure

Let us put the pieces together in the dual-path architecture that runs through this whole series. In a 2N facility there are two complete, independent systems, A and B, each capable of supporting 100 % of the load. A dual-corded server connects to both and manages the transfer itself: here no STS is needed, because the redundancy is in the equipment itself. That is the ideal situation. The STS appears where that redundancy does not exist in the equipment: to give two sources to a single-corded load, or to transfer between two UPS outputs a set of loads that cannot have dual supply.

The mapping onto the Tier levels is illustrative. Tier II uses an ATS at the generator point and little else. Tier III has two independent paths but only one active at a time, and employs STS units at PDU or rack level to transfer between paths during maintenance. Tier IV keeps both paths active at once; dual cording eliminates the rack STS, but the STS remains present at transfer points between redundant UPS modules and for single-corded loads.

And here a design warning that connects with the philosophy of the previous articles: the switch itself can be a single point of failure. An STS that serves critical load and fails leaves that load with no alternative; a single UPS upstream of the STS is another weak point. That is why serious designs feed the STS from two independent UPS units, give the STS redundant controllers and drivers plus thermal monitoring, and, as a golden rule, specify a wrap-around maintenance bypass, with mechanical interlocking, for any STS or ATS that serves critical load. Replicating the sources and leaving the switch without redundancy is, once again, solving half the problem.

The maintenance bypass deserves emphasis because it is what makes possible the concurrent maintainability that defines Tier III and IV. Without it, testing or repairing a switch would mean leaving the load it serves unprotected; with it, the power is routed through an interlocked alternative path while the switch is isolated, tested and returned to service, all without a blink. There is a practical nuance in specifying it: if the two sources of the STS come from the same utility (a dual bus off the same feed), they are naturally synchronised and the STS can do make-before-break; if they are independent, you have to specify explicitly an STS with break-before-make capability, because it will have to transfer between sources that do not share phase. These details, invisible on a block diagram, are what separate a design that works from one that fails at the first real switching operation.

Towards solid state

Switching is living through its own revolution, pushed by AI. The corporate move says it all: Schneider Electric bought Vertiv’s transfer switch business (ASCO, a reference ATS manufacturer) for 1.25 billion USD, a sign of how strategic this link has become. The market for datacenter ATS and switchgear is projected to go from some 4 billion USD in 2025 to more than 10 billion in 2035.

The underlying technical novelty is the solid-state circuit breaker (SSCB), enabled by wide-bandgap semiconductors (silicon carbide and gallium nitride). It responds to faults in microseconds, isolating them so fast that the main bus voltage barely sags and the rest of the datacenter keeps computing without noticing. It is protection at the speed AI demands. The segment is attracting capital and innovation strongly: there are dedicated funding rounds to scale next-generation SSCBs for buildings and AI datacenters, and the major power semiconductor manufacturers flag it as one of the key pieces of the transition to 800 V distribution. It is a generational change: moving from protecting with mechanics, contacts that open, to protecting with electronics, which switch without arcing and without wear, finally aligning the speed of protection with that of the load.

Because that is the challenge that changes everything: AI loads swing brutally. An individual GPU oscillates 50-75 % of its power in milliseconds; at rack level, from 120 to 150 kW; and aggregated at gigawatt scale, system ramps exceed 1000 MW per second, stressing both switching capacity and grid stability. Added to the transition towards 800 V DC distribution, which NVIDIA proposes for its next architectures and which shifts the protection paradigm towards the DC world and SSCBs, the switching of an AI datacenter in 2026 looks less and less like that of five years ago. The referee has to be ever faster.

Another trend reshaping switching is integration with batteries and microgrids. As storage systems (BESS) enter the chain, to shave peaks, provide grid services or reduce generator hours, the switching plane stops being a simple “grid or generator” and starts orchestrating several simultaneous sources: grid, on-site generation, batteries and, where they exist, fuel cells. Grid-forming architectures with bidirectional flow can sustain the load without starting the generator for short events, leaving it only for extended outages or low battery state of charge. That multi-source orchestration, in milliseconds and with the intelligence to always pick the healthiest source, is the immediate future of the power chain’s referee. One reliability detail that is not minor: all this intelligence has to be protected from itself, giving the controllers redundancy so that the brain of the switching does not become, in its turn, a single point of failure.

Key takeaways

Switching is the referee that decides, instant by instant, where the power comes from. The master rule is the 10-20 ms window that server power supplies tolerate: the electromechanical ATS (60-200 ms) exceeds it and needs UPS backup, while the solid-state STS (2-4 ms) fits inside it and allows transfers without the load noticing. Closed transition avoids the gap at the price of synchronising and agreeing with the utility; the maintenance bypass guarantees that the switch itself is not a single point of failure; and the AI era, with its load swings and its jump to 800 V DC, is pushing this whole link towards solid state.

If you had to keep one idea only, it would be this: switching is not about choosing a source, but about doing it within a window of a few milliseconds, with the right technology at each point of the hierarchy, a robust ATS above and a fast STS below, without letting the referee itself become the link that fails. The rest are details, but details that are paid for dearly on the day of the first real switching operation.

One last component remains, the one that covers the gap the ATS cannot bridge and the one that first absorbs the stress of AI’s pulsing loads: the UPS. The fourth and final article closes the series with the sprinter of the power chain.

Sources

NFM Consulting, Static Transfer Switch vs ATS for Data Centers — https://nfmconsulting.com/knowledge/data-center-sts-vs-ats/
Eaton, Automatic Transfer Switch Fundamentals — https://www.eaton.com/us/en-us/products/low-voltage-power-distribution-control-systems/automatic-transfer-switches/automatic-transfer-switch-fundamentals.html
Eaton, Bypass-Isolation Contactor-Type Automatic Transfer Switches — https://www.eaton.com/us/en-us/catalog/low-voltage-power-distribution-controls-systems/bypass-isolation-contactor-type-automatic-transfer-switches.html
LayerZero, Static Transfer Switches — https://www.layerzero.com/products/static-transfer-switches/
ABB, Automatic Transfer Switching for Power Systems in Data Centers (UL 1008) — https://search.abb.com/library/Download.aspx?DocumentID=9AKK108468A1350&LanguageCode=en
Consulting-Specifying Engineer, Understanding transfer switch operation — https://www.csemag.com/understanding-transfer-switch-operation/
Power-Eng, Keep an Open Mind About Closed Transition — https://www.power-eng.com/om/keep-an-open-mind-about-closed-transition/
ASCO / Schneider, NEC Requirements for Emergency Power Transfer Switching — https://cdn2.hubspot.net/hubfs/4333126/Files/Technical%20Information/ASCO/White%20Papers/ATS/ASCO%20national_electrical_code_requirements_for_emergency_power_transfer_switching_147492_0.pdf
Generator Source, Paralleling Switchgear Explained — https://generatorsource.com/industries-served/data-centers/paralleling-switchgear-explained-how-we-power-hyperscale-data-center-growth/
Socomec, Data Center Redundancy: Definition, Reliability, Best Practices — https://www.socomec.us/en-us/solutions/business/data-centers/data-center-redundancy-definition-reliability-best-practices
Data Center Knowledge, Schneider Buys Vertiv’s Transfer Switch Business (ASCO) for 1.25B USD — https://www.datacenterknowledge.com/business/schneider-buys-vertiv-s-transfer-switch-business-for-1-25b
NVIDIA Technical Blog, 800 V HVDC Architecture for AI Factories — https://developer.nvidia.com/blog/nvidia-800-v-hvdc-architecture-will-power-the-next-generation-of-ai-factories/
arXiv, Operational Risks in Grid Integration of Large Data Center Loads — https://arxiv.org/pdf/2510.05437
Uptime Institute, AI to trigger radical overhaul of data center electrification — https://intelligence.uptimeinstitute.com/resource/ai-trigger-radical-overhaul-data-center-electrification

Datacenter physical infrastructure (4/4): the UPS and the ride-through

Mon, 22 Jun 2026 17:00:00 +0200

We close the series with the component we have been promising in every article: the UPS, the uninterruptible power supply. It is the sprinter of the power chain. While the generator is the long-distance runner that sustains the facility for hours and the switching is the referee that decides the source, the UPS covers the most critical instant of all: the seconds that pass between the grid flickering and the generator taking over. Without it, none of the earlier pieces is any use, because the outage would already have brought the load down before the generator managed to start.

The ten-second bridge

Let us recall the sequence we saw in the first article. When the grid goes away, the generator has to start, reach its rated speed and voltage and accept the load; NFPA 110, for Level 1 and Type 10 systems, gives it up to 10 seconds for all of that, starting cold, with no preheating. During those 10 seconds, and the few more that the switching takes to transfer, the load cannot be left without power. What sustains it is the UPS.

That is its primary function: the ride-through, the bridge. But it is not the only one. An online UPS also acts as a permanent filter: it isolates the load from grid anomalies (voltage dips, overvoltages, transients, harmonics, frequency variations) and delivers a clean, regenerated waveform at all times. It conditions as much as it backs up. That is why the UPS is the sprinter and the guardian of the quality of the power that reaches the servers.

It is worth underlining that the vast majority of UPS “interventions” are not total grid outages but micro-incidents: voltage dips of a few cycles caused by a switching operation on the grid, the start-up of a large neighbouring load or a distant fault that the utility clears in milliseconds. Those disturbances, too brief for the generator to start, are precisely the ones the UPS absorbs without anyone noticing, and they are hundreds of times more frequent than a real blackout. The UPS therefore works every day even though the grid has never “gone away”: its value lies not only in the dramatic outage, but in the constant trickle of imperfections it filters out.

The consequence of this narrow role is counter-intuitive: the UPS has very little autonomy on purpose. Since it only has to cover the start-up of the generator, its batteries are sized for 5 to 15 minutes, not for hours. More would be expensive and useless: the long-distance runner is the generator, and asking the sprinter to run a marathon wastes money and space on batteries that will never be needed.

The topologies: VFI, VI and VFD

IEC 62040-3 classifies static UPS units by how much they isolate the output from anomalies at the input, and the three classes are worth knowing:

VFI (Voltage and Frequency Independent), known as double-conversion online, is the mission-critical standard. Power always passes through two conversions, a rectifier from AC to DC and an inverter from DC to AC, so that the output is entirely independent of whatever happens at the input: the load lives on a waveform built anew. It is the Tier III and IV topology.

VI (Voltage Independent), or line-interactive, regulates voltage variations with a transformer or an AVR, but lets the input frequency through. It is more efficient and cheaper, suitable for less demanding loads or edge environments.

VFD (Voltage and Frequency Dependent), or standby/offline, regulates nothing in normal operation: the load goes straight to the grid and the UPS only steps in when there is a failure. It is the one in a home computer, not in a datacenter.

There is also delta conversion, an online topology that routes a good part of the power directly from input to output and processes only the difference, gaining efficiency in steady state while keeping the behaviour of an online unit when a fault occurs.

The table summarises when to use each one:

Topology	IEC 62040-3 class	Isolation	Typical efficiency	Use
Double-conversion online	VFI	Total (voltage and frequency)	97-98.1 %	Mission critical, Tier III/IV
Delta conversion	VFI	Total under fault	~98 %	Mission critical with an efficiency focus
Line-interactive	VI	Voltage only	High	Medium loads, edge
Standby / offline	VFD	None in normal operation	Very high	Not suitable for a datacenter

The great historic reproach against double conversion was its efficiency: processing all the energy twice costs losses. But that argument has aged. Equipment from 2025 reaches figures that were once unthinkable: the Riello Multi Power2, with silicon carbide semiconductors, declares up to 98.1 % in double conversion; the Vertiv Liebert APM2, up to 97.5 %; and Schneider’s modular range, around 99 % with its high-efficiency modes. The double-conversion penalty, put at around 92 % in older documentation, is history today.

Eco-mode: the last point of efficiency

Even so, there is one more step. In eco-mode (or high-efficiency mode), the UPS feeds the load directly through the bypass, without going through the rectifier or the inverter in normal operation, and reserves double conversion for when the grid fails. Efficiency then rises to 98-99 %. In a 2N installation, where there is twice the equipment, that couple of efficiency points translates into a very real saving in energy, and in PUE.

The classic problem with eco-mode is the instant of the fault: on detecting the anomaly, the UPS has to transfer from bypass to double conversion, and that change takes between 1 and 16 milliseconds, during which the load is momentarily exposed to the faulty grid. It is the same server tolerance window we saw in the article on switching, and that is why the risk is real though bounded: a typical server rides out those milliseconds, but an unlucky transient, an overcurrent when voltage is restored after a dip, can trip protections. That is why mission-critical operators were historically reluctant. The manufacturers’ answer has been advanced eco-mode, multi-mode, predictive, with commercial names such as Schneider’s eConversion or Vertiv’s Dynamic Online, which keeps the Class 1 performance classification (the same as double conversion) while removing a good part of the transfer risk. Operating-cost pressure is pushing its adoption, previously unthinkable on critical loads.

The flywheel: backup without batteries

Not all ride-through is done with batteries. The flywheel UPS stores energy kinetically: a rotor spins at high speed, from 3,000 to 10,000 revolutions per minute, and its inertia delivers power during the interruption. Its autonomy is 10 to 30 seconds, which seems little until we remember that those are, exactly, the seconds the generator takes to start. To act as a bridge to the generator, it is more than enough.

Its advantages speak for themselves: with no batteries, there is no battery maintenance, no periodic replacements, no thermal management of a chemical bank and none of the associated risk; it takes up far less space and tolerates a wide temperature range, with an efficiency that reaches 98 %. Its limit is the short autonomy, so it is often combined with batteries in hybrid systems: the flywheel absorbs the short, frequent events (the most numerous ones) and the battery provides the autonomy for the few outages that drag on. This combination, as we shall see, fits the pulsing profile of AI loads particularly well.

That suitability for AI is no accident. Most grid quality events are extremely brief, voltage dips of a few cycles, and require no autonomy, only an instant response; a flywheel (or, along the same lines, a supercapacitor) handles them without spending a single battery cycle. Since batteries suffer under frequent cycling, discharging the flywheel for the short events and reserving the battery for the long outages extends the life of the chemical bank and reduces its maintenance. The Uptime Institute itself points out that manufacturers are researching flywheels, supercapacitors and advanced chemistries precisely in order to better tolerate the frequent, short, high-power discharges and recharges that AI loads impose. In the AI era, the chain’s sprinter also has to be a sprinter that recovers fast.

Batteries: lead gives way to lithium

Where there are batteries, the great transition of 2025-2026 is the one from lead to lithium. VRLA lead-acid batteries, the standard for decades, are giving ground to lithium iron phosphate (LFP) at a notable speed: lithium already takes around 40 % of datacenter backup installations, and up to 55 % among the hyperscalers.

The reasons are compelling. LFP lithium tolerates a wider thermal range, withstands many more cycles, of the order of 3,000 against the 200-500 of lead, and that extends its service life to 12-15 years against the 3-6 of VRLA. It takes up less space, recharges faster and, despite its higher initial cost, offers a ten-year total cost of ownership around 39 % lower. LFP chemistry is preferred precisely for its thermal stability and its resistance to thermal runaway, and it always comes accompanied by a battery management system (BMS) watching over it. As an intermediate option, thin plate pure lead (TPPL) persists, which recharges faster than classic VRLA and reaches long lifetimes, attractive in certain cost profiles and at low temperatures.

There is one figure that connects batteries directly with our world: in Schneider’s tests with real hyperscaler profiles, above 110 % load lead batteries have difficulty with high-frequency changes, while lithium handles them without trouble. Put another way, AI’s oscillating profile does not merely favour lithium: it almost demands it.

The move to lithium is not free of considerations. The energy density that makes it attractive is also what obliges it to be treated with respect: although LFP chemistry is much more stable than the NMC of electric vehicles and resists thermal runaway better, a lithium battery bank of several MWh concentrates a great deal of energy in a small space, which has led to specific fire detection and suppression regulations and to careful design of the room. The BMS is not an accessory: it watches every cell, balances the charge and disconnects on an anomaly, and its reliability is part of the reliability of the UPS. Against this, the appeal of thin plate pure lead (TPPL) as an intermediate step is that it keeps the operational and safety familiarity of lead while cutting back its worst defects. The choice between lithium and lead is, ultimately, not only about cost per kWh: it is about total cost, about space, about safety and about how it fits with the load profile.

Sizing: little time, unity power factor

Sizing a UPS has two peculiarities. The first, already mentioned, is the short autonomy: since only the start-up of the generator (some 30 seconds) plus its stabilisation (a couple of minutes) has to be covered, a minimum of about 5 minutes is enough, and 10-15 are recommended for margin. Battery autonomy is calculated, in essence, as the stored energy divided by the power demanded:

$$t_{\text{runtime}} = \frac{E_{\text{battery}}}{P_{\text{load}}}$$

The second peculiarity is the power factor. As we saw in the first article, apparent and active power are related by the power factor, and older UPS units, specified at 0.8, “stranded” up to a fifth of their rated capacity. Modern datacenter UPS units operate at unity power factor: they deliver all their kVA as useful kW. It is a quiet but important improvement when it comes to calculating how much real load a unit supports. And UPS sizing does not end with itself: the generator, upstream, has to be capable of feeding the UPS load plus the recharging of its batteries, another piece of coordination that the first article already anticipated.

An example lands the calculation. To sustain a load of 500 kW for 12 minutes, of the order of 100 kWh of useful energy has to be stored in the batteries (500 kW times 0.2 hours), plus the margins for ageing, depth of discharge and inverter efficiency, which in practice push that figure up considerably. It seems a lot for “only” 12 minutes, but remember that the short autonomy is deliberate: if the generator has not started in that time, the problem is no longer the UPS but the generator, and that is why both are tested together. The temptation to “buy more batteries just in case” is usually a bad deal: it costs more, takes up space and does not solve the scenario that really matters, which is getting the long-distance runner started.

The modular architecture

The way a UPS is built has changed as much as its chemistry. The modular UPS, with hot-swap power blocks, has displaced the monolithic one because it allows capacity and redundancy to be added incrementally, paying as you grow, and a module to be serviced or replaced without shutting the system down (Schneider calls it Live Swap). Internally it offers N+1 redundancy: if one module fails, the others take on its load without the system going down.

At whole-system level, the configurations reproduce the redundancy nomenclature we saw in the first article, applied to the UPS: from capacity (no redundancy) up to 2N (two independent systems, each capable of 100 %, equivalent to Tier IV availability), by way of parallel redundant (N+1), isolated redundant or catcher (a standby UPS that picks up the load if another fails) and distributed redundant. In the dual-path architecture, each path A and B has its own UPS, and the STS units we saw in the previous article transfer between their outputs for the loads that need it. All the pieces of the series fit together here.

It is worth understanding the nuance of each configuration, because it defines cost and behaviour under a fault. Parallel redundant (N+1) shares the load among several UPS units on a common bus; if one goes down, the others absorb its share, but they all serve the same load, so that a fault on the common bus affects them all. Isolated redundant or catcher keeps a standby UPS waiting that “catches” the load of any main system that fails, an efficient solution when there are several systems to protect with a single backup. 2N is the most robust and the most expensive: two systems that share absolutely nothing, each sized at 100 %, so that not even a bus fault can affect both. The choice, like all redundancy, is a balance between the cost of duplicating and the cost of unavailability, the same calculation that runs through this entire series.

The market follows this evolution. The recent launches, the new-generation Eaton 93PM with lithium, Schneider’s Galaxy VL and VXL ranges, the Riello Multi Power2 with silicon carbide, share three traits: modularity, lithium and efficiencies above 97 % in double conversion. The datacenter UPS market moves, according to analysts, between USD 4 billion and USD 9 billion in 2025 with high-single-digit annual growth, pushed precisely by AI demand.

The AI challenge: loads that pulse

And so we reach the challenge that is rewriting UPS design, and that sums up why this series matters for an inference factory. AI loads are not constant: they pulse. In a training cluster, thousands of GPUs draw power almost in unison, following the steps of the computation, and the power oscillates every one or two seconds. The Uptime Institute’s analysis from late 2025 puts numbers on it: in the worst case, the difference between the minimum and the maximum consumption can exceed 100 % at system level, the load doubles in milliseconds, and with the most recent hardware the swings reach 150 %, with overshoots of more than 10 % above specification. In GB200 NVL72 racks, sudden rises from 60-70 kW to more than 150 kW have been observed, above even the 132 kW maximum specification.

The UPS is the first piece of equipment to take that stress, and if it is not designed for it a load drop can occur. Schneider tested its Galaxy range with real hyperscaler profiles at 40 °C and made it withstand 1 to 100 % with no drops, up to 10 minutes of overload at 125 % and one minute at 150 %. Uptime, for its part, recommends specific mitigations: mixing AI loads with more stable loads and sharing generators to damp the fluctuations; choosing UPS configurations with more capacitance and greater redundancy, even N+2, to absorb the peaks; and leaning on the accelerator’s own power-smoothing tools, such as the power smoothing NVIDIA builds into Blackwell. And manufacturers are researching supercapacitors, advanced chemistries and flywheels precisely because they tolerate the frequent, short, high-power discharges and recharges that characterise AI better than a conventional battery does.

It is worth pausing on why the UPS is the first one affected. When thousands of GPUs go from their minimum draw to their maximum in a millisecond, someone has to cover that sudden demand instantly, and that someone is the UPS, because neither the grid nor the generator reacts that fast. If the UPS is not designed with enough capacitance and margin, its output voltage sags under the step and can drag the load down with it. That is why the Uptime Institute recommends, for AI loads, configurations with more redundancy, even N+2 instead of N+1, not so much for fault tolerance as for the capacity to absorb the peaks without repeatedly overloading the modules. It is a change of mindset: UPS redundancy stops being sized only against failure and starts being sized against the dynamics of the load as well. And upstream, the generator, which reacts more slowly still, is the link most stressed by these oscillations, which leads to recommendations such as mixing AI loads with stable loads and sharing generation to damp them.

Density adds another turn of the screw: with racks scaling from 100 kW towards 600 kW and the megawatt under discussion, traditional low-voltage UPS units are reaching their ceiling, and medium-voltage UPS architectures and centralised UPS systems with grid-scale batteries (BESS) are emerging. That convergence between the UPS and grid storage also opens up a new role: the UPS as a resource that provides services to the electricity grid, peak shaving, frequency regulation, reserve, blurring the boundary between datacenter backup and the grid itself. A battery bank that spends 99.9 % of its time waiting for an outage that almost never comes is an expensive, immobile asset; turning it into a resource that shaves consumption peaks or sells reserve to the grid while it is not needed for backup changes its economic equation. Manufacturers such as Vertiv already offer utility-grade storage systems integrated with the datacenter for this dual use. The challenge is that the UPS must never compromise its primary mission, being available for the outage, in order to provide services to the grid; the design has to guarantee that the energy needed for the ride-through is always held in reserve. It is worth not losing sight, however, of the systemic risks this introduces: during a minor grid fault in Virginia in 2025, several datacenters disconnected at the same time and caused a 1.5 GW drop, a reminder that, at this scale, datacenter physical infrastructure is already critical grid infrastructure.

Takeaways, and the close of the series

The UPS is the sprinter that covers the ten-second bridge until the generator starts, and at the same time the guardian that delivers clean power at all times. It is sized with little autonomy on purpose, it is built increasingly modular and with lithium, and it faces a new and formidable challenge: AI’s pulsing loads, which turn it into the first line of defence against oscillations that double the load in milliseconds.

And as in each of the previous articles, the conclusion returns to operations: a UPS is chosen well on its topology and its chemistry, but it earns its keep when, the outage having arrived, it transfers without a flicker. That demands testing the batteries, discharging them for real, measuring their actual autonomy, not trusting the nominal age; rehearsing the transfer coordinated with the generator; and watching every cell with the BMS. A battery bank that nobody has discharged under load is, like the generator that nobody has tested, an unverified promise. The whole power chain, from the utility feed to the chip, is only as strong as its least-tested link.

With this we close the physical infrastructure series. We began with the complete power chain, from the utility feed to the rack; we went on to the generators that sustain the facility when the grid goes away; we looked at the switching that arbitrates between sources; and we finish with the UPS that covers the critical instant. Four links of one and the same chain whose conclusion is always the same: in an inference factory, the GPUs are the expensive, visible asset, but it is the electrical infrastructure, silent, without glamour, measured in megawatts and milliseconds, that decides whether that asset works or shuts down. Designing it well, sizing it with slack for AI’s transients and, above all, testing it relentlessly, is what separates a factory that meets its SLA from one that learns the hard way on the day of the first outage.

Sources

EarthSafe / Cummins, The 10-second start — NFPA 110 Type 10 — https://www.earthsafe.com/sites/default/files/pdf/esrf-cummins-life-safety-gen-10-second-start.pdf
Fuji Electric, Classification of UPS (IEC 62040-3) — https://www.india.fujielectric.com/resources/technical-guide/classification-of-ups
DCD, Exploring Static UPS systems — https://www.datacenterdynamics.com/en/analysis/exploring-static-ups-systems/
DCD, Riello UPS Modular Multi Power2 raises efficiency — https://www.datacenterdynamics.com/en/product-news/riello-upss-modular-multi-power2-raises-efficiency-to-unparalleled-heights/
Vertiv, Dynamic Online UPS — https://www.vertiv.com/en-anz/solutions/dynamic-power/dynamic-online-ups/
Schneider Electric, eConversion high-efficiency UPS mode — https://www.se.com/ww/en/work/products/product-launch/econversion-high-efficiency-ups-mode/
Consulting-Specifying Engineer, Evaluating UPS system efficiency — https://www.csemag.com/evaluating-ups-system-efficiency/
Industrial Monitor Direct, Flywheel UPS vs Battery — https://industrialmonitordirect.com/blogs/knowledgebase/resolving-industrial-power-outages-with-flywheel-ups
Introl, UPS & Battery Systems for AI Data Centers (2025) — https://introl.com/blog/ups-battery-systems-lithium-ion-ai-data-center-2025
Vertiv, EnergyCore Lithium-Ion Battery Cabinet — https://www.vertiv.com/en-us/products-catalog/critical-power/uninterruptible-power-supplies-ups/vertiv-energycore-lithium-ion-battery-cabinet/
EnerSys, Why Modern Data Centers Need a New Approach to UPS Batteries (TPPL) — https://www.enersys.com/en/blog-articles/why-modern-data-centers-need-a-new-approach-to-ups-batteries/
Schneider Electric, AI-tolerant UPSs: first line of defense — https://blog.se.com/datacenter/2025/05/20/ai-tolerant-ups-first-line-defense-data-center-resilience/
Schneider Electric, Comparing UPS System Design Configurations (WP) — https://download.schneider-electric.com/files?p_Doc_Ref=SPD_SADE-5TPL8X_EN
Uptime Institute, Electrical considerations with large AI compute (2025) — https://journal.uptimeinstitute.com/electrical-considerations-with-large-ai-compute/
Vertiv, EnergyCore Grid utility-grade BESS — https://www.vertiv.com/en-us/about/news-and-events/corporate-news/vertiv-launches-utility-grade-energy-storage-system-to-accelerate-data-center-interconnection-and-grid-resilience-in-north-america/
MarketsandMarkets, Data Center UPS Market — https://www.marketsandmarkets.com/PressReleases/data-center-ups.asp

Energy in TCO and in regulation: cost, PUE, CSRD and the new European reporting obligation for datacenters

Tue, 16 Jun 2026 10:00:00 +0200

Notation: amounts in N € or N USD; decimals with a point; thousands separated by a comma. The dollar symbol is not used (it is a formula delimiter). Generic example hardware (4×H100 SXM node, 3.6 kW IT). No real infrastructure.

TL;DR

A 4×H100 SXM node at Spanish industrial prices (0.116 €/kWh, PUE 1.54) spends ~5,475 € on energy per year: 6–8 % of total opex when staff and maintenance dominate, but up to 18–22 % of opex in clusters with a solar PPA and cheap colocation. Energy is not the main TCO lever; it is the only TCO variable that European regulation requires you to measure, publish and certify.

Since September 2024, every datacenter with IT demand ≥ 500 kW in the EU is obliged to report PUE, WUE, ERF and REF to the ReportENER database (Delegated Regulation EU 2024/1364). From August 2027 an automatic sustainability label will be generated from that data. Companies subject to CSRD have to publish Scope 1, 2 and 3 in their sustainability report, along with emissions intensity, which turns the gCO₂/token ratio into a corporate accounting figure, not just an engineering one.

Track context

This article is C8 of the energy pillar. The previous ones:

The full TCO model that this article extends is in:

For the context of technical security compliance:

Technical controls: ENS, ISO 42001 and the EU AI Act

Energy as a TCO component

The reference model

The full TCO model of the on-premise cluster is developed in A8. The reference node: 4×H100 SXM5 80 GB, IT draw ~3.6 kW, 8-node cluster, 3-year amortisation.

Annual opex per node in the base scenario (8-node cluster, Spain):

Opex item	€/year per node	% of total opex
Staff (0.5 FTE pro rata)	7,500	26.8 %
Maintenance / support / failures	9,000	32.1 %
Rack colocation (Spain, high density)	6,000	21.4 %
Energy (PUE 1.54; 0.116 €/kWh)	5,475	19.6 %
Total opex	~28,000	100 %

The annual energy cost formula

$$\text{annual energy cost} = P_{\text{IT}} \times \text{PUE} \times 8{,}760\,\text{h} \times p_{\text{kWh}}$$

where $P_{\text{IT}}$ is the power of the IT equipment (kW), PUE the infrastructure multiplier, and $p_{\text{kWh}}$ the price of electricity (€/kWh).

For the reference node:

$$\text{energy/year} = 3.6\,\text{kW} \times 1.54 \times 8{,}760\,\text{h} \times 0.116\,\frac{\text{EUR}}{\text{kWh}} \approx 5{,}475\,\text{EUR}$$

Sensitivity to energy price and to PUE

Variation in the price of energy has a moderate impact on total TCO when hardware capex dominates:

Energy price (€/kWh)	Energy/year (€)	% opex	€/GPU-hour (70 % util., base)	Var. vs base
0.034 (Spanish solar PPA Q3 2025)	1,604	6.8 %	3.00	−12 %
0.116 (Spanish industrial, base)	5,475	19.6 %	3.39	reference
0.160 (European industrial average)	7,550	25.4 %	3.54	+4 %
0.200 (high tariff, no PPA)	9,437	30.2 %	3.67	+8 %

Variation in PUE is also moderate in total TCO but relevant in the absolute cost of energy and in regulatory reporting:

PUE	Energy/year (€, 0.116 €/kWh)	% opex	€/GPU-hour (70 % util.)
1.15 (liquid cooling)	2,166	8.4 %	3.21
1.20 (modern datacenter)	2,259	8.7 %	3.23
1.48 (datacenters <5 years, Uptime 2025)	3,490	13.1 %	3.33
1.54 (global average, Uptime 2025)	5,475	19.6 %	3.39
1.80 (legacy colocation)	6,400	23.2 %	3.47

The difference between PUE 1.15 and PUE 1.80 in €/GPU-hour is only +8 %, because capex still dominates, but in absolute energy consumed the difference is ×2.96: 2,166 € against 6,400 € a year per node. That difference in absolute energy is exactly what the regulatory framework is going to measure, publish and, in the future, classify.

Energy as a fraction of total TCO

Annual TCO per node includes amortised capex + opex:

$$\text{annual TCO per node} = \frac{\text{capex}}{\text{years}} + \text{annual opex} = 55{,}300\,\text{EUR} + 28{,}000\,\text{EUR} = 83{,}300\,\text{EUR}$$

With these numbers, energy represents:

$$\frac{5{,}475}{83{,}300} \approx 6.6\,\%\ \text{of the all-in TCO}$$

In the low-opex scenario (solar PPA, own datacenter, no colocation):

$$\frac{1{,}604}{68{,}300} \approx 2.3\,\%\ \text{of the TCO}$$

In the high-opex scenario (open-market tariff, expensive colocation):

$$\frac{9{,}437}{130{,}300} \approx 7.2\,\%\ \text{of the TCO}$$

The weight of energy in all-in TCO ranges from 2 % to 7 %. The more capex dominates, the lower the weight of energy. Energy matters more in TCO the longer the amortisation and the lower the relative capex, that is, in scenarios of cheaper hardware or long amortisation periods.

European regulatory framework 2024-2026

Map of the rules

The European framework on datacenters and energy is built from four regulatory layers with different scopes of application, required metrics and effective dates:

Rule	Legal basis	Main obligation	Who it applies to	From when
EED 2023/1791, Art. 12	Directive (EU) 2023/1791	Annual reporting of energy and sustainability KPIs to an EU database	Datacenters ≥ 500 kW IT in the EU	15 Sep. 2024 (1st report)
Delegated Reg. 2024/1364	Art. 33(3) EED	Specifies KPIs, formats and reporting deadlines (Phase 1 of the assessment scheme)	Datacenters ≥ 500 kW IT in the EU	Published 17 May 2024; in force
EU rating scheme (Phase 2, 2026 draft)	Art. 33(3) EED	Automatic sustainability label generated by the ReportENER database	Datacenters that report under 2024/1364	Automatic label from 15 Aug. 2027
CSRD / ESRS E1	Directive 2022/2464/EU + Delegated Regulation (EU) 2023/2772	Sustainability report: Scope 1-2-3 emissions, emissions intensity, transition plan	Large EU companies (>500 employees, from FY 2024); expanding progressively	Financial years from 1 Jan. 2024 (first reports 2025); listed SMEs from 2026
PNIEC 2023-2030 (Spain)	National Integrated Energy and Climate Plan	Energy efficiency 44 %, renewables 81 % in electricity; datacenters included in the target; Royal Decree in progress (consultation Sep. 2025)	Spain	2025-2030

Energy Efficiency Directive: EED 2023/1791

Directive (EU) 2023/1791, published in the OJEU of 20 September 2023 (L 231), recasts the earlier Directive 2012/27/EU. Article 12 introduces for the first time a reporting obligation specific to datacenters at European scale.

Threshold of application: datacenters with an electrical power demand of installed IT systems ≥ 500 kW. Member States may lower this threshold (Germany lowered it to 300 kW; France to 100 kW).

Types of datacenter covered: (a) enterprise datacenters, operated by a company solely for its own IT needs; (b) colocation datacenters, where the customer installs and manages its own equipment; (c) co-hosting or multi-tenant datacenters, where the customer accesses shared infrastructure.

What must be reported: 18 key performance indicators (KPIs) that include installed IT demand, total energy consumption, water consumption, reused waste heat, renewable energy consumed, and the four sustainability indicators: PUE, WUE, ERF and REF.

Deadlines:

First report (2023 data): before 15 September 2024.
Second report (2024 data): before 15 May 2025.
Subsequent annual reports: before 15 May each year.

Delegated Regulation (EU) 2024/1364

Commission Delegated Regulation (EU) 2024/1364 of 14 March 2024 (OJEU L, 17 May 2024) constitutes Phase 1 of establishing the Union’s common assessment regime for datacenters. It specifies:

The performance indicators and the information that operators must report to the European ReportENER database.
The calculation formulas for the four sustainability KPIs (PUE, WUE, ERF, REF).
The reporting format and the classification categories by datacenter type and size.

The Commission has made the ReportENER database available as the single entry point for reporting, or via the equivalent national systems of each Member State.

Phase 2: rating scheme and sustainability label

In March 2026 the European Commission published the public consultation on the draft second Delegated Regulation on the datacenter sustainability rating regime, Phase 2 of the scheme. The consultation was open until 23 April 2026.

The draft establishes that the ReportENER database will automatically generate an electronic sustainability label for every datacenter that has reported under Regulation 2024/1364. The label will show:

PUE and WUE classes based on indices calculated under Annex I of the draft.
REF (renewable energy factor) and ERF (energy reuse factor).
Information on grid functions (demand response capability).

Date of first issue of labels: 15 August 2027, and every year thereafter.

The draft also introduces amendments to Regulation 2024/1364, among them: the possibility of voluntary reporting for datacenters <500 kW, the removal of data traffic indicators from the list of mandatory KPIs, and the relaxation of reporting obligations for colocation operators (reflecting the difficulty of obtaining tenant data for confidentiality reasons).

The Commission’s technical report of October 2025 anticipates that the minimum performance standards (MPS) will be finalised between late 2027 and early 2028, and that they will progressively become mandatory for new builds and significant refurbishments.

CSRD and ESRS E1: corporate emissions reporting

The Corporate Sustainability Reporting Directive (CSRD, Directive 2022/2464/EU) requires companies in scope to publish a sustainability report drawn up under the European Sustainability Reporting Standards (ESRS). The ESRS E1 standard covers climate change.

Temporal scope of application:

Large companies (>500 employees, already subject to NFRD): financial year 2024 data, report published in 2025.
Other large companies (>250 employees or >40 M € turnover or >20 M € balance sheet): financial year 2025 data, report in 2026.
Listed SMEs: financial year 2026 data, report in 2027 (with the possibility of an extension).

What ESRS E1 requires in relation to datacenters and energy:

E1-4: GHG emission reduction targets in absolute value and in intensity (e.g. gCO₂eq per unit of product or service).
E1-5: Energy consumption and mix (renewable vs non-renewable energy, energy intensity).
E1-6: Gross GHG emissions Scope 1, Scope 2 (location-based and market-based) and Scope 3, without netting off offsets; emissions intensity.
E1-7: Removals and carbon projects (relevant for offsets that do not replace reduction).

The amended version of ESRS E1 (EFRAG Exposure Draft of July 2025, submitted to the European Commission in November 2025) expands the disclosure requirements from 9 to 11 and is expected to apply from January 2027.

Relevance for datacenter operators: a company that runs an on-premise GPU cluster for its own AI workloads has those datacenters within the perimeter of its Scope 1 (if it has its own generators) and Scope 2 (purchased electricity). If it contracts cloud services or colocation, those datacenters fall into its Scope 3 (category 1 or 11 under the GHG Protocol).

Spain: PNIEC context and the Royal Decree in progress

Spain’s National Integrated Energy and Climate Plan (PNIEC) 2023-2030, updated in September 2024 (MITECO), sets targets of 81 % renewables in electricity and 44 % energy efficiency for 2030. Datacenters are included among the sectors under intensive efficiency monitoring.

In August-September 2025, MITECO opened the public consultation on the draft Royal Decree regulating the energy efficiency and sustainability of datacenters, transposing it from Art. 12 of EED 2023/1791. The Royal Decree introduces additional requirements for datacenters with total installed rated power above 1 MW: an obligation to use the waste heat generated to supply climate control services or other heat applications.

The deadline for incorporating Art. 12 of the EED expired on 11 October 2025. The Royal Decree is awaiting the opinion of the Council of State.

Regulatory metrics: definitions and formulas

PUE — Power Usage Effectiveness

Defined in ISO/IEC 30134-2:2018 (European equivalent EN 50600-4-2). It is the central metric of the regulatory framework:

$$\text{PUE} = \frac{E_{\text{DC}}}{E_{\text{IT}}}$$

where $E_{\text{DC}}$ is the total energy consumed by the datacenter (including cooling, electrical distribution, lighting and losses) and $E_{\text{IT}}$ is the energy consumed by the IT equipment (servers, storage, network).

PUE = 1.0: theoretical perfection (all the energy goes to compute).
PUE = 1.54: 2025 global average (Uptime Institute, 15th annual survey).
PUE = 1.10–1.15: hyperscaler range.
PUE ≥ 1.80: legacy, unoptimised facilities.

Regulation 2024/1364 requires annual PUE to be measured and reported. The Phase 2 draft introduces PUE classes (A to D or similar) for the sustainability label.

WUE — Water Usage Effectiveness

Defined in ISO/IEC 30134-9:2022:

$$\text{WUE} = \frac{W_{\text{in}}}{E_{\text{IT}}}$$

where $W_{\text{in}}$ is the datacenter’s total water consumption (m³), and the result is expressed in m³/MWh. ReportENER data (2024-2025) shows EU average values between 0.07 and 1.28 m³/MWh depending on Member State, with an EU average of ~0.58 m³/MWh.

WUE is particularly relevant in water-stressed regions, such as southern and inland Spain, where evaporative cooling consumes drinking water.

ERF — Energy Reuse Factor

Defined in ISO/IEC 30134-6:2021:

$$\text{ERF} = \frac{E_{\text{reuse}}}{E_{\text{DC}}}$$

where $E_{\text{reuse}}$ is the energy (waste heat) exported and reused outside the datacenter (e.g. district heating networks, industrial processes). ERF varies between 0 (no reuse) and 1 (all the energy reused, theoretically). The Phase 2 draft proposes minimum ERF thresholds for new facilities.

REF — Renewable Energy Factor

$$\text{REF} = \frac{E_{\text{res,tot}}}{E_{\text{DC}}}$$

where $E_{\text{res,tot}}$ is the total renewable energy consumed (own generation + PPAs + Guarantees of Origin), divided by the datacenter’s total consumption. The Commission’s proposal points to targets of 75–100 % REF by 2030.

CUE — Carbon Usage Effectiveness

Defined in ISO/IEC 30134-8:2022. Not included in the reporting obligation of Regulation 2024/1364, but identified as a future KPI in the Commission’s technical report (Oct. 2025):

$$\text{CUE} = \frac{G_{\text{DC}}}{E_{\text{IT}}}$$

where $G_{\text{DC}}$ is the datacenter’s total GHG emissions (gCO₂eq) and the result is expressed in gCO₂eq/kWh IT. It is equivalent to the product $\text{PUE} \times I_{\text{grid}}$ when only location-based Scope 2 emissions are considered.

Master table: rule × what it requires × metric/threshold × date

Rule	What it requires	Metrics / thresholds	Date of application	To whom
EED 2023/1791, Art. 12	Annual reporting of KPIs to an EU database (ReportENER)	PUE, WUE, ERF, REF + 14 additional KPIs (energy, water, waste heat, renewables)	15 Sep. 2024 (1st report, 2023 data); 15 May annually from 2025	Datacenters ≥ 500 kW IT in the EU
Delegated Reg. (EU) 2024/1364	Technical specification of KPIs and reporting format (Phase 1 of the scheme)	PUE = E DC/E IT; WUE = W in/E IT; ERF = E reuse/E DC; REF = E res/E DC	In force since 17 May 2024	Datacenters ≥ 500 kW IT in the EU
EU rating scheme, Phase 2 (2026 draft)	Automatic electronic sustainability label with PUE and WUE classes	PUE and WUE classes A-D (thresholds in Annex I of the draft; pending final adoption)	Automatic labels from 15 Aug. 2027	Datacenters that have reported under 2024/1364
Minimum performance standards (MPS, expected 2027-2028)	Mandatory minimum PUE, REF and ERF thresholds for new builds and refurbishments	PUE targets differentiated by climate and size; REF ≥ 75-100 % by 2030; minimum ERF for new facilities	Finalisation expected 2027-2028; progressively mandatory	New and refurbished datacenters
CSRD / ESRS E1	Sustainability report: GHG emissions Scope 1, 2 (location- and market-based) and 3; emissions intensity; transition plan	Scope 1+2+3 mandatory in tCO₂eq; intensity in gCO₂eq/functional unit	Data from 1 Jan. 2024 (>500 employees); from 1 Jan. 2025 (>250 employees)	Large EU companies subject to CSRD
PNIEC 2023-2030 / Spanish Royal Decree (in progress)	Energy efficiency and sustainability of datacenters; obligation to use waste heat	Renewables target 81 % by 2030; datacenters ≥ 1 MW: waste heat for climate control or other applications	Royal Decree pending approval (consultation Sep. 2025)	Datacenters in Spain

How Wh/token and gCO₂/token connect with the required reporting

The conversion chain

The technical inference efficiency figure, J/token or Wh/token, measured with tools such as DCGM, NVML or nvidia-smi, becomes a regulatory reporting figure through the following chain:

$$E_{\text{token}} = \frac{P_{\text{IT}}}{T} \quad \text{[Wh/tok]}$$ $$E_{\text{DC/token}} = E_{\text{token}} \times \text{PUE} \quad \text{[Wh/tok adjusted to the datacenter]}$$ $$\text{gCO}_{2}\text{/token} = E_{\text{DC/token}} \times I_{\text{grid}} \quad \text{[gCO}_2\text{eq/tok]}$$

where $T$ is throughput (tok/s), $P_{\text{IT}}$ the power of the IT node, and $I_{\text{grid}}$ the carbon intensity of the grid (gCO₂eq/kWh). The full methodology is in C6.

For the reference node (4×H100 SXM, $P_{\text{IT}} = 3.6\,\text{kW}$, indicative throughput 1,000 tok/s, PUE 1.54):

$$E_{\text{DC/token}} = \frac{3.6\,\text{kW} \times 1.54}{1{,}000\,\text{tok/s}} = 5.544\,\text{J/tok} = 1.54 \times 10^{-3}\,\text{Wh/tok}$$

From Wh/token to the regulatory KPI: aggregation

What regulatory reporting (Regulation 2024/1364) requires is not per-token ratios but annual totals:

$E_{\text{IT}}$: total energy of the IT equipment over the year (kWh).
$E_{\text{DC}}$: total datacenter energy over the year (kWh), including cooling and distribution.
$W_{\text{in}}$: total water consumption (m³).
$E_{\text{reuse}}$: energy (waste heat) exported and reused (kWh).
$E_{\text{res,tot}}$: renewable energy consumed (kWh).

The four sustainability KPIs are calculated from those totals. The link between the per-token ratio and regulatory reporting is:

$$E_{\text{IT,annual}} = \sum_{\text{inferences}} E_{\text{token}} \times N_{\text{tokens}} + E_{\text{idle}} \quad \text{[kWh/year]}$$

The $E_{\text{token}}$ figure is needed in order to attribute total IT consumption to inference workloads (vs idle, vs fine-tuning), which allows the energy intensity of inference to be calculated as a fraction of the datacenter’s total consumption, a figure relevant to ESRS E1 reporting of emissions intensity.

From Wh/token to gCO₂/token as a CSRD accounting figure

Under ESRS E1-6, companies subject to CSRD must report GHG emissions with intensity per functional unit (E1-4). For a company running an on-premise LLM, the natural functional unit is the token generated, the query answered or the user served (under the Green Software Foundation’s SCI specification, ISO/IEC 21031:2022).

gCO₂/token becomes a corporate accounting figure like this:

Technical measurement: J/token per node (DCGM/NVML).
PUE adjustment: Wh/token at datacenter level = J/token × PUE / 3,600.
Carbon intensity: gCO₂/token = Wh/token × $I_{\text{grid}}$ / 1,000.
Annual scaling: tCO₂eq/year = gCO₂/token × annual N tokens / 10⁶ / 10⁶.
ESRS E1-6 reporting: location-based Scope 2 emissions (with national annual average $I_{\text{grid}}$) and market-based (with $I_{\text{grid}}$ according to renewable energy contracts), in tCO₂eq/year + intensity in gCO₂eq/token.

The location-based / market-based distinction from the GHG Protocol Scope 2 Guidance is mandatory under ESRS E1-6: both methods must be reported.

Worked example: from J/token to tCO₂eq/year

Reference node (4×H100 SXM, PUE 1.54, throughput 1,000 tok/s, 70 % utilisation, Spanish average grid intensity 2024 ~108 gCO₂/kWh):

$$E_{\text{DC/token}} = 1.54 \times 10^{-3}\,\text{Wh/tok}$$ $$\text{gCO}_{2}\text{/tok} = 1.54 \times 10^{-3} \times 108 \approx 0.166\,\text{gCO}_{2}\text{eq/tok}$$

Tokens generated per year (1,000 tok/s × 70 % util. × 8,760 h × 3,600 s/h):

$$N_{\text{tokens/year}} = 1{,}000 \times 0.70 \times 8{,}760 \times 3{,}600 \approx 2.2 \times 10^{10}\,\text{tok}$$

Annual location-based Scope 2 emissions per node:

$$\text{tCO}_{2}\text{eq/year} = \frac{0.166 \times 2.2 \times 10^{10}}{10^{12}} \approx 3.65\,\text{tCO}_{2}\text{eq}$$

For an 8-node cluster: ~29.2 tCO₂eq/year. With a solar PPA (effective intensity ~20 gCO₂/kWh market-based): ~5.4 tCO₂eq/year. The difference between the two methods, location-based vs market-based, is ×5.4, and both have to appear in the CSRD report.

Scenario	Grid intensity (gCO₂/kWh)	gCO₂/tok	tCO₂eq/year (1 node, 70 % util.)	ESRS method
Spanish average 2024 (location-based)	108	0.166	3.65	Location-based mandatory
Spanish solar PPA (market-based)	~20	0.031	0.68	Market-based mandatory
French average 2025 (location-based)	19.6	0.030	0.66	Location-based mandatory
German average 2025 (location-based)	328	0.505	11.1	Location-based mandatory

The 500 kW threshold and on-premise AI datacenters

Which datacenters does it affect in practice?

A 4×H100 SXM5 node draws ~3.6 kW IT. The 500 kW IT threshold is equivalent to ~139 nodes of 4×H100 SXM5. For a smaller cluster, the regulatory threshold looks distant.

There are, however, three routes by which it applies to smaller on-premise AI infrastructure:

Route	Explanation
Shared datacenter	If the GPU nodes sit in a larger corporate datacenter that exceeds 500 kW IT in total (not only GPUs), the datacenter as a whole is subject to reporting, and it is the datacenter operator who reports. The end user receives the PUE of its facility.
Threshold lowered by a Member State	France lowered the threshold to 100 kW IT (~28 4×H100 nodes). Germany set it at 300 kW (~83 nodes). In those countries, medium-sized AI clusters are already subject to reporting.
CSRD independent of the datacenter threshold	The CSRD obligation applies to the company according to its size, not to the datacenter. A company subject to CSRD that operates even a single GPU node is obliged to include the Scope 2 emissions of that node in its sustainability report.

PUE as an operational figure and as a regulatory figure

PUE is simultaneously:

An operational figure for the TCO model (multiplier of energy cost and of gCO₂/token).
A regulatory KPI subject to mandatory reporting under Regulation 2024/1364.
A classification criterion in the future Phase 2 sustainability label.

The reference technical standard for measurement is ISO/IEC 30134-2:2018, which defines four levels of PUE measurement depending on where IT power is measured (from the rack to the incoming transformers). Regulation 2024/1364 specifies that PUE measurement for reporting to ReportENER must follow this standard.

For a typical on-premise facility, PUE has not historically been measured with the granularity the standard demands. The first compliance step is to instrument the datacenter with power meters at the points specified by ISO/IEC 30134-2.

Sensitivity of gCO₂/token to PUE and energy price

The following table shows the variation in gCO₂/token for the reference node (4×H100 SXM, 1,000 tok/s, Spain) as a function of PUE and of the energy scenario (which determines the effective carbon intensity):

PUE	Grid intensity (gCO₂/kWh)	gCO₂/1,000 tok	gCO₂/1M tok	Var. vs base
1.15 (liquid, Spain solar)	50	0.064	63.9	−55 %
1.15 (liquid, Spain average)	108	0.138	138.0	−2 %
1.54 (global average, Spain average)	108	0.166	166.3	reference
1.54 (global average, Spain night gas)	220	0.339	338.8	+104 %
1.80 (legacy, Spain average)	108	0.194	194.4	+17 %
1.80 (legacy, Spain night gas)	220	0.396	396.0	+138 %

Grid intensity has more impact than PUE on gCO₂/token: going from Spanish solar (50 gCO₂/kWh) to Spanish night-time gas (220 gCO₂/kWh) multiplies emissions by ×4.4, while going from PUE 1.15 to 1.80 multiplies them only by ×1.57. The combination of a dirty grid plus a high PUE is the worst: ×6.2 against the base case.

For location-based CSRD reporting, the intensity used is the national annual average (or regional if more granular data exists), not the hourly figure. For market-based reporting, the effective intensity depends on the energy contract (PPA, Guarantees of Origin). Carbon-aware temporal shifting (moving load to hours of lower intensity) reduces real emissions but does not directly change location-based reporting, which uses the annual grid average.

Connection between operator metrics and regulator obligations

The map of required data

Technical figure	How it is obtained	Use in TCO	Use in regulatory reporting
$P_{\text{IT}}$ (W)	DCGM / NVML / IPMI	Energy cost formula	$E_{\text{IT,annual}}$ for PUE, WUE, ERF, REF
Throughput (tok/s)	Engine metrics (vLLM, TGI)	€/token, Wh/token	Energy intensity of inference
PUE (measured)	Meters on PDUs and IT equipment (ISO/IEC 30134-2)	Multiplier in the TCO formula	Mandatory ReportENER KPI
Annual $I_{\text{grid}}$ (gCO₂/kWh)	REE / MITECO / national factors	gCO₂/token for carbon-aware	Emission factor for location-based Scope 2 ESRS E1-6
Hourly $I_{\text{grid}}$	ElectricityMaps / Carbon Aware SDK	Carbon-aware scheduling	Location-based Scope 2 with hourly granularity (best practice)
$W_{\text{in}}$ (m³)	Datacenter water meters	Cost of water (where applicable)	WUE for ReportENER
$E_{\text{reuse}}$ (kWh)	Heat export meters	Possible waste heat revenue	ERF for ReportENER
$E_{\text{res,tot}}$ (kWh)	Energy bills with a renewable attribute / GOs	Energy cost with a PPA	REF for ReportENER; market-based Scope 2 for ESRS E1-6
Hardware embodied carbon (gCO₂eq)	Supplier LCA (AWS methodology, where available)	—	Scope 3 cat. 1 for ESRS E1-6

The minimum measurement system for compliance

For an on-premise datacenter subject to the obligations of Regulation 2024/1364 and to CSRD, the minimum measurement system needed is:

A power meter at the datacenter entry point (total datacenter energy, $E_{\text{DC}}$).
Power meters on the IT racks (IT equipment energy, $E_{\text{IT}}$): in practice, metered PDUs or the sum of DCGM/NVML measurements.
A water meter for the datacenter cooling circuit ($W_{\text{in}}$).
A waste heat export meter if the datacenter has a recovery system ($E_{\text{reuse}}$).
A record of energy bills with a breakdown of renewable energy (PPAs, GOs) for calculating REF and market-based Scope 2.

Without this measurement system, reporting to ReportENER cannot be correct and CSRD reporting lacks the data base it needs.

Minimum Performance Standards: the 2027-2028 outlook

The European Commission’s technical report (Assessment, July 2025, published on the EU Op Portal) and EUDCA’s monitoring anticipate that the Minimum Performance Standards (MPS) for datacenters will be defined in the 2027-2028 period with the following expected characteristics (not finalised):

KPI	MPS proposed for new builds	MPS proposed for existing facilities	Horizon
PUE	≤ 1.3 (cold climate); ≤ 1.4 (temperate/warm climate)	Staged targets differentiated by age	2028+
REF	≥ 75 % renewables in 2025; ≥ 100 % by 2030	Progressive, following the CNDCP roadmap	2025-2030
ERF	Minimum threshold for new facilities with usable heat output	Feasibility assessment requirement	2028+
WUE	No general threshold proposed yet; focus on potable vs non-potable water	Monitoring and reporting	2027+

These values are indicative and come from the consultation process; the definitive values will appear in the second or third Delegated Regulation adopted under Art. 33(3) of the EED.

Sources

EUR-Lex · Directiva (UE) 2023/1791 del Parlamento Europeo y del Consejo, de 13 de septiembre de 2023, relativa a la eficiencia energética (EED refundida) — https://eur-lex.europa.eu/legal-content/ES/TXT/?uri=OJ:JOL_2023_231_R_0001
EUR-Lex · Reglamento Delegado (UE) 2024/1364 de la Comisión, de 14 de marzo de 2024, relativo a la primera fase del establecimiento de un régimen de evaluación común de la Unión para centros de datos — https://eur-lex.europa.eu/eli/reg_del/2024/1364/oj/eng
BOE / DOUE · Texto en español del Reglamento Delegado (UE) 2024/1364 — https://www.boe.es/buscar/doc.php?id=DOUE-L-2024-80715
Comisión Europea · Commission adopts EU-wide scheme rating sustainability of data centres (15 mar. 2024) — https://energy.ec.europa.eu/news/commission-adopts-eu-wide-scheme-rating-sustainability-data-centres-2024-03-15_en
Comisión Europea · Rating scheme for data centres in the EU: Commission launches call for feedback (26 mar. 2026) — https://energy.ec.europa.eu/news/rating-scheme-data-centres-eu-commission-launches-call-feedback-2026-03-27_en
DLA Piper · EU Commission proposes new data centre sustainability rating label (20 abr. 2026) — https://www.dlapiper.com/en-us/insights/blogs/environment-health-safety-and-product-compliance/2026/eu-commission-proposes-new-data-centre-sustainability-rating-label
RE24 Energy · What EU New Data Centre Rating Scheme Means for Operators (dic. 2025) — https://re24.energy/blog-what-eu-new-data-centre-rating-scheme-means/
Comisión Europea · Assessment of next steps to promote the energy performance and sustainability of data centres in the EU (2nd Technical Report, jul. 2025) — https://op.europa.eu/o/opportal-service/download-handler?identifier=83be4c3e-5c79-11f0-a9d0-01aa75ed71a1&format=PDF&language=en&productionSystem=cellar
Comisión Europea · Assessment of next steps — October 2025 follow-up — https://op.europa.eu/o/opportal-service/download-handler?identifier=29aa14ed-a4ba-11f0-a7c5-01aa75ed71a1&format=PDF&language=en&productionSystem=cellar
Comisión Europea · Minimum performance standards for EU data centres (preparatory study) — https://energy.ec.europa.eu/resources/preparatory-studies/minimum-performance-standards-eu-data-centres_en
MITECO · Centros de datos: normativa y reporte (España) — https://www.miteco.gob.es/en/energia/eficiencia/centros-de-datos/normativa.html
MITECO · PNIEC 2023-2030 (actualización sept. 2024) — https://www.miteco.gob.es/content/dam/miteco/es/energia/files-1/pniec-2023-2030/PNIEC_2024_240924.pdf
Ashurst · Draft Royal Decree regulating energy efficiency and sustainability of data centres in Spain (sept. 2025) — https://www.ashurst.com/es-es/insights/draft-royal-decree-regulating-energy-efficiency-and-sustainability-of-data-centres-in-spain/
EFRAG · Amended ESRS E1 Exposure Draft (jul. 2025) — https://www.efrag.org/sites/default/files/media/document/2025-07/Amended_ESRS_Exposure_Draft_July_2025_ESRS_E1_0.pdf
Normative · ESRS E1 Explained: CSRD Climate Disclosure (2026) — https://normative.io/insight/esrs-e1/
CSRD Pro · Reporting Cloud Services and Data Centres for CSRD — https://www.csrdpro.com/en/report-cloud-services-data-centres-csrd/
Uptime Institute · Global Data Center Survey 2025 (PUE medio global 1,54) — https://datacenter.uptimeinstitute.com/rs/711-RIA-145/images/2025.Annual.Survey.Report.pdf
SOCOMEC · Energy Efficiency Directive: Essential Guidelines for Data Center Compliance — https://emea.socomec.com/en/news/eu-energy-efficiency-directive-what-new-rules-mean-data-centres
ISO/IEC 30134-2:2018 · Power Usage Effectiveness (PUE) — https://www.iso.org/standard/63451.html
ISO/IEC 30134-6:2021 · Energy Reuse Factor (ERF) — https://www.iso.org/standard/75728.html
ISO/IEC 30134-8:2022 · Carbon Usage Effectiveness (CUE) — https://www.iso.org/standard/82562.html
ISO/IEC 30134-9:2022 · Water Usage Effectiveness (WUE) — https://www.iso.org/standard/82563.html
Green Software Foundation · Software Carbon Intensity (SCI) Specification (ISO/IEC 21031:2022) — https://sci.greensoftware.foundation/
White & Case · Data centres and energy consumption: evolving EU regulatory landscape and outlook for 2026 — https://www.whitecase.com/insight-alert/data-centres-and-energy-consumption-evolving-eu-regulatory-landscape-and-outlook-2026
Datacenter Forum · New reporting requirements for data center operators introduced by the Delegated Regulation (EU) 2024/1364 — https://datacenter-forum.ro/en/new-reporting-requirements-for-data-center-operators-introduced-by-the-delegated-regulation-eu-2024-1364/

Levers for spending fewer watts per token: a quantified catalogue for LLM inference

Tue, 16 Jun 2026 09:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

TL;DR

Seven independent levers act on the numerator or the denominator of the identity $\text{J/token} = \text{power (W)} / \text{throughput (tok/s)}$. Applied in order of cost-benefit: quantisation (FP16→FP8) cuts J/token by 30–50% without changing hardware; power capping to 70% of TDP reduces power by 30% with less than a 10% drop in throughput on memory-bound loads; DVFS in the decode phase saves a further 22–45%; continuous batching lowers J/token by 25–40% against unoptimised stacks; speculative decoding trims up to 29% off J/token at low batch; prefix caching eliminates the prefill recompute for repeated prompts; and scheduling to clean hours shifts the carbon footprint without changing physical consumption. Starting reference: a vLLM FP16 stack on an H100 SXM (700 W TDP) with Llama-3 70B produces ~0.39 J/token under realistic batch loads; an unoptimised stack (PyTorch baseline) starts 2–6× higher. The levers stack; the combined effect can reach a 73% reduction against the unoptimised baseline (arXiv 2504.17674).

The reference identity

$$\text{J/token} = \frac{\overline{P}\ [\text{W}]}{\text{throughput}\ [\text{tok/s}]}$$

where $\overline{P}$ is the average power over the measurement window (see the methodology in energy per token). The levers act on one of the two factors:

They reduce $\overline{P}$: power capping, DVFS, scheduling to low-intensity hours.
They raise throughput (the denominator): quantisation, continuous batching, speculative decoding, prefix caching, chunked prefill, model selection.
Both at once: quantisation (smaller weight footprint → a larger batch fits in VRAM → higher throughput at similar or lower power).

The expanded formula including infrastructure overhead:

$$\text{J/token}_{\text{effective}} = \frac{\overline{P}_{\text{GPU}} + \overline{P}_{\text{rest of node}}}{\text{throughput}_{\text{tok/s}}} \times \text{PUE}$$

For the lever calculations the work is done on $\overline{P}_{\text{GPU}}$ for clarity; the PUE factor (see energy per token) amplifies or attenuates the effect of each lever proportionally.

Master table: levers × impact × cost × tool

Lever	J/token reduction (%)	Effect on latency	Effect on quality	Main OSS tool	Source
Quantisation FP16→FP8	30–50%	−8–33% TPOT (improvement)	<1% degradation on benchmarks	vLLM, TRT-LLM, llm-compressor	arXiv 2504.17674, baseten.co
Quantisation FP16→INT4 (AWQ/GPTQ)	45–60%	latency improves (lower BW)	1–3% degradation depending on model	AutoAWQ, llm-compressor, vLLM	arXiv 2504.17674
Continuous vs static batching	25–40% vs PyTorch	+5× throughput (improvement)	no impact	vLLM, TGI, TRT-LLM	arXiv 2504.17674, TokenPowerBench
Power capping 70% TDP	20–30% power	<10% throughput (decode)	no impact	nvidia-smi, DCGM, Zeus	arXiv 2604.11391, NERSC docs
DVFS decode-phase	22–45% in decode	1–6% total latency	no impact	nvidia-smi, GreenLLM, DVFS-GPT	arXiv 2501.08219, HAL 05190051
Speculative decoding	up to 29% (batch≤16)	TTFT improves	no demonstrable impact	vLLM, TRT-LLM	arXiv 2602.09113
Prefix caching	50–90% on reusable prefill	TTFT −50–80% (repeated prompts)	no impact	vLLM, SGLang	vLLM docs
Chunked prefill	~10–15% on mixed loads	more stable TTFT	no impact	vLLM (SARATHI)	arXiv 2308.16369
SLM vs dense LLM selection	60–80% (7B vs 70B model)	low latency	degradation varies by task	Ollama, vLLM, llama.cpp	arXiv 2504.17674
Scheduling to clean hours	0% J/token, up to −70% CO₂	no impact on the load	no impact	Kueue, Volcano, cron	ACM SLIT, arXiv 2507.09942

Note: the percentages are against the effective J/token of the load, not against idle board power. The effects are always measured under conditions of equal output quality (same model, same serving stack, same prompt distribution).

Lever 1: quantisation

Mechanics

Quantisation reduces the numerical precision of the weights (and optionally the activations), which shortens the time spent reading weights from HBM per generated token. Decode is memory-bandwidth-bound: the bottleneck is reading the weights, not computing. Reducing bit width frees bandwidth for higher throughput or allows a larger batch in VRAM.

Format	Bits/weight	BW reduction vs FP32	J/token vs FP16	Tool
FP32	32	1× (reference)	+100% (worse)	—
BF16/FP16	16	2×	reference	vLLM native
FP8 (W8A8)	8	~4×	−30–50%	vLLM, TRT-LLM, llm-compressor
INT4 AWQ (W4A16)	4	~8×	−45–60%	AutoAWQ, llm-compressor
INT4 GPTQ (W4A16)	4	~8×	−40–55%	AutoGPTQ, vLLM

Concrete reference figures:

Llama-3 405B on H100, FP16→FP8: −30% J/token measured; TPOT improves by 33% (Baseten, baseten.co/blog/33-faster-llm-inference-with-fp8-quantization).
The vLLM+CUDA graphs stack vs the PyTorch baseline: 2–6× more efficient in J/token (arXiv 2504.17674).
Llama-65B on V100 FP16: 3–4 J/token. Llama-3 70B on H100 FP8: 0.39 J/token. Combined hardware+quantisation reduction: ~90% (MLCommons, arXiv 2504.17674).

Quality cost

FP8 quantisation with calibration produces <1% degradation on standard benchmarks (MMLU, HumanEval). INT4 AWQ can produce 1–3% degradation depending on the model and the task; INT4 GPTQ without fine calibration, up to 2–5%. Validate with LLM energy leaderboards and your own quality benchmarks.

OSS tooling

# vLLM with FP8 (load-time quantisation, H100/A100)
vllm serve meta-llama/Llama-3-70B-Instruct \
 --quantization fp8 \
 --dtype auto

# llm-compressor: offline quantisation to W4A16 (AWQ)
python -m llmcompressor.transformers.compression.helpers \
 --model meta-llama/Llama-3-70B-Instruct \
 --recipe w4a16_awq.yaml \
 --output-dir llama-70b-w4a16/

Lever 2: continuous batching

Mechanics

Static batching keeps the batch fixed: if one request finishes early, the slot stays empty until the end of the batch. Continuous batching inserts new requests as soon as a slot frees up, keeping the GPU always at the highest possible utilisation.

The energy effect: GPU power is almost constant (the GPU is always active); the throughput grows. The result is a larger denominator at similar power:

$$\text{J/token}_{\text{continuous batch}} = \frac{\overline{P}}{\text{throughput}_{\uparrow}}$$

Measured figures (arXiv 2504.17674, TokenPowerBench arXiv 2512.03024):

vLLM with PagedAttention vs unoptimised Transformers: −25–40% J/token on high batch loads.
Online continuous batching vs static offline: a further −5% of J/token (small but consistent).
Batch 32→256 tokens: −25% J/token on Llama-3 70B (kernel economies of scale).
Throughput: ×5 against static batching stacks at equivalent load.

The efficiency knee on the H100 sits at around 256–512 tokens of batch; beyond that, J/token flattens or rises slightly.

OSS tooling

vLLM, TGI (HuggingFace Text Generation Inference), TRT-LLM. The relevant flag in vLLM: --max-num-seqs (maximum concurrent sequences); tuning it to the available VRAM maximises the effective batch.

Lever 3: power capping

Mechanics

The power cap sets a ceiling on instantaneous power (W). The GPU respects it by cutting the clock (internal DVFS). The efficiency trick: LLM decode is memory-bandwidth-bound; the GPU is already not at usefully full clock, and lowering the TDP reduces power without degrading throughput proportionally.

The power cap vs throughput curve (H100 SXM)

Figures from (arXiv 2604.11391) and NERSC (docs.nersc.gov/jobs/power-capping):

Power cap (W)	% of TDP (700 W)	Relative throughput (memory-bound)	Efficiency (tok/W)
700 W	100%	100%	1.00×
560 W	80%	~95%	~1.19×
490 W	70%	~90–95%	~1.27× (sweet spot)
420 W	60%	~80–85%	~1.33× (latency rises)
350 W	50%	~75% (the H100 reaches maximum BW at 350 W memory-bound)	~1.50×
200 W	29%	~50%	frequency saturation

Key finding from arXiv 2604.11391: the H100 reaches its maximum effective memory bandwidth with only ~350 W on memory-bound loads (STriad benchmark), against a TDP of 700 W. Going from 700 W to 490 W (−30% power) produces less than a 10% drop in decode throughput: J/token improves by ~20–27%.

For compute-bound loads (heavy prefill), the sweet spot shifts to 400–500 W, where performance scales almost linearly with the cap; from 500 W to 700 W, the gain is only ~10% extra throughput for each additional 100 W.

J/token formula with a power cap

$$\text{J/token}(P_{\text{cap}}) = \frac{P_{\text{cap}} \times \eta(P_{\text{cap}})}{\text{throughput}_{\text{base}} \times f(P_{\text{cap}})}$$

where $\eta(P_{\text{cap}})$ is the fraction of power actually consumed under the cap (on memory-bound loads, the GPU no longer consumes the whole cap), and $f(P_{\text{cap}})$ is the resulting fraction of throughput. In the H100’s 490–560 W zone, $\eta \approx 0{.}9$ and $f \approx 0{.}92\text{–}0{.}95$: the numerator falls more than the denominator, so J/token improves.

Commands

# Set the power cap to 490 W on GPU 0 (H100 SXM, 700 W TDP)
sudo nvidia-smi -i 0 -pl 490

# Check the applied limit and the power draw in real time
nvidia-smi --query-gpu=power.limit,power.draw --format=csv -l 1

# With DCGM: read DCGM_FI_DEV_POWER_USAGE in Prometheus
# and DCGM_FI_DEV_POWER_LIMIT to confirm the cap is active

# Apply the cap to every GPU on the node (systemd/cron script)
for i in $(seq 0 3); do
 sudo nvidia-smi -i $i -pl 490
done

With DCGM in production, the cap can be set via dcgmi policy --set powercap:490 or through the NVIDIA GPU Operator (the powerLimit field in the DevicePlugin). Zeus (ml.energy/zeus) can sweep caps automatically and measure the resulting J/token per configuration.

Lever 4: DVFS (Dynamic Voltage and Frequency Scaling)

Mechanics

DVFS is finer-grained than a power cap: it allows the SM frequency to be set directly (nvidia-smi -lgc to lock the GPU clock) or left for the driver to adjust dynamically according to available power and load. In LLM inference, the asymmetry between phases is the key fact:

Prefill: compute-bound. Benefits from maximum clock. Sensitive to frequency reduction.
Decode: memory-bandwidth-bound. The SM clock barely matters; what rules is HBM bandwidth.

This asymmetry is what LLM-specific DVFS systems exploit:

System	Energy saving	Latency penalty	Hardware	Source
DVFS-GPT (HAL)	up to 32% at iso-latency	~1–2%	A100, RTX 4090	HAL 05190051
EcoInfer	up to 25.4% (average 21.5%)	<5%	A100	arXiv MDPI Electronics
DualScale (disaggregated prefill/decode)	22–45% with SLO	1–6%	H100	arXiv 2602.18755
GreenLLM (SLO-aware DVFS)	18–30%	within the SLO	—	arXiv 2508.16449

The paper arXiv 2501.08219 measures that reducing the GPU frequency from 2,842 MHz to 180 MHz achieves an average saving of 42% with only a 1–6% latency increase in the decode phase, because decode is memory-bound and the SM frequency is not the bottleneck.

Commands (manual DVFS)

# Lock the SM clock at 1,200 MHz (H100 boost: 1,980 MHz) for decode
sudo nvidia-smi -i 0 -lgc 1200,1200

# Restore the dynamic clock
sudo nvidia-smi -i 0 -rgc

# Persistent mode (needed for the lock to survive between jobs)
sudo nvidia-smi -pm 1

For automatic DVFS tuned to the SLO (target TPOT), systems such as GreenLLM or DVFS-GPT measure the time of each iteration and adjust the frequency in real time, without overrunning the latency budget.

Lever 5: model selection (SLM, MoE vs dense)

The effect of scale

J/token scales with model size, but sublinearly. For Llama-3 (arXiv 2504.17674):

Model	Parameters	Relative J/token	Factor vs 1B
Llama-3 1B	1B	1.0×	reference
Llama-3 8B	8B	~2.3×	×2.3 (not ×8)
Llama-3 70B	70B	~7.3×	×7.3 (not ×70)
Llama-3 405B	405B	~40×	×40 (not ×405)

J/token grows ~7.3× from 1B to 70B, while parameters grow ×70: hardware efficiency (caches, tensor parallelism, kernels) cushions the scaling. Even so, moving from 70B to 8B cuts J/token by ~68% if the quality is sufficient for the task.

MoE vs dense

MoE models activate only a subset of experts per token (typically 2 of 8–64), which reduces active compute. But there is an implementation trap:

OLMoE-1B-7B (MoE, 1B active) vs OLMo-1B (dense): the MoE consumes 54.24% more energy per token than the dense model of the same active size (arXiv 2504.17674), because the fused expert kernels are ~19–63% slower than an equivalent dense GEMM.
OLMoE-1B-7B vs OLMo-7B (dense): here the MoE is indeed more efficient (7B active → only 1B active, a ~7× reduction in active FLOPs per token).

The rule: MoE is efficient when the comparison is equal total parameters, fewer active (for example Mixtral 8×7B vs dense Llama-3 70B). It is not efficient when the comparison is equal active parameters per token.

Model selection table by use case

Use case	Recommended model	Efficiency rationale
Simple chat, entity extraction	SLM 7–8B FP8	−68% J/token vs 70B
RAG with long contexts	70B FP8 or MoE 8×7B	optimal quality/J at K>2K tokens
Complex code, reasoning	70B FP8 or 405B INT4	quality is non-negotiable
Classification, embeddings	dedicated model (e.g. E5, BGE)	orders of magnitude less J
Deferrable offline batch	large MoE (DeepSeek, Mixtral)	high quality, batch throughput

Lever 6: speculative decoding

Mechanics

Speculative decoding uses a small draft model to propose several tokens, which the target model verifies in parallel in a single forward pass. If the tokens are accepted, K tokens are generated per step instead of 1: potentially K× throughput.

The energy efficiency condition

The J/token benefit depends on batch size (arXiv 2602.09113):

Batch size	Effect on J/token	Reason
1–16 (low batch)	−up to 29%	the draft model fills the waiting time; verification is cheap
32–64 (medium batch)	~0% (neutral)	verification overhead cancels out the gain
128+ (high batch)	+25% (worse)	GPU saturated; the draft model adds overhead without saving

Conclusion: speculative decoding is a J/token lever only in a low-latency regime (batch≤16); in high-throughput serving it is counterproductive in J/token terms (although it reduces TPOT latency).

OSS tooling

vLLM supports speculative decoding with --speculative-model and --num-speculative-tokens. TRT-LLM does too. The draft model must come from the same family (for example Llama-3 8B as draft for Llama-3 70B).

Lever 7: prefix caching and chunked prefill

Prefix caching

The prefill KV-cache of a shared prompt (system prompt, few-shot examples, document context) is identical across requests. Storing and reusing it eliminates the prefill recompute for that fraction of the prompt.

Energy impact: prefill is compute-bound and draws power spikes. Removing the repeated prefill removes those spikes. In scenarios with a fixed 1,000-token system prompt and 200-token user prompts, the saving is ~1,000/(1,000+200) = ~83% of prefill energy. In total J/token, the effect depends on the length of the shared prompt against the generation length.

Enabling it in vLLM:

vllm serve meta-llama/Llama-3-70B-Instruct \
 --enable-prefix-caching \
 --max-num-seqs 256

Chunked prefill (SARATHI)

Chunked prefill splits the prefill into fixed-size chunks and processes each chunk together with decode tokens from other requests. The benefit: the prefill of a long prompt no longer blocks the decode of other requests (reducing TPOT spikes), which allows higher GPU utilisation over time.

In terms of total J/token, chunked prefill reduces the node’s peak power (fewer prefill spikes) and improves average utilisation, which translates into ~10–15% less J/token on mixed loads with long prompts (arXiv 2308.16369).

# Enable chunked prefill in vLLM (chunk size: 512 tokens)
vllm serve meta-llama/Llama-3-70B-Instruct \
 --enable-chunked-prefill \
 --max-num-batched-tokens 512

Lever 8: scheduling to clean hours

This lever does not reduce physical J/token, but it does reduce the gCO₂/token of deferrable load. The mechanics: moving training, document ingestion, batch re-ranking and fine-tuning to the hours of lowest grid carbon intensity.

Impact figures for temporal shifting (ACM SLIT, arXiv 2507.09942):

CO₂ reduction from temporal shifting: up to 34.7% for flexible loads.
With carbon-aware reinforcement learning (Eco-Orchestrator): up to a 70% offset in integrated deployments.
In Spain, the hourly variation in intensity is ~80–250 gCO₂/kWh: a factor of ~3× within the same day (see from watt to carbon).

The practical implementation uses Kueue or Volcano on Kubernetes, with a grid intensity metric (the ElectricityMaps API or esios) as the scheduling signal. Loads are marked with a carbon-deadline annotation (the maximum tolerable wait) and the scheduler places them in the minimum-intensity hours within that window.

# Kueue annotation for carbon-aware scheduling
metadata:
 annotations:
 kueue.x-k8s.io/carbon-aware: "true"
 kueue.x-k8s.io/carbon-deadline: "8h"

The power cap vs throughput curve: the sweet spot

Reading the chart: the power curve (solid line) falls faster than the throughput curve (dashed line) in the 490–560 W zone. Above 560 W, each additional 70 W gives only ~5% extra throughput; below 420 W, throughput starts falling faster than power. The efficiency sweet spot (minimum J/token) sits at 490–560 W for decode-bound loads.

For compute-bound loads (pure prefill), the sweet spot shifts to 500–600 W: prefill saturates SM compute and needs more power to sustain throughput.

How to measure the effect and validate quality

Measurement stack

The full stack for measuring the effect of each lever:

What to measure	Tool	Metric
Real-time GPU power	DCGM (`DCGM_FI_DEV_POWER_USAGE`)	W per GPU, 100ms
J/token in production	Kepler + vLLM tokens	`rate(kepler_joules) / rate(vllm_tokens)`
J/token on the bench	Zeus (`ZeusMonitor`)	J measured during the benchmark
Throughput	vLLM metrics (`vllm:generation_tokens_total`)	tok/s
TPOT latency	vLLM (`vllm:e2e_request_latency`)	ms/tok
Output quality	lm-evaluation-harness	MMLU, HumanEval, etc.

See the full stack in energy tooling.

Measurement protocol per lever

# Real-time J/token (Kepler + vLLM in Prometheus)
sum(rate(kepler_container_joules_total{container="vllm"}[5m]))
 /
sum(rate(vllm_generation_tokens_total[5m]))

# Average power per GPU (DCGM)
avg by (gpu) (DCGM_FI_DEV_POWER_USAGE)

# Check that the power cap is active
avg by (gpu) (DCGM_FI_DEV_POWER_LIMIT)

With Zeus, the configuration sweep is automatic:

from zeus.monitor import ZeusMonitor

monitor = ZeusMonitor(gpu_indices=[0, 1, 2, 3])

with monitor.begin_window("fp8_inference"):
 # run the benchmark with vLLM FP8
 run_benchmark(model="llama-3-70b-fp8", num_tokens=10_000)

result = monitor.end_window("fp8_inference")
print(f"J/token: {result.total_energy / 10_000:.4f}")

Quality validation

Validating that the lever does not degrade quality is done with lm-evaluation-harness (github.com/EleutherAI/lm-evaluation-harness):

# Compare MMLU between FP16 and FP8
lm_eval --model vllm \
 --model_args pretrained=meta-llama/Llama-3-70B-Instruct,dtype=float16 \
 --tasks mmlu --num_fewshot 5 --output_path ./results/fp16/

lm_eval --model vllm \
 --model_args pretrained=meta-llama/Llama-3-70B-Instruct,quantization=fp8 \
 --tasks mmlu --num_fewshot 5 --output_path ./results/fp8/

The MMLU score difference between FP16 and FP8 with calibration is typically <1%. Without calibration it can be 2–4%.

Pareto decision table (energy / latency / quality)

Lever	J/token	TPOT (latency)	Quality	Implementation cost	Reversibility
FP8 quantisation	↓↓↓	↓ (improves)	~ (−<1%)	Low (1 flag)	High
INT4 AWQ quantisation	↓↓↓↓	↓ (improves)	↓ (−1–3%)	Medium (recalibrate)	High
Continuous batching	↓↓↓	~ (neutral on TPOT)	~	Low (new engine)	High
Power cap 70% TDP	↓↓	↑ <10%	~	Very low (1 command)	Immediate
DVFS decode	↓↓↓	↑ <6%	~	Medium (script + tuning)	High
Speculative decoding	↓ (low batch)	↓ (improves TTFT)	~	Medium (draft model)	High
Prefix caching	↓↓ (prefill)	↓↓ (TTFT improves)	~	Low (1 flag)	High
Chunked prefill	↓	↑ stable TPOT	~	Low (1 flag)	High
SLM (7B vs 70B)	↓↓↓↓	↓↓ (improves a lot)	↓↓ (−variable)	High (quality re-eval)	Medium
MoE (vs dense, same active size)	↑ (+54%)	↑	~	—	—
Scheduling to clean hours	~ J/token, ↓↓ CO₂	~ (for deferrable)	~	Medium (scheduler)	High

Key: ↓↓↓↓ strong improvement · ↓↓↓ significant improvement · ↓↓ moderate improvement · ↓ slight improvement · ~ neutral · ↑ worsens · ↑↑ worsens significantly.

Stacking levers: the combined effect

The levers are mostly orthogonal and they stack. The maximum combined impact documented in (arXiv 2504.17674):

“Correct application of the relevant inference optimisations can reduce total energy consumption by up to 73% against unoptimised baselines.”

Recommended sequence by cost-benefit:

Optimised engine (vLLM, TRT-LLM) with continuous batching: −25–40% immediately.
FP8 quantisation (if the hardware supports it: H100, A100 with limited support): a further −30–50% on the remaining J/token.
Power cap at 70% TDP: −20–30% power with <10% of throughput. Low cost.
Prefix caching: turn it on if there are system prompts or repeated contexts. No cost.
Chunked prefill: if the load has long variable prompts.
DVFS in decode: a further −22–45% if the SLO allows it.
Speculative decoding: only if the batch is low (<16) and TTFT latency matters.
Model selection: move to an SLM if the quality is sufficient (evaluate with lm-evaluation-harness on your specific task).

Quantified example on 4×H100 SXM (2,800 W node TDP), Llama-3 70B, chat load:

Configuration	Power (W)	Throughput (tok/s)	J/token
PyTorch FP16, static batching	~2,100	~400	5.25
vLLM FP16, continuous batching	~2,200	~1,800	1.22
vLLM FP8, continuous batching	~2,100	~2,400	0.875
+ power cap 490 W/GPU (1,960 W node)	~1,960	~2,160	0.907
+ prefix caching (40% hit rate)	~1,960	~2,600	0.754
+ DVFS decode 1,200 MHz	~1,500	~2,400	0.625

The values are illustrative estimates based on the cited papers applied to a 4×H100 node. Real J/token depends on the prompt distribution, the effective batch and the temperature of the models. Measuring it with Zeus or DCGM+Kepler is mandatory.

Reference hardware: TDP and power cap ranges

GPU	TDP (W)	Supported cap range	Decode sweet spot
H100 SXM 80GB	700	200–700 W	~490–560 W
H100 PCIe 80GB	350	100–350 W	~245–280 W
A100 SXM 80GB	400	100–400 W	~280–320 W
A100 PCIe 80GB	300	100–300 W	~210–240 W
L40S 48GB	350	100–350 W	~245–280 W
RTX 5090 (reference, PCIe)	575	~100–575 W	~400–460 W

For the exact ranges: nvidia-smi -q -d POWER | grep -E "Min|Max|Current".

Executive summary by role

Role	First action	Second action	Key tool
Cluster operator	`nvidia-smi -pl [70% TDP]` on every node	Enable persistent mode	nvidia-smi, DCGM
Serving engineer	Migrate to vLLM FP8 with continuous batching	Enable prefix caching	vLLM, llm-compressor
Platform architect	Select the smallest model that meets the quality SLO	Design a carbon-based scheduling policy	lm-evaluation-harness, Kueue
MLOps team	Set up a power cap sweep with Zeus	Instrument J/token in Prometheus	Zeus, DCGM exporter

See the full measurement stack in energy tooling and the J/token methodology in energy per token. For the context of reference energy leaderboards and benchmarks, see LLM energy leaderboards. The connection with full TCO is in GPU utilisation as FinOps.

Sources

arXiv 2504.17674 · Energy Considerations of LLM Inference and Efficiency Optimizations (CMU/Hugging Face, 2025) — https://arxiv.org/pdf/2504.17674
arXiv 2604.11391 · Architectural Trade-offs: power-capping NVIDIA H100 and H200 (FAU Erlangen, 2026) — https://arxiv.org/html/2604.11391v1
arXiv 2512.03024 · TokenPowerBench: Benchmarking the Power Consumption of LLM Inference — https://arxiv.org/pdf/2512.03024
arXiv 2602.09113 · Benchmarking the Energy Savings with Speculative Decoding Strategies — https://arxiv.org/pdf/2602.09113
arXiv 2501.08219 · Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling — https://arxiv.org/abs/2501.08219
arXiv 2602.18755 · DualScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFS — https://arxiv.org/pdf/2602.18755
arXiv 2508.16449 · GreenLLM: SLO-Aware Dynamic Frequency Scaling for Energy-Efficient LLM Serving — https://arxiv.org/pdf/2508.16449
HAL 05190051 · DVFS-GPT: Dynamic Voltage and Frequency Scaling for Energy-Efficient LLMs — https://hal.science/hal-05190051
arXiv 2308.16369 · SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — https://arxiv.org/pdf/2308.16369
arXiv 2507.09942 · Green-LLM: Optimal Workload Allocation for Environmentally-Aware Distributed Inference — https://arxiv.org/pdf/2507.09942
arXiv 2505.06371 · The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization — https://arxiv.org/html/2505.06371v1
Zeus Project (ml.energy) — https://ml.energy/zeus/
Zeus · PyTorch blog: Deep Learning Energy Measurement and Optimization — https://pytorch.org/blog/zeus/
Baseten · 33% faster LLM inference with FP8 quantization — https://www.baseten.co/blog/33-faster-llm-inference-with-fp8-quantization/
NVIDIA · Managing Power Capping (DGX H100/H200 User Guide) — https://docs.nvidia.com/dgx/dgxh100-user-guide/power-capping.html
NERSC · GPU Power Capping documentation — https://docs.nersc.gov/jobs/power-capping/
MLCommons · MLPerf Inference v5.1 results — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
vLLM · Disaggregated Prefill documentation — https://docs.vllm.ai/en/latest/features/disagg_prefill/
Microsoft Research · Characterizing Power Management Opportunities for LLMs in the Cloud (ASPLOS 2024) — https://www.microsoft.com/en-us/research/wp-content/uploads/2024/03/GPU_Power_ASPLOS_24.pdf
EcoInfer (MDPI Electronics) · Optimizing Energy Efficiency with Latency Guarantees Through Iteration-Level GPU Frequency Control — https://doi.org/10.3390/electronics15102139

From watt to carbon, honestly: PUE, grid intensity and gCO₂ per token

Tue, 16 Jun 2026 08:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter). Generic example hardware (4×H100 SXM 80 GB); no real infrastructure. Emission figures in gCO₂eq/kWh (CO₂ equivalents).

TL;DR

A model that consumes 2 J/token at the board (4×H100 SXM, PUE 1.45) emits between $0.08$ and $1.05$ gCO₂ per 1,000 tokens depending on the region and the hour: $0.08$ in nuclear France in the small hours, $0.22$ in solar Spain at midday, $0.72$ in Spain on night-time gas, and $1.05$ in Germany on coal. The difference, a factor of ×13, is set by three variables external to the hardware: the datacenter PUE (Uptime Institute 2025: global average 1.54; hyperscale 1.10–1.15), the grid carbon intensity (which varies in real time, from 5 to 600+ gCO₂/kWh depending on zone and hour), and the type of emissions being counted (location-based, market-based, marginal). Moving batch work to hours of lower carbon intensity, temporal carbon-aware shifting, cuts the emissions of deferrable work by between 20 % and 65 % without touching the hardware. The SCI specification (Software Carbon Intensity, Green Software Foundation, ISO/IEC 21031:2022) formalises the metric as $\text{SCI} = (E \times I + M) / R$, where $M$ captures the embodied carbon of the hardware, the part that almost no LLM inference analysis includes today.

Track context

This article is C6 of the energy pillar. The previous ones:

The starting point is the J/token already measured (methodology in C2, tools in C3). This article converts that number into gCO₂/token.

PUE: the first multiplier

Definition

PUE (Power Usage Effectiveness) measures what fraction of the total energy consumed by a datacenter actually reaches the IT equipment:

$$\text{PUE} = \frac{\text{total datacenter energy (W)}}{\text{IT equipment energy (W)}}$$

A PUE of 1.0 would be perfect (all energy going to compute). A PUE of 2.0 means that for every watt the GPU consumes, another watt is consumed in cooling, electrical distribution, lighting and conversion losses. PUE turns board power, the figure DCGM or NVML measure, into the total datacenter power drawn from the grid.

Current values

The Uptime Institute Global Data Center Survey 2025 (15th annual edition) reports:

Segment	Typical PUE	Source
Global average	1.54 (6th consecutive year unchanged)	Uptime Institute 2025
Hyperscale (Google, Meta, Microsoft, Amazon)	1.10–1.15	Uptime Institute 2025
Colocation / enterprise	1.58–1.80	Uptime Institute 2025
Unoptimised on-premise	1.6–2.0	industry, various
Theoretical limit (cold air, favourable climate)	1.10–1.20	ASHRAE / industry

The stagnation of average PUE at 1.54 reflects the fact that legacy facilities, with inefficient cooling, keep operating alongside the new ones. For a typical on-premise platform with no specific investment in efficiency, assuming a PUE between 1.4 and 1.6 is reasonable. Facilities with direct liquid cooling can reach 1.10–1.20.

How it inflates J/token

If the reference node (4×H100 SXM) consumes 3,200 W at full load (GPU + CPU + DRAM + fans):

PUE	Total datacenter power	Inflation factor over board
1.10	3,520 W	+10 %
1.40	4,480 W	+40 %
1.54 (global average)	4,928 W	+54 %
1.80 (enterprise legacy)	5,760 W	+80 %

The energy per token on a leaderboard, measured at GPU or node level, does not include PUE. The impact: going from PUE 1.10 (hyperscale) to PUE 1.54 (global average) multiplies effective energy consumption by 1.40×. It is the same GPU, the same model, the same throughput; the difference comes from the facility.

Grid carbon intensity

What it measures and why it varies

The carbon intensity of the electricity grid expresses how many grams of CO₂eq are emitted per kilowatt-hour generated, as a function of the source mix (nuclear, wind, solar, gas, coal, hydro). It is the most heterogeneous variable in the calculation: it varies by country, by region within a country, and by hour of the day.

The reference OSS tools are:

Tool	Signal type	Coverage	Access
ElectricityMaps	Hourly average (production-based and flow-traced)	60+ countries in real time	API (limited free plan); OSS on GitHub
WattTime	Hourly marginal (MOER)	US + growing global coverage	API; free data since 2025 (REsurety-WattTime partnership)
Carbon Aware SDK (Green Software Foundation)	Wrapper over ElectricityMaps and WattTime	Same sources	OSS, Apache 2.0; github.com/Green-Software-Foundation/carbon-aware-sdk

ElectricityMaps (formerly electricityMap) publishes carbon intensity in gCO₂eq/kWh hour by hour via API and on its interactive map (app.electricitymaps.com). WattTime publishes the marginal operating emissions rate (MOER), useful for calculating the causal impact of adding or removing load. The GSF Carbon Aware SDK integrates both signals as interchangeable backends.

Country data (annual averages 2024–2025)

Country	Average intensity 2024 (gCO₂eq/kWh)	Approximate hourly range	Source
Spain	~108	30–250	Nowtricity / REE 2025
France	~21.7 (2024); ~19.6 (2025)	5–80	RTE Bilan Électrique 2025
Germany	~363 (2024); ~328 (2025)	100–600	Fraunhofer ISE 2025
EU average	~213	—	Ember European Electricity Review 2025

France has the lowest intensity in continental Europe thanks to its nuclear fleet (>90 % low-carbon generation in 2025, according to RTE). Spain closed 2024 with a historic record: 56.8 % renewable generation, with an average intensity of 108 gCO₂/kWh, among the lowest in the non-nuclear EU. Germany, in accelerated transition but still dependent on lignite and gas, has three times Spain’s average intensity.

Hourly variation: the figure that matters most for scheduling

Within a single country, the hourly variation in carbon intensity is enormous. Data from the ElectricityMaps 2025 grid review for Spain:

Time of day	Typical condition	Estimated intensity (gCO₂eq/kWh)
Solar midday (11:00–15:00, summer)	Solar covers >40 % of demand	30–80
Night-time trough (02:00–06:00, summer)	Zero solar; variable wind	80–180
Demand peak (19:00–21:00, winter)	Gas as the marginal plant	180–280
Cold windless small hours (winter)	Gas + combined cycle dominate	220–300

The price-carbon correlation in Spain is strong and negative (coefficient −0.70 in 2025 according to ElectricityMaps): cheap hours are also the cleanest, because both are driven by renewable penetration. The 477 hours with negative prices in 2025 were also the hours of lowest carbon intensity.

Spain vs France vs Germany: impact on gCO₂/token

To anchor the difference in operational terms, with the reference node (4×H100 SXM, indicative throughput 1,000 tok/s under load, PUE 1.45):

Energy per token at the datacenter:

$$E_{\text{DC}} = \frac{3{,}200\ \text{W} \times 1{.}45}{1{,}000\ \text{tok/s}} = 4{.}64\ \text{J/tok} = 1{.}29 \times 10^{-3}\ \text{Wh/tok}$$

Country / hour	Intensity (gCO₂/kWh)	gCO₂ per 1,000 tokens	gCO₂ per 1M tokens
France (2025 average)	19.6	0.025	25.3
Spain solar (summer midday)	~50	0.064	64.5
Spain 2024 average	~108	0.140	139.3
Spain night-time gas (winter)	~220	0.284	283.8
Germany 2025 average	~328	0.423	423.0
Germany coal peak	~550	0.709	709.0

The range is ×28 between the French average and the German peak. Between the two hourly extremes in Spain (solar vs winter night-time gas) the difference is ×4.4. The choice of country and of execution hour moves carbon per token more than any hardware or engine optimisation within that range.

Scope 2 and Scope 3: what counts and what does not

GHG Protocol scopes

The GHG Protocol defines three scopes for corporate emissions:

Scope	Definition	Datacenter example
Scope 1	Direct emissions from owned or controlled sources	Emergency diesel generators
Scope 2	Indirect emissions from purchased electricity consumption	Electricity consumption of the cluster
Scope 3	Other indirect emissions in the value chain	Hardware manufacturing (embodied carbon)

For an on-premise inference platform, Scope 2 is the dominant item in operation; upstream Scope 3 (manufacturing of servers and GPUs) is typically the second largest item over the hardware life cycle.

Scope 2: location-based vs market-based

The GHG Protocol Scope 2 Guidance (2015, under public revision 2025–2027) establishes two accounting methods, both mandatory in corporate reporting:

Method	What it measures	How it is obtained
Location-based	Actual intensity of the electricity grid where the facility sits	Emission factor of the national or regional grid (published by operators such as REE, RTE)
Market-based	Intensity according to the contractual instruments for energy purchase	Guarantees of Origin (GoO), PPAs or tariffs with a renewable attribute

The revision under public consultation (October 2025) proposes requiring, for the first time, hourly matching for the market-based method: buying 100 MWh of solar over a year is not enough if it is consumed in hours without sun. The aim is for market-based to reflect physical electricity and not just accounting certificates.

For an inference cluster, the location-based method with hourly intensity (not an annual average) is the most informative for carbon-aware operation: it shows in real time what emissions each token generates.

Marginal vs average emissions

This distinction, documented by ElectricityMaps and WattTime, is fundamental for evaluating the real impact of scheduling decisions:

Type	Definition	When to use
Average	The share of all grid emissions attributable to the consumer according to their quota	Location-based Scope 2 carbon reporting
Marginal (MOER)	Emissions of the generator that would respond to a marginal increase in load	Evaluating the causal impact of adding or moving load

Textbook example from ElectricityMaps: in a grid with 50 % wind (0 gCO₂/kWh) and 50 % gas (500 gCO₂/kWh), the average intensity is 250 gCO₂/kWh, but the marginal factor is ~500 gCO₂/kWh (gas is the plant that responds to the demand increase). Reducing load avoids 500 gCO₂/kWh, even though the Scope 2 report records only 250 gCO₂/kWh.

Marginal and average must never be mixed in the same accounting: they represent different attribution paradigms (the GHG Protocol and ElectricityMaps both say so explicitly).

Scope 3: the embodied carbon of the hardware

Embodied carbon is the CO₂eq footprint generated during manufacturing, transport and end of life of the hardware. For GPUs and high-performance servers, this item is significant:

Item	Indicative value	Source
Service life of a datacenter server	3–5 years	AWS / industry
Service life of the datacenter facility	15–20 years	industry
Amortisation methodology	Embodied carbon / years of service life	GHG Protocol / AWS methodology 2025
Embodied carbon of an AI server (indicative)	1,000–3,000 kgCO₂eq (cradle-to-gate)	AWS Embodied Carbon methodology

Embodied carbon is amortised over the service life of the hardware: a server with 1,500 kgCO₂eq of embodied carbon and a 4-year service life contributes 375 kgCO₂eq/year of Scope 3, regardless of how much it is used. A high utilisation rate reduces the embodied carbon per token (the same hardware produces more functional value).

None of the public LLM energy leaderboards (HF AI Energy Score, ML.ENERGY, MLPerf Power) include Scope 3 in their metrics. The SCI specification incorporates it explicitly.

The full conversion: from J/token to gCO₂/token

The equation

$$\text{gCO}_{2}\text{/token} = \frac{E_{\text{board}}\ [\text{Wh/tok}] \times \text{PUE} \times I_{\text{grid}}\ [\text{gCO}_{2}\text{/kWh}]}{1{,}000}$$

where:

$E_{\text{board}}$ is the energy per token measured at GPU/node level (without PUE)
$\text{PUE}$ multiplies by the datacenter overhead
$I_{\text{grid}}$ is the grid carbon intensity at the moment of execution
Dividing by 1,000 converts Wh to kWh

For the reference node (4×H100 SXM, throughput 1,000 tok/s, GPU+CPU+RAM consumption 3,200 W, PUE 1.45):

$$E_{\text{board}} = \frac{3{,}200\ \text{W}}{1{,}000\ \text{tok/s}} = 3{.}2\ \text{J/tok} = 8{.}9 \times 10^{-4}\ \text{kWh/tok}$$

Then, applying PUE 1.45:

$$E_{\text{DC}} = 8{.}9 \times 10^{-4} \times 1{.}45 = 1{.}29 \times 10^{-3}\ \text{kWh/tok}$$

And the emissions:

$$\text{gCO}_{2}/\text{tok} = 1{.}29 \times 10^{-3} \times I_{\text{grid}}$$

Full table by region and hour

Scenario	PUE	Grid intensity (gCO₂/kWh)	gCO₂/1,000 tok	gCO₂/1M tok
France 2025 average	1.45	19.6	0.025	25.3
Spain solar (summer midday)	1.45	50	0.064	64.5
Spain 2024 average	1.45	108	0.139	139.3
Spain night-time gas (winter)	1.45	220	0.284	283.8
Germany 2025 average	1.45	328	0.423	423.0
Germany coal peak	1.45	550	0.709	709.0
Spain solar + hyperscale PUE	1.12	50	0.050	50.0
Spain night-time gas + legacy PUE	1.80	220	0.354	354.0

The PUE factor matters more in high-intensity scenarios: going from PUE 1.12 to 1.80 at the German coal peak moves the result from ~617 to ~860 gCO₂/1M tok (+39 %). In solar Spain, the same PUE change moves it from 50 to 79 gCO₂/1M tok; the low grid intensity cushions the impact of poor PUE.

Carbon-aware shifting: temporal and spatial

Principle and tools

Carbon-aware computing means running flexible load (batch, training, fine-tuning, document ingestion) when and where the grid carbon intensity is lower. The reference OSS tool is the Carbon Aware SDK (Green Software Foundation, Apache 2.0): a wrapper that queries ElectricityMaps or WattTime and returns the current and forecast intensity for a zone and time window.

The SDK distinguishes two types of shifting:

Type	Description	Applicable to
Temporal	Delaying or advancing execution within the same datacenter to a lower-carbon hour	Batch, training, deferrable tasks with a loose deadline
Spatial	Moving the load to a geographic region with lower intensity at that moment	Multi-cloud or multi-datacenter; requires model/data replication

For an on-premise platform in a single datacenter, only temporal shifting is immediately applicable. Spatial shifting requires multi-site infrastructure.

Quantifying the temporal saving

Recent research (arXiv 2512.07799, 2512.08725; Microsoft Carbon-Aware Computing Whitepaper) quantifies the saving from temporal shifting of batch load:

Study / case	Emissions saving	Conditions
Batch load with 24 h shifting (simulation, several grids)	20–40 % emissions reduction	24 h window, grid with a renewable-fossil mix
ML training with shifting against grid forecast (Microsoft)	~30 %	Azure, shifting against forecast intensity
Greedy heuristics vs optimum (arXiv 2512.07799)	≥ 90 % of the optimum	Simple one-migration or greedy deferral
UBS + Microsoft, Azure Batch, 24 h window	Validated with the Carbon Aware SDK	Real temporal shifting in production

For Spain, where the price-carbon correlation is −0.70, shifting to solar hours cuts both electricity cost and emissions per token at the same time. In the 477 hours with negative prices recorded in 2025 (ElectricityMaps / REE), carbon intensity was also at a minimum. The coincidence between clean hour and cheap hour means that carbon-aware scheduling has no economic opportunity cost in Spain: optimising carbon and cost point to the same time window.

Quantification for the reference node

For a 1-hour batch task on the reference node (4×H100 SXM, 4,640 W total with PUE 1.45), generating ~3,600 million tokens per hour:

Scenario	Intensity (gCO₂/kWh)	Emissions per batch hour	Saving vs non-deferred execution
No shifting (Spain average)	108	501 gCO₂	—
Shifted to a solar hour (50 gCO₂/kWh)	50	232 gCO₂	−54 %
Shifted to a night-time gas hour (220 gCO₂/kWh)	220	1,021 gCO₂	+104 % (worst moment)

The range between the worst and the best moment to run that batch is ×4.4 in emissions. Shifting to the cleanest hour available in a 24-hour window in Spain typically avoids between 40 % and 60 % of the emissions of deferrable work.

The SCI standard: Software Carbon Intensity

Specification

Software Carbon Intensity (SCI) is a Green Software Foundation specification, published as standard ISO/IEC 21031:2022 and maintained on GitHub (github.com/Green-Software-Foundation/sci). It defines a rate of carbon emissions per functional unit of software:

$$\text{SCI} = \frac{(E \times I) + M}{R}$$

where:

Symbol	Meaning	Units
$E$	Energy consumed by the software system (servers, network, user devices)	kWh
$I$	Carbon intensity of the energy consumed (location-based or another declared basis)	gCO₂eq/kWh
$M$	Embodied carbon of the hardware, amortised by time of use and utilisation percentage	gCO₂eq
$R$	Functional unit (the denominator that normalises the rate)	per API call, per token, per user, etc.

For an LLM inference platform, $R$ is naturally the output token or the query. SCI in gCO₂eq/token therefore includes the three components that most current analyses omit: PUE implicit in $E$, hourly intensity in $I$, and embodied carbon in $M$.

Key difference from the simple calculation

The calculation $E \times I$ (energy × grid intensity) is operational Scope 2. SCI adds $M$, the Scope 3 of the hardware, which has a different structure:

$$M = \frac{\text{total embodied carbon (gCO2eq)}}{\text{service life (hours)}} \times \frac{\text{hours of use}}{1} \times \frac{\text{allocated resources}}{\text{total server resources}}$$

For an AI server with 1,500 kgCO₂eq of embodied carbon, a 4-year service life (~35,040 hours) and 80 % utilisation:

$$M_{\text{hour}} = \frac{1{,}500{,}000\ \text{gCO2eq}}{35{,}040\ \text{h}} \times 0{.}80 \approx 34\ \text{gCO2eq/h}$$

With 3,600 million tokens per hour:

$$M_{\text{per token}} \approx \frac{34}{3{.}6 \times 10^{9}} \approx 9{.}4 \times 10^{-9}\ \text{gCO2eq/tok}$$

At this scale, embodied carbon per token is negligible against operational Scope 2 during operation. However, $M$ becomes dominant in two cases: underutilised hardware (low utilisation amplifies the cost per token) and hardware with a very short life cycle (chips replaced every 2 years instead of 4).

SCI as the reference metric for the track

The SCI specification and its AI extension (github.com/Green-Software-Foundation/sci-ai) are being adopted as the reference metric for reporting the carbon footprint of AI systems in companies subject to CSRD. It provides a common denominator, gCO₂eq per functional unit, that allows architectures, regions and scheduling strategies to be compared on the same basis.

Limits of the calculation, and honesty

No gCO₂/token calculation captures the full reality. The limits of the model set out in this article:

Limit	Description
Annual average vs hourly intensity	Using the country’s annual average as the emission factor gives a stable but fictitious result. Intensity varies ×4–10× within a day. For location-based Scope 2 reporting with hourly data, the difference can be 30–50 % relative to the average.
Static PUE	PUE varies with outdoor temperature (cooling works harder in summer) and with IT load. The declared PUE is an average; on hot days it can be 10–20 % worse.
Constant throughput	The calculation uses throughput at full load. In production, with variable demand, consumption does not scale linearly with throughput; efficiency per token gets worse at low load.
Partial Scope 3	Embodied carbon here covers only the IT hardware. It does not include the construction of the datacenter building, the manufacture of cooling equipment, or the energy supply chain.
National emission factors	The per-country values are annual averages. France has low intraday variation (nuclear dominates); Spain and Germany have high variation. The national factor is useful for order-of-magnitude comparisons, not for operational scheduling.
Embodied carbon of GPUs	Lifecycle assessment data for AI chips such as the H100 are not public. Estimates rely on die-area models, which carry ±30–50 % uncertainty.
Spatial shifting omitted	This article covers temporal shifting only. Spatial shifting (moving load between regions) can add another 20–40 % of reduction but requires multi-site infrastructure.
Cooling water and other impacts	gCO₂eq does not capture cooling water consumption (WUE, Water Usage Effectiveness), another environmental impact indicator relevant to arid zones such as southern Spain.

Sources

Uptime Institute · Global Data Center Survey 2025 (PUE medio global 1,54, 15.ª edición) — https://uptimeinstitute.com/resources/research-and-reports/uptime-institute-global-data-center-survey-results-2025
Uptime Institute Blog · Global PUEs — are they going anywhere? — https://journal.uptimeinstitute.com/global-pues-are-they-going-anywhere/
Uptime Institute Blog · Large data centers are mostly more efficient — https://journal.uptimeinstitute.com/large-data-centers-are-mostly-more-efficient-analysis-confirms/
ElectricityMaps · Electricity Grid Review 2025: Spain (precio, intensidad, correlación precio-carbono −0,70) — https://www.electricitymaps.com/grid-in-review-2025/spain
ElectricityMaps · Marginal emissions — what they are, and when to use them (actualizado enero 2025) — https://www.electricitymaps.com/resources/publications/marginal-emissions-introduction
ElectricityMaps · Interactive map and API — https://app.electricitymaps.com
WattTime · Average vs marginal emissions — https://watttime.org/data-science/data-signals/average-vs-marginal/
WattTime · REsurety-WattTime free global marginal emissions platform (marzo 2025) — https://watttime.org/news-and-insights/resurety-and-watttime-announce-release-of-free-electricity-marginal-carbon-platform/
Red Eléctrica de España (REE) · Renewable energies generated 56 % of Spain’s electricity mix in 2024 — https://www.ree.es/en/press-office/press-release/news/press-release/2025/01/renewable-energies-generated-56-per-cent-spains-electricity-mix-2024
Nowtricity · CO₂ emissions per kWh in Spain (media 2024: 108 gCO₂eq/kWh) — https://www.nowtricity.com/country/spain/
RTE · Bilan électrique 2025 — GHG Emissions (Francia 2025: 19,6 gCO₂eq/kWh) — https://analysesetdonnees.rte-france.com/en/annual-review-2025/ghg-emissions
RTE · Bilan électrique 2024 key findings (Francia 2024: 21,7 gCO₂eq/kWh) — https://assets.rte-france.com/analyse-et-donnees/2025-11/RTE%20-%20Annual%20electricity%20review%202024%20-%20Key%20findings.pdf
Fraunhofer ISE · German Public Electricity Generation in 2025 (Alemania 2025: ~328 gCO₂/kWh) — https://www.ise.fraunhofer.de/en/press-media/press-releases/2026/german-public-electricity-generation-in-2025-wind-and-solar-power-take-the-lead.html
Ember · European Electricity Review 2025 (intensidad UE media: 213 gCO₂/kWh) — https://ember-energy.org/latest-insights/european-electricity-review-2025/five-years-of-progress/
GHG Protocol · Scope 2 Standard Advances (consulta pública 2025, hourly matching, market-based revisions) — https://ghgprotocol.org/blog/scope-2-standard-advances-isb-approves-consultation-market-and-location-based-revisions
GHG Protocol · Public Consultation Scope 2 (octubre 2025) — https://ghgprotocol.org/sites/default/files/2025-10/GHG-Protocol-Scope2-Public-Consultation.pdf
Green Software Foundation · Software Carbon Intensity (SCI) Specification — https://sci.greensoftware.foundation/
Green Software Foundation · SCI GitHub (ISO/IEC 21031:2022) — https://github.com/Green-Software-Foundation/sci
Green Software Foundation · SCI for AI — https://github.com/Green-Software-Foundation/sci-ai
Green Software Foundation · Carbon Aware SDK (Apache 2.0) — https://github.com/Green-Software-Foundation/carbon-aware-sdk
Green Software Foundation · Carbon-Aware Computing Whitepaper — UBS/Microsoft Azure Batch case — https://greensoftware.foundation/articles/carbon-aware-computing-whitepaper-how-ubs-succeeded-in-measuring-and-reducing-car/
arXiv 2512.07799 · Quantifying the Carbon Reduction of DAG Workloads: A Job Shop Scheduling Perspective — https://arxiv.org/abs/2512.07799
arXiv 2512.08725 · Spatio-Temporal Shifting to Reduce Carbon, Water, and Land-Use Footprints of Cloud Workloads — https://arxiv.org/abs/2512.08725
AWS · How AWS estimates embodied emissions of IT hardware (Scope 3 methodology, 2025) — https://aws.amazon.com/blogs/infrastructure-sustainability/how-aws-estimates-embodied-emissions-of-it-hardware-the-science-and-technology-behind-the-latest-customer-carbon-footprint-methodology/
CodeCarbon · Track and reduce CO₂ emissions from your local computing — https://codecarbon.io/

LLM energy leaderboards: comparing models by Wh/token and choosing on efficiency

Tue, 16 Jun 2026 04:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter). Generic example hardware; no real infrastructure.

TL;DR

There are three OSS leaderboards with public data and documented methodology for comparing the energy efficiency of LLMs in inference: Hugging Face AI Energy Score (166 models, Wh/query on H100, 1–5 star scale, launched February 2025), ML.ENERGY Leaderboard v3 (University of Michigan, J/token per task, Zeus tooling, December 2025) and MLPerf Power (samples/joule certified with a physical Yokogawa WT310E power meter). The three measure different dimensions and are not directly interchangeable. The available data shows that reasoning models consume up to 700× more energy than their non-reasoning equivalents; that MoE models consume roughly 3× fewer J/token than a dense model with equivalent active parameters; and that INT4 quantisation cuts consumption by up to 79 % against FP16 under favourable conditions. The inference engine (vLLM vs Transformers) can move the result another 25–40 %. Without fixing hardware, engine, batch size and task, no comparison between leaderboards is valid.

Track context

This article is C5 of the energy pillar. The base context:

The fundamentals of quantisation are a prerequisite for the quantisation section of this article.

The three leaderboards: technical data sheet

1 · Hugging Face AI Energy Score

Field	Detail
URL	huggingface.co/AIEnergyScore · huggingface.co/spaces/AIEnergyScore/Leaderboard
Organisation	Hugging Face (Sasha Luccioni et al.), with Salesforce and Cohere as initial partners
Launch	February 2025 (AI Action Summit, Paris); v2 December 2025
Models indexed	166 (v1, Feb 2025); +39 new ones in v2 (Dec 2025)
Tasks measured	10 tasks: text generation, summarisation, classification, image generation, ASR, audio generation, translation, question answering, reasoning (added in v2)
Unit of measurement	Wh (watt-hour) per 1,000 queries of the task
Reference hardware	NVIDIA H100 exclusively (a single GPU for class A/B models; multiple for class C)
Measurement tooling	CodeCarbon (GPU energy) + the `ai-energy-benchmarks` package (OSS, PyPI)
Rating system	1–5 stars per task: quintiles of the energy range; ⭐⭐⭐⭐⭐ = the 20 % most efficient
Reference batch size	Batch size = 1 (does not reflect production with aggressive batching)
Access for proprietary models	Yes, via an audited Docker container
Update frequency	No fixed cadence; v1 Feb 2025, v2 Dec 2025
Project licence	Apache 2.0 (repository github.com/huggingface/AIEnergyScore)

Scope of the metric. The AI Energy Score measures GPU energy only (CodeCarbon); it does not capture CPU, DRAM or system overhead. The Wh/1k-queries unit covers the whole run time (prefill + decode + framework overhead), but at batch = 1. The results are therefore comparable between models under the same test conditions, but not extrapolable to a production environment with real concurrency without correction.

Model class (the project’s internal classification):

Class	Definition
A	Fits on a consumer GPU (≤ ~24 GB VRAM)
B	Requires a cloud GPU (≥ 40 GB VRAM)
C	Requires multiple GPUs

2 · ML.ENERGY Leaderboard

Field	Detail
URL	ml.energy/leaderboard
Organisation	Symbiotic Lab, University of Michigan (Mosharaf Chowdhury, Jae-Won Chung et al.)
Reference paper	arXiv 2505.06371 — “The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization” (NeurIPS 2025 D&B, Spotlight)
Current version	v3.0 (December 2025)
Measurement tooling	Zeus (github.com/ml-energy/zeus) via NVML + RAPL; measurement overhead in single-digit ms
Unit of measurement	J/token (energy per output token generated) and total energy per complete response
Reference hardware	NVIDIA A100 80 GB and H100 SXM (declared per submission; varies between models)
Tasks measured	6 tasks: chat (text conversation), reasoning, code generation, summarisation, visual question answering, video generation
Normalisation	Mean energy per complete response (prefill + decode). Output J/token is also reported. The task is stated explicitly because it determines the output length
Scope of the measurement	GPU via NVML + CPU/DRAM via RAPL; not a power meter at the wall
Models covered	~40 architectures in the NeurIPS 2025 version; the web leaderboard is updated with more
Licence	Apache 2.0 (zeus: github.com/ml-energy/zeus); MIT (benchmark: github.com/ml-energy/benchmark)
Update frequency	Continuous on the web leaderboard; the paper is a point-in-time snapshot

Zeus as tooling. Zeus is the measurement engine of the ML.ENERGY Leaderboard and also a standalone package (pip install zeus-ml). It supports NVIDIA GPU (NVML), AMD GPU (ROCm), CPU (RAPL), DRAM (RAPL), Apple Silicon and NVIDIA Jetson. The ZeusMonitor adds measurement overhead in single-digit milliseconds. Since May 2025 it has been a PyTorch ecosystem project. MIT licence.

3 · MLPerf Power

The full data sheet is in the C4 article. A summary of the points relevant for comparison with the previous two:

Field	Detail
URL	mlcommons.org/benchmarks/inference-datacenter/
Organisation	MLCommons Power Working Group (>20 orgs)
Unit of measurement	samples/joule (throughput/mean power) = the inverse of J/sample
Hardware	Complete node measured at the wall (AC); Yokogawa WT310E analyser (±0.1 % of reading)
LLM tasks	GPT-J 6B, Llama 2 70B, Mixtral 8×7B (from v5.0)
Granularity	Complete node (GPU + CPU + RAM + fans + PSU losses); no attribution to individual workloads
Node overhead over GPU	25–45 % of total consumption in submissions with a physical analyser
Licence of the corpus	Public results on GitHub (mlcommons/inference_results_vX.Y); PTDaemon requires MLCommons membership

Comparison of the three leaderboards

Dimension	HF AI Energy Score	ML.ENERGY Leaderboard	MLPerf Power
Unit	Wh/1k-queries	Output J/token	samples/J (complete node)
Fixed hardware	H100 (all models)	A100/H100 (varies)	Depends on the submitter
Measurement	CodeCarbon (GPU)	Zeus NVML+RAPL	Physical AC power meter (Yokogawa)
System coverage	GPU only	GPU + CPU + DRAM	Complete node including fans and PSU
Batch size	1	Varies by task	Per LoadGen scenario
Models covered	166+ (text, image, audio)	~40 generative LLMs	Few (GPT-J, Llama 2, Mixtral)
Proprietary models	Yes (audited Docker)	No (OSS only)	Yes (MLCommons members)
External certification	No	No	Yes (SPEC PTDaemon)
Frequency	Point-in-time (v1, v2)	Continuous	Half-yearly (MLPerf rounds)
Licence	Apache 2.0	Apache 2.0 / MIT	Public results; PTDaemon: membership

Incompatibility between leaderboards. The three measure different dimensions: Wh/query ≠ J/token ≠ node samples/J. A direct comparison requires converting units and assuming that the hardware, the engine and the task are equivalent, which is rarely the case across leaderboards.

How energy per token is measured and normalised

The base identity is developed in the C2 article:

$$E_{\text{token}} \,[\text{J/tok}] = \frac{\bar{P} \,[\text{W}]}{\text{throughput} \,[\text{tok/s}]}$$

To compare models against each other, every factor other than the model must be fixed:

Factor	Effect if it varies	How to fix it
Hardware	H100 vs A100 vs L40S changes the result by 2–4×	Declare the exact hardware; compare only within the same HW
Inference engine	vLLM vs Transformers: 25–40 % difference in J/token	Fix the engine and the version
Batch size / concurrency	Batch 1 vs batch 32: throughput rises but so does power; the ratio varies	Declare the batch size; compare within the same regime
Model precision	FP16 vs INT8 vs INT4: up to −79 % energy	Declare the precision; do not mix
Response length	A query with 50 tokens ≠ one with 500	Use a fixed dataset or normalise per token
Measurement window	Including warm-up or idle inflates the numerator	Align the power window with the token window (see C2)

Conversion formula Wh/query ↔ J/token:

$$E_{\text{J/tok}} = \frac{E_{\text{Wh/query}} \times 3600}{\bar{n}_{\text{tokens/query}}}$$

Example: if a model consumes 0.05 Wh/query (= 180 J/query) and generates an average of 200 tokens per query:

$$E_{\text{J/tok}} = \frac{0{.}05 \times 3600}{200} = \frac{180}{200} = 0{.}9 \,\text{J/tok}$$

AI Energy Score data: concrete examples

The v2 data (December 2025, H100 hardware, batch = 1, text generation task with reasoning on/off):

Model	Active params	Reasoning	GPU Wh/1k queries	Stars (text-gen)
DistilGPT-2	82 M	—	1.31	⭐⭐⭐⭐⭐
SmolLM3-3B	3 B	Off	18.35	⭐⭐⭐⭐
SmolLM3-3B	3 B	On	12,791.22	⭐
Phi-4-reasoning-plus	15 B	Off	18.42	⭐⭐⭐⭐
Phi-4-reasoning-plus	15 B	On	9,461.61	⭐
DeepSeek-R1-Distill-Llama-70B	70 B	Off	49.53	⭐⭐⭐
DeepSeek-R1-Distill-Llama-70B	70 B	On	7,626.53	⭐

Source: Hugging Face AI Energy Score v2 (Dec 2025).

The reasoning multiplier. The energy increase when reasoning is switched on ranges from ×154 (DeepSeek-R1-Distill-Llama-70B) to ×697 (SmolLM3-3B). The direct cause: reasoning models generate between 300 and 800 times more tokens than their non-reasoning equivalents (internal chains of thought). The mean across the v2 corpus is ×30 extra energy for reasoning.

Newer models are not always more efficient. Of the 14 comparable models (no reasoning, no MoE, similar size) between the Feb 2025 and Dec 2025 cohorts: 8 out of 14 had equal or higher energy. The range runs from 3 % of the reference model’s energy to almost 2×. Parameter scale is no longer enough to estimate efficiency.

ML.ENERGY Leaderboard data: J/token by family

The data from the arXiv 2505.06371 paper and from leaderboard v3 (A100/H100 hardware, vLLM as the reference engine):

Scale within one family (Llama 3):

Size	Params	Relative J/token (base = 1B)	Params/energy ratio
Llama 3 · 1B	1 B	1.0×	—
Llama 3 · 8B	8 B	~2.1×	8× params → 2.1× energy
Llama 3 · 70B	70 B	~7.3×	70× params → 7.3× energy

The sublinearity (70× params → 7.3× energy, not 70×) reflects that inference energy is dominated by memory bandwidth (memory-bandwidth bound), not by raw FLOPs.

Dense vs MoE:

Model	Type	Total params	Active params/token	Relative J/token
Llama 3 · 8B	Dense	8 B	8 B	1.0×
Mixtral 8×7B	MoE (top-2)	47 B	~13 B	~0.33×
Llama 3 · 70B	Dense	70 B	70 B	~3.5×

The MoE activates only 2 of 8 experts per token. Mixtral 8×7B consumes roughly ⅓ of the J/token of a dense model with 8B active, at a quality comparable to a dense model of larger scale. Routing overhead and holding all the experts in memory cancel out part of the theoretical gain.

Effect of the task (ML.ENERGY v3, same model):

Task	Energy multiplier per response (vs chat)
Chat (text conversation)	1× (reference)
Summarisation	~2–4×
Code generation	~3–6×
Reasoning	~25×
Image + text	1.1–5.2×
Video + text	1.3–15.0×

Reasoning uses ~10× more tokens per response, and the extra memory of the chain of thought reduces the effective batch size, raising energy per token through memory pressure.

Effect of quantisation on energy per token

Data on NVIDIA H100 hardware, Llama 3 family (arXiv 2508.16712 and arXiv 2504.03360):

Precision	Energy reduction vs FP16	Condition
FP16	reference (0 %)	—
BF16	~0 % (iso-energy)	Same hardware and engine
FP8	−25 to −35 %	H100/H200 with native hardware support
INT8	−23 to −44 % (median ~39 %)	Depends on batch size; more at low batches
INT4 (AWQ / GPTQ)	−50 to −79 %	Requires hardware with efficient low-precision support

Warning. On GPUs without native INT4 hardware support (or with suboptimal dequantisation kernels), quantisation can increase latency and energy per token rather than reduce them, because of run-time dequantisation overhead. The benefit of quantisation is real on H100/A100 with TensorRT-LLM or a well-configured llama.cpp, but it is not guaranteed with any engine.

Quantisation and throughput: the memory reduction per model frees VRAM, which allows larger batch sizes. At a larger batch, throughput rises more than power does, cutting J/token further still. The net effect can exceed the direct reduction in energy per operation.

Effect of the inference engine

The engine is a variable that model-level leaderboards tend to fix, but which in production is a decision of your own. Data from published comparisons (vLLM, TensorRT-LLM, naive Transformers, A100):

Engine	Relative J/token vs Transformers baseline
Transformers (naive, not optimised)	1.0× (reference)
vLLM (PagedAttention, continuous batching)	−25 to −35 %
TensorRT-LLM (optimised NVIDIA kernels, FP8)	−35 to −45 %
llama.cpp (hybrid CPU/GPU, INT4)	Variable; −30 to −60 % depending on hardware

Moving from naive Transformers to TensorRT-LLM can cut energy per token by more than moving from a 70B model to an 8B one of the same origin. The choice of engine is a first-order lever on energy efficiency.

Limits of energy leaderboards

Limit	Description
Hardware dependence	A ranking on H100 is not valid on A100 or L40S without correction. The hierarchy of models can change from one piece of hardware to another.
Engine dependence	The results are valid only for the engine they were measured with. A model that is ×2 more efficient on the leaderboard can fall behind if a slower engine is used.
Artificial batch size	AI Energy Score uses batch = 1. In production with aggressive batching, the efficiency relationship between large and small models changes: the large ones scale better with batch.
Training not captured	All the leaderboards measure inference only. The energy cost of training (which can exceed that of inference by 1,000× over the life of the model) is out of scope.
Incompatibility between leaderboards	Wh/query, J/token and samples/J measure different things. Converting between them requires knowing the mean output length, which varies by task and dataset.
Partial system coverage	AI Energy Score and ML.ENERGY measure GPU (+CPU/DRAM with Zeus); they do not capture the overhead of the complete system (PSU losses, fans, interconnect). MLPerf Power does, but covers few models.
Data latency	The leaderboards publish results months after the tests. New hardware (H200, B100, B200) may have no data available at the time of the decision.
No PUE	None of the three includes datacenter PUE. For real TCO, the leaderboard J/token must be multiplied by your own PUE.

Decision table: choosing a model on energy efficiency

The selection criteria in order, without recommendation prose:

Criterion	Question	Action
Task with reasoning	Does the task require step-by-step reasoning?	Yes → multiply the model’s base energy by ×30–700 before comparing. If there is a non-reasoning alternative with sufficient quality, prefer it.
Size vs minimum quality	What is the minimum acceptable quality for the task?	Consult quality benchmarks (see B7 when available). Pick the smallest model that clears the quality threshold; energy grows sublinearly with size.
Dense vs MoE	Does the hardware have enough memory for the full MoE?	If yes: the active-equivalent MoE consumes ~3× fewer J/token than the equivalent dense model. If not: paging or offload eats the gain.
Precision	Does the hardware have native FP8/INT4 support?	H100/H200: native FP8 (−30 %). With TensorRT-LLM: INT4 AWQ (−50 to −79 %). Without native support: stay on FP16 or BF16 until validated with your own benchmark.
Inference engine	Is the optimal engine for the hardware being used?	Measure with C3. If the engine is not optimised, changing engine can cut energy more than changing model.
Consult the leaderboard	Is the task covered by AI Energy Score or ML.ENERGY?	Filter by: same task, same hardware class, reasoning explicitly off/on. Do not compare models from different hardware classes or different engines.
Validate on your own hardware	Are the leaderboard results on the same HW as yours?	Always validate with Zeus or DCGM on your own hardware before taking the final decision. The leaderboard is a reference, not a prediction.

Quick signals table:

Signal	Effect on energy	Data source
Switching reasoning on	×30–700	AI Energy Score v2
Going from dense 8B to dense 70B	~×3.5	ML.ENERGY Leaderboard v3
Going from dense 8B to active-equivalent 8B MoE	~×0.33 (−67 %)	ML.ENERGY v3
FP16 → INT4 (compatible hardware)	−50 to −79 %	arXiv 2508.16712, 2504.03360
Naive Transformers → TensorRT-LLM FP8	−35 to −45 %	TokenPowerBench, ML.ENERGY
PUE 1.0 → PUE 1.5	+50 % on real datacenter energy	MLPerf Power (scope)

Reference data: energy in a generic node (4×H100 SXM)

Generic example hardware, to anchor the leaderboard values to a real node:

Parameter	Indicative value
TDP 4×H100 SXM 80 GB	4 × 700 W = 2,800 W (GPU only)
Complete-node system power (at the wall)	~3,500–5,000 W depending on load
Non-GPU overhead over GPU	25–45 %
J/token Llama 3 70B FP16, vLLM, batch 8	~1–3 J/tok (indicative, A100/H100)
J/token Llama 3 8B FP16, vLLM, batch 8	~0.3–0.7 J/tok (indicative)
J/token Mixtral 8×7B FP16, vLLM, batch 8	~0.4–0.8 J/tok (indicative)
Energy per 1M tokens (Llama 3 70B, PUE 1.4)	~0.5–1.2 kWh

The J/token values are indicative and depend heavily on batch size, prompt length, prefill/decode ratio and engine version. For certified values, consult the MLPerf Power submissions (mlcommons.org).

For the alternative reference node (4×A100 PCIe 80 GB, TDP ~300 W each):

Parameter	Indicative value
TDP 4×A100 PCIe	4 × 300 W = 1,200 W (GPU only)
Complete-node system power	~1,500–2,000 W
J/token Llama 3 70B FP16, vLLM	~2–5 J/tok (indicative; higher because of lower HBM bandwidth vs SXM)

How to use the leaderboards in practice

A decision flow based on the public data available:

Step	Action	Resource
1	Identify the dominant task of the workload	—
2	Consult AI Energy Score filtered by task and hardware class	huggingface.co/spaces/AIEnergyScore/Leaderboard
3	Note the models with ⭐⭐⭐⭐ or ⭐⭐⭐⭐⭐ on the task	Wh/1k-queries as a relative reference
4	Cross-check with ML.ENERGY for the J/token of each candidate	ml.energy/leaderboard
5	If any model is in MLPerf Power (Llama 2, GPT-J, Mixtral), consult the certified samples/J	mlcommons.org/benchmarks/inference-datacenter/
6	Select the 2–3 candidates with the best energy/quality ratio	—
7	Measure on your own hardware with Zeus or DCGM	github.com/ml-energy/zeus
8	Multiply the measured J/token by the datacenter PUE	J/token × PUE = effective J/token in the datacenter
9	Work out the electricity cost per token at the contracted price	See C2

Sources

Hugging Face · AI Energy Score · organización y leaderboard — https://huggingface.co/AIEnergyScore
Hugging Face · Announcing AI Energy Score Ratings (Luccioni et al., feb. 2025) — https://huggingface.co/blog/sasha/announcing-ai-energy-score
Hugging Face · AI Energy Score v2: Refreshed Leaderboard, now with Reasoning (Luccioni, Gamazaychikov, dic. 2025) — https://huggingface.co/blog/sasha/ai-energy-score-v2
Hugging Face · AIEnergyScore GitHub (Apache 2.0) — https://github.com/huggingface/AIEnergyScore
ML.ENERGY Initiative · Leaderboard — https://ml.energy/leaderboard
ML.ENERGY Initiative · Blog: Diagnosing Inference Energy Consumption with the ML.ENERGY Leaderboard v3.0 (dic. 2025) — https://ml.energy/blog/measurement/energy/diagnosing-inference-energy-consumption-with-the-mlenergy-leaderboard-v30/
arXiv 2505.06371 · The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization (Chung et al., NeurIPS 2025 D&B Spotlight) — https://arxiv.org/abs/2505.06371
ML.ENERGY Initiative · Zeus: Deep Learning Energy Measurement and Optimization — https://ml.energy/zeus/
GitHub ml-energy/zeus (MIT) — https://github.com/ml-energy/zeus
PyTorch Blog · Zeus: Deep Learning Energy Measurement and Optimization — https://pytorch.org/blog/zeus/
University of Michigan CSE · Power-hungry AI: Researchers evaluate energy consumption across models — https://cse.engin.umich.edu/stories/power-hungry-ai-researchers-evaluate-energy-consumption-across-models
arXiv 2512.03024 · TokenPowerBench: Benchmarking the Power Consumption of LLM Inference (dic. 2024) — https://arxiv.org/abs/2512.03024
arXiv 2508.16712 · Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective — https://arxiv.org/abs/2508.16712
arXiv 2504.03360 · Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency — https://arxiv.org/abs/2504.03360
Epoch AI · AI Energy Use: Data & Research — https://epoch.ai/topics/energy
MLCommons · MLPerf Inference Datacenter benchmark results — https://mlcommons.org/benchmarks/inference-datacenter/
arXiv 2410.12032 · MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems (Tschand et al., 2024) — https://arxiv.org/abs/2410.12032
Coalition for Sustainable AI · AI Energy Score as best practice in benchmarking — https://www.sustainableaicoalition.org/ai-energy-score-a-standardized-approach-to-evaluating-ai-model-energy-efficiency/

MLPerf Power: the standard energy-efficiency benchmark for on-premise ML systems

Mon, 15 Jun 2026 06:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

MLPerf Power is the only standardised benchmark for measuring the energy efficiency of complete ML systems. It is run by the MLCommons Power Working Group and backed by more than 20 organisations. It measures power at the wall (AC wall power) for the whole System Under Test, GPUs, CPUs, memory, interconnect and fans included, with a SPEC PTDaemon certified power analyser (Yokogawa WT310/WT5000) and the public tooling in mlcommons/power-dev. The central metric for datacenter inference is samples/joule (or tokens/joule for LLMs); for latency, energy per stream. The public corpus adds up to 1,841 reproducible measurements from 60 systems (590 datacenter, 792 edge, 447 tiny, 12 training). GPT-J and Llama 2 show improvements of more than 100× in samples/joule between the first and the latest available rounds. The direct comparison with the software-based measurement of post C3 (DCGM/NVML/RAPL/Kepler) shows that MLPerf Power offers maximum accuracy and reproducibility at the cost of dedicated hardware (~3,000 USD for the Yokogawa 310E) and of the requirement to be an MLCommons member to access PTDaemon; software measurement is continuous, needs no extra hardware, but carries a wider error bar and no external validation.

What MLPerf Power is and how it fits into the MLCommons ecosystem

MLCommons is the consortium of more than 100 organisations (NVIDIA, Google, Intel, Dell, AMD, Meta and others) that maintains the MLPerf benchmarks: Training, Inference (Datacenter and Edge), Tiny, HPC, Storage, Client and Automotive. The Power Working Group is the specific group that extends each of those benchmarks by adding the energy dimension (MLCommons Power Working Group).

MLPerf Power is not a standalone benchmark: it is an energy measurement layer overlaid on the existing performance benchmarks. For a submission to be valid with power, it must first satisfy the performance rules (MLPerf Inference, Training or Tiny) and then the additional power measurement rules.

Coupling with MLPerf Inference

The integration most relevant to on-premise inference is with MLPerf Inference Datacenter, which defines:

Scenarios: Server (latency, queries per second with a request queue) and Offline (pure throughput, batch with no latency constraint). Power is measured in both.
Benchmarks: ResNet-50, BERT, RNN-T, 3D-UNET, RetinaNet, DLRM, GPT-J 6B, Llama 2 70B (from v4.0), Mixtral 8×7B (from v5.0).
Divisions: closed (model and preprocessing fixed, only runtime optimisation allowed) and open (modifications to the model permitted).

Power is measured during the performance phase, not during accuracy or compliance. The same run that produces the performance log produces the power log: reporting the highest performance of three runs and the lowest power of another three is forbidden (MLPerf Inference Power Measurement Rules, §5.9).

Coupling with MLPerf Training and Tiny

MLPerf Training: large-scale power measurement (multi-node, 10K+ GPUs) does not use an external analyser, which would be impracticable. Node telemetry (IPMI/Redfish) plus an estimate for the interconnect network is used instead. The metric is energy to train (J or kWh). In the v4.0 submissions with power, the systems running Llama 2 70B Training range from single nodes to hundreds, revealing the non-linear scaling of energy (arXiv 2410.12032).
MLPerf Tiny: microcontroller systems (from 5.64 mW). Specialised micro-power instrumentation is used, with hardware pins to demarcate the start and end of the inference. The metric is energy per inference (J), not samples/joule.

Measurement methodology: certified wall power

The fundamental principle: the whole SUT, at the wall

Rule number one is absolute: power must be measured at system level, that is, including every component the benchmark activates: host processors, accelerators, memory, disks, fans, internal interconnect. Measuring the GPUs alone is not enough (MLPerf Inference Power Measurement Rules, §5.1):

“The power consumption must be measured at the system level, i.e. including all components that are sensitized by LoadGen e.g. the host processor on which LoadGen runs, accelerators, memory, fans, etc.”

Power is measured in AC (alternating current), at the SUT input, upstream of the PSUs. Any battery or energy storage between the mains socket and the system PSUs is forbidden.

Measurement hardware: SPEC-certified power analyser

MLPerf Power requires a power analyser certified by SPEC PTDaemon (official SPEC list). The most widespread in submissions is the Yokogawa WT310E (~3,000 USD), which connects to the director over USB (Linux) or Ethernet/serial (Windows). Voltage is measured in parallel and current in series with the SUT power line.

Relevant specifications of the Yokogawa WT310E:

Power accuracy: 0.1 % of reading + 0.1 % of range
Measurement range: µW to MW (the WT5000 reaches industrial installations)
Update rate: from 50 ms

For systems with more than one channel, or multi-PSU nodes, multi-analyser configurations are allowed. The ranging rule: first a ranging run is performed with the range in Auto mode to determine the maximum current and voltage values; the testing runs use fixed ranges based on those peaks, which maximises accuracy within the range. Auto mode is not permitted in the testing runs.

SPEC PTDaemon: the daemon that orchestrates the measurement

PTDaemon (Power Thermal Daemon) is the SPEC tool that handles communication with the analyser. MLCommons is licensed to use it within the MLPerf Power flow. Access requires MLCommons membership and signing the corresponding EULA (MLPerf Power FAQ).

The measurement flow is as follows:

The process step by step (mlcommons/power-dev):

NTP sync between Director and SUT to align timestamps.
Ranging run: power with ranges in Auto; the analyser determines the current and voltage peaks.
Testing run: fixed ranges; LoadGen runs the benchmark; the Director records power with timestamps; the SUT records the performance log with start/end timestamps of the execution phase.
Post-processing: the result summariser cross-references power log and performance log by timestamp to compute the mean power over the execution window.
A minimum of 60 seconds of valid power data. If the workload finishes before 60 s, it is run in a loop until that threshold is reached.

What is and is not inside the SUT

The SUT includes everything the benchmark activates:

Inside the SUT	Outside the SUT
GPUs / accelerators	PDU shared with other systems
Host CPU and RAM	Datacenter cooling infrastructure (PUE)
Disks / NVMe if the benchmark uses them	Management network (BMC, out of band)
Node fans and cooling systems	Datacenter network switches (not the node’s)
Internal interconnect (NVLink, PCIe)
Node PSUs

The paper makes it explicit: PUE is out of scope for MLPerf Power by design (arXiv 2410.12032, §III-C). MLPerf measures the efficiency of the ML system, not that of the datacenter. Including PUE would obscure the differences between systems by mixing hardware efficiency with building efficiency.

Metrics: formulas and definitions

Throughput benchmarks (Datacenter / Offline / Edge)

For throughput benchmarks, Offline and Server in datacenter plus some edge scenarios, the energy efficiency metric is:

$$\eta = \frac{\text{throughput (samples/s)}}{\text{mean power (W)}} \quad \Rightarrow \quad \left[\frac{\text{samples}}{\text{J}}\right]$$

where mean power is computed over the window of the execution phase of the performance run. The reciprocal gives energy per sample:

$$E_{\text{sample}} = \frac{\text{mean power (W)}}{\text{throughput (samples/s)}} \quad \left[\text{J/sample}\right]$$

For LLM benchmarks (GPT-J, Llama 2), where the output varies in length, a “sample” is one complete query (prompt + response). Energy per output token is obtained by dividing by the number of tokens generated, which varies per query and must be reported or estimated.

Latency benchmarks (Single Stream / Tiny)

For latency benchmarks, Single Stream in edge and tiny, where the processing time is fixed, the metric is the inverse of energy per inference:

$$\eta_{\text{latency}} = \frac{1}{E_{\text{inference}}} \quad \left[\frac{1}{\text{J}}\right]$$

The paper treats both metrics (samples/J and 1/J) as comparable within their category, although they are not interchangeable between categories.

System power and run energy

The three quantities that appear in every power submission:

Quantity	Definition	Unit
System power	mean of the AC power samples over the execution window	W
Run energy	mean power × window duration	J
Energy efficiency	throughput / system power = samples/J	samples/J

The run energy formula:

$$E_{\text{run}} = \bar{P} \times \Delta t = \frac{\sum_{i} P_i \cdot \Delta t_i}{\Delta t_{\text{total}}} \times \Delta t_{\text{total}} \quad [\text{J}]$$

where $ P_i $ are the analyser samples and $ \Delta t_i $ the intervals between samples, both inside the window demarcated by the LoadGen timestamps.

How to read an MLPerf Power submission

Divisions and availability categories

Submissions inherit the divisions of MLPerf Inference:

Division	Constraints	Comparability
Closed	Fixed model, fixed preprocessing, only the runtime and the hardware are optimised	High: submissions directly comparable with each other
Open	Modifications to the model allowed (aggressive quantisation, distillation, pruning)	Low: each system may use a different model

Within closed, there are system availability categories:

Category	Definition
Available	System commercially available at the closing date
Preview	Announced but not commercially available
RDI (Research, Development, Internal)	Systems for internal or experimental use

Only Available systems in closed are comparable without caveats. A Preview system with better energy than an Available one from another company is not a valid comparison for purchasing decisions.

What comparability MLPerf Power gives and what it does not

It gives:

An objective comparison of the energy efficiency of complete nodes (not just the GPU) under a standardised ML workload.
Verified reproducibility: the logs are published on GitHub and anyone can review them.
Trends over time between rounds: how much the efficiency of each hardware family improves version to version.
A basis for comparing heterogeneous systems running the same workload under the same rules.

It does not give:

An isolated GPU-to-GPU comparison (it measures the complete node, not the card alone).
Broad hardware coverage: the number of submissions with power is small. In v4.0 only four companies (Dell, Fujitsu, NVIDIA, Qualcomm) delivered power numbers for datacenter. In v5.1 there were two power submissions (Lenovo datacenter + GATEOverflow edge).
Representativeness of your own workloads: the benchmarks are fixed and may not match the distribution of your real load (batch size, prompt length, prefill/decode ratio).
Datacenter cooling data: PUE, liquid vs air, are out of scope.
Real-time comparison: results are published months after the closing date.

Structure of a submission

Every power submission publishes the following in the MLCommons results repository:

<division>/<submitter>/measurements/<system>/
├── analyzer_table.md # Analyser configuration (model, ranges)
├── power_settings.md # Power management configuration of the SUT
results/<system>/<benchmark>/<scenario>/
├── mlperf_log_summary.txt # Performance results (LoadGen)
├── spl.txt # Power log (power, current, voltage, timestamps)
└── ...

The spl.txt files contain the raw analyser readings with timestamps, which makes it possible to verify alignment with the LoadGen window.

Data: scale, trends and reference hardware

The corpus: 1,841 measurements from 60 systems

The MLPerf Power submissions database covers (arXiv 2410.12032):

Category	Submissions	Power range
Datacenter (Inference)	590	~200 W – ~10 kW per node
Edge (Inference)	792	~10 W – several kW
Tiny (Inference)	447	5.64 mW – hundreds of mW
Training / HPC	12	up to 500 kW (measured); ~10 MW estimated in HPC

The total range runs from microwatts to megawatts, 9 orders of magnitude, which means no single methodology applies across every segment.

Energy efficiency improvements: LLMs lead

The evolution of the samples/joule metric between rounds, normalised to the first submission of each model (arXiv 2410.12032, §V-A):

Workload	Category	Cumulative improvement (samples/J)
GPT-J 6B	Datacenter	>100×
Llama 2 70B	Datacenter	>100×
RetinaNet	Datacenter	Largest among the classic models
BERT-99.0	Edge	~4×
RNN-T	Edge	~4×
ResNet-50	Edge	~1.5×
ResNet-50	Tiny	>1,000×
Other Tiny	Tiny	79× – 596×

The >100× improvements in datacenter LLMs reflect the massive industry attention on software optimisation (FP8 kernels, FlashAttention, speculation) and hardware (tensor core architectures, HBM3). Tiny models show even larger gains in relative terms thanks to the low starting point and the progress in specialised chips.

On-premise reference hardware

For a node of 4×H100 SXM 80 GB (the track’s generic hardware), the typical values in MLPerf Power:

Parameter	Indicative value
Declared TDP 4×H100 SXM	4 × 700 W = 2,800 W (GPU only)
System power (complete node) measured at the wall	~3,500 – 5,000 W depending on load
CPU + RAM + fans overhead over GPU	25 – 45 % of the total
Efficiency: Llama 2 70B offline (closed)	a function of the runtime; order of magnitude ~0.1 – 1 tokens/J

The key point: the “node overhead”, the difference between what the analyser measures at the wall and what NVML reports for the GPUs, is 25–45 % of total consumption. A GPU-only measurement underestimates the real system consumption by that margin.

For the A100 PCIe 80 GB (4× per node), system consumption is lower (GPU TDP ~300 W each, node ~1,500–2,000 W), with better efficiency per watt but lower absolute throughput. The L40S (4× per node) has an intermediate profile: TDP ~350 W each, good FP8 inference performance, node consumption ~1,800–2,500 W.

Contrast with the software measurement of post C3

Post C3 of this track covers the DCGM/NVML/RAPL/Kepler stack in production. Here is the data-driven comparison against MLPerf Power:

Dimension	Software (DCGM/NVML/RAPL/Kepler)	MLPerf Power (analyser + PTDaemon)
What it measures	On-device sensors (GPU, CPU, DRAM)	AC wall power (the whole node)
Coverage	GPU + CPU + DRAM (≠ the wall)	Complete node including fans, PSU overhead
Node overhead	Not captured by default	Included in the measurement
Accuracy	±5–15 % vs power meter (Kepler estimate); NVML ~3–5 % on the GPU	±0.1 % reading + ±0.1 % range (Yokogawa WT310E)
Frequency / continuity	Continuous in production (sub-second)	Only during the benchmark run
Extra hardware	None (uses system sensors)	SPEC-certified analyser (~3,000 USD)
Attribution per workload	Per pod/MIG with Kepler	No: it measures the system, not the individual workload
External certification	No	Yes (SPEC PTDaemon + MLCommons review)
Data access	Real time in Prometheus	Public logs on GitHub after the round
Recommended use	Continuous monitoring in production	Certified comparative benchmarking

The operational conclusion: the two approaches are complementary, not competing. The DCGM/Kepler stack gives the continuous per-pod view for production; MLPerf Power gives the certified ground truth for hardware comparisons with reproducibility guarantees.

The myth of measuring only the GPU: data from the paper

The arXiv 2410.12032 paper devotes section §III-C to dismantling the myths of power measurement in ML systems. The one most relevant to on-premise inference is Myth 1:

“A common misconception is that measuring the power consumption of specific ML components, such as accelerators or GPUs, is adequate to assess system efficiency. In reality, overall system power consumption is crucial. Different components are active at various stages of ML workloads with varying duty cycles.”

The data from submissions with a wall analyser versus NVML readings confirms that the overhead of the node’s non-GPU components (CPU, memory, disks, fans, PSU losses) ranges between 25 and 45 % of total consumption depending on the system. An efficiency comparison based on NVML readings alone systematically underestimates consumption and can distort the ranking between systems with different CPU/GPU proportions.

Myth 2 (TDP and PSU as power proxies) is also relevant: the TDP of an H100 SXM is 700 W per card, but real consumption in inference with Llama 2 70B depends on the load, the prompt length and the batch size. The MLPerf Power data shows that systems routinely run well below TDP in online inference scenarios (server scenario), where latency limits throughput and the GPU is not at 100 % utilisation.

Limitations of the current submissions

The limitations are not of the method but of the coverage:

Few submissions with power in datacenter. In v4.0 only 4 companies delivered power numbers for datacenter; in v5.1, one (Lenovo). Performance benchmarking has dozens of submitters; energy benchmarking has few. This limits the representativeness of the corpus for comparing on-premise hardware.
Heterogeneous SUTs. Each submitter defines its own SUT: number of GPUs, server, interconnect, software stack. A Dell system with 4×H100 is not directly comparable to a Supermicro one with 8×H100, even if both use the same model. Platform differences (NVLink vs PCIe, server vs rack) are captured but not separated out.
Fixed workloads. The MLPerf benchmarks are representative but they are not your load. The distribution of prompt lengths, the prefill/decode ratio and your users’ batch size may differ significantly from the MLPerf synthetic datasets. The benchmark’s samples/J result is a reference, not a prediction of your workload.
Publication latency. Results are published months after the submission deadline. New hardware (H200, B100, B200) may have no power results available at the moment of the purchasing decision.
No PUE. MLPerf Power does not include datacenter efficiency. Two identical systems in two datacenters with PUE 1.1 and 1.5 give the same MLPerf Power result but a very different electricity cost. For TCO, the MLPerf J/sample must be multiplied by your own datacenter PUE.

Usage flow for an on-premise platform

For anyone who wants to use MLPerf Power as a purchasing reference or as their own benchmark, the practical flow:

Step	Action	Resource
1	Consult the publicly available submissions	mlcommons.org/benchmarks/inference-datacenter
2	Filter by model (Llama 2 70B, GPT-J), scenario (Offline/Server) and division (Closed/Available)	GitHub mlcommons/inference_results_vX.Y
3	Compare samples/J between systems with similar hardware	power log + performance summary
4	Adjust for your own PUE	samples/J × (1/PUE) for real energy in the datacenter
5	If you want to measure your own hardware: buy a Yokogawa WT310E, join MLCommons, get access to PTDaemon	docs.mlcommons.org/inference/power
6	Calibrate the software stack (DCGM/Kepler) against the analyser once	see post C3

Cross-links of the energy track

This article is C4 of the energy pillar. The related articles:

C1 — State of the art: energy benchmarking of LLM frameworks: inventory of tools and overview of the field.
C2 — Energy per token: methodology and the Spanish electricity market: the J/token identity and how the electricity price multiplies it.
C3 — Measurement tooling in production: Kepler, DCGM and a practical stack: the continuous stack for production that MLPerf Power complements.
Benchmarking track — state of the art of frameworks: context of the performance benchmarking that MLPerf Power extends into energy.

Sources

MLCommons Power Working Group — https://mlcommons.org/working-groups/benchmarks/power/
arXiv 2410.12032 · MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI (Tschand et al., 2024) — https://arxiv.org/abs/2410.12032
MLPerf Inference Power Measurement Rules v2.0 (power_measurement.adoc) — https://github.com/mlcommons/inference_policies/blob/master/power_measurement.adoc
MLCommons power-dev · repositorio público de herramientas de medición — https://github.com/mlcommons/power-dev
MLPerf Inference Power Measurement Documentation (MLCFlow) — https://docs.mlcommons.org/inference/power/
SPEC PTDaemon · lista de dispositivos certificados — https://open.spec.org/power/docs/specpower-device_list/
MLCommons · MLPerf Inference v1.0 con las primeras mediciones de potencia (abril 2021) — https://mlcommons.org/2021/04/mlperf-inference-v1-0-results-with-first-power-measurements/
MLCommons · MLPerf Inference v4.1 results (agosto 2024) — https://mlcommons.org/2024/08/mlperf-inference-v4-1-results/
MLCommons · MLPerf Inference v5.1 results (septiembre 2025) — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
MLCommons · MLPerf Power benchmark presentado en IEEE HPCA 2025 — https://mlcommons.org/2025/03/ml-commons-power-hpca/
SPEC Updates PTDaemon Interface (GlobeNewswire, febrero 2024) — https://www.globenewswire.com/news-release/2024/02/22/2833367/0/en/SPEC-Updates-PTDaemon-Interface-to-Meet-Evolving-Industry-Requirements.html
Yokogawa WT310E Power Analyzer — https://tmi.yokogawa.com/us/solutions/products/power-analyzers/

Measuring energy in production: Kepler, DCGM and the practical stack (accuracy and overhead)

Sun, 14 Jun 2026 05:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Third article of the energy track (C3). C1 made the conceptual inventory of the tools; here we get practical: how the measurement stack is deployed on Kubernetes, where each piece takes its figures from, how accurate it is against a power meter, how much overhead it adds, and how energy is attributed per pod and per MIG instance. It is the equivalent of B2/B3 for energy: to stop talking about what each tool measures and start building the bench that measures it. With commands; no recommendations, only the mechanics.

The measurement stack on Kubernetes

To measure energy per workload on a GPU cluster, the practical stack has four pieces, and the good news is that three of them you already have if you do observability:

The new piece compared with a normal observability stack is Kepler, which adds energy attribution per pod. DCGM already gives GPU power; Prometheus stores it; Grafana draws it. It is reused infrastructure, not a new system, in line with GPU observability via DCGM.

DCGM exporter: the base of GPU power

GPU power, the dominant source in LLM inference, comes from NVIDIA’s dcgm-exporter (part of the GPU Operator), which exposes DCGM metrics to Prometheus. The key metric is DCGM_FI_DEV_POWER_USAGE (instantaneous power in W), alongside utilisation (DCGM_FI_DEV_GPU_UTIL) and memory. With that, energy is the integral of power over time across the workload’s window. DCGM gives power at card level; to split it per pod or per MIG instance you need a layer on top, which is Kepler.

Kepler in depth: eBPF and the sources

Kepler (Kubernetes-based Efficient Power Level Exporter) is a Prometheus exporter that measures energy at container, pod and node level, using eBPF to keep overhead low and scientific methods to improve accuracy (Kepler · GitHub). Its strength is that it combines several power sources:

Source	What it covers
RAPL	CPU and DRAM
NVML	NVIDIA GPU power
ACPI	platform power management
Redfish / IPMI	platform power (BMC)
Regression models	estimation when there is no real metric

For the GPU, the dcgmEndpoint parameter points at the dcgm-exporter metrics URL, which enables power attribution per MIG instance (Kepler · configuration). That is: DCGM gives the card’s power, Kepler splits it between the pods (and MIG partitions) that use it, via eBPF and models. The result is per-pod energy metrics in Prometheus, ready for Grafana, usable for scheduling or scaling, consumption reporting and visualisation (Kepler · GitHub).

Deployment: a DaemonSet on top of Prometheus

The deployment is a DaemonSet (one Kepler per node), pointing at the existing Prometheus and at the dcgm-exporter:

# Helm, pointing at the dcgm-exporter for GPU attribution
helm install kepler kepler/kepler \
 --namespace kepler --create-namespace \
 --set dcgmEndpoint="http://dcgm-exporter:9400/metrics"

Prerequisites: a kernel with eBPF, the dcgm-exporter running (from the GPU Operator) and Prometheus scraping. Once deployed, Kepler exposes metrics such as kepler_container_joules_total per pod, queried with PromQL and drawn in Grafana. Energy per workload stops being a manual estimate and becomes a continuous time series, just like cost with OpenCost.

PromQL queries: energy per pod

With Kepler exporting, these are the queries that build the panels. The exact names vary by version; check your /metrics:

# Cumulative energy per pod (joules)
kepler_container_joules_total

# Mean power per namespace over the last hour (W)
sum by (namespace) (rate(kepler_container_joules_total[1h]))

# GPU power per node (from DCGM, W)
sum by (node) (DCGM_FI_DEV_POWER_USAGE)

# Energy per token: power ÷ throughput (with a gateway/engine metric)
sum(rate(kepler_container_joules_total[5m])) / sum(rate(vllm_generation_tokens_total[5m]))

The last one closes the circle with the cost track: it crosses Kepler’s energy with the engine/gateway tokens to give J/token in real time, per workload. With that in Grafana you have energy per pod, per namespace and per token, continuously, the energy equivalent of the OpenCost cost report.

A calibration procedure

Since Kepler estimates, it is worth calibrating once per node type. A simple procedure:

Power meter at the node input (or a PDU/BMC reading), measuring the node’s real power.
Controlled load: a stable sweep at a known power (the engine at 80 % GPU, say).
Compare: the node’s sum of kepler_container_joules_total plus idle against the meter reading over the same window.
Adjust Kepler’s factor or model if the deviation is noticeable, and document the residual error.
Repeat for each node type (calibration is not transferable between different hardware).

With that residual error documented, Kepler’s continuous figure has a known error bar, which is exactly what a CSRD auditor or a technical committee will ask for. Measuring without calibrating gives a number with no credibility; calibrating once turns the continuous estimate into a defensible figure.

Accuracy: estimation vs power meter

The data honesty of the energy track: Kepler estimates (eBPF + models + counters), it does not measure with a power meter. Accuracy depends on the source: with NVML/DCGM and RAPL available, the figure is fairly faithful (it reads hardware counters); when it falls back on regression models (with no real metric), it is an estimate with more error. Good practice: calibrate against a physical power meter at least once per node type, adjust, and use Kepler as a continuous proxy. The meter is the point-in-time ground truth; Kepler, the continuous series calibrated against it. For a CSRD report that an auditor may challenge, that initial calibration is what gives the continuous figure credibility.

Overhead: the cost of measuring

Measuring energy has its own compute cost, worth budgeting for. In a Kubernetes environment, the overhead of the monitoring system is split between the Prometheus server and the targets: the Prometheus server consumes ~150 % of CPU (1.5 cores) and ~15 GB of RAM, while the targets (node exporter and dcgm-exporter) together use ~35 % of CPU and ~300 MB of RAM (arXiv · Container-level Energy Observability). Kepler uses eBPF precisely to keep its own overhead low. The reading: the bulk of the cost of measuring is in Prometheus (which you probably already have for observability), not in the energy exporters. Adding Kepler to an existing observability stack is marginal; standing up Prometheus just for this would be expensive. That is why measuring energy is nearly free if you already observe, and a project if you do not.

Zeus: for the bench and for optimisation, not for continuous production

Zeus (ml.energy) plays a different role: it is not a continuous production exporter, it is the bench tool for measuring and optimising the energy of one specific workload (a benchmark, a training run), with power capping and frequency selection. It fits into the benchmarking flow (alongside GuideLLM/AIPerf): during a sweep, Zeus measures the energy per configuration and lets you test the effect of power capping on performance. The practical division: Kepler/DCGM in production (continuous per-pod attribution), Zeus on the bench (fine measurement + optimisation of one workload), CodeCarbon if you want carbon context integrated into an ML pipeline.

CodeCarbon and Scaphandre in practice

The other two pieces, with their specific role:

CodeCarbon integrates inside the code of an ML pipeline (a decorator or a context manager wrapping the training or the inference), and returns the energy and the CO₂eq of that block, using nvidia-smi for the GPU and estimating CPU/RAM. It fits when you want the carbon figure per experiment or per ML job, integrated into the data flow, not at infrastructure level. It is the closest to the power meter among the Python tools.
Scaphandre is a metrology agent in Rust that reads RAPL and gives process/VM power with raw hardware accuracy. For CPU it is very accurate; for GPU it has to be combined with NVML. It fits when you need the most faithful CPU metrology, or a lightweight agent independent of Kubernetes.

The division with Kepler: Kepler attributes energy per pod in the cluster (the infrastructure view); CodeCarbon gives carbon per ML job (the pipeline view); Scaphandre gives raw CPU metrology. They do not compete; they cover different layers, and a mature platform uses whichever matches the question.

Choosing the stack by objective

Objective	Tool	Why
Continuous per-pod attribution (production)	Kepler + DCGM	eBPF, per pod/MIG, into Prometheus
Fine measurement + optimisation (bench)	Zeus	measures and optimises (power capping)
Carbon in an ML pipeline	CodeCarbon	integrated CO₂eq, close to the power meter
Raw CPU accuracy (RAPL)	Scaphandre	hardware metrology
Calibration / ground truth	power meter	the standard everything is calibrated against

It is not “one tool”, it is a stack by objective: continuous production (Kepler/DCGM), bench (Zeus), reporting (CodeCarbon), calibration (power meter). Mixing them according to the phase is what gives a complete and credible energy measurement.

From figure to action

Measuring is the start; the value is in acting. With energy per pod in Prometheus:

Reporting (CSRD): the carbon footprint per workload, calibrated, ready for regulatory reporting.
Energy/carbon-aware scheduling: move deferrable load to hours or nodes of lower intensity (connects with the hourly carbon of the Spanish grid).
Power capping: with Zeus, trim peak power with little performance loss, lowering the W in the numerator of J/token.
Optimisation: identify pods with poor energy efficiency (a lot of energy per token) and attack them with quantisation or batching.

The measurement stack is not a panel to look at: it is the input to scheduling, optimisation and reporting decisions. Just as OpenCost closes the cost loop, Kepler+DCGM closes the energy one.

From energy to carbon: the final layer

The energy figure (kWh) becomes carbon by multiplying by the grid intensity, and here the practical stack connects with the hourly source. In Spain, the intensity is published by Red Eléctrica (esios) and aggregated by ElectricityMaps, hour by hour. The pattern:

# Carbon per workload: energy (kWh) × grid intensity (gCO2/kWh)
(sum by (namespace) (increase(kepler_container_joules_total[1h])) / 3.6e6)
 * on() group_left() grid_carbon_intensity_gco2_kwh

where grid_carbon_intensity_gco2_kwh is a metric you feed from esios/ElectricityMaps (an exporter or a periodic push). With that, the Grafana panel shows gCO₂ per namespace and per token, hour by hour, not an annual average. And since Spanish intensity varies a lot (from ~80 in solar hours to ~250 on gas-heavy nights), that hourly resolution is what enables carbon-aware scheduling: moving the deferrable work to the clean hours lowers the measured carbon. The measurement stack does not only report carbon; with the hourly figure, it allows it to be reduced.

Energy efficiency KPIs to watch

KPI	What it indicates	Where from
J/token per model	energy efficiency	Kepler ÷ engine tokens
Mean power per pod	consumption per workload	`rate(kepler_container_joules_total)`
gCO₂/token (hourly)	real footprint	energy × grid intensity
Idle energy	waste	power of pods with no traffic
Effective PUE	DC overhead	total energy ÷ compute energy

These five turn the energy panel into a decision tool: which model is more efficient, which pod wastes, when it pays to schedule. Without them the stack is a pretty panel; with them, it is the input to optimisation and reporting.

Example: energy attribution on a multi-tenant cluster

With Kepler splitting per pod, an energy report per team on the example node (8×H100, 5.6 kW of boards, ~7.84 kW with PUE), over one month:

Team	Energy/month (kWh)	gCO₂ (ES grid ~160)	Mean J/token
A · chat-prod	~3,100	~496 kg	~2.2 J/tok
B · batch	~1,400	~224 kg	~2.8 J/tok
C · experimentation	~900	~144 kg	~6.5 J/tok
idle	~250	~40 kg	—

What it reveals: A is efficient (FP8, high utilisation → 2.2 J/token); C is inefficient (6.5 J/token, underused GPU), just as it dragged the idle cost in the FinOps track. The carbon column takes the figure into the per-team CSRD report. And the idle row (~250 kWh, ~40 kg CO₂) is pure wasted energy. Without Kepler, that split does not exist: you would only see the node’s total consumption, with no idea whose each watt is. With it, energy is attributable and actionable, exactly like cost with OpenCost. In fact, the two panels together (OpenCost cost + Kepler energy) give the complete picture of efficiency per team.

Energy and carbon-aware scheduling

The highest-return action the stack enables: scheduling by energy and carbon. With energy per pod and hourly grid intensity in Prometheus, two policies:

By carbon: move the deferrable load (training, ingestion, overnight batch) to the hours of lowest grid intensity. In Spain, the hours of peak solar (~80 gCO₂/kWh) against gas-heavy nights (~250). The same workload emits ~3× less when scheduled at the right hour.
By efficiency: consolidate workloads to eliminate idle (a powered GPU with no work is wasted energy), and direct each workload to the hardware most efficient for it.

Both use the same data (energy per pod + hourly intensity) that the stack already exposes, and they connect with cluster scheduling (Kueue/Volcano). It is the difference between measuring carbon (and reporting it) and reducing it (by scheduling). The C3 stack enables both.

Deployment checklist

Step	Action	Verification
1	dcgm-exporter (GPU Operator)	`DCGM_FI_DEV_POWER_USAGE` in Prometheus
2	Kepler DaemonSet, kernel with eBPF	`kepler_container_joules_total` per pod
3	`dcgmEndpoint` pointing at the exporter	GPU/MIG attribution works
4	Grid intensity metric (esios)	`grid_carbon_intensity` in Prometheus
5	Calibration vs power meter (once per node type)	error bar documented
6	Grafana panels (J/token, carbon, idle)	KPIs visible

Step 5 (calibration) is the one most often skipped and the one that gives the figure credibility; step 3 (dcgmEndpoint) is the one most often forgotten, and without it the GPU is not attributed.

Energy by phase: prefill vs decode

A level of detail the stack allows and that guides optimisation: power is not constant during a request. Prefill (processing the prompt) is compute-bound and draws a lot of power in a burst; decode (generating tokens) is memory-bound and draws a lower peak but for longer. With fine DCGM sampling that bimodal profile is visible, and it has consequences:

A workload with long prompts (a lot of prefill) has a different power profile from one with long generation (a lot of decode), even if they produce the same tokens.
Power capping affects each phase differently: trimming peak power mostly touches the prefill.

For most cases the mean J/token is enough; to optimise energy efficiency thoroughly, separating prefill and decode consumption (aligning the DCGM window with each phase) reveals where the spend is. It is the energy equivalent of separating TTFT and TPOT in benchmarking, the same principle, measuring the two phases separately because they behave differently.

State of the art: the LLM resolution gap

An honest note on the limits of current tooling. The general-purpose tools (Kepler, CodeCarbon, Scaphandre) measure well at machine, process and pod level, but they lack LLM-specific resolution: out of the box they do not distinguish prefill from decode energy, nor energy per token by phase, nor the energy cost of a reasoning block against a normal response. It is an open gap in the field: per-pod attribution exists, but attribution per token and per inference phase has to be built by crossing power (DCGM/Kepler) with engine metrics (vLLM exposes tokens per phase). The 2026 state of the art is a mature stack for infrastructure and an emerging one for LLM-specific resolution, exactly the crossing that the reproducible harness (S4) has to build by hand. Acknowledging it avoids promising a per-token accuracy that the general-purpose tools do not deliver on their own.

Limits and traps (data-driven)

Uncalibrated estimation. Kepler estimates; without calibrating against a power meter at least once, the figure can carry a noticeable error, especially when it falls back on regression models.
Forgetting the dcgmEndpoint. Without pointing Kepler at the dcgm-exporter, GPU (and MIG) attribution does not work; it is the key parameter for GPU.
Accounting for overhead badly. The bulk of the cost of measuring is in Prometheus; if you stand it up just for energy, budget for it (1.5 cores, 15 GB).
Confusing bench and production. Zeus measures one workload finely; Kepler attributes continuously in production. Using one for the other gives the wrong figure.
Misaligned windows. For energy per token, the power window and the token window must coincide; if they do not, the J/token does not correspond.

With the measurement stack in place, the energy track has its instrumentation; the following articles (C6 carbon, C8 TCO and regulation) build on this data. The foundation is this: a continuous per-pod measurement, calibrated, on top of the observability infrastructure you already have.

Closing

Measuring the energy of an AI cluster in production does not require a new system: it requires adding Kepler, per-pod attribution via eBPF, to the DCGM, Prometheus and Grafana stack you already have for observability. That is the central idea of C3: the cost of measuring is almost entirely in the Prometheus you already pay for, and Kepler turns it from “card power” into “energy per pod, per team and per token”. With two cautions that separate a credible figure from a decorative panel: calibrate once against a power meter (to have an error bar that survives an audit) and point the dcgmEndpoint (without which the GPU is not attributed). Done that way, energy stops being the soft axis of sustainability and becomes an actionable time series: the J/token per model that exposes inefficiency, the hourly carbon that enables clean scheduling, the idle that gets recovered, and the CSRD report that holds up. For the sovereign platform, the Kepler+DCGM stack closes the energy loop just as OpenCost closes the cost one, and together, with GuideLLM’s goodput, they give the three faces of the dashboard measured on the same iron, in euros, joules and grams of CO₂.

Sources

Kepler · GitHub (CNCF, eBPF, por pod/nodo) — https://github.com/sustainable-computing-io/kepler
Kepler · configuración (fuentes, dcgmEndpoint para MIG) — https://github.com/sustainable-computing-io/kepler/blob/main/docs/user/configuration.md
CNCF · explorando Kepler (consumo de aplicaciones cloud) — https://www.cncf.io/blog/2023/10/11/exploring-keplers-potentials-unveiling-cloud-application-power-consumption/
Red Hat · Kepler: monitorización eficiente de energía para Kubernetes — https://next.redhat.com/2023/08/22/introducing-kepler-efficient-power-monitoring-for-kubernetes/
arXiv · Container-level Energy Observability in Kubernetes (overhead) — https://arxiv.org/pdf/2504.10702
NVIDIA · dcgm-exporter — https://github.com/NVIDIA/dcgm-exporter
Zeus Project (ml.energy) — https://ml.energy/zeus/

Energy per token in Spain: methodology, the electricity market and the cases of speculation

Sun, 14 Jun 2026 03:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The whole article is focused on the Spanish electricity market (OMIE). The dollar symbol is not used (on this site it is a formula delimiter).

What this article covers

This is the second article of the energy track (C2), and deliberately longer, because it puts together two things that are almost never told together: the technical methodology of measuring energy per token, and the reality of the Spanish electricity market that puts a price on that energy. For an inference platform in Spain, the J/token figure is half the story; the other half is that those kilowatt-hours are bought in a market that is marginal-priced, volatile and with documented cases of manipulation. Anyone who sizes the electricity cost of a cluster without understanding how the price is formed in OMIE, and why it has tripled in a year, is building 30–50 % of the TCO on sand. This article covers the measurement, the market, the cases of speculation and the strategies for not being left exposed. No partisan opinions: data, market mechanics and documented facts, with their sources.

The identity: energy per token, and why the price rules

Energy per token comes from the identity set out in the opening article:

$$\text{energy per token (J)} = \frac{\text{average power (W)}}{\text{throughput (tok/s)}}$$

And the electricity cost per token is that energy multiplied by the price of electricity:

$$\text{electricity cost per token} = \text{energy per token (kWh)} \times \text{PUE} \times \text{price (€/kWh)}$$

Here is the point of this article: the first two factors (energy and PUE) are controlled by your engineering; the third, the price, is set by the Spanish electricity market, and it is the one that moves most. The same workload, with the same efficiency, can cost three times as much in electricity depending on the month, not because of the technology but because of OMIE. That is why measuring energy per token well is necessary but not sufficient: you have to understand the price variable that multiplies it.

Methodology: measuring energy per token without fooling yourself

Before the market, the measurement. As we saw in the energy introduction, the energy-per-token figure has four instrumentation decisions that change the result:

Decision	Trap	Good practice
Sampling frequency	coarse sampling misses the prefill peaks	sample finely (sub-second)
Board vs node	the GPU alone underestimates by 10–20 %	measure the whole node or declare the limit
Time window	including warm-up or shutdown distorts the J/token	align the window with the measured workload
Idle baseline	attributing or subtracting changes the number	decide and document it

The golden rule: the same window for the power and for the tokens. If DCGM integrates the power of one interval and the benchmark tool counts the tokens of another, the J/token means nothing. With that settled, you already have the energy per token; the rest of the article is about what multiplies it.

The Spanish electricity market: how the price is formed

In Spain (and Portugal), the wholesale price of electricity is set by OMIE (Operador del Mercado Ibérico de Energía), which runs the day-ahead market and sets the price for each hour (OMIE). The system is marginal-priced (pay-as-clear): each hour supply (the plants, ordered from cheapest to most expensive) and demand are matched, and the price for everyone is that of the last plant to enter, the marginal one. That marginal price is the same across all of Spain (except the Canaries) and applies to all agents regardless of their initial bid (Energía y Sociedad).

The consequence, central to understanding the bill: when gas is the marginal plant (the usual case in hours without sun or wind), renewables, nuclear and hydro, dirt cheap to produce, are paid the price of gas. That difference between production cost and price received is what are called “windfall profits”, the heart of the debate about market design. For the buyer of electricity, a datacenter for example, it means that the price paid does not reflect the cost of generating, but that of the most expensive technology needed that hour.

The volatility: the price that triples in a year

The marginal-priced market, tied to gas in the expensive hours, produces enormous volatility. The 2026 data illustrates it: the Spanish wholesale price closed May 2026 with an average of 54 €/MWh, more than triple the 16.92 €/MWh of the same month in 2025 (Merca2). Tripling in a year, without your consumption or your efficiency changing.

And within a single day, the dispersion is brutal because of the effect of renewables: solar and wind enter at prices close to zero, and in the midday hours of May 2026 prices reached negative levels in several sessions (Merca2). The structural context: wind and solar went from 26 % of generation in 2019 to more than 40 % in the first half of 2024, and over that period the wholesale price was more than 40 % lower than it would have been with 2019 renewable levels (Banco de España).

Reference	Value	Source
Wholesale May 2026	54 €/MWh	Merca2
Wholesale May 2025	16.92 €/MWh	Merca2
Year-on-year variation	+219 % (×3.2)	derived
Solar hours 2026	prices close to 0 or negative	Merca2
Renewables in generation	26 % (2019) → >40 % (2024)	Banco de España

The reading for a cluster: the electricity part of the TCO is not a fixed number, it is a variable with a range of 3× year on year and from zero-price hours to peak hours within the day. Sizing the energy cost with an annual average price and forgetting about it is ignoring the biggest risk in the model.

The structure of the bill: wholesale is not what you pay

It is worth not confusing the wholesale price (the OMIE pool price, the one the headlines talk about) with what a company actually pays. On top of the wholesale price come:

Component	What it is	Nature
Energy (wholesale)	the OMIE price hour by hour	variable, volatile
Access tolls	use of the transmission and distribution networks	regulated
Charges	system costs (past renewables, etc.)	regulated
Taxes	IEE (special electricity tax) + VAT	fiscal

The result: the industrial retail price a datacenter pays is well above the wholesale price, since tolls, charges and taxes can double the price of pure energy. That is why this article always distinguishes between the wholesale price (~0.054 €/kWh in May 2026) and industrial retail (~0.12 €/kWh): for the TCO the second one matters, but the first is the one that moves and drags the second with it. A large company can access the wholesale market more directly (via a supplier or a representative), reducing the mark-up, but taking on the exposure to volatility that comes with it.

There is also the PVPC (Precio Voluntario al Pequeño Consumidor), the regulated tariff indexed to the wholesale price for small consumers; a datacenter does not use PVPC, but its existence explains why the volatility of the pool reaches public opinion and political debate directly, and, with it, the regulation that ends up affecting everyone.

The 2021–2022 crisis: the origin of the problem

Today’s volatility is not new: it originates in the gas crisis of 2021–2022. When the price of gas shot up (the marginal plant in many hours), the marginal-pricing system passed that price on to all electricity, including that generated by cheap renewables, nuclear and hydro. The result was wholesale prices that were several times the historical levels and massive “windfall profits” for generators of cheap technologies that were paid the price of gas. That episode is what motivated both the Iberian exception (capping gas in price formation) and the European debate about redesigning the market, and it explains why, years later, the price is still a first-order risk variable for anyone consuming a lot of megawatt-hours. For a multi-year TCO model, the lesson is that the Spanish electricity price has shown it can multiply within months through external factors (gas geopolitics), and that no assumption of a “stable” price survives an episode like that unless it is covered by contract.

The cases of speculation and manipulation

The volatility of marginal pricing creates incentives to manipulate the price, and in Spain there are documented cases with final CNMC fines. The largest sanctions on utilities for manipulating the wholesale market (Nada es Gratis):

Company	CNMC fine	Concept
Iberdrola	25 million €	manipulation of the wholesale price via hydro plants
Naturgy	19.5 million €	manipulation of the wholesale market
Endesa	5.8 million €	manipulation of the wholesale market

The Iberdrola case is the most illustrative of how marginal pricing is gamed: between 30 November and 23 December 2013, the company raised the prices of several hydro plants so that they would not clear despite the high prices in the day-ahead market (Nada es Gratis). The mechanism: withhold cheap water (not bidding it, or bidding it at a prohibitive price) so that the clearing needs a more expensive plant, pushing up the marginal price paid to your whole portfolio. In a pay-as-clear system, withdrawing cheap capacity raises the price of everything you sell that hour; the incentive to manipulate is embedded in the design.

OMIE has supervision mechanisms and carries out audits to prevent manipulation (Energía y Sociedad), and the CNMC sanctions it when it is detected; the fines above are the proof that it happens. For the buyer, the relevant point is not to assign blame, but to recognise that the price paid is formed in a market where manipulation is possible and has happened, and that therefore exposure to that price is a risk to manage, not a figure to accept.

The other speculation: hoarding grid capacity (and the new 2026 law)

There is a second form of speculation, distinct from price manipulation, that affects directly anyone who wants to build a datacenter: the hoarding of access and connection permits to the grid. For years, many developers reserved connection capacity at grid nodes for projects (mostly renewable) that had no real intention or capacity to build, blocking that capacity for firm projects and speculating on the value of the permit. Since an inference platform is a demand facility that needs its connection point, this bottleneck hits it squarely: grid capacity is a scarce and hoarded resource.

The regulatory response of 2026 targets exactly that. The Government (Ministry for the Ecological Transition) finalised a set of rules, within the framework of Royal Decree-Law 7/2026, to free up grid capacity and fight speculation, ensuring that firm requests can connect (El Periódico de la Energía, Fieldfisher). The central mechanism, with effect from 22 March 2026:

Element	Detail
Who pays	holders of access and connection permits for demand facilities
What they pay	a monthly charge for “reserving” capacity until entering operation
How it is calculated	transmission and distribution tolls × a “k factor”
Modulation	by voltage level and by the delay in commissioning
k factor	to be set by resolution of the Secretary of State for Energy

The logic: if reserving capacity costs every month and the cost grows with the delay, hoarding stops being free and the speculator releases the capacity it is not going to use. For a real datacenter project the implication is twofold: (1) that charge on the reserved capacity has to be budgeted during construction, and (2) there is a strong incentive to build quickly, since every month of delay in starting up adds to the charge. In other words, the new law turns commissioning time into an explicit cost. For the TCO model and the timetable of the proposal, this is a figure to build in from the start: grid capacity is no longer reserved “just in case” for free, and the construction schedule now has a bill attached.

The design debate: why marginal pricing rewards speculation

The underlying problem is not just that some companies manipulate: it is that the market design rewards doing it. In a pay-as-clear system (everyone is paid the marginal price), a generator with a lot of cheap capacity has a perverse incentive: withdrawing part of its cheap capacity raises the marginal price it is paid for all the capacity it does sell. If you control enough market share, the extra revenue from raising the price of your whole portfolio can exceed what you lose by not selling the withdrawn capacity. That is exactly what the CNMC sanctioned in the hydro case.

The design alternatives under debate:

Design	How it pays	Effect on the incentive to speculate
Pay-as-clear (current)	everyone is paid the marginal price	high: withdrawing capacity raises the price of everything
Pay-as-bid	each one is paid its own bid	lower incentive, but strategic bidding
Long-term contracts / PPA	fixed bilateral price	takes volume out of the pool, reduces exposure

There is no technical consensus that pay-as-bid is better (it introduces its own problems of strategic bidding), and the reform of the European electricity market has opted more for promoting long-term contracts (PPA and CfD) than for changing the pool mechanism. For the buyer, the practical conclusion is the same as for the regulator: taking your volume out of the pool through forward contracts is the most direct way of not being exposed to the marginal price, and to whoever manipulates it.

The Iberian exception: a patch with an expiry date

Faced with the gas price crisis of 2021–2022, the European Commission granted Spain and Portugal an exception allowing them to cap the price of the gas used to produce electricity (market context). The “Iberian exception” partially decoupled the electricity price from gas, containing the peaks. It is relevant for a multi-year cost projection because it is a regulatory mechanism with limited duration: any 36-month TCO model has to consider what happens to the price if the exception changes or expires. Regulation is another price variable, not just the market.

What it means for the electricity cost of a cluster

Putting together energy per token (technical) with the Spanish price (market), the electricity cost per token of an example node (8×H100, 0.78 kWh per million tokens with PUE) according to the price scenario:

Price scenario (Spain)	€/kWh	Electricity cost / 1M tokens
Solar hour / negative price	~0	~0
May 2025 (low wholesale)	0.017	~0.013 €
May 2026 (average wholesale)	0.054	~0.042 €
Industrial retail (with tolls and taxes)	~0.12	~0.094 €
Peak hour / crisis	0.20+	~0.16 €

The range is more than 10× between the solar hour and the peak hour, and 3× year on year at the level of the average. Over a fleet consuming megawatt-hours a year, that variability is the difference between a comfortable TCO and a suffocated one, and it does not depend on your technology but on when and how you buy the energy.

Strategies for not being left exposed

This is where the technical figure turns into a decision. Four levers for managing exposure to the Spanish price, from the least to the most structural:

Strategy	What it does	Trade-off
Indexed tariff	you pay the wholesale price hour by hour	cheap on average, maximum exposure to volatility
Fixed / forward tariff (OMIP)	price locked by contract	predictability, you pay a premium for the insurance
PPA with a renewable plant	you buy directly from a solar/wind farm	stable price and low carbon, requires volume and term
Solar self-consumption	you generate on site	you cover the sunlight hours almost free, capex and space
Temporal scheduling	move shiftable load to cheap hours	only applies to the non-urgent (batch, fine-tuning)

The sensible combination for an inference platform in Spain: a PPA or a fixed tariff for the base load (online inference, which cannot be moved), solar self-consumption to cover the midday hours almost free, and scheduling of the shiftable work (training, overnight batch) towards the off-peak or peak-solar hours. That turns the volatility of marginal pricing from risk into opportunity: the same zero-price hours that sink generators’ revenue are the cheap hours for whoever consumes.

Solar self-consumption: the numbers for a cluster

Spain is one of the best places in Europe for solar, and for a datacenter self-consumption has a clear cost logic. The numbers, for illustration:

Parameter	Reference value
Solar output in Spain	~1,400–1,600 kWh per kWp per year
Capex of an industrial installation	~600–900 € per kWp
Panel service life	~25–30 years
Electricity cost avoided (retail)	~0.12 €/kWh

For a 100 kWp installation (warehouse roof): it generates ~150 MWh/year, with a capex of ~70,000–90,000 €. At 0.12 €/kWh avoided, it saves ~18,000 €/year, with a payback of ~4–6 years and then ~20 years of almost free energy. The snag: a cluster consumes 24/7 and solar only produces by day, so self-consumption covers a fraction, the midday hours, not everything. But those midday hours are precisely the ones with low or negative wholesale prices, so self-consumption and the market complement each other: at night you buy cheap (off-peak), by day you generate. The avoided cost is not only that of the energy, it is that of not being exposed to the non-existent solar peak and to daytime volatility. For a cluster with roof space available, self-consumption is the most solid structural lever on electricity cost in Spain.

Annual electricity TCO: three scenarios for a cluster in Spain

Putting it all together, the annual electricity cost of an 8×H100 node (~7.84 kW with PUE, ~68,700 kWh/year at 100 % utilisation) according to the purchasing strategy, in Spain:

Strategy	Effective €/kWh	Annual electricity cost	Exposure
Indexed (unhedged)	~0.08 (variable)	~5,500 € ±a lot	maximum to volatility
Fixed / renewable PPA	~0.06 (stable)	~4,120 €	low, predictable
PPA + solar self-consumption	~0.045 (mixed)	~3,090 €	low + minimal carbon

The difference between the worst and the best strategy is ~2,400 €/year per node, before counting the risk: the indexed option can shoot up in a bad month, while PPA+solar is stable and low. Over a fleet of dozens of nodes and a 36-month horizon, the choice of how the energy is bought moves the TCO as much as a hardware decision, and it is a lever decided with contracts, not with engineering. This number feeds directly into the TCO model of the FinOps track (A8) and the sizing of the proposal.

PPAs: the central lever (and its small print in 2026)

Of all the strategies, the PPA (Power Purchase Agreement, a long-term energy purchase contract) is the most important piece for a datacenter in Spain today, and it deserves a section of its own because its current situation is exceptionally favourable… with a catch.

What a PPA is, in short

A PPA is a bilateral, long-term contract (typically 10–12 years in Spain) between a consumer and a renewable generator, at a locked price per MWh. It takes your volume out of the OMIE pool: instead of paying the marginal price hour by hour (with its volatility and its exposure to manipulation), you pay the contract price. There are two families: physical (you receive the energy from that farm) and financial/virtual (a swap covering the difference against the market); and two profile forms: pay-as-produced (you pay for what the farm generates, when it generates, cheap but intermittent) and baseload (a guaranteed flat profile, more expensive but firm).

Prices today: historic lows

Spain is living a unique moment in PPA prices. Solar PPAs fell to a historic low of 32.5 €/MWh in the fourth quarter of 2025, with the most competitive percentiles below 30 €/MWh, half the French level (pv-magazine, El Periódico de la Energía). Wind sits at around 50–60 €/MWh. To put it in context: a solar PPA at 30 €/MWh is ~0.030 €/kWh, less than half the industrial retail price (~0.12 €/kWh) and well below the volatile wholesale price.

Technology (PPA, Spain 2026)	Price	Reference
Solar photovoltaic	~32.5 €/MWh (percentiles <30)	pv-magazine
Wind	~50–60 €/MWh	market
Industrial retail (for comparison)	~120 €/MWh	—

Why so cheap: cannibalisation

The fall is no accident: it is the cannibalisation of photovoltaics. The more solar is installed, the more midday hours there are with wholesale prices close to zero or negative, which sinks the value captured by solar (captured prices have fallen below 15 €/MWh), and with it the price at which farms sign PPAs (Energía Estratégica). With more than 40 GW competing for 30 GW of demand, there is an excess of solar supply that cheapens the PPA. For the industrial consumer it is a historic opportunity; for the solar developer, a crisis of profitability, hence the talk of the “closing of the solar PPA window” and of the new Royal Decree seeking to restore industrial competitiveness.

Spain, European leader in PPAs

This is no marginal phenomenon: in 2024 Spain was the country with the most PPAs signed in Europe for the sixth consecutive year, with 47 agreements for 4.66 GW of renewable capacity (market sources). The market is deep, liquid and mature, which makes it easier for a datacenter to find a counterparty.

The catch for a datacenter: annual energy ≠ firm power

Here is the small print that decides the architecture. A cheap solar PPA offsets energy over the year, but does not guarantee physical power every hour. And an AI datacenter cannot stop because there is no wind one January night or because the solar curve falls at dusk (El Periódico de la Energía). This is the difference between energy (kWh per year, which the solar PPA covers dirt cheap) and firm power (kW available for all 8,760 hours, which the sun does not provide). The sector sums it up like this: Spain is now forced to talk about firm energy (Revista Cloud).

The consequence for the design of the proposal: a solar PPA at 30 €/MWh is the ideal cost floor for the sunlight hours, but it has to be combined to cover the rest: a hybrid PPA (solar + wind, which complement each other over time), baseload for the flat part, or storage (batteries, which RDL 7/2026 also promotes) to move energy from the sunlight hours to the night. The sensible energy architecture of a Spanish datacenter in 2026 is not “a solar PPA”, it is a cheap solar PPA as the daytime base + firmness for the hours without sun (wind, baseload, batteries or the grid as backup). Anyone who signs only the cheap solar PPA and discovers in production that they have no firm power in the small hours will have optimised the cost of the energy and ignored that of availability, which for an inference platform with an SLO is non-negotiable.

The context: the datacenter boom in Spain

All of this is happening in the middle of a boom: Spain is positioning itself as the epicentre of southern Europe in datacenters, with investments of billions and installed capacity doubling towards 2026 (DCD). The appeal combines cheap renewable PPAs, industrial land more accessible than in other European digital capitals, and EU jurisdiction. The constraint that tempers it is exactly the one in C2: the cost of energy is low but firm power and grid capacity are the bottlenecks, and both are managed with the combination of a hybrid PPA, storage and a connection schedule that does not fall foul of the hoarding charge.

The sovereign and carbon angle

For a sovereign proposal, Spain has a double advantage worth quantifying. First, carbon: the Spanish grid runs at around ~150–170 gCO₂/kWh (a mix of renewables and gas), well below Germany (~363) though above nuclear France (~20–60). Second, growing renewables (>40 % of generation), which pull the price and the carbon down in the sunlight hours. The argument for the proposal: a cluster in Spain, with solar self-consumption and a renewable PPA, can achieve at the same time a low and predictable electricity cost and a competitive carbon per token, within EU jurisdiction. The volatility of marginal pricing is not only a risk: managed well with contracting and scheduling, it is a cost lever that a hyperscaler in another region does not exploit in the same way.

Carbon, moreover, is variable by the hour: in peak-solar hours the intensity of the Spanish grid drops, so moving shiftable load to those hours reduces both the cost and the carbon per token at once. The same scheduling strategy serves both axes.

Guarantees of Origin: carbon has its market too

One nuance that connects price speculation with carbon reporting: in Spain (and the EU) there is a market in Guarantees of Origin (GoO), certificates attesting that a quantity of energy was produced from a renewable source. A company can buy GoOs to declare that its consumption is “green” in its market-based carbon reporting, even though physically the electricity it received from the grid had the average intensity of the mix. This creates two ways of accounting for carbon, with different implications:

Method	What it counts	How the figure is brought down
By location (location-based)	the real intensity of the Spanish grid that hour	generate/consume in clean hours; location
By market (market-based)	according to the GoOs/PPAs contracted	buy certificates or sign a PPA

Location-based accounting rewards consuming when the Spanish grid is clean (solar hours); market-based allows the green attribute to be “bought” with GoOs. CSRD reporting normally requires both. The word of caution: the GoO market is relatively cheap, and buying GoOs without a real PPA behind them is a way of dressing up the figure without changing the physical electricity you consume, the cost equivalent of accounting greenwashing. For an honest sovereign proposal, the robust route is a real PPA with a Spanish renewable farm (which does change the physical origin and stabilises the price) plus location-based measurement that rewards scheduling to clean hours, rather than accumulating certificates.

Hourly intensity: the Red Eléctrica figure

Spain publishes the carbon intensity of the grid hour by hour (via Red Eléctrica / esios and ElectricityMaps), and it varies enormously over the day: on a sunny spring afternoon it can drop to ~80–100 gCO₂/kWh, and on a cold, windless night, with gas cycles, exceed ~250. That hourly figure is what makes carbon-aware scheduling possible: moving training, document ingestion or overnight batch to the hours of lowest intensity reduces the carbon per token of the shiftable work at no technical cost. It is the same lever as price-based scheduling, since the cheap hours and the clean ones largely coincide in Spain, because both are set by the excess of solar and wind. Measuring carbon per token “on an annual average” wastes this information; measuring it hour by hour and scheduling accordingly is what turns the figure into a real reduction.

Energy per token, with the Spanish price, step by step

Closing the circle with a complete calculation for the example node, in Spain:

Energy per token: 5,600 W (8×H100) ÷ 2,800 tok/s = 2 J/token (board); ×PUE 1.4 = 2.8 J/token effective.
Energy per 1M tokens: 0.78 kWh (with PUE).
Electricity cost per 1M tokens by contracting model:
- Indexed in a solar hour (~0 €/kWh): ~0 €.
- Renewable PPA (~0.05 €/kWh stable): ~0.039 €.
- Industrial retail (~0.12 €/kWh): ~0.094 €.
Carbon per 1M tokens: 0.78 kWh × ~160 gCO₂/kWh = ~125 gCO₂ (and less in solar hours).

The message: the J/token is fixed (your engineering sets it); the electricity cost per token varies more than 10× depending on how and when you buy the energy in Spain. That is why this article devotes more space to the market than to the measurement: measuring is 30 % of the problem; buying well is 70 %.

Limits and traps (data-driven)

A misleading annual average price. Using an annual average hides the 3× year-on-year volatility and the 10× intraday range. Model scenarios, not a point.
Indexed by default. The indexed tariff is cheap on average but leaves you exposed to every peak and every episode of manipulation. For base load, hedge it.
Ignoring the Iberian exception. It is a regulatory mechanism with an expiry date; a 36-month TCO has to consider it changing.
Forgetting hourly carbon. Carbon and price fall at the same time in solar hours; scheduling the shiftable work improves both axes.
Confusing wholesale with what you pay. Industrial retail adds tolls and taxes on top of the wholesale price; use your contract price, not the pool price.

With the Spanish price understood, the energy track continues towards measurement tools in depth (C3) and energy in TCO and regulation (C8). But the lesson of C2 is that, in Spain, the energy cost of AI is not decided in the GPU alone: it is decided in how you buy the electricity in a market that is marginal-priced, volatile and watched for manipulation.

Closing

Energy per token is a number that engineering fixes with precision, so many joules per token, measured in a window aligned with the throughput, and that the Spanish electricity market multiplies by a factor that changes more than 10× depending on the hour and 3× depending on the year. That second half, the price one, hardly ever appears in technical AI articles, and it is what decides 30–50 % of the TCO of a cluster in Spain. The price is formed in a marginal-priced market where cheap renewables are paid the price of gas, where volatility has tripled the bill in a year, and where there are final CNMC fines, Iberdrola 25 M€, Naturgy 19.5 M€, Endesa 5.8 M€, that show that manipulation is not theoretical. For a sovereign inference platform in Spain, the conclusion the data supports is twofold: measure energy per token rigorously, and buy the electricity with your head — a PPA or fixed tariff for the base load, solar self-consumption for the sunlight hours, and scheduling of the shiftable work to the cheap, clean hours. Spain offers abundant sun, growing renewables and a reasonable average carbon figure; well contracted, its energy is a competitive advantage, and badly contracted, the biggest risk in the model. The J/token is yours to decide; the euro per token, the market’s, unless you take it out of the market with a contract.

Sources

OMIE · Operador del Mercado Ibérico de Energía — https://www.omie.es/
OMIE · precio del mercado diario — https://www.omie.es/es/market-results/daily/daily-market/day-ahead-price
Energía y Sociedad · importancia y precios de OMIE — https://www.energiaysociedad.es/omie-cual-es-su-importancia-para-el-sistema-electrico-espanol-y-portugues/
Nada es Gratis · manipulación del precio en el mercado eléctrico español (multas CNMC) — https://nadaesgratis.es/jose-luis-ferreira/manipulacion-del-precio-en-el-mercado-electrico-espanol
Merca2 · el precio de la luz en mayo 2026 (54 €/MWh, ×3 interanual) — https://www.merca2.es/2026/05/30/precio-luz-mayo-2026-disparo-2386473/
Banco de España · el sector eléctrico ante el alza del gas (renovables y precio) — https://www.bde.es/f/webbe/SES/Secciones/Publicaciones/PublicacionesSeriadas/DocumentosOcasionales/23/Fich/do2316.pdf
El Periódico de la Energía · normativa para liberar capacidad de red y combatir la especulación — https://elperiodicodelaenergia.com/el-gobierno-ultima-una-normativa-para-liberar-la-capacidad-de-las-redes-electricas-y-combatir-la-especulacion/
Fieldfisher · Real Decreto-ley 7/2026, paquete regulatorio de renovables — https://www.fieldfisher.com/es-es/locations/espana/actualidad/nuevo-paquete-regulatorio-energias-renovables
Bird & Bird · almacenamiento energético en el RDL 7/2026 — https://www.twobirds.com/es/insights/2026/spain/almacenamiento-energetico-real-decreto-ley-7-2026
pv-magazine · caída récord de los PPAs fotovoltaicos a 32,5 €/MWh — https://www.pv-magazine.es/2026/02/04/la-canibalizacion-provoca-una-caida-record-de-los-precios-de-los-ppas-fotovoltaicos-en-espana-hasta-los-325-e-mwh/
El Periódico de la Energía · PPAs solares cerca de 30 €/MWh (mitad que Francia) — https://elperiodicodelaenergia.com/la-canibalizacion-acerca-el-precio-de-los-ppas-solares-a-30-mwh-en-espana-la-mitad-que-en-francia/
Energía Estratégica · cierre de la ventana de PPAs solares y el nuevo RD — https://www.energiaestrategica.com/espana-atraviesa-un-cierre-de-la-ventana-de-ppas-solares-mientras-el-nuevo-real-decreto-busca-aliviar-la-industria/
El Periódico de la Energía · la gran decisión energética de los data centers (energía firme) — https://elperiodicodelaenergia.com/la-gran-decision-energetica-de-los-data-centers
Revista Cloud · los centros de datos obligan a España a hablar de energía firme — https://revistacloud.com/los-centros-de-datos-obligan-a-espana-a-hablar-de-energia-firme
DCD · boom de data centers en España hacia 2026 — https://www.datacenterdynamics.com/es/features/data-centers-en-espa%C3%B1a-hacia-2026-un-boom-de-90000-millones-en-proyectos/

Cost, performance and energy: the three axes of LLM inference and the identity that ties them together

Sat, 13 Jun 2026 03:30:00 +0200

Notation: amounts in euros (N €), decimals with a point. Cost and energy references are European (France, Germany, Spain), because the aim is to defend a sovereign European architecture; when a source quotes prices in dollars it is flagged as “USD”. The dollar symbol is not used (on this site it is a formula delimiter).

What this series is

This is the first instalment of a series of 28 data articles (not opinion pieces) about the three dimensions that decide whether a sovereign European LLM platform is sustainable: cost (FinOps), performance (benchmarking) and energy. The aim is to have a body of verifiable figures, with their methodology, to support architecture decisions with numbers rather than intuition. This first article sets the frame: what each axis measures, how they link to one another, an end-to-end worked example with European data and the 2026 references that will be used as a base.

The quantitative thesis, in one sentence: the three axes are not independent. Throughput (tokens per second) is the common denominator of cost per token and energy per token, and energy is, on top of that, between 30 % and 50 % of the cost itself. Optimising one moves the other two. And in Europe, where the cluster sits (France, Germany, Spain) changes both the electricity cost and the carbon per token at once.

The three axes and their metrics

Axis	Question	Key metrics	Unit	Tool family
Cost (FinOps)	How much does each unit of work cost?	cost/1M tokens (CPM), cost/request, €/GPU-hour, utilisation	€/1M tok, €/h	OpenCost, Kubecost, CloudZero
Performance	How much work per second and at what latency?	TTFT, TPOT/ITL, throughput, goodput, P50/P99	tok/s, ms	GuideLLM, AIPerf, LLMPerf, MLPerf Inference
Energy	How many watts per unit of work?	energy/token, average power, PUE, carbon	J/tok, Wh, kWh, gCO₂	Zeus, CodeCarbon, MLPerf Power, Kepler

Each axis has its own in-depth introductory article in this same batch; here only how they fit together matters. It is worth breaking each one down a little more before crossing them.

Cost axis in detail

Metric	Definition	Why it matters
CPM (cost/1M tokens)	cluster cost ÷ tokens produced	the comparable unit on-prem vs cloud
Cost per request	cost attributed to one complete request	billing by real usage
€/GPU-hour	hourly cost of a GPU (amortised or rented)	the basis of any calculation
Utilisation	fraction of the GPU actually used	the lever that moves everything else
Fixed vs variable cost	amortised capex vs energy/operations	defines the break-even point

Performance axis in detail

Metric	Definition	Dominant phase
TTFT	latency to the first token	prefill
TPOT / ITL	time between output tokens	decode
Throughput	tokens or requests per second of the system	both
Goodput	throughput that meets the SLO (not the raw figure)	the honest number
P95 / P99	latency percentiles (not the mean)	the real experience

Energy axis in detail

Metric	Definition	Unit
Average power	average instantaneous consumption	W
Energy per token	energy per generated token	J/token
Total energy	integral of power over time	Wh, kWh
PUE	datacenter overhead (cooling, losses)	ratio ≥ 1
Carbon	energy × grid intensity	gCO₂eq

The identity that ties the three axes together: throughput

The cost per million tokens (CPM) is the cluster’s cost divided by what it produces:

$$\text{CPM} = \frac{\text{cluster cost per hour}}{\text{throughput (tok/s)} \times 3600 / 10^6}$$

Energy per token has exactly the same form, with power in the numerator:

$$\text{energy per token (J)} = \frac{\text{average power (W)}}{\text{throughput (tok/s)}}$$

The two share the same denominator: throughput. Hence the key operational consequence: every performance improvement that raises tokens/s lowers both cost per token and energy per token at the same time. These are not three separate optimisations; throughput moves them together. That is why performance benchmarking (axis 2) is, at bottom, also a FinOps tool and an energy efficiency tool.

The CPM↔throughput relationship is direct: the higher the throughput, the lower the CPM. It is the identity on which the whole cost analysis of the series is built.

Worked example: unit economics of an 8×H100 node in Europe

To see the identity in action, an end-to-end calculation on an example node (8×H100 SXM, Llama 3.1 70B FP16 served with vLLM), with European prices. Order-of-magnitude figures; the methodology is what matters.

Starting data:

Parameter	Value	Source
European cloud rental (Scaleway, per H100)	2.73 €/h	Scaleway
→ 8×H100 node on-demand	~21.84 €/h	8 × 2.73
Amortised on-prem cost (estimate)	~10–13 €/h	capex 24–36 months + opex
Throughput	2,800 tok/s	reference benchmark
Power per H100 (TDP)	up to 700 W	NVIDIA
Reference PUE	1.4	typical of a datacenter

Cost per million tokens. At 2,800 tok/s, hourly throughput is 2,800 × 3,600 / 10⁶ = 10.08 million tokens/hour. Therefore:

$$\text{CPM (cloud)} = \frac{21{.}84}{10{.}08} \approx 2{.}17 \text{ €/1M tok}$$

With the amortised on-prem cost (~11 €/h) it drops to ~1.09 €/1M tok, the difference between renting from a provider and amortising your own iron.

Energy per token. The power of the 8 cards is 8 × 700 = 5,600 W (board only). Over 2,800 tok/s:

$$\text{energy per token} = \frac{5,600}{2,800} = 2 \text{ J/token (board)}$$

Over a million tokens that is 2 × 10⁶ J = 0.56 kWh (GPU only). Applying PUE 1.4: ~0.78 kWh per million tokens. At French wholesale prices (~0.058 €/kWh) that is ~0.045 € of electricity per million tokens; in Germany (~0.091 €/kWh), ~0.071 €, some 57 % more, because of the grid, not the model.

The link. If a serving optimisation raised throughput from 2,800 to 4,200 tok/s (+50 %), the CPM would fall to ~1.45 €/1M tok and energy per token to ~1.33 J, both at once, through the same denominator. That is the identity, with numbers.

Energy is inside the cost

The second link: energy is not an axis separate from cost, it is a large component of it. 2026 reference data:

Figure	Value	Source
Electricity as a share of TCO (5 years)	30–40 % (>50 % at high density 24/7)	Spheron
Day-ahead wholesale France (Mar-2026)	~58 €/MWh (0.058 €/kWh)	TradingEconomics
Day-ahead wholesale Spain	~54 €/MWh (0.054 €/kWh)	TradingEconomics
Day-ahead wholesale Germany	~91 €/MWh (0.091 €/kWh)	TradingEconomics
Electricity cost/token at 80 % util	¼ of that at 20 % util	Spheron

A methodological warning: the wholesale price (day-ahead) is not what a company pays; industrial retail adds grid charges and taxes (roughly ×1.5–2). But the relative ordering holds: France and Spain, cheap; Germany, expensive.

The third link, utilisation, closes the triangle: power is consumed even when the GPU is idle, so spreading the fixed cost (energy included) over more tokens (more throughput, more occupancy) lowers all three axes at once. The electricity cost per token at 80 % utilisation is a quarter of what it is at 20 %.

Sensitivity analysis: what moves the cost per token

Four levers dominate the CPM, and it is worth knowing their elasticity before defending any figure:

Lever	Change	Effect on CPM	Mechanism
Throughput	+50 % tok/s	−33 % CPM	direct denominator
Utilisation	20 % → 80 %	up to −75 % on the electricity portion	spreads fixed cost
Location (energy)	Germany → France	lowers the electricity portion (~37 % less €/kWh)	wholesale FR < DE
Amortise vs rent	21.84 → 11 €/h	−50 % CPM	removes the provider’s margin
Quantisation (FP8/INT4)	+throughput, −VRAM/KV	lowers CPM and J/token	same identity

The reading: the CPM is not a number, it is a function of utilisation, throughput, location (energy price) and ownership model. Any on-prem vs cloud comparison that does not fix those assumptions is not comparable, and that is exactly the job of the synthesis article.

Simplified TCO model (on-premise)

The real on-premise cost is not the GPU alone: it is amortised capex + opex, over a hardware cycle of 24–36 months. Reference structure of an inference node:

Component	Type	Typical weight	Notes
GPUs	capex	the bulk	amortise over 24–36 months
Server, network, BMC	capex	medium	NICs, switching, out-of-band management
Electricity	opex	30–50 % of TCO	depending on country (FR/ES cheap, DE expensive) and PUE
Cooling	opex	inside the PUE	PUE 1.2–1.6 typical
Maintenance and support	opex	medium	spares, contracts, RMA
Staff and operations	opex	variable	the hidden cost that cloud rental includes

The difference between cloud rental (21.84 €/h in the example) and the real amortised cost (~10–13 €/h) is this breakdown: the provider charges a margin for absorbing capex, cooling and operations. The break-even point appears when sustained volume exceeds ~2M tokens/day, with typical amortisation in 6–12 months (Spheron). Below that threshold, renting usually wins; above it, amortising. Article A8 develops the full model and synthesis S3 connects it with sizing.

Reference: the price of a GPU-hour in European sovereign cloud

The other side of the on-prem comparison is the price of renting the GPU. For a sovereign European platform, what matters is not AWS/GCP/Azure (subject to the US CLOUD Act), but the European providers:

Provider	H100 (€/hour)	Location	Sovereignty
Scaleway	from 2.73	Paris, Warsaw	EU jurisdiction, exempt from US CLOUD Act
OVHcloud	+40–45 % vs Scaleway	France, Germany	EU jurisdiction, exempt from US CLOUD Act
Hyperscalers (reference)	3–7 USD	global	subject to US CLOUD Act

The data: Scaleway offers the H100 from 2.73 €/h in European datacenters (Paris, Warsaw), with models and data under EU jurisdiction, immune to extraterritorial regulation; OVHcloud serves it from France and Germany, ~40–45 % more expensive than Scaleway as of March 2026. Both are exempt from the US CLOUD Act, the foundational sovereignty requirement for data subject to GDPR (Scaleway, DeployBase).

The implication for the proposal: the comparison is not only cost, it is cost + sovereignty. An on-prem cluster of your own or a European cloud keep the data under EU jurisdiction; a US hyperscaler does not, however cheap the GPU-hour comes out. That is the leg that the cost axis on its own does not capture, and that synthesis S2 quantifies.

Energy and carbon reference: France, Germany, Spain

Here the choice of European country changes both the electricity cost and the carbon per token. It is the most important table for a sovereign proposal:

Country	Wholesale (€/MWh, Mar-2026)	Grid carbon (gCO₂/kWh)	Profile
France	~58	~20–60	nuclear: cheap and very clean
Spain	~54	~150–170	renewables + gas: cheap, medium carbon
Germany	~91	~363 (2024 average)	coal/gas + renewables: expensive and dirty

Sources: day-ahead wholesale prices (TradingEconomics), German carbon ~363 gCO₂/kWh (2024 average, Statista via Wikipedia), nuclear ~17–35 gCO₂e/kWh (EEA), Spain ~167 gCO₂/kWh (2020). The PUE multiplies all of that energy: at 1.4, for every kWh of GPU the datacenter spends 1.4 kWh with cooling included.

The calculation, on the example node (0.78 kWh per million tokens, with PUE): in France (~40 gCO₂/kWh) it is ~31 gCO₂ per million tokens; in Germany (~363 gCO₂/kWh), ~283 gCO₂ per million, a factor of ~9×. That is, the same load served in France emits of the order of 9× less carbon per token than in Germany, and costs ~37 % less in electricity, without touching a single line of the stack, purely because of the country’s grid. Siting the cluster in France or Spain is at once a cost lever, a carbon lever and a compliance lever. Hourly grid intensity is obtained from ElectricityMaps, the reference source for these calculations.

Scale context: global datacenters consumed ~460 TWh in 2025 (≈1.8 % of the world’s electricity), and the IEA projects passing 800 TWh in 2028, which is why energy efficiency per token stops being a detail and becomes a design constraint, and why carbon enters the European regulatory equation.

Sovereignty and regulation: the axis that does not appear in the CPM

For a European platform there is a fourth factor that the three numerical axes do not capture, and that can invalidate the lowest cost figure: jurisdiction. Three fronts that turn “the cheapest GPU-hour” into “the cheapest GPU-hour that complies”:

Front	What it requires	Architectural implication
GDPR + US CLOUD Act	personal data under a jurisdiction accessible to US authorities is exposed	European cloud (Scaleway/OVHcloud) or on-prem keep the data in the EU
EU AI Act	obligations for high-risk systems (traceability, control, documentation)	on-prem control makes auditability and compliance easier
CSRD / emissions reporting	reporting the carbon footprint (scope 2)	a low-carbon grid (France) improves the reported figure

The key point for the proposal: a US hyperscaler can offer a competitive GPU-hour, but it puts the data under the US CLOUD Act, which for data subject to GDPR is a compliance risk, not a saving. That is why this series’ comparison is not made in USD against US providers: it is made in euros, against options that keep data sovereignty, on-prem of your own or European cloud. Cost, energy and carbon are measurable axes; sovereignty is the constraint that decides which options even enter the table. This links with the mapping of the EU AI Act onto the on-premise architecture and with the technical controls ENS × ISO 42001 × EU AI Act.

The operational conclusion: France and Spain not only come out cheaper and cleaner than Germany; being EU jurisdiction, they meet the sovereignty constraint that a US hyperscaler does not meet at any price. Three advantages that reinforce each other, and that the proposal can defend with the same table.

Methodology: how to read (and not fool yourself with) these figures

Five warnings that apply to all the data in this series:

Board vs node. The GPU’s power (700 W per H100) is the floor; the complete node (CPU, NICs, power supplies, fans) consumes more. Always compare the same boundary.
Wholesale vs retail. The day-ahead price is not what the company pays; industrial retail adds grid charges and taxes. Use whichever matches your contract.
Measured vs estimated. A wattmeter figure and an nvidia-smi estimate are not the same thing; the difference is quantified in the energy articles.
Amortisation assumptions. The on-prem cost/hour depends on the hardware cycle (24 vs 36 months) and on the assumed utilisation; changing those assumptions moves the CPM more than any serving optimisation.
Comparability. Two CPM or J/token figures are only comparable if model, precision, context length, load and country match. Most public comparisons (and in USD) do not fix those assumptions, which is why the series reproduces the ones that matter with its own harness and European data.

The combined scorecard

The destination of the series is a single board where each model/configuration/location is scored on all the axes at once, to decide on a Pareto frontier rather than on a single number. An illustrative example (order-of-magnitude figures) on the same 8×H100 node:

Candidate	CPM (€/1M)	Goodput (tok/s)	Energy (J/tok)	Carbon (gCO₂/1M)	Sovereignty
A · 70B FP16, on-prem France	~1.09	~2,800	~2.0	low (grid ~40 gCO₂/kWh)	EU
B · 70B FP8, on-prem France	~0.75	~4,000	~1.4	lower	EU
C · Scaleway on-demand	~2.17	~2,800	n/a	low (FR)	EU
D · US hyperscaler	~3.0	~2,800	n/a	depends on region	not EU

The reading: B dominates A on the three intrinsic axes thanks to FP8. C (European cloud) is more expensive per token but shifts capex and operations, while keeping sovereignty. D (hyperscaler) may be competitive on cost but loses sovereignty, an axis that for GDPR data is not negotiable. No isolated number decides: the choice comes from seeing the whole row.

The trap the board avoids: optimising a single axis. Each cell will be filled in by an article in the series with measured data.

Where each axis is measured: from design to production

Each axis is measured at three different moments, with different tools and increasing precision. Confusing them is a common source of figures that do not add up:

Axis	Design (estimate)	Benchmark (controlled)	Production (continuous)
Cost	datasheet, GPU-hour price	CPM measured over synthetic load	OpenCost/Kubecost per team and token
Performance	theoretical throughput (roofline)	GuideLLM/AIPerf with SLO	OTel/Langfuse over real traffic
Energy	TDP × hours	Zeus/MLPerf Power on the bench	Kepler/DCGM per pod in the cluster

The rule: a design figure serves for sizing, but not for defending a proposal; for that you need a reproducible benchmark figure, and to operate, a continuous production one. When someone disputes a number in the proposal, the answer is not “a blog says so”, but “here is the bench, here is the methodology, reproduce it”.

Glossary of units and conversions

Quantity	Unit	Conversion / reference
Energy	joule (J), watt-hour (Wh)	1 Wh = 3,600 J; 1 kWh = 3.6 MJ = 3,600,000 J
Power	watt (W)	1 W = 1 J/s; H100 ≈ 700 W TDP
Energy per token	J/token	= average power (W) ÷ throughput (tok/s)
Throughput	tok/s	tokens generated per second by the system
Cost per token	€/1M tok (CPM)	= cost/hour ÷ (tok/s × 3,600 / 10⁶)
Latency	ms	TTFT (first token), TPOT/ITL (between tokens)
PUE	ratio (≥1)	total DC energy ÷ compute energy
Carbon	gCO₂eq	= energy (kWh) × PUE × grid intensity (gCO₂/kWh)

A useful conversion to keep in your head: at a constant 700 W, an H100 consumes ~0.7 kWh per hour, ~16.8 kWh a day and ~6,130 kWh a year (board only, no PUE). Multiplied by the PUE and your country’s energy price (FR ~0.058, ES ~0.054, DE ~0.091 €/kWh wholesale), that number is the basis of the electricity portion of the TCO.

How the rest of the series fits together

The 28 articles group into four tracks; this is the map:

Track	What it contributes	Closes with
FinOps (8)	measuring and allocating cost, TCO model	cost/token comparable on-prem vs European cloud
LLM benchmarking (8)	measuring performance reproducibly	Pareto frontier of engines
Energy (8)	measuring watts and carbon per token	energy in the TCO and in European regulation
Synthesis (4)	crossing the three axes	scorecard, on-prem vs cloud, sizing, harness

The three articles that follow this one in the batch introduce each axis in depth: their frameworks and the state of the art. From there, each article adds a layer of data to the final scorecard.

The thread running through all 28 instalments is the one in this article: cost, performance and energy are a single problem with three faces, tied together by throughput and by the energy that lives inside the cost, and in Europe, by the country where the cluster sits. Anyone who optimises one face while ignoring the others will end up with a pretty number and a platform that does not hold up. The work of the series is to produce the figures, measured, dated, European and reproducible, that allow a sovereign architecture to be defended before any committee with the only answer that closes a technical debate: the data and the bench to reproduce it.

Sources

Scaleway · H100 GPU instance (€/h price, EU sovereignty) — https://www.scaleway.com/en/h100/
DeployBase · OVHcloud GPU Pricing (European sovereignty) — https://deploybase.ai/articles/ovhcloud-gpu-pricing
TradingEconomics · electricity price France / Germany / Spain — https://tradingeconomics.com/france/electricity-price
EEA · emission intensity of electricity in Europe — https://www.eea.europa.eu/en/analysis/indicators/greenhouse-gas-emission-intensity-of-1
Electricity sector in Germany (carbon ~363 gCO₂/kWh) — https://en.wikipedia.org/wiki/Electricity_sector_in_Germany
Spheron · AI Inference Power & Cost Economics (2026) — https://www.spheron.network/blog/ai-inference-cost-economics-2026/
NVIDIA · H100 (TDP/specs) — https://www.nvidia.com/en-us/data-center/h100/

Energy benchmarking in AI: frameworks, metrics and state of the art (tool by tool)

Sat, 13 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The energy and carbon references are European (France, Germany, Spain), since this is a sovereign proposal. The dollar sign is not used (on this site it is a formula delimiter).

What this introduction covers

Fourth article in the data series and deep dive on the energy axis. Measuring the watts of an AI workload looks like a sustainability detail, but it is at once a cost lever (electricity is 30–50 % of TCO), an axis of sovereignty (carbon per token depends on the country) and, increasingly, a regulatory obligation (emissions reporting). This article inventories the metrics, where each figure physically comes from, the data sheet for each measurement framework, and how to go from watt to carbon with European figures. No recommendations; only data and methodology, because in energy, even more than in performance, the measurement method changes the result.

Why measuring energy matters (with numbers)

Three quantified reasons, connecting with the opening article:

Reason	Figure	Implication
Cost	electricity = 30–50 % of TCO	measuring energy is measuring almost half the cost
Sovereignty/carbon	France ~9× fewer gCO₂/token than Germany	the country of the cluster is a lever
Scale/regulation	datacenters ~460 TWh in 2025 (≈1.8 % worldwide), >800 TWh projected for 2028	emissions reporting (CSRD) mandatory

At 700 W per H100, an 8-card node draws of the order of 5.6 kW from the boards alone, ~49,000 kWh a year. Applying PUE 1.4 that becomes ~68,700 kWh/year, and there the country decides the bill: at the French wholesale price (~0.058 €/kWh) that is ~3,980 €/year of electricity per node; in Germany (~0.091 €/kWh), ~6,250 €/year, 57 % more for the same iron and the same work. Multiplied across a fleet, it is a first-order cost line, and a carbon footprint that, in Europe, has to be measured and reported. Energy stopped being an engineering detail and became a business and compliance variable.

The energy metrics

Metric	Definition	Unit
Mean power	average instantaneous consumption	W
Energy per token	energy consumed per generated token	J/token
Energy per inference	energy per request	µJ – Wh
Total energy	integral of power over time	Wh, kWh
PUE (Power Usage Effectiveness)	datacenter overhead (cooling, losses)	ratio ≥ 1
Carbon	energy × grid intensity	gCO₂eq

The two base identities of the whole axis:

$$\text{energy per token (J)} = \frac{\text{mean power (W)}}{\text{throughput (tok/s)}}$$ $$\text{carbon} = \text{energy (kWh)} \times \text{PUE} \times \text{grid intensity (gCO}_2\text{/kWh)}$$

The first has the same shape as cost per token: it shares the throughput denominator, so raising tokens/s lowers energy per token. The second brings in the two multipliers external to the compute, the datacenter PUE and the country’s grid intensity, which can change carbon per token by an order of magnitude without touching the stack.

Worked example: energy per token of an 8×H100 node

To anchor the identities with numbers, the end-to-end calculation on the example node (8×H100, Llama 3.1 70B FP16, vLLM at 2,800 tok/s):

Step	Calculation	Result
Board power	8 × 700 W	5,600 W
Energy per token (boards)	5,600 ÷ 2,800	2.0 J/token
Energy per token (with PUE 1.4)	2.0 × 1.4	2.8 J/token
Energy per 1M tokens (boards)	2 × 10⁶ J	0.56 kWh
Energy per 1M tokens (with PUE)	0.56 × 1.4	0.78 kWh

From there come the electricity cost and the carbon, by country:

Country	Electricity cost / 1M tok	Carbon / 1M tok
France (0.058 €/kWh; ~40 gCO₂/kWh)	~0.045 €	~31 gCO₂
Spain (0.054 €/kWh; ~160 gCO₂/kWh)	~0.042 €	~125 gCO₂
Germany (0.091 €/kWh; ~363 gCO₂/kWh)	~0.071 €	~283 gCO₂

And the link with performance: if an optimisation (FP8, say) raised throughput to 4,200 tok/s, energy per token would drop to ~1.33 J/token (boards) and everything below it in the same proportion. Electricity cost and carbon per token fall with throughput, by the identity. Note: these are board + PUE figures; the complete node (CPU, NICs, PSUs) consumes somewhat more, and energy measured with a power meter can differ from energy estimated with nvidia-smi.

Where the figure comes from: the measurement layers

Not every energy figure is equally reliable. The figure can come from four layers, with increasing accuracy and decreasing context:

The GPU is the dominant source in LLM inference, and its power is read from a hardware counter (NVML/nvidia-smi, aggregated by DCGM, the same basis as GPU observability). RAPL gives CPU and DRAM. The eBPF + model layer (Kepler) splits the total watt between containers and pods without instrumenting the application. And the physical power meter is the ground truth against which estimates are calibrated: no software estimate is better than the meter, only more convenient.

How it is instrumented (and where the error creeps in)

Four instrumentation decisions explain most of the discrepancies between energy measurements:

Sampling rate. nvidia-smi/DCGM read power at intervals; sampling that is too coarse misses the peaks and underestimates the energy. It must be fine enough to capture the power variation between prefill (high) and decode (lower).
Board vs node. GPU (board) power does not include CPU, NICs, fans or PSU losses (typically an extra 10–20 %). Comparing a board energy figure with a complete-node one is comparing different things.
Idle baseline. A powered GPU with no work already draws power. Is that idle attributed to the workload or discounted? The decision changes energy per token, and idle matters: it is real energy that someone pays for.
Time window. Energy is the integral of power over time; the measurement window has to line up exactly with the workload (without counting warm-up or shutdown), or the number does not correspond to the work measured.

These decisions, as in performance benchmarking, mean that the measurement method matters as much as the system measured. A J/token without a stated sampling rate, board/node boundary and idle treatment is not comparable.

Frameworks, tool by tool

Zeus (ml.energy, University of Michigan)

What it does: measure and optimise the energy of deep learning workloads. Method: NVML counters. Scope: NVIDIA and AMD GPUs, CPU, DRAM, Apple Silicon and NVIDIA Jetson. Licence: OSS; academic backing (NSDI'23 paper). Differentiator: it does not only measure, it also optimises (power capping, frequency selection) to cut energy without significant loss of performance (Zeus Project, NSDI'23). It is the reference tool when the goal is to lower J/token, not just to measure them.

CodeCarbon

What it does: estimates the energy and the CO₂eq of compute processes. Method: reads the GPU with nvidia-smi and estimates CPU/RAM. Scope: CPU + GPU + RAM. Licence: OSS (Python). Differentiator: it integrates into ML flows and CI pipelines, and it is the one that comes closest to the power meter among the Python tools (comparison). It brings carbon context out of the box. Limit: a high-level estimate, less exact than direct hardware measurement.

CarbonTracker

What it does: tracks energy and carbon at GPU run time. Scope: GPU only, it measures neither CPU nor memory. Licence: OSS. Differentiator: lightweight, easy to add to a training/inference script. Limit: partial coverage (no CPU/DRAM), so it undercounts the total system energy.

Scaphandre

What it does: a power metrology agent written in Rust. Method: RAPL (reads /proc, splits joules by CPU ticks). Scope: process or VM, with the raw accuracy of the hardware layer. Licence: OSS. Differentiator: the highest direct accuracy among the general-purpose tools, at the cost of bringing no carbon context (it is pure metrology) (Scaphandre · GitHub). Limit: focused on CPU/RAPL; for GPU it has to be combined with NVML.

Kepler (CNCF)

What it does: a Prometheus exporter that measures energy at container, pod and node level in Kubernetes. Method: eBPF + ML models to estimate consumption per workload. Language: Go + C. Licence: Apache 2.0 (the sustainable-computing-io project) (Kepler · GitHub). Differentiator: it is the cloud-native option for attributing energy per pod without instrumenting the app, just as OpenCost attributes cost. Limit: attribution via eBPF + model is estimation, not direct measurement; its accuracy depends on the model.

MLPerf Power (MLCommons)

What it does: a standardised benchmark of energy efficiency, from µW to MW. Maintainer: MLCommons. Differentiator: cross-vendor comparability of efficiency, with a dataset spanning several years and workload versions (MLPerf Power, arXiv 2410.12032). It is to energy what MLPerf Inference is to performance: the comparison standard between vendors.

Comparison table

Tool	Method	Scope	Licence	Accuracy / note
Zeus	NVML / counters	NV+AMD GPU, CPU, DRAM, Apple, Jetson	OSS	measures and optimises DL energy
CodeCarbon	nvidia-smi + estimation	CPU+GPU+RAM, CO₂eq	OSS (Python)	the closest to the power meter among Python tools
CarbonTracker	GPU runtime	GPU only	OSS	lightweight; no CPU/mem
Scaphandre	RAPL (/proc)	process/VM	OSS (Rust)	raw accuracy; no carbon context
Kepler	eBPF + model	container/pod/node (K8s)	Apache 2.0 (CNCF)	cloud-native; model-based estimation
MLPerf Power	standardised measurement	µW to MW	MLCommons	cross-vendor comparability

Accuracy: who to believe

The difference between tools is not a nuance: Scaphandre offers the raw accuracy of the hardware layer (RAPL), while the Python ones such as CodeCarbon give higher-level, carbon-aware estimates, with more context but less direct exactness; among the Python tools, CodeCarbon is the one that comes closest to the power meter, followed by CarbonTracker, with variability across infrastructures (comparison). The rule: for sizing and design, a software estimate will do; to defend a figure before a committee or an auditor, calibrate against a power meter at least once, and use the software tool as a continuous proxy. The measured figure beats the estimated one; the rest is convenience.

Energy leaderboards and benchmarks

To compare efficiency between models without building your own test bench:

Resource	What it provides
MLPerf Power	standardised efficiency measurement, from µW to MW
HF AI Energy Score	comparable per-model energy efficiency ratings (HF)
ML.ENERGY Benchmark	automated measurement and optimisation of inference energy
awesome-green-ai	curated index of Green AI resources and tools (repo)

These resources give the order of magnitude of the energy per token of each model, useful for choosing a model on efficiency at the design stage. But they do not replace measurement on your hardware and your workload: a leaderboard measures on one specific bench, not on your node.

From watt to carbon: France, Germany, Spain

Here the energy axis turns into a sovereignty argument. Carbon per token comes from multiplying the energy by the country’s grid intensity, which in Europe varies by almost two orders of magnitude:

Country	Wholesale (€/MWh, Mar 2026)	Grid carbon (gCO₂/kWh)	Profile
France	~58	~20–60	nuclear: cheap and very clean
Spain	~54	~150–170	renewables + gas: cheap, medium carbon
Germany	~91	~363 (2024 average)	coal/gas + renewables: expensive and dirty

Sources: wholesale prices (TradingEconomics), Germany’s carbon ~363 gCO₂/kWh (2024 average, Wikipedia), nuclear ~17–35 gCO₂e/kWh (EEA), Spain ~167 gCO₂/kWh (2020). Hourly intensity comes from ElectricityMaps.

The calculation, on the example node (0.78 kWh per million tokens, with PUE): in France (~40 gCO₂/kWh) it is ~31 gCO₂ per million tokens; in Germany (~363 gCO₂/kWh), ~283 gCO₂, a factor of ~9×, without touching a line of the stack. Placing the cluster in France or Spain is at once a cost lever, a carbon lever and a compliance lever (EU AI Act, controls ENS × 42001 × EU AI Act).

Energy efficiency levers

Once it is measured, it can be brought down. The levers, ordered by the identity J/token = W ÷ throughput (lower the numerator or raise the denominator):

Lever	Mechanism	Effect
Quantisation (FP8/INT4)	more tokens per GPU-hour, less VRAM/KV	raises throughput → lowers J/token
Batching / continuous batching	amortises the fixed power cost	raises throughput → lowers J/token
Power capping (Zeus)	limits GPU power	lowers W with little loss of throughput
Efficiency-aware scheduling	consolidate workloads, power down idle GPUs	eliminates idle consumption
Location (country)	cleaner grid	lowers carbon per token
Avoiding over-reasoning	fewer reasoning tokens that add nothing	less compute → less energy

Quantisation (see quantisation for inference) is the highest-return lever because it moves all three axes at once: it raises throughput, lowers cost and lowers energy per token. Zeus’s power capping is energy-specific: it trims peak power with little performance loss, lowering the W in the numerator. And idle, the GPU powered but not working, is pure wasted energy that scheduling recovers.

From design to production: where energy is measured

As with the other axes, energy is measured at three moments, with different tools and increasing accuracy:

Moment	Tool	What it gives
Design	TDP × hours (datasheet)	ceiling estimate for sizing
Benchmark	Zeus, MLPerf Power, power meter	energy per token measured on a controlled bench
Production	Kepler (eBPF), DCGM	continuous per-pod consumption in the cluster

The design figure (TDP) serves to size the utility feed and the cooling, but it overestimates (the GPU is rarely at TDP 100 % of the time). The benchmark figure is the one used for the proposal’s cost and carbon per token, because it is measured and reproducible. And the production figure (Kepler/DCGM) closes the loop: it checks that real consumption matches the budget and feeds the footprint report. The three have to be traceable to each other, which is what the harness in article S4 does.

Energy in TCO and European regulation

The closing of the axis: energy is not only sustainability, it is cost and compliance.

Cost: being 30–50 % of TCO, every J/token improvement translates directly into euros. In the example, moving from Germany (~0.091 €/kWh) to France (~0.058 €/kWh) lowers the electricity share by ~37 % on top of the carbon.
Regulation (CSRD): European companies must report their carbon footprint (scope 2). A low-carbon grid (France) directly improves that reported figure, and measuring energy per token stops being optional and becomes a reporting requirement.
EU AI Act: efficiency and energy consumption form part of the documentation expected of AI systems, which makes energy measurement part of the technical file.

One reporting nuance worth knowing: the electricity footprint is scope 2 (indirect emissions from purchased energy), and it can be accounted for by location (the real intensity of the country’s grid) or by market (according to the guarantees of origin contracted). Location-based accounting is the one that genuinely rewards putting the cluster on a clean grid such as the French one; market-based accounting allows green energy to be “bought” through certificates. Knowing which one your CSRD report requires changes which lever (physical location vs contract) lowers the figure, and both start from measuring energy per token, which is what this axis provides.

In other words: measuring energy covers at once a first-order cost line, the carbon and sovereignty argument, and a regulatory obligation. For a sovereign European proposal, the energy axis is not the “nice sustainability bit”: it is one of the three columns with a number attached.

Checklist for a reproducible energy measurement

For an energy or carbon figure to be defensible before a committee or an auditor, it has to come with its context. The minimum to fix and publish:

What to fix	Why
Tool + version	each layer (RAPL, NVML, eBPF) and tool measures differently
Sampling rate	coarse sampling misses peaks
Board vs node boundary	±10–20 % depending on whether CPU/PSUs are included
PUE used	it multiplies all the compute energy
Source and time of the grid intensity	carbon varies ~9× by country and by hour
Idle treatment	attributed or discounted changes the J/token
Model, precision and workload	energy depends on what is served and how
Calibration vs power meter	at least once, to validate the estimate

The rule: if you cannot hand over this table alongside the gCO₂/token figure, the figure is not auditable. The reproducible harness of article S4 records all these parameters so that the energy figure is as traceable as the cost and performance ones, and so that the reported footprint (CSRD) holds up under audit.

State of the art 2026

LLM-specific resolution is missing: general-purpose tools measure well at machine or process level, but they lack LLM-specific resolution (energy per token, per prefill/decode phase), and the simulators lack emissions modelling, an open gap in the field.
eBPF as a trend (Kepler) for attributing energy per pod without instrumenting the app, in parallel to how OpenCost attributes cost.
Standardisation of comparison via MLPerf Power and HF AI Energy Score.
Optimisation, not only measurement (Zeus): the field is moving from “how much do I spend” to “how do I spend less without losing performance”.

Limits and traps (data-driven)

Estimation vs measurement. An nvidia-smi figure is not a power meter. Calibrate against hardware at least once before defending a number.
Partial coverage. CarbonTracker measures GPU only; Scaphandre, mostly CPU/RAPL. Add the layers or you undercount the total energy.
Forgetting PUE. Board energy does not include cooling or losses; without PUE you underestimate the real consumption and carbon.
Carbon without a country. A gCO₂/token without a stated grid (and hour) means nothing: it varies ~9× between France and Germany.
Confusing energy with sustainability. Here it is cost and compliance, with a number; treating it as a “green” extra is losing one of the three columns of the proposal.

That closes the run of introductions: cost, performance and energy, each with its frameworks and its methodology. The rest of the series goes tool by tool through to the dashboard that underpins the sovereign architecture decision, with European figures, measured and reproducible.

Closing

Energy is the axis easiest to dismiss as “the green one” and the one that hides the most: it is almost half the cost, it defines the footprint Europe requires to be reported, and its carbon per token changes ~9× depending on whether the cluster sits in France or in Germany. Measuring it properly, with the right layer, the right sampling, PUE included and the country’s grid, turns a soft sustainability argument into a hard column of the proposal, with a number and a source. And the sovereign conclusion stands on its own with the data in this article: an inference platform in France or Spain is at once cheaper, cleaner and compliant with European jurisdiction than the same platform on a US hyperscaler, and that is demonstrated with measured J/token, gCO₂/kWh by country and euros, not with intentions. The energy axis does not decorate the proposal: it closes it.

Sources

Zeus Project (ml.energy, UMich) — https://ml.energy/zeus/
Zeus · USENIX NSDI'23 — https://www.usenix.org/system/files/nsdi23-you.pdf
Kepler (CNCF, eBPF) · GitHub — https://github.com/sustainable-computing-io/kepler
Scaphandre · GitHub — https://github.com/hubblo-org/scaphandre
CarbonTracker · GitHub — https://github.com/saintslab/carbontracker
MLPerf Power (arXiv 2410.12032) — https://arxiv.org/html/2410.12032v2
HF AI Energy Score — https://huggingface.github.io/AIEnergyScore/
awesome-green-ai — https://github.com/samuelrince/awesome-green-ai
TradingEconomics · precio electricidad Francia/Alemania/España — https://tradingeconomics.com/france/electricity-price
EEA · intensidad de emisiones de la electricidad en Europa — https://www.eea.europa.eu/en/analysis/indicators/greenhouse-gas-emission-intensity-of-1
Electricity sector in Germany (carbono ~363 gCO₂/kWh) — https://en.wikipedia.org/wiki/Electricity_sector_in_Germany

Technical figure	How it is obtained	Use in TCO	Use in regulatory reporting
\(P_{\text{IT}}\) (W)	DCGM / NVML / IPMI	Energy cost formula	\(E_{\text{IT,annual}}\) for PUE, WUE, ERF, REF
Throughput (tok/s)	Engine metrics (vLLM, TGI)	€/token, Wh/token	Energy intensity of inference
PUE (measured)	Meters on PDUs and IT equipment (ISO/IEC 30134-2)	Multiplier in the TCO formula	Mandatory ReportENER KPI
Annual \(I_{\text{grid}}\) (gCO₂/kWh)	REE / MITECO / national factors	gCO₂/token for carbon-aware	Emission factor for location-based Scope 2 ESRS E1-6
Hourly \(I_{\text{grid}}\)	ElectricityMaps / Carbon Aware SDK	Carbon-aware scheduling	Location-based Scope 2 with hourly granularity (best practice)
\(W_{\text{in}}\) (m³)	Datacenter water meters	Cost of water (where applicable)	WUE for ReportENER
\(E_{\text{reuse}}\) (kWh)	Heat export meters	Possible waste heat revenue	ERF for ReportENER
\(E_{\text{res,tot}}\) (kWh)	Energy bills with a renewable attribute / GOs	Energy cost with a PPA	REF for ReportENER; market-based Scope 2 for ESRS E1-6
Hardware embodied carbon (gCO₂eq)	Supplier LCA (AWS methodology, where available)	—	Scope 3 cat. 1 for ESRS E1-6

Symbol	Meaning	Units
\(E\)	Energy consumed by the software system (servers, network, user devices)	kWh
\(I\)	Carbon intensity of the energy consumed (location-based or another declared basis)	gCO₂eq/kWh
\(M\)	Embodied carbon of the hardware, amortised by time of use and utilisation percentage	gCO₂eq
\(R\)	Functional unit (the denominator that normalises the rate)	per API call, per token, per user, etc.

Energy on lo0 — Tech Blog

Measuring the power of a GPU: NVML, DCGM and the sampling errors that invalidate your watts

TL;DR

What the hardware exposes

NVML: power

NVML: the energy counter

NVML: power limits

DCGM: the equivalent fields

The real accuracy of the sensor

Update period and averaging window

The steady-state error

The response to transients

The two ways of obtaining energy

Why 1 Hz destroys a 200 ms inference

Aliasing with periodic loads

What the GPU counter does not measure

The tax of keeping a context open

CPU and DRAM: RAPL

The software stack

Procedure for making a Wh/token auditable

Subtracting idle, or not

Power capping: what it costs and what it saves

Reference figures

Cross-links from the energy track

See also

Sources

Datacenter physical infrastructure (1/4): the power chain

Why power is the foundation of everything

The chain, link by link

The A/B dual path

Redundancy: the language of the “N"s

The Uptime Institute Tier levels

The metrics you really have to understand

kW versus kVA and the power factor

PUE: the efficiency of the whole facility

Availability and the nines

A sizing example

The AI earthquake: rack density

High-density distribution and the jump to 800 V DC

The grid as the limit: siting and on-site generation

The facility is only as reliable as its operation

For an inference factory

See also

Sources

Datacenter physical infrastructure (2/4): the generators

The generator’s role: long-duration backup

Starting in 10 seconds

The ratings: why “standby” is not enough

Sizing it right: the devil in the transients

Diesel, gas, hydrogen and batteries

Emissions: the tension with regulation

The fuel: autonomy and degradation

Paralleling: several sets as one

Maintenance and testing: the enemy is low load

Key takeaways

See also

Sources

Datacenter physical infrastructure (3/4): the switching

The referee of the chain

The window that decides everything

The ATS: robust and slow

The STS: fast and electronic

Open or closed transition

The sequence of a power cut, step by step

The regulatory framework

Synchronising in order to close

A/B architecture and the single point of failure

Towards solid state

Key takeaways

See also

Sources

Datacenter physical infrastructure (4/4): the UPS and the ride-through

The ten-second bridge

The topologies: VFI, VI and VFD

Eco-mode: the last point of efficiency

The flywheel: backup without batteries

Batteries: lead gives way to lithium

Sizing: little time, unity power factor

The modular architecture

The AI challenge: loads that pulse