Energy benchmarking in AI: frameworks, metrics and state of the art (tool by tool)
Contents
Notation: amounts in euros (N €), decimals with a point. The energy and carbon references are European (France, Germany, Spain), since this is a sovereign proposal. The dollar sign is not used (on this site it is a formula delimiter).
What this introduction covers
Fourth article in the data series and deep dive on the energy axis. Measuring the watts of an AI workload looks like a sustainability detail, but it is at once a cost lever (electricity is 30–50 % of TCO), an axis of sovereignty (carbon per token depends on the country) and, increasingly, a regulatory obligation (emissions reporting). This article inventories the metrics, where each figure physically comes from, the data sheet for each measurement framework, and how to go from watt to carbon with European figures. No recommendations; only data and methodology, because in energy, even more than in performance, the measurement method changes the result.
Why measuring energy matters (with numbers)
Three quantified reasons, connecting with the opening article:
| Reason | Figure | Implication |
|---|---|---|
| Cost | electricity = 30–50 % of TCO | measuring energy is measuring almost half the cost |
| Sovereignty/carbon | France ~9× fewer gCO₂/token than Germany | the country of the cluster is a lever |
| Scale/regulation | datacenters ~460 TWh in 2025 (≈1.8 % worldwide), >800 TWh projected for 2028 | emissions reporting (CSRD) mandatory |
At 700 W per H100, an 8-card node draws of the order of 5.6 kW from the boards alone, ~49,000 kWh a year. Applying PUE 1.4 that becomes ~68,700 kWh/year, and there the country decides the bill: at the French wholesale price (~0.058 €/kWh) that is ~3,980 €/year of electricity per node; in Germany (~0.091 €/kWh), ~6,250 €/year, 57 % more for the same iron and the same work. Multiplied across a fleet, it is a first-order cost line, and a carbon footprint that, in Europe, has to be measured and reported. Energy stopped being an engineering detail and became a business and compliance variable.
The energy metrics
| Metric | Definition | Unit |
|---|---|---|
| Mean power | average instantaneous consumption | W |
| Energy per token | energy consumed per generated token | J/token |
| Energy per inference | energy per request | µJ – Wh |
| Total energy | integral of power over time | Wh, kWh |
| PUE (Power Usage Effectiveness) | datacenter overhead (cooling, losses) | ratio ≥ 1 |
| Carbon | energy × grid intensity | gCO₂eq |
The two base identities of the whole axis:
$$\text{energy per token (J)} = \frac{\text{mean power (W)}}{\text{throughput (tok/s)}}$$ $$\text{carbon} = \text{energy (kWh)} \times \text{PUE} \times \text{grid intensity (gCO}_2\text{/kWh)}$$The first has the same shape as cost per token: it shares the throughput denominator, so raising tokens/s lowers energy per token. The second brings in the two multipliers external to the compute, the datacenter PUE and the country’s grid intensity, which can change carbon per token by an order of magnitude without touching the stack.
Worked example: energy per token of an 8×H100 node
To anchor the identities with numbers, the end-to-end calculation on the example node (8×H100, Llama 3.1 70B FP16, vLLM at 2,800 tok/s):
| Step | Calculation | Result |
|---|---|---|
| Board power | 8 × 700 W | 5,600 W |
| Energy per token (boards) | 5,600 ÷ 2,800 | 2.0 J/token |
| Energy per token (with PUE 1.4) | 2.0 × 1.4 | 2.8 J/token |
| Energy per 1M tokens (boards) | 2 × 10⁶ J | 0.56 kWh |
| Energy per 1M tokens (with PUE) | 0.56 × 1.4 | 0.78 kWh |
From there come the electricity cost and the carbon, by country:
| Country | Electricity cost / 1M tok | Carbon / 1M tok |
|---|---|---|
| France (0.058 €/kWh; ~40 gCO₂/kWh) | ~0.045 € | ~31 gCO₂ |
| Spain (0.054 €/kWh; ~160 gCO₂/kWh) | ~0.042 € | ~125 gCO₂ |
| Germany (0.091 €/kWh; ~363 gCO₂/kWh) | ~0.071 € | ~283 gCO₂ |
And the link with performance: if an optimisation (FP8, say) raised throughput to
4,200 tok/s, energy per token would drop to ~1.33 J/token (boards) and everything below it in
the same proportion. Electricity cost and carbon per token fall with throughput, by the
identity. Note: these are board + PUE figures; the complete node (CPU, NICs, PSUs)
consumes somewhat more, and energy measured with a power meter can differ from energy
estimated with nvidia-smi.
Where the figure comes from: the measurement layers
Not every energy figure is equally reliable. The figure can come from four layers, with increasing accuracy and decreasing context:
The GPU is the dominant source in LLM inference, and its power is read from a hardware counter (NVML/nvidia-smi, aggregated by DCGM, the same basis as GPU observability). RAPL gives CPU and DRAM. The eBPF + model layer (Kepler) splits the total watt between containers and pods without instrumenting the application. And the physical power meter is the ground truth against which estimates are calibrated: no software estimate is better than the meter, only more convenient.
How it is instrumented (and where the error creeps in)
Four instrumentation decisions explain most of the discrepancies between energy measurements:
- Sampling rate.
nvidia-smi/DCGM read power at intervals; sampling that is too coarse misses the peaks and underestimates the energy. It must be fine enough to capture the power variation between prefill (high) and decode (lower). - Board vs node. GPU (board) power does not include CPU, NICs, fans or PSU losses (typically an extra 10–20 %). Comparing a board energy figure with a complete-node one is comparing different things.
- Idle baseline. A powered GPU with no work already draws power. Is that idle attributed to the workload or discounted? The decision changes energy per token, and idle matters: it is real energy that someone pays for.
- Time window. Energy is the integral of power over time; the measurement window has to line up exactly with the workload (without counting warm-up or shutdown), or the number does not correspond to the work measured.
These decisions, as in performance benchmarking, mean that the measurement method matters as much as the system measured. A J/token without a stated sampling rate, board/node boundary and idle treatment is not comparable.
Frameworks, tool by tool
Zeus (ml.energy, University of Michigan)
What it does: measure and optimise the energy of deep learning workloads. Method: NVML counters. Scope: NVIDIA and AMD GPUs, CPU, DRAM, Apple Silicon and NVIDIA Jetson. Licence: OSS; academic backing (NSDI'23 paper). Differentiator: it does not only measure, it also optimises (power capping, frequency selection) to cut energy without significant loss of performance (Zeus Project, NSDI'23). It is the reference tool when the goal is to lower J/token, not just to measure them.
CodeCarbon
What it does: estimates the energy and the CO₂eq of compute processes. Method: reads the
GPU with nvidia-smi and estimates CPU/RAM. Scope: CPU + GPU + RAM. Licence: OSS (Python).
Differentiator: it integrates into ML flows and CI pipelines, and it is the one that comes
closest to the power meter among the Python tools (comparison).
It brings carbon context out of the box. Limit: a high-level estimate, less exact than direct
hardware measurement.
CarbonTracker
What it does: tracks energy and carbon at GPU run time. Scope: GPU only, it measures neither CPU nor memory. Licence: OSS. Differentiator: lightweight, easy to add to a training/inference script. Limit: partial coverage (no CPU/DRAM), so it undercounts the total system energy.
Scaphandre
What it does: a power metrology agent written in Rust. Method: RAPL (reads /proc,
splits joules by CPU ticks). Scope: process or VM, with the raw accuracy of the hardware
layer. Licence: OSS. Differentiator: the highest direct accuracy among the general-purpose
tools, at the cost of bringing no carbon context (it is pure metrology)
(Scaphandre · GitHub). Limit: focused on CPU/RAPL;
for GPU it has to be combined with NVML.
Kepler (CNCF)
What it does: a Prometheus exporter that measures energy at container, pod and node level
in Kubernetes. Method: eBPF + ML models to estimate consumption per workload. Language:
Go + C. Licence: Apache 2.0 (the sustainable-computing-io project) (Kepler · GitHub).
Differentiator: it is the cloud-native option for attributing energy per pod without
instrumenting the app, just as OpenCost attributes cost. Limit: attribution via eBPF + model
is estimation, not direct measurement; its accuracy depends on the model.
MLPerf Power (MLCommons)
What it does: a standardised benchmark of energy efficiency, from µW to MW. Maintainer: MLCommons. Differentiator: cross-vendor comparability of efficiency, with a dataset spanning several years and workload versions (MLPerf Power, arXiv 2410.12032). It is to energy what MLPerf Inference is to performance: the comparison standard between vendors.
Comparison table
| Tool | Method | Scope | Licence | Accuracy / note |
|---|---|---|---|---|
| Zeus | NVML / counters | NV+AMD GPU, CPU, DRAM, Apple, Jetson | OSS | measures and optimises DL energy |
| CodeCarbon | nvidia-smi + estimation | CPU+GPU+RAM, CO₂eq | OSS (Python) | the closest to the power meter among Python tools |
| CarbonTracker | GPU runtime | GPU only | OSS | lightweight; no CPU/mem |
| Scaphandre | RAPL (/proc) | process/VM | OSS (Rust) | raw accuracy; no carbon context |
| Kepler | eBPF + model | container/pod/node (K8s) | Apache 2.0 (CNCF) | cloud-native; model-based estimation |
| MLPerf Power | standardised measurement | µW to MW | MLCommons | cross-vendor comparability |
Accuracy: who to believe
The difference between tools is not a nuance: Scaphandre offers the raw accuracy of the hardware layer (RAPL), while the Python ones such as CodeCarbon give higher-level, carbon-aware estimates, with more context but less direct exactness; among the Python tools, CodeCarbon is the one that comes closest to the power meter, followed by CarbonTracker, with variability across infrastructures (comparison). The rule: for sizing and design, a software estimate will do; to defend a figure before a committee or an auditor, calibrate against a power meter at least once, and use the software tool as a continuous proxy. The measured figure beats the estimated one; the rest is convenience.
Energy leaderboards and benchmarks
To compare efficiency between models without building your own test bench:
| Resource | What it provides |
|---|---|
| MLPerf Power | standardised efficiency measurement, from µW to MW |
| HF AI Energy Score | comparable per-model energy efficiency ratings (HF) |
| ML.ENERGY Benchmark | automated measurement and optimisation of inference energy |
| awesome-green-ai | curated index of Green AI resources and tools (repo) |
These resources give the order of magnitude of the energy per token of each model, useful for choosing a model on efficiency at the design stage. But they do not replace measurement on your hardware and your workload: a leaderboard measures on one specific bench, not on your node.
From watt to carbon: France, Germany, Spain
Here the energy axis turns into a sovereignty argument. Carbon per token comes from multiplying the energy by the country’s grid intensity, which in Europe varies by almost two orders of magnitude:
| Country | Wholesale (€/MWh, Mar 2026) | Grid carbon (gCO₂/kWh) | Profile |
|---|---|---|---|
| France | ~58 | ~20–60 | nuclear: cheap and very clean |
| Spain | ~54 | ~150–170 | renewables + gas: cheap, medium carbon |
| Germany | ~91 | ~363 (2024 average) | coal/gas + renewables: expensive and dirty |
Sources: wholesale prices (TradingEconomics), Germany’s carbon ~363 gCO₂/kWh (2024 average, Wikipedia), nuclear ~17–35 gCO₂e/kWh (EEA), Spain ~167 gCO₂/kWh (2020). Hourly intensity comes from ElectricityMaps.
The calculation, on the example node (0.78 kWh per million tokens, with PUE): in France (~40 gCO₂/kWh) it is ~31 gCO₂ per million tokens; in Germany (~363 gCO₂/kWh), ~283 gCO₂, a factor of ~9×, without touching a line of the stack. Placing the cluster in France or Spain is at once a cost lever, a carbon lever and a compliance lever (EU AI Act, controls ENS × 42001 × EU AI Act).
Energy efficiency levers
Once it is measured, it can be brought down. The levers, ordered by the identity J/token = W ÷ throughput (lower the numerator or raise the denominator):
| Lever | Mechanism | Effect |
|---|---|---|
| Quantisation (FP8/INT4) | more tokens per GPU-hour, less VRAM/KV | raises throughput → lowers J/token |
| Batching / continuous batching | amortises the fixed power cost | raises throughput → lowers J/token |
| Power capping (Zeus) | limits GPU power | lowers W with little loss of throughput |
| Efficiency-aware scheduling | consolidate workloads, power down idle GPUs | eliminates idle consumption |
| Location (country) | cleaner grid | lowers carbon per token |
| Avoiding over-reasoning | fewer reasoning tokens that add nothing | less compute → less energy |
Quantisation (see quantisation for inference) is the highest-return lever because it moves all three axes at once: it raises throughput, lowers cost and lowers energy per token. Zeus’s power capping is energy-specific: it trims peak power with little performance loss, lowering the W in the numerator. And idle, the GPU powered but not working, is pure wasted energy that scheduling recovers.
From design to production: where energy is measured
As with the other axes, energy is measured at three moments, with different tools and increasing accuracy:
| Moment | Tool | What it gives |
|---|---|---|
| Design | TDP × hours (datasheet) | ceiling estimate for sizing |
| Benchmark | Zeus, MLPerf Power, power meter | energy per token measured on a controlled bench |
| Production | Kepler (eBPF), DCGM | continuous per-pod consumption in the cluster |
The design figure (TDP) serves to size the utility feed and the cooling, but it overestimates (the GPU is rarely at TDP 100 % of the time). The benchmark figure is the one used for the proposal’s cost and carbon per token, because it is measured and reproducible. And the production figure (Kepler/DCGM) closes the loop: it checks that real consumption matches the budget and feeds the footprint report. The three have to be traceable to each other, which is what the harness in article S4 does.
Energy in TCO and European regulation
The closing of the axis: energy is not only sustainability, it is cost and compliance.
- Cost: being 30–50 % of TCO, every J/token improvement translates directly into euros. In the example, moving from Germany (~0.091 €/kWh) to France (~0.058 €/kWh) lowers the electricity share by ~37 % on top of the carbon.
- Regulation (CSRD): European companies must report their carbon footprint (scope 2). A low-carbon grid (France) directly improves that reported figure, and measuring energy per token stops being optional and becomes a reporting requirement.
- EU AI Act: efficiency and energy consumption form part of the documentation expected of AI systems, which makes energy measurement part of the technical file.
One reporting nuance worth knowing: the electricity footprint is scope 2 (indirect emissions from purchased energy), and it can be accounted for by location (the real intensity of the country’s grid) or by market (according to the guarantees of origin contracted). Location-based accounting is the one that genuinely rewards putting the cluster on a clean grid such as the French one; market-based accounting allows green energy to be “bought” through certificates. Knowing which one your CSRD report requires changes which lever (physical location vs contract) lowers the figure, and both start from measuring energy per token, which is what this axis provides.
In other words: measuring energy covers at once a first-order cost line, the carbon and sovereignty argument, and a regulatory obligation. For a sovereign European proposal, the energy axis is not the “nice sustainability bit”: it is one of the three columns with a number attached.
Checklist for a reproducible energy measurement
For an energy or carbon figure to be defensible before a committee or an auditor, it has to come with its context. The minimum to fix and publish:
| What to fix | Why |
|---|---|
| Tool + version | each layer (RAPL, NVML, eBPF) and tool measures differently |
| Sampling rate | coarse sampling misses peaks |
| Board vs node boundary | ±10–20 % depending on whether CPU/PSUs are included |
| PUE used | it multiplies all the compute energy |
| Source and time of the grid intensity | carbon varies ~9× by country and by hour |
| Idle treatment | attributed or discounted changes the J/token |
| Model, precision and workload | energy depends on what is served and how |
| Calibration vs power meter | at least once, to validate the estimate |
The rule: if you cannot hand over this table alongside the gCO₂/token figure, the figure is not auditable. The reproducible harness of article S4 records all these parameters so that the energy figure is as traceable as the cost and performance ones, and so that the reported footprint (CSRD) holds up under audit.
State of the art 2026
- LLM-specific resolution is missing: general-purpose tools measure well at machine or process level, but they lack LLM-specific resolution (energy per token, per prefill/decode phase), and the simulators lack emissions modelling, an open gap in the field.
- eBPF as a trend (Kepler) for attributing energy per pod without instrumenting the app, in parallel to how OpenCost attributes cost.
- Standardisation of comparison via MLPerf Power and HF AI Energy Score.
- Optimisation, not only measurement (Zeus): the field is moving from “how much do I spend” to “how do I spend less without losing performance”.
Limits and traps (data-driven)
- Estimation vs measurement. An
nvidia-smifigure is not a power meter. Calibrate against hardware at least once before defending a number. - Partial coverage. CarbonTracker measures GPU only; Scaphandre, mostly CPU/RAPL. Add the layers or you undercount the total energy.
- Forgetting PUE. Board energy does not include cooling or losses; without PUE you underestimate the real consumption and carbon.
- Carbon without a country. A gCO₂/token without a stated grid (and hour) means nothing: it varies ~9× between France and Germany.
- Confusing energy with sustainability. Here it is cost and compliance, with a number; treating it as a “green” extra is losing one of the three columns of the proposal.
That closes the run of introductions: cost, performance and energy, each with its frameworks and its methodology. The rest of the series goes tool by tool through to the dashboard that underpins the sovereign architecture decision, with European figures, measured and reproducible.
Closing
Energy is the axis easiest to dismiss as “the green one” and the one that hides the most: it is almost half the cost, it defines the footprint Europe requires to be reported, and its carbon per token changes ~9× depending on whether the cluster sits in France or in Germany. Measuring it properly, with the right layer, the right sampling, PUE included and the country’s grid, turns a soft sustainability argument into a hard column of the proposal, with a number and a source. And the sovereign conclusion stands on its own with the data in this article: an inference platform in France or Spain is at once cheaper, cleaner and compliant with European jurisdiction than the same platform on a US hyperscaler, and that is demonstrated with measured J/token, gCO₂/kWh by country and euros, not with intentions. The energy axis does not decorate the proposal: it closes it.
See also
- LLM energy efficiency leaderboards — the J/token rankings where the numbers these tools measure land: which leaderboard to use, how to read it and what biases it carries.
- From watt to carbon: PUE, grid intensity and the real cost of a token — how to convert the J/token measured with Zeus or Kepler into gCO₂eq using the datacenter PUE and the country’s electricity grid intensity.
- Energy efficiency levers in LLM inference — which configuration changes reduce the J/token these tools report: quantization, batching, engine, KV cache precision.
Sources
- Zeus Project (ml.energy, UMich) — https://ml.energy/zeus/
- Zeus · USENIX NSDI'23 — https://www.usenix.org/system/files/nsdi23-you.pdf
- Kepler (CNCF, eBPF) · GitHub — https://github.com/sustainable-computing-io/kepler
- Scaphandre · GitHub — https://github.com/hubblo-org/scaphandre
- CarbonTracker · GitHub — https://github.com/saintslab/carbontracker
- MLPerf Power (arXiv 2410.12032) — https://arxiv.org/html/2410.12032v2
- HF AI Energy Score — https://huggingface.github.io/AIEnergyScore/
- awesome-green-ai — https://github.com/samuelrince/awesome-green-ai
- TradingEconomics · precio electricidad Francia/Alemania/España — https://tradingeconomics.com/france/electricity-price
- EEA · intensidad de emisiones de la electricidad en Europa — https://www.eea.europa.eu/en/analysis/indicators/greenhouse-gas-emission-intensity-of-1
- Electricity sector in Germany (carbono ~363 gCO₂/kWh) — https://en.wikipedia.org/wiki/Electricity_sector_in_Germany