Kepler on lo0 — Tech Blog

Measuring energy in production: Kepler, DCGM and the practical stack (accuracy and overhead)

Sun, 14 Jun 2026 05:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Third article of the energy track (C3). C1 made the conceptual inventory of the tools; here we get practical: how the measurement stack is deployed on Kubernetes, where each piece takes its figures from, how accurate it is against a power meter, how much overhead it adds, and how energy is attributed per pod and per MIG instance. It is the equivalent of B2/B3 for energy: to stop talking about what each tool measures and start building the bench that measures it. With commands; no recommendations, only the mechanics.

The measurement stack on Kubernetes

To measure energy per workload on a GPU cluster, the practical stack has four pieces, and the good news is that three of them you already have if you do observability:

The new piece compared with a normal observability stack is Kepler, which adds energy attribution per pod. DCGM already gives GPU power; Prometheus stores it; Grafana draws it. It is reused infrastructure, not a new system, in line with GPU observability via DCGM.

DCGM exporter: the base of GPU power

GPU power, the dominant source in LLM inference, comes from NVIDIA’s dcgm-exporter (part of the GPU Operator), which exposes DCGM metrics to Prometheus. The key metric is DCGM_FI_DEV_POWER_USAGE (instantaneous power in W), alongside utilisation (DCGM_FI_DEV_GPU_UTIL) and memory. With that, energy is the integral of power over time across the workload’s window. DCGM gives power at card level; to split it per pod or per MIG instance you need a layer on top, which is Kepler.

Kepler in depth: eBPF and the sources

Kepler (Kubernetes-based Efficient Power Level Exporter) is a Prometheus exporter that measures energy at container, pod and node level, using eBPF to keep overhead low and scientific methods to improve accuracy (Kepler · GitHub). Its strength is that it combines several power sources:

Source	What it covers
RAPL	CPU and DRAM
NVML	NVIDIA GPU power
ACPI	platform power management
Redfish / IPMI	platform power (BMC)
Regression models	estimation when there is no real metric

For the GPU, the dcgmEndpoint parameter points at the dcgm-exporter metrics URL, which enables power attribution per MIG instance (Kepler · configuration). That is: DCGM gives the card’s power, Kepler splits it between the pods (and MIG partitions) that use it, via eBPF and models. The result is per-pod energy metrics in Prometheus, ready for Grafana, usable for scheduling or scaling, consumption reporting and visualisation (Kepler · GitHub).

Deployment: a DaemonSet on top of Prometheus

The deployment is a DaemonSet (one Kepler per node), pointing at the existing Prometheus and at the dcgm-exporter:

# Helm, pointing at the dcgm-exporter for GPU attribution
helm install kepler kepler/kepler \
 --namespace kepler --create-namespace \
 --set dcgmEndpoint="http://dcgm-exporter:9400/metrics"

Prerequisites: a kernel with eBPF, the dcgm-exporter running (from the GPU Operator) and Prometheus scraping. Once deployed, Kepler exposes metrics such as kepler_container_joules_total per pod, queried with PromQL and drawn in Grafana. Energy per workload stops being a manual estimate and becomes a continuous time series, just like cost with OpenCost.

PromQL queries: energy per pod

With Kepler exporting, these are the queries that build the panels. The exact names vary by version; check your /metrics:

# Cumulative energy per pod (joules)
kepler_container_joules_total

# Mean power per namespace over the last hour (W)
sum by (namespace) (rate(kepler_container_joules_total[1h]))

# GPU power per node (from DCGM, W)
sum by (node) (DCGM_FI_DEV_POWER_USAGE)

# Energy per token: power ÷ throughput (with a gateway/engine metric)
sum(rate(kepler_container_joules_total[5m])) / sum(rate(vllm_generation_tokens_total[5m]))

The last one closes the circle with the cost track: it crosses Kepler’s energy with the engine/gateway tokens to give J/token in real time, per workload. With that in Grafana you have energy per pod, per namespace and per token, continuously, the energy equivalent of the OpenCost cost report.

A calibration procedure

Since Kepler estimates, it is worth calibrating once per node type. A simple procedure:

Power meter at the node input (or a PDU/BMC reading), measuring the node’s real power.
Controlled load: a stable sweep at a known power (the engine at 80 % GPU, say).
Compare: the node’s sum of kepler_container_joules_total plus idle against the meter reading over the same window.
Adjust Kepler’s factor or model if the deviation is noticeable, and document the residual error.
Repeat for each node type (calibration is not transferable between different hardware).

With that residual error documented, Kepler’s continuous figure has a known error bar, which is exactly what a CSRD auditor or a technical committee will ask for. Measuring without calibrating gives a number with no credibility; calibrating once turns the continuous estimate into a defensible figure.

Accuracy: estimation vs power meter

The data honesty of the energy track: Kepler estimates (eBPF + models + counters), it does not measure with a power meter. Accuracy depends on the source: with NVML/DCGM and RAPL available, the figure is fairly faithful (it reads hardware counters); when it falls back on regression models (with no real metric), it is an estimate with more error. Good practice: calibrate against a physical power meter at least once per node type, adjust, and use Kepler as a continuous proxy. The meter is the point-in-time ground truth; Kepler, the continuous series calibrated against it. For a CSRD report that an auditor may challenge, that initial calibration is what gives the continuous figure credibility.

Overhead: the cost of measuring

Measuring energy has its own compute cost, worth budgeting for. In a Kubernetes environment, the overhead of the monitoring system is split between the Prometheus server and the targets: the Prometheus server consumes ~150 % of CPU (1.5 cores) and ~15 GB of RAM, while the targets (node exporter and dcgm-exporter) together use ~35 % of CPU and ~300 MB of RAM (arXiv · Container-level Energy Observability). Kepler uses eBPF precisely to keep its own overhead low. The reading: the bulk of the cost of measuring is in Prometheus (which you probably already have for observability), not in the energy exporters. Adding Kepler to an existing observability stack is marginal; standing up Prometheus just for this would be expensive. That is why measuring energy is nearly free if you already observe, and a project if you do not.

Zeus: for the bench and for optimisation, not for continuous production

Zeus (ml.energy) plays a different role: it is not a continuous production exporter, it is the bench tool for measuring and optimising the energy of one specific workload (a benchmark, a training run), with power capping and frequency selection. It fits into the benchmarking flow (alongside GuideLLM/AIPerf): during a sweep, Zeus measures the energy per configuration and lets you test the effect of power capping on performance. The practical division: Kepler/DCGM in production (continuous per-pod attribution), Zeus on the bench (fine measurement + optimisation of one workload), CodeCarbon if you want carbon context integrated into an ML pipeline.

CodeCarbon and Scaphandre in practice

The other two pieces, with their specific role:

CodeCarbon integrates inside the code of an ML pipeline (a decorator or a context manager wrapping the training or the inference), and returns the energy and the CO₂eq of that block, using nvidia-smi for the GPU and estimating CPU/RAM. It fits when you want the carbon figure per experiment or per ML job, integrated into the data flow, not at infrastructure level. It is the closest to the power meter among the Python tools.
Scaphandre is a metrology agent in Rust that reads RAPL and gives process/VM power with raw hardware accuracy. For CPU it is very accurate; for GPU it has to be combined with NVML. It fits when you need the most faithful CPU metrology, or a lightweight agent independent of Kubernetes.

The division with Kepler: Kepler attributes energy per pod in the cluster (the infrastructure view); CodeCarbon gives carbon per ML job (the pipeline view); Scaphandre gives raw CPU metrology. They do not compete; they cover different layers, and a mature platform uses whichever matches the question.

Choosing the stack by objective

Objective	Tool	Why
Continuous per-pod attribution (production)	Kepler + DCGM	eBPF, per pod/MIG, into Prometheus
Fine measurement + optimisation (bench)	Zeus	measures and optimises (power capping)
Carbon in an ML pipeline	CodeCarbon	integrated CO₂eq, close to the power meter
Raw CPU accuracy (RAPL)	Scaphandre	hardware metrology
Calibration / ground truth	power meter	the standard everything is calibrated against

It is not “one tool”, it is a stack by objective: continuous production (Kepler/DCGM), bench (Zeus), reporting (CodeCarbon), calibration (power meter). Mixing them according to the phase is what gives a complete and credible energy measurement.

From figure to action

Measuring is the start; the value is in acting. With energy per pod in Prometheus:

Reporting (CSRD): the carbon footprint per workload, calibrated, ready for regulatory reporting.
Energy/carbon-aware scheduling: move deferrable load to hours or nodes of lower intensity (connects with the hourly carbon of the Spanish grid).
Power capping: with Zeus, trim peak power with little performance loss, lowering the W in the numerator of J/token.
Optimisation: identify pods with poor energy efficiency (a lot of energy per token) and attack them with quantisation or batching.

The measurement stack is not a panel to look at: it is the input to scheduling, optimisation and reporting decisions. Just as OpenCost closes the cost loop, Kepler+DCGM closes the energy one.

From energy to carbon: the final layer

The energy figure (kWh) becomes carbon by multiplying by the grid intensity, and here the practical stack connects with the hourly source. In Spain, the intensity is published by Red Eléctrica (esios) and aggregated by ElectricityMaps, hour by hour. The pattern:

# Carbon per workload: energy (kWh) × grid intensity (gCO2/kWh)
(sum by (namespace) (increase(kepler_container_joules_total[1h])) / 3.6e6)
 * on() group_left() grid_carbon_intensity_gco2_kwh

where grid_carbon_intensity_gco2_kwh is a metric you feed from esios/ElectricityMaps (an exporter or a periodic push). With that, the Grafana panel shows gCO₂ per namespace and per token, hour by hour, not an annual average. And since Spanish intensity varies a lot (from ~80 in solar hours to ~250 on gas-heavy nights), that hourly resolution is what enables carbon-aware scheduling: moving the deferrable work to the clean hours lowers the measured carbon. The measurement stack does not only report carbon; with the hourly figure, it allows it to be reduced.

Energy efficiency KPIs to watch

KPI	What it indicates	Where from
J/token per model	energy efficiency	Kepler ÷ engine tokens
Mean power per pod	consumption per workload	`rate(kepler_container_joules_total)`
gCO₂/token (hourly)	real footprint	energy × grid intensity
Idle energy	waste	power of pods with no traffic
Effective PUE	DC overhead	total energy ÷ compute energy

These five turn the energy panel into a decision tool: which model is more efficient, which pod wastes, when it pays to schedule. Without them the stack is a pretty panel; with them, it is the input to optimisation and reporting.

Example: energy attribution on a multi-tenant cluster

With Kepler splitting per pod, an energy report per team on the example node (8×H100, 5.6 kW of boards, ~7.84 kW with PUE), over one month:

Team	Energy/month (kWh)	gCO₂ (ES grid ~160)	Mean J/token
A · chat-prod	~3,100	~496 kg	~2.2 J/tok
B · batch	~1,400	~224 kg	~2.8 J/tok
C · experimentation	~900	~144 kg	~6.5 J/tok
idle	~250	~40 kg	—

What it reveals: A is efficient (FP8, high utilisation → 2.2 J/token); C is inefficient (6.5 J/token, underused GPU), just as it dragged the idle cost in the FinOps track. The carbon column takes the figure into the per-team CSRD report. And the idle row (~250 kWh, ~40 kg CO₂) is pure wasted energy. Without Kepler, that split does not exist: you would only see the node’s total consumption, with no idea whose each watt is. With it, energy is attributable and actionable, exactly like cost with OpenCost. In fact, the two panels together (OpenCost cost + Kepler energy) give the complete picture of efficiency per team.

Energy and carbon-aware scheduling

The highest-return action the stack enables: scheduling by energy and carbon. With energy per pod and hourly grid intensity in Prometheus, two policies:

By carbon: move the deferrable load (training, ingestion, overnight batch) to the hours of lowest grid intensity. In Spain, the hours of peak solar (~80 gCO₂/kWh) against gas-heavy nights (~250). The same workload emits ~3× less when scheduled at the right hour.
By efficiency: consolidate workloads to eliminate idle (a powered GPU with no work is wasted energy), and direct each workload to the hardware most efficient for it.

Both use the same data (energy per pod + hourly intensity) that the stack already exposes, and they connect with cluster scheduling (Kueue/Volcano). It is the difference between measuring carbon (and reporting it) and reducing it (by scheduling). The C3 stack enables both.

Deployment checklist

Step	Action	Verification
1	dcgm-exporter (GPU Operator)	`DCGM_FI_DEV_POWER_USAGE` in Prometheus
2	Kepler DaemonSet, kernel with eBPF	`kepler_container_joules_total` per pod
3	`dcgmEndpoint` pointing at the exporter	GPU/MIG attribution works
4	Grid intensity metric (esios)	`grid_carbon_intensity` in Prometheus
5	Calibration vs power meter (once per node type)	error bar documented
6	Grafana panels (J/token, carbon, idle)	KPIs visible

Step 5 (calibration) is the one most often skipped and the one that gives the figure credibility; step 3 (dcgmEndpoint) is the one most often forgotten, and without it the GPU is not attributed.

Energy by phase: prefill vs decode

A level of detail the stack allows and that guides optimisation: power is not constant during a request. Prefill (processing the prompt) is compute-bound and draws a lot of power in a burst; decode (generating tokens) is memory-bound and draws a lower peak but for longer. With fine DCGM sampling that bimodal profile is visible, and it has consequences:

A workload with long prompts (a lot of prefill) has a different power profile from one with long generation (a lot of decode), even if they produce the same tokens.
Power capping affects each phase differently: trimming peak power mostly touches the prefill.

For most cases the mean J/token is enough; to optimise energy efficiency thoroughly, separating prefill and decode consumption (aligning the DCGM window with each phase) reveals where the spend is. It is the energy equivalent of separating TTFT and TPOT in benchmarking, the same principle, measuring the two phases separately because they behave differently.

State of the art: the LLM resolution gap

An honest note on the limits of current tooling. The general-purpose tools (Kepler, CodeCarbon, Scaphandre) measure well at machine, process and pod level, but they lack LLM-specific resolution: out of the box they do not distinguish prefill from decode energy, nor energy per token by phase, nor the energy cost of a reasoning block against a normal response. It is an open gap in the field: per-pod attribution exists, but attribution per token and per inference phase has to be built by crossing power (DCGM/Kepler) with engine metrics (vLLM exposes tokens per phase). The 2026 state of the art is a mature stack for infrastructure and an emerging one for LLM-specific resolution, exactly the crossing that the reproducible harness (S4) has to build by hand. Acknowledging it avoids promising a per-token accuracy that the general-purpose tools do not deliver on their own.

Limits and traps (data-driven)

Uncalibrated estimation. Kepler estimates; without calibrating against a power meter at least once, the figure can carry a noticeable error, especially when it falls back on regression models.
Forgetting the dcgmEndpoint. Without pointing Kepler at the dcgm-exporter, GPU (and MIG) attribution does not work; it is the key parameter for GPU.
Accounting for overhead badly. The bulk of the cost of measuring is in Prometheus; if you stand it up just for energy, budget for it (1.5 cores, 15 GB).
Confusing bench and production. Zeus measures one workload finely; Kepler attributes continuously in production. Using one for the other gives the wrong figure.
Misaligned windows. For energy per token, the power window and the token window must coincide; if they do not, the J/token does not correspond.

With the measurement stack in place, the energy track has its instrumentation; the following articles (C6 carbon, C8 TCO and regulation) build on this data. The foundation is this: a continuous per-pod measurement, calibrated, on top of the observability infrastructure you already have.

Closing

Measuring the energy of an AI cluster in production does not require a new system: it requires adding Kepler, per-pod attribution via eBPF, to the DCGM, Prometheus and Grafana stack you already have for observability. That is the central idea of C3: the cost of measuring is almost entirely in the Prometheus you already pay for, and Kepler turns it from “card power” into “energy per pod, per team and per token”. With two cautions that separate a credible figure from a decorative panel: calibrate once against a power meter (to have an error bar that survives an audit) and point the dcgmEndpoint (without which the GPU is not attributed). Done that way, energy stops being the soft axis of sustainability and becomes an actionable time series: the J/token per model that exposes inefficiency, the hourly carbon that enables clean scheduling, the idle that gets recovered, and the CSRD report that holds up. For the sovereign platform, the Kepler+DCGM stack closes the energy loop just as OpenCost closes the cost one, and together, with GuideLLM’s goodput, they give the three faces of the dashboard measured on the same iron, in euros, joules and grams of CO₂.

Sources

Kepler · GitHub (CNCF, eBPF, por pod/nodo) — https://github.com/sustainable-computing-io/kepler
Kepler · configuración (fuentes, dcgmEndpoint para MIG) — https://github.com/sustainable-computing-io/kepler/blob/main/docs/user/configuration.md
CNCF · explorando Kepler (consumo de aplicaciones cloud) — https://www.cncf.io/blog/2023/10/11/exploring-keplers-potentials-unveiling-cloud-application-power-consumption/
Red Hat · Kepler: monitorización eficiente de energía para Kubernetes — https://next.redhat.com/2023/08/22/introducing-kepler-efficient-power-monitoring-for-kubernetes/
arXiv · Container-level Energy Observability in Kubernetes (overhead) — https://arxiv.org/pdf/2504.10702
NVIDIA · dcgm-exporter — https://github.com/NVIDIA/dcgm-exporter
Zeus Project (ml.energy) — https://ml.energy/zeus/

Energy benchmarking in AI: frameworks, metrics and state of the art (tool by tool)

Sat, 13 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The energy and carbon references are European (France, Germany, Spain), since this is a sovereign proposal. The dollar sign is not used (on this site it is a formula delimiter).

What this introduction covers

Fourth article in the data series and deep dive on the energy axis. Measuring the watts of an AI workload looks like a sustainability detail, but it is at once a cost lever (electricity is 30–50 % of TCO), an axis of sovereignty (carbon per token depends on the country) and, increasingly, a regulatory obligation (emissions reporting). This article inventories the metrics, where each figure physically comes from, the data sheet for each measurement framework, and how to go from watt to carbon with European figures. No recommendations; only data and methodology, because in energy, even more than in performance, the measurement method changes the result.

Why measuring energy matters (with numbers)

Three quantified reasons, connecting with the opening article:

Reason	Figure	Implication
Cost	electricity = 30–50 % of TCO	measuring energy is measuring almost half the cost
Sovereignty/carbon	France ~9× fewer gCO₂/token than Germany	the country of the cluster is a lever
Scale/regulation	datacenters ~460 TWh in 2025 (≈1.8 % worldwide), >800 TWh projected for 2028	emissions reporting (CSRD) mandatory

At 700 W per H100, an 8-card node draws of the order of 5.6 kW from the boards alone, ~49,000 kWh a year. Applying PUE 1.4 that becomes ~68,700 kWh/year, and there the country decides the bill: at the French wholesale price (~0.058 €/kWh) that is ~3,980 €/year of electricity per node; in Germany (~0.091 €/kWh), ~6,250 €/year, 57 % more for the same iron and the same work. Multiplied across a fleet, it is a first-order cost line, and a carbon footprint that, in Europe, has to be measured and reported. Energy stopped being an engineering detail and became a business and compliance variable.

The energy metrics

Metric	Definition	Unit
Mean power	average instantaneous consumption	W
Energy per token	energy consumed per generated token	J/token
Energy per inference	energy per request	µJ – Wh
Total energy	integral of power over time	Wh, kWh
PUE (Power Usage Effectiveness)	datacenter overhead (cooling, losses)	ratio ≥ 1
Carbon	energy × grid intensity	gCO₂eq

The two base identities of the whole axis:

$$\text{energy per token (J)} = \frac{\text{mean power (W)}}{\text{throughput (tok/s)}}$$ $$\text{carbon} = \text{energy (kWh)} \times \text{PUE} \times \text{grid intensity (gCO}_2\text{/kWh)}$$

The first has the same shape as cost per token: it shares the throughput denominator, so raising tokens/s lowers energy per token. The second brings in the two multipliers external to the compute, the datacenter PUE and the country’s grid intensity, which can change carbon per token by an order of magnitude without touching the stack.

Worked example: energy per token of an 8×H100 node

To anchor the identities with numbers, the end-to-end calculation on the example node (8×H100, Llama 3.1 70B FP16, vLLM at 2,800 tok/s):

Step	Calculation	Result
Board power	8 × 700 W	5,600 W
Energy per token (boards)	5,600 ÷ 2,800	2.0 J/token
Energy per token (with PUE 1.4)	2.0 × 1.4	2.8 J/token
Energy per 1M tokens (boards)	2 × 10⁶ J	0.56 kWh
Energy per 1M tokens (with PUE)	0.56 × 1.4	0.78 kWh

From there come the electricity cost and the carbon, by country:

Country	Electricity cost / 1M tok	Carbon / 1M tok
France (0.058 €/kWh; ~40 gCO₂/kWh)	~0.045 €	~31 gCO₂
Spain (0.054 €/kWh; ~160 gCO₂/kWh)	~0.042 €	~125 gCO₂
Germany (0.091 €/kWh; ~363 gCO₂/kWh)	~0.071 €	~283 gCO₂

And the link with performance: if an optimisation (FP8, say) raised throughput to 4,200 tok/s, energy per token would drop to ~1.33 J/token (boards) and everything below it in the same proportion. Electricity cost and carbon per token fall with throughput, by the identity. Note: these are board + PUE figures; the complete node (CPU, NICs, PSUs) consumes somewhat more, and energy measured with a power meter can differ from energy estimated with nvidia-smi.

Where the figure comes from: the measurement layers

Not every energy figure is equally reliable. The figure can come from four layers, with increasing accuracy and decreasing context:

The GPU is the dominant source in LLM inference, and its power is read from a hardware counter (NVML/nvidia-smi, aggregated by DCGM, the same basis as GPU observability). RAPL gives CPU and DRAM. The eBPF + model layer (Kepler) splits the total watt between containers and pods without instrumenting the application. And the physical power meter is the ground truth against which estimates are calibrated: no software estimate is better than the meter, only more convenient.

How it is instrumented (and where the error creeps in)

Four instrumentation decisions explain most of the discrepancies between energy measurements:

Sampling rate. nvidia-smi/DCGM read power at intervals; sampling that is too coarse misses the peaks and underestimates the energy. It must be fine enough to capture the power variation between prefill (high) and decode (lower).
Board vs node. GPU (board) power does not include CPU, NICs, fans or PSU losses (typically an extra 10–20 %). Comparing a board energy figure with a complete-node one is comparing different things.
Idle baseline. A powered GPU with no work already draws power. Is that idle attributed to the workload or discounted? The decision changes energy per token, and idle matters: it is real energy that someone pays for.
Time window. Energy is the integral of power over time; the measurement window has to line up exactly with the workload (without counting warm-up or shutdown), or the number does not correspond to the work measured.

These decisions, as in performance benchmarking, mean that the measurement method matters as much as the system measured. A J/token without a stated sampling rate, board/node boundary and idle treatment is not comparable.

Frameworks, tool by tool

Zeus (ml.energy, University of Michigan)

What it does: measure and optimise the energy of deep learning workloads. Method: NVML counters. Scope: NVIDIA and AMD GPUs, CPU, DRAM, Apple Silicon and NVIDIA Jetson. Licence: OSS; academic backing (NSDI'23 paper). Differentiator: it does not only measure, it also optimises (power capping, frequency selection) to cut energy without significant loss of performance (Zeus Project, NSDI'23). It is the reference tool when the goal is to lower J/token, not just to measure them.

CodeCarbon

What it does: estimates the energy and the CO₂eq of compute processes. Method: reads the GPU with nvidia-smi and estimates CPU/RAM. Scope: CPU + GPU + RAM. Licence: OSS (Python). Differentiator: it integrates into ML flows and CI pipelines, and it is the one that comes closest to the power meter among the Python tools (comparison). It brings carbon context out of the box. Limit: a high-level estimate, less exact than direct hardware measurement.

CarbonTracker

What it does: tracks energy and carbon at GPU run time. Scope: GPU only, it measures neither CPU nor memory. Licence: OSS. Differentiator: lightweight, easy to add to a training/inference script. Limit: partial coverage (no CPU/DRAM), so it undercounts the total system energy.

Scaphandre

What it does: a power metrology agent written in Rust. Method: RAPL (reads /proc, splits joules by CPU ticks). Scope: process or VM, with the raw accuracy of the hardware layer. Licence: OSS. Differentiator: the highest direct accuracy among the general-purpose tools, at the cost of bringing no carbon context (it is pure metrology) (Scaphandre · GitHub). Limit: focused on CPU/RAPL; for GPU it has to be combined with NVML.

Kepler (CNCF)

What it does: a Prometheus exporter that measures energy at container, pod and node level in Kubernetes. Method: eBPF + ML models to estimate consumption per workload. Language: Go + C. Licence: Apache 2.0 (the sustainable-computing-io project) (Kepler · GitHub). Differentiator: it is the cloud-native option for attributing energy per pod without instrumenting the app, just as OpenCost attributes cost. Limit: attribution via eBPF + model is estimation, not direct measurement; its accuracy depends on the model.

MLPerf Power (MLCommons)

What it does: a standardised benchmark of energy efficiency, from µW to MW. Maintainer: MLCommons. Differentiator: cross-vendor comparability of efficiency, with a dataset spanning several years and workload versions (MLPerf Power, arXiv 2410.12032). It is to energy what MLPerf Inference is to performance: the comparison standard between vendors.

Comparison table

Tool	Method	Scope	Licence	Accuracy / note
Zeus	NVML / counters	NV+AMD GPU, CPU, DRAM, Apple, Jetson	OSS	measures and optimises DL energy
CodeCarbon	nvidia-smi + estimation	CPU+GPU+RAM, CO₂eq	OSS (Python)	the closest to the power meter among Python tools
CarbonTracker	GPU runtime	GPU only	OSS	lightweight; no CPU/mem
Scaphandre	RAPL (/proc)	process/VM	OSS (Rust)	raw accuracy; no carbon context
Kepler	eBPF + model	container/pod/node (K8s)	Apache 2.0 (CNCF)	cloud-native; model-based estimation
MLPerf Power	standardised measurement	µW to MW	MLCommons	cross-vendor comparability

Accuracy: who to believe

The difference between tools is not a nuance: Scaphandre offers the raw accuracy of the hardware layer (RAPL), while the Python ones such as CodeCarbon give higher-level, carbon-aware estimates, with more context but less direct exactness; among the Python tools, CodeCarbon is the one that comes closest to the power meter, followed by CarbonTracker, with variability across infrastructures (comparison). The rule: for sizing and design, a software estimate will do; to defend a figure before a committee or an auditor, calibrate against a power meter at least once, and use the software tool as a continuous proxy. The measured figure beats the estimated one; the rest is convenience.

Energy leaderboards and benchmarks

To compare efficiency between models without building your own test bench:

Resource	What it provides
MLPerf Power	standardised efficiency measurement, from µW to MW
HF AI Energy Score	comparable per-model energy efficiency ratings (HF)
ML.ENERGY Benchmark	automated measurement and optimisation of inference energy
awesome-green-ai	curated index of Green AI resources and tools (repo)

These resources give the order of magnitude of the energy per token of each model, useful for choosing a model on efficiency at the design stage. But they do not replace measurement on your hardware and your workload: a leaderboard measures on one specific bench, not on your node.

From watt to carbon: France, Germany, Spain

Here the energy axis turns into a sovereignty argument. Carbon per token comes from multiplying the energy by the country’s grid intensity, which in Europe varies by almost two orders of magnitude:

Country	Wholesale (€/MWh, Mar 2026)	Grid carbon (gCO₂/kWh)	Profile
France	~58	~20–60	nuclear: cheap and very clean
Spain	~54	~150–170	renewables + gas: cheap, medium carbon
Germany	~91	~363 (2024 average)	coal/gas + renewables: expensive and dirty

Sources: wholesale prices (TradingEconomics), Germany’s carbon ~363 gCO₂/kWh (2024 average, Wikipedia), nuclear ~17–35 gCO₂e/kWh (EEA), Spain ~167 gCO₂/kWh (2020). Hourly intensity comes from ElectricityMaps.

The calculation, on the example node (0.78 kWh per million tokens, with PUE): in France (~40 gCO₂/kWh) it is ~31 gCO₂ per million tokens; in Germany (~363 gCO₂/kWh), ~283 gCO₂, a factor of ~9×, without touching a line of the stack. Placing the cluster in France or Spain is at once a cost lever, a carbon lever and a compliance lever (EU AI Act, controls ENS × 42001 × EU AI Act).

Energy efficiency levers

Once it is measured, it can be brought down. The levers, ordered by the identity J/token = W ÷ throughput (lower the numerator or raise the denominator):

Lever	Mechanism	Effect
Quantisation (FP8/INT4)	more tokens per GPU-hour, less VRAM/KV	raises throughput → lowers J/token
Batching / continuous batching	amortises the fixed power cost	raises throughput → lowers J/token
Power capping (Zeus)	limits GPU power	lowers W with little loss of throughput
Efficiency-aware scheduling	consolidate workloads, power down idle GPUs	eliminates idle consumption
Location (country)	cleaner grid	lowers carbon per token
Avoiding over-reasoning	fewer reasoning tokens that add nothing	less compute → less energy

Quantisation (see quantisation for inference) is the highest-return lever because it moves all three axes at once: it raises throughput, lowers cost and lowers energy per token. Zeus’s power capping is energy-specific: it trims peak power with little performance loss, lowering the W in the numerator. And idle, the GPU powered but not working, is pure wasted energy that scheduling recovers.

From design to production: where energy is measured

As with the other axes, energy is measured at three moments, with different tools and increasing accuracy:

Moment	Tool	What it gives
Design	TDP × hours (datasheet)	ceiling estimate for sizing
Benchmark	Zeus, MLPerf Power, power meter	energy per token measured on a controlled bench
Production	Kepler (eBPF), DCGM	continuous per-pod consumption in the cluster

The design figure (TDP) serves to size the utility feed and the cooling, but it overestimates (the GPU is rarely at TDP 100 % of the time). The benchmark figure is the one used for the proposal’s cost and carbon per token, because it is measured and reproducible. And the production figure (Kepler/DCGM) closes the loop: it checks that real consumption matches the budget and feeds the footprint report. The three have to be traceable to each other, which is what the harness in article S4 does.

Energy in TCO and European regulation

The closing of the axis: energy is not only sustainability, it is cost and compliance.

Cost: being 30–50 % of TCO, every J/token improvement translates directly into euros. In the example, moving from Germany (~0.091 €/kWh) to France (~0.058 €/kWh) lowers the electricity share by ~37 % on top of the carbon.
Regulation (CSRD): European companies must report their carbon footprint (scope 2). A low-carbon grid (France) directly improves that reported figure, and measuring energy per token stops being optional and becomes a reporting requirement.
EU AI Act: efficiency and energy consumption form part of the documentation expected of AI systems, which makes energy measurement part of the technical file.

One reporting nuance worth knowing: the electricity footprint is scope 2 (indirect emissions from purchased energy), and it can be accounted for by location (the real intensity of the country’s grid) or by market (according to the guarantees of origin contracted). Location-based accounting is the one that genuinely rewards putting the cluster on a clean grid such as the French one; market-based accounting allows green energy to be “bought” through certificates. Knowing which one your CSRD report requires changes which lever (physical location vs contract) lowers the figure, and both start from measuring energy per token, which is what this axis provides.

In other words: measuring energy covers at once a first-order cost line, the carbon and sovereignty argument, and a regulatory obligation. For a sovereign European proposal, the energy axis is not the “nice sustainability bit”: it is one of the three columns with a number attached.

Checklist for a reproducible energy measurement

For an energy or carbon figure to be defensible before a committee or an auditor, it has to come with its context. The minimum to fix and publish:

What to fix	Why
Tool + version	each layer (RAPL, NVML, eBPF) and tool measures differently
Sampling rate	coarse sampling misses peaks
Board vs node boundary	±10–20 % depending on whether CPU/PSUs are included
PUE used	it multiplies all the compute energy
Source and time of the grid intensity	carbon varies ~9× by country and by hour
Idle treatment	attributed or discounted changes the J/token
Model, precision and workload	energy depends on what is served and how
Calibration vs power meter	at least once, to validate the estimate

The rule: if you cannot hand over this table alongside the gCO₂/token figure, the figure is not auditable. The reproducible harness of article S4 records all these parameters so that the energy figure is as traceable as the cost and performance ones, and so that the reported footprint (CSRD) holds up under audit.

State of the art 2026

LLM-specific resolution is missing: general-purpose tools measure well at machine or process level, but they lack LLM-specific resolution (energy per token, per prefill/decode phase), and the simulators lack emissions modelling, an open gap in the field.
eBPF as a trend (Kepler) for attributing energy per pod without instrumenting the app, in parallel to how OpenCost attributes cost.
Standardisation of comparison via MLPerf Power and HF AI Energy Score.
Optimisation, not only measurement (Zeus): the field is moving from “how much do I spend” to “how do I spend less without losing performance”.

Limits and traps (data-driven)

Estimation vs measurement. An nvidia-smi figure is not a power meter. Calibrate against hardware at least once before defending a number.
Partial coverage. CarbonTracker measures GPU only; Scaphandre, mostly CPU/RAPL. Add the layers or you undercount the total energy.
Forgetting PUE. Board energy does not include cooling or losses; without PUE you underestimate the real consumption and carbon.
Carbon without a country. A gCO₂/token without a stated grid (and hour) means nothing: it varies ~9× between France and Germany.
Confusing energy with sustainability. Here it is cost and compliance, with a number; treating it as a “green” extra is losing one of the three columns of the proposal.

That closes the run of introductions: cost, performance and energy, each with its frameworks and its methodology. The rest of the series goes tool by tool through to the dashboard that underpins the sovereign architecture decision, with European figures, measured and reproducible.

Closing

Energy is the axis easiest to dismiss as “the green one” and the one that hides the most: it is almost half the cost, it defines the footprint Europe requires to be reported, and its carbon per token changes ~9× depending on whether the cluster sits in France or in Germany. Measuring it properly, with the right layer, the right sampling, PUE included and the country’s grid, turns a soft sustainability argument into a hard column of the proposal, with a number and a source. And the sovereign conclusion stands on its own with the data in this article: an inference platform in France or Spain is at once cheaper, cleaner and compliant with European jurisdiction than the same platform on a US hyperscaler, and that is demonstrated with measured J/token, gCO₂/kWh by country and euros, not with intentions. The energy axis does not decorate the proposal: it closes it.

Sources

Zeus Project (ml.energy, UMich) — https://ml.energy/zeus/
Zeus · USENIX NSDI'23 — https://www.usenix.org/system/files/nsdi23-you.pdf
Kepler (CNCF, eBPF) · GitHub — https://github.com/sustainable-computing-io/kepler
Scaphandre · GitHub — https://github.com/hubblo-org/scaphandre
CarbonTracker · GitHub — https://github.com/saintslab/carbontracker
MLPerf Power (arXiv 2410.12032) — https://arxiv.org/html/2410.12032v2
HF AI Energy Score — https://huggingface.github.io/AIEnergyScore/
awesome-green-ai — https://github.com/samuelrince/awesome-green-ai
TradingEconomics · precio electricidad Francia/Alemania/España — https://tradingeconomics.com/france/electricity-price
EEA · intensidad de emisiones de la electricidad en Europa — https://www.eea.europa.eu/en/analysis/indicators/greenhouse-gas-emission-intensity-of-1
Electricity sector in Germany (carbono ~363 gCO₂/kWh) — https://en.wikipedia.org/wiki/Electricity_sector_in_Germany

Kepler on lo0 — Tech Blog

Measuring energy in production: Kepler, DCGM and the practical stack (accuracy and overhead)

What this article covers

The measurement stack on Kubernetes

DCGM exporter: the base of GPU power

Kepler in depth: eBPF and the sources

Deployment: a DaemonSet on top of Prometheus

PromQL queries: energy per pod

A calibration procedure

Accuracy: estimation vs power meter

Overhead: the cost of measuring

Zeus: for the bench and for optimisation, not for continuous production

CodeCarbon and Scaphandre in practice

Choosing the stack by objective

From figure to action

From energy to carbon: the final layer

Energy efficiency KPIs to watch

Example: energy attribution on a multi-tenant cluster

Energy and carbon-aware scheduling

Deployment checklist

Energy by phase: prefill vs decode

State of the art: the LLM resolution gap

Limits and traps (data-driven)

Closing

See also

Sources

Energy benchmarking in AI: frameworks, metrics and state of the art (tool by tool)

What this introduction covers

Why measuring energy matters (with numbers)

The energy metrics

Worked example: energy per token of an 8×H100 node

Where the figure comes from: the measurement layers

How it is instrumented (and where the error creeps in)

Frameworks, tool by tool

Zeus (ml.energy, University of Michigan)

CodeCarbon

CarbonTracker

Scaphandre

Kepler (CNCF)

MLPerf Power (MLCommons)

Comparison table

Accuracy: who to believe

Energy leaderboards and benchmarks

From watt to carbon: France, Germany, Spain

Energy efficiency levers

From design to production: where energy is measured

Energy in TCO and European regulation

Checklist for a reproducible energy measurement

State of the art 2026

Limits and traps (data-driven)

Closing

See also

Sources