--- title: "Measuring energy in production: Kepler, DCGM and the practical stack (accuracy and overhead)" url: https://blog.lo0.es/en/posts/measuring-energy-in-production-kepler-dcgm-stack/ markdown: https://blog.lo0.es/en/posts/measuring-energy-in-production-kepler-dcgm-stack/index.md lang: en date: 2026-06-14 lastmod: 2026-09-13 author: "José Manuel Román" description: "A practical deep dive into energy tooling: how the measurement stack is deployed on Kubernetes (DCGM exporter + Kepler eBPF + Prometheus + Grafana), which sources each piece takes its figures from, how accurate it is against a power meter, how much overhead it adds, how energy is attributed per pod and per MIG, and how to choose the stack. With commands and sources." tags: ["energy","kepler","dcgm","ebpf","prometheus","zeus","kubernetes","carbon","data-series"] translations: es: https://blog.lo0.es/posts/herramientas-energia-deploy-precision-overhead/index.md --- # Measuring energy in production: Kepler, DCGM and the practical stack (accuracy and overhead) > Notation: amounts in **euros (N €)**, decimals with a point. The dollar sign is not used > (on this site it is a formula delimiter). ## What this article covers Third article of the energy track (C3). C1 made the conceptual inventory of the tools; here we get practical: how the measurement stack is deployed on Kubernetes, where each piece takes its figures from, how accurate it is against a power meter, how much overhead it adds, and how energy is attributed per pod and per MIG instance. It is the equivalent of B2/B3 for energy: to stop talking about what each tool measures and start building the bench that measures it. With commands; no recommendations, only the mechanics. --- ## The measurement stack on Kubernetes To measure energy per workload on a GPU cluster, the practical stack has four pieces, and the good news is that three of them you already have if you do observability: > **Diagram:** Energy measurement stack on Kubernetes: DCGM exporter and Kepler export to Prometheus, which Grafana visualises, with the power meter as calibration — [see the HTML version](https://blog.lo0.es/en/posts/measuring-energy-in-production-kepler-dcgm-stack/) The new piece compared with a normal observability stack is Kepler, which adds **energy attribution per pod**. DCGM already gives GPU power; Prometheus stores it; Grafana draws it. It is reused infrastructure, not a new system, in line with [GPU observability via DCGM](https://blog.lo0.es/en/posts/gpu-observability-llm-inference-twelve-dcgm-vllm-metrics/index.md). --- ## DCGM exporter: the base of GPU power GPU power, the dominant source in LLM inference, comes from NVIDIA's dcgm-exporter (part of the GPU Operator), which exposes DCGM metrics to Prometheus. The key metric is `DCGM_FI_DEV_POWER_USAGE` (instantaneous power in W), alongside utilisation (`DCGM_FI_DEV_GPU_UTIL`) and memory. With that, energy is the integral of power over time across the workload's window. DCGM gives power at card level; to split it per pod or per MIG instance you need a layer on top, which is Kepler. --- ## Kepler in depth: eBPF and the sources Kepler (Kubernetes-based Efficient Power Level Exporter) is a Prometheus exporter that measures energy at container, pod and node level, using eBPF to keep overhead low and scientific methods to improve accuracy ([Kepler · GitHub](https://github.com/sustainable-computing-io/kepler)). Its strength is that it combines several power sources: | Source | What it covers | |---|---| | **RAPL** | CPU and DRAM | | **NVML** | NVIDIA GPU power | | **ACPI** | platform power management | | **Redfish / IPMI** | platform power (BMC) | | **Regression models** | estimation when there is no real metric | For the GPU, the `dcgmEndpoint` parameter points at the dcgm-exporter metrics URL, which enables power attribution per MIG instance ([Kepler · configuration](https://github.com/sustainable-computing-io/kepler/blob/main/docs/user/configuration.md)). That is: DCGM gives the card's power, Kepler splits it between the pods (and MIG partitions) that use it, via eBPF and models. The result is per-pod energy metrics in Prometheus, ready for Grafana, usable for scheduling or scaling, consumption reporting and visualisation ([Kepler · GitHub](https://github.com/sustainable-computing-io/kepler)). > **Diagram:** Kepler combines RAPL, NVML/DCGM, ACPI and Redfish, and with eBPF splits the energy per pod and per MIG instance — [see the HTML version](https://blog.lo0.es/en/posts/measuring-energy-in-production-kepler-dcgm-stack/) --- ## Deployment: a DaemonSet on top of Prometheus The deployment is a DaemonSet (one Kepler per node), pointing at the existing Prometheus and at the dcgm-exporter: ```bash # Helm, pointing at the dcgm-exporter for GPU attribution helm install kepler kepler/kepler \ --namespace kepler --create-namespace \ --set dcgmEndpoint="http://dcgm-exporter:9400/metrics" ``` Prerequisites: a kernel with eBPF, the dcgm-exporter running (from the GPU Operator) and Prometheus scraping. Once deployed, Kepler exposes metrics such as `kepler_container_joules_total` per pod, queried with PromQL and drawn in Grafana. Energy per workload stops being a manual estimate and becomes a continuous time series, just like cost with OpenCost. --- ## PromQL queries: energy per pod With Kepler exporting, these are the queries that build the panels. The exact names vary by version; check your `/metrics`: ```promql # Cumulative energy per pod (joules) kepler_container_joules_total # Mean power per namespace over the last hour (W) sum by (namespace) (rate(kepler_container_joules_total[1h])) # GPU power per node (from DCGM, W) sum by (node) (DCGM_FI_DEV_POWER_USAGE) # Energy per token: power ÷ throughput (with a gateway/engine metric) sum(rate(kepler_container_joules_total[5m])) / sum(rate(vllm_generation_tokens_total[5m])) ``` The last one closes the circle with the cost track: it crosses Kepler's energy with the engine/gateway tokens to give J/token in real time, per workload. With that in Grafana you have energy per pod, per namespace and per token, continuously, the energy equivalent of the OpenCost cost report. --- ## A calibration procedure Since Kepler estimates, it is worth calibrating once per node type. A simple procedure: 1. **Power meter at the node input** (or a PDU/BMC reading), measuring the node's real power. 2. **Controlled load**: a stable sweep at a known power (the engine at 80 % GPU, say). 3. **Compare**: the node's sum of `kepler_container_joules_total` plus idle against the meter reading over the same window. 4. **Adjust** Kepler's factor or model if the deviation is noticeable, and document the residual error. 5. **Repeat** for each node type (calibration is not transferable between different hardware). With that residual error documented, Kepler's continuous figure has a known error bar, which is exactly what a CSRD auditor or a technical committee will ask for. Measuring without calibrating gives a number with no credibility; calibrating once turns the continuous estimate into a defensible figure. --- ## Accuracy: estimation vs power meter The data honesty of the energy track: Kepler estimates (eBPF + models + counters), it does not measure with a power meter. Accuracy depends on the source: with NVML/DCGM and RAPL available, the figure is fairly faithful (it reads hardware counters); when it falls back on regression models (with no real metric), it is an estimate with more error. Good practice: calibrate against a physical power meter at least once per node type, adjust, and use Kepler as a continuous proxy. The meter is the point-in-time ground truth; Kepler, the continuous series calibrated against it. For a CSRD report that an auditor may challenge, that initial calibration is what gives the continuous figure credibility. --- ## Overhead: the cost of measuring Measuring energy has its own compute cost, worth budgeting for. In a Kubernetes environment, the overhead of the monitoring system is split between the Prometheus server and the targets: the Prometheus server consumes ~150 % of CPU (1.5 cores) and ~15 GB of RAM, while the targets (node exporter and dcgm-exporter) together use ~35 % of CPU and ~300 MB of RAM ([arXiv · Container-level Energy Observability](https://arxiv.org/pdf/2504.10702)). Kepler uses eBPF precisely to keep its own overhead low. The reading: the bulk of the cost of measuring is in Prometheus (which you probably already have for observability), not in the energy exporters. Adding Kepler to an existing observability stack is marginal; standing up Prometheus just for this would be expensive. That is why measuring energy is nearly free if you already observe, and a project if you do not. --- ## Zeus: for the bench and for optimisation, not for continuous production Zeus (ml.energy) plays a different role: it is not a continuous production exporter, it is the bench tool for measuring and optimising the energy of one specific workload (a benchmark, a training run), with power capping and frequency selection. It fits into the benchmarking flow (alongside GuideLLM/AIPerf): during a sweep, Zeus measures the energy per configuration and lets you test the effect of power capping on performance. The practical division: Kepler/DCGM in production (continuous per-pod attribution), Zeus on the bench (fine measurement + optimisation of one workload), CodeCarbon if you want carbon context integrated into an ML pipeline. --- ## CodeCarbon and Scaphandre in practice The other two pieces, with their specific role: - **CodeCarbon** integrates inside the code of an ML pipeline (a decorator or a context manager wrapping the training or the inference), and returns the energy and the CO₂eq of that block, using `nvidia-smi` for the GPU and estimating CPU/RAM. It fits when you want the carbon figure per experiment or per ML job, integrated into the data flow, not at infrastructure level. It is the closest to the power meter among the Python tools. - **Scaphandre** is a metrology agent in Rust that reads RAPL and gives process/VM power with raw hardware accuracy. For CPU it is very accurate; for GPU it has to be combined with NVML. It fits when you need the most faithful CPU metrology, or a lightweight agent independent of Kubernetes. The division with Kepler: Kepler attributes energy per pod in the cluster (the infrastructure view); CodeCarbon gives carbon per ML job (the pipeline view); Scaphandre gives raw CPU metrology. They do not compete; they cover different layers, and a mature platform uses whichever matches the question. --- ## Choosing the stack by objective | Objective | Tool | Why | |---|---|---| | Continuous per-pod attribution (production) | **Kepler + DCGM** | eBPF, per pod/MIG, into Prometheus | | Fine measurement + optimisation (bench) | **Zeus** | measures and optimises (power capping) | | Carbon in an ML pipeline | **CodeCarbon** | integrated CO₂eq, close to the power meter | | Raw CPU accuracy (RAPL) | **Scaphandre** | hardware metrology | | Calibration / ground truth | **power meter** | the standard everything is calibrated against | It is not "one tool", it is a stack by objective: continuous production (Kepler/DCGM), bench (Zeus), reporting (CodeCarbon), calibration (power meter). Mixing them according to the phase is what gives a complete and credible energy measurement. --- ## From figure to action Measuring is the start; the value is in acting. With energy per pod in Prometheus: - **Reporting (CSRD)**: the carbon footprint per workload, calibrated, ready for regulatory reporting. - **Energy/carbon-aware scheduling**: move deferrable load to hours or nodes of lower intensity (connects with the hourly carbon of the Spanish grid). - **Power capping**: with Zeus, trim peak power with little performance loss, lowering the W in the numerator of J/token. - **Optimisation**: identify pods with poor energy efficiency (a lot of energy per token) and attack them with quantisation or batching. The measurement stack is not a panel to look at: it is the input to scheduling, optimisation and reporting decisions. Just as OpenCost closes the cost loop, Kepler+DCGM closes the energy one. --- ## From energy to carbon: the final layer The energy figure (kWh) becomes carbon by multiplying by the grid intensity, and here the practical stack connects with the hourly source. In Spain, the intensity is published by **Red Eléctrica (esios) and aggregated by ElectricityMaps**, hour by hour. The pattern: ```promql # Carbon per workload: energy (kWh) × grid intensity (gCO2/kWh) (sum by (namespace) (increase(kepler_container_joules_total[1h])) / 3.6e6) * on() group_left() grid_carbon_intensity_gco2_kwh ``` where `grid_carbon_intensity_gco2_kwh` is a metric you feed from esios/ElectricityMaps (an exporter or a periodic push). With that, the Grafana panel shows **gCO₂ per namespace and per token, hour by hour**, not an annual average. And since Spanish intensity varies a lot (from ~80 in solar hours to ~250 on gas-heavy nights), that hourly resolution is what enables carbon-aware scheduling: moving the deferrable work to the clean hours lowers the measured carbon. The measurement stack does not only report carbon; with the hourly figure, it **allows it to be reduced**. --- ## Energy efficiency KPIs to watch | KPI | What it indicates | Where from | |---|---|---| | **J/token per model** | energy efficiency | Kepler ÷ engine tokens | | **Mean power per pod** | consumption per workload | `rate(kepler_container_joules_total)` | | **gCO₂/token (hourly)** | real footprint | energy × grid intensity | | **Idle energy** | waste | power of pods with no traffic | | **Effective PUE** | DC overhead | total energy ÷ compute energy | These five turn the energy panel into a decision tool: which model is more efficient, which pod wastes, when it pays to schedule. Without them the stack is a pretty panel; with them, it is the input to optimisation and reporting. --- ## Example: energy attribution on a multi-tenant cluster With Kepler splitting per pod, an energy report per team on the example node (8×H100, 5.6 kW of boards, ~7.84 kW with PUE), over one month: | Team | Energy/month (kWh) | gCO₂ (ES grid ~160) | Mean J/token | |---|---|---|---| | A · chat-prod | ~3,100 | ~496 kg | ~2.2 J/tok | | B · batch | ~1,400 | ~224 kg | ~2.8 J/tok | | C · experimentation | ~900 | ~144 kg | ~6.5 J/tok | | idle | ~250 | ~40 kg | — | What it reveals: A is efficient (FP8, high utilisation → 2.2 J/token); C is inefficient (6.5 J/token, underused GPU), just as it dragged the idle cost in the FinOps track. The carbon column takes the figure into the per-team CSRD report. And the idle row (~250 kWh, ~40 kg CO₂) is pure wasted energy. Without Kepler, that split does not exist: you would only see the node's total consumption, with no idea whose each watt is. With it, energy is attributable and actionable, exactly like cost with OpenCost. In fact, the two panels together (OpenCost cost + Kepler energy) give the complete picture of efficiency per team. --- ## Energy and carbon-aware scheduling The highest-return action the stack enables: scheduling by energy and carbon. With energy per pod and hourly grid intensity in Prometheus, two policies: - **By carbon**: move the deferrable load (training, ingestion, overnight batch) to the hours of lowest grid intensity. In Spain, the hours of peak solar (~80 gCO₂/kWh) against gas-heavy nights (~250). The same workload emits ~3× less when scheduled at the right hour. - **By efficiency**: consolidate workloads to eliminate idle (a powered GPU with no work is wasted energy), and direct each workload to the hardware most efficient for it. Both use the same data (energy per pod + hourly intensity) that the stack already exposes, and they connect with cluster scheduling (Kueue/Volcano). It is the difference between measuring carbon (and reporting it) and reducing it (by scheduling). The C3 stack enables both. --- ## Deployment checklist | Step | Action | Verification | |---|---|---| | 1 | dcgm-exporter (GPU Operator) | `DCGM_FI_DEV_POWER_USAGE` in Prometheus | | 2 | Kepler DaemonSet, kernel with eBPF | `kepler_container_joules_total` per pod | | 3 | `dcgmEndpoint` pointing at the exporter | GPU/MIG attribution works | | 4 | Grid intensity metric (esios) | `grid_carbon_intensity` in Prometheus | | 5 | Calibration vs power meter (once per node type) | error bar documented | | 6 | Grafana panels (J/token, carbon, idle) | KPIs visible | Step 5 (calibration) is the one most often skipped and the one that gives the figure credibility; step 3 (`dcgmEndpoint`) is the one most often forgotten, and without it the GPU is not attributed. --- ## Energy by phase: prefill vs decode A level of detail the stack allows and that guides optimisation: power is not constant during a request. Prefill (processing the prompt) is compute-bound and draws a lot of power in a burst; decode (generating tokens) is memory-bound and draws a lower peak but for longer. With fine DCGM sampling that bimodal profile is visible, and it has consequences: - A workload with long prompts (a lot of prefill) has a different power profile from one with long generation (a lot of decode), even if they produce the same tokens. - **Power capping** affects each phase differently: trimming peak power mostly touches the prefill. For most cases the mean J/token is enough; to optimise energy efficiency thoroughly, separating prefill and decode consumption (aligning the DCGM window with each phase) reveals where the spend is. It is the energy equivalent of separating TTFT and TPOT in benchmarking, the same principle, measuring the two phases separately because they behave differently. --- ## State of the art: the LLM resolution gap An honest note on the limits of current tooling. The general-purpose tools (Kepler, CodeCarbon, Scaphandre) measure well at machine, process and pod level, but they **lack LLM-specific resolution**: out of the box they do not distinguish prefill from decode energy, nor energy per token by phase, nor the energy cost of a reasoning block against a normal response. It is an open gap in the field: per-pod attribution exists, but attribution **per token and per inference phase** has to be built by crossing power (DCGM/Kepler) with engine metrics (vLLM exposes tokens per phase). The 2026 state of the art is a mature stack for infrastructure and an emerging one for LLM-specific resolution, exactly the crossing that the reproducible harness (S4) has to build by hand. Acknowledging it avoids promising a per-token accuracy that the general-purpose tools do not deliver on their own. --- ## Limits and traps (data-driven) 1. **Uncalibrated estimation.** Kepler estimates; without calibrating against a power meter at least once, the figure can carry a noticeable error, especially when it falls back on regression models. 2. **Forgetting the `dcgmEndpoint`.** Without pointing Kepler at the dcgm-exporter, GPU (and MIG) attribution does not work; it is the key parameter for GPU. 3. **Accounting for overhead badly.** The bulk of the cost of measuring is in Prometheus; if you stand it up just for energy, budget for it (1.5 cores, 15 GB). 4. **Confusing bench and production.** Zeus measures one workload finely; Kepler attributes continuously in production. Using one for the other gives the wrong figure. 5. **Misaligned windows.** For energy per token, the power window and the token window must coincide; if they do not, the J/token does not correspond. With the measurement stack in place, the energy track has its instrumentation; the following articles (C6 carbon, C8 TCO and regulation) build on this data. The foundation is this: a continuous per-pod measurement, calibrated, on top of the observability infrastructure you already have. ## Closing Measuring the energy of an AI cluster in production does not require a new system: it requires adding Kepler, per-pod attribution via eBPF, to the DCGM, Prometheus and Grafana stack you already have for observability. That is the central idea of C3: the cost of measuring is almost entirely in the Prometheus you already pay for, and Kepler turns it from "card power" into "energy per pod, per team and per token". With two cautions that separate a credible figure from a decorative panel: calibrate once against a power meter (to have an error bar that survives an audit) and point the `dcgmEndpoint` (without which the GPU is not attributed). Done that way, energy stops being the soft axis of sustainability and becomes an actionable time series: the J/token per model that exposes inefficiency, the hourly carbon that enables clean scheduling, the idle that gets recovered, and the CSRD report that holds up. For the sovereign platform, the Kepler+DCGM stack closes the energy loop just as OpenCost closes the cost one, and together, with GuideLLM's goodput, they give the three faces of the dashboard measured on the same iron, in euros, joules and grams of CO₂. ## See also - [MLPerf Power](https://blog.lo0.es/en/posts/mlperf-power-energy-efficiency-benchmark/index.md) — the standard measurement at the wall (PTDaemon), complementary to software instrumentation (DCGM/RAPL/Kepler). - [Measuring the power of a GPU: NVML, DCGM and sampling errors](https://blog.lo0.es/en/posts/measuring-gpu-power-nvml-dcgm-sampling-errors/index.md) — the sensor metrology that feeds all these tools: averaging window, cumulative counter and what falls outside the counter. - [LLM energy efficiency leaderboards](https://blog.lo0.es/en/posts/llm-energy-leaderboards-wh-per-token/index.md) — where the J/token these tools produce end up: the public rankings and how to interpret them without being misled by the setup. - [From watt to carbon: PUE, grid intensity and the real cost of a token](https://blog.lo0.es/en/posts/watt-to-carbon-pue-grid-intensity-gco2-per-token/index.md) — the conversion step from measured watts to gCO₂eq and to euros, applying datacenter PUE and the country's electricity grid intensity. - [Energy efficiency levers in LLM inference](https://blog.lo0.es/en/posts/levers-fewer-watts-per-token-llm-inference/index.md) — the optimisations that reduce the consumption these tools measure: quantization, batching, engine and power schedulers. ## Sources - Kepler · GitHub (CNCF, eBPF, por pod/nodo) — https://github.com/sustainable-computing-io/kepler - Kepler · configuración (fuentes, dcgmEndpoint para MIG) — https://github.com/sustainable-computing-io/kepler/blob/main/docs/user/configuration.md - CNCF · explorando Kepler (consumo de aplicaciones cloud) — https://www.cncf.io/blog/2023/10/11/exploring-keplers-potentials-unveiling-cloud-application-power-consumption/ - Red Hat · Kepler: monitorización eficiente de energía para Kubernetes — https://next.redhat.com/2023/08/22/introducing-kepler-efficient-power-monitoring-for-kubernetes/ - arXiv · Container-level Energy Observability in Kubernetes (overhead) — https://arxiv.org/pdf/2504.10702 - NVIDIA · dcgm-exporter — https://github.com/NVIDIA/dcgm-exporter - Zeus Project (ml.energy) — https://ml.energy/zeus/