Dcgm on lo0 — Tech Blog

Measuring the power of a GPU: NVML, DCGM and the sampling errors that invalidate your watts

Sat, 29 Aug 2026 08:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

TL;DR

A GPU power sensor does not return what almost everyone believes. The NVML documentation is explicit: on Ampere except GA100, and on later architectures including H100, nvmlDeviceGetPowerUsage returns power averaged over a one-second interval, not instantaneous power. The University of Oxford study presented at SC24, covering more than 70 GPUs across 25 models checked against an external 1 mΩ meter, adds the fact that breaks most published measurements: on the A100 and H100 the sensor only samples 25 % of the time (a 25 ms window within a 101 ms period), so during the remaining 75 % the GPU may be consuming something radically different. Integrating power naively over nine real benchmarks, the mean error was 39.27 %; applying good practice brought it down to 4.89 %. The operational consequence is short: if the hardware is Volta or later, use the accumulated energy counter (nvmlDeviceGetTotalEnergyConsumption, in millijoules) and subtract, rather than integrating samples. And the GPU counter is not the bill: on measured 8× H100 nodes, the node reaches 8.4 kW against 5.6 kW of summed GPU TDP, node idle is 1.8 kW, and with an average PUE of 1.54 the NVML watts are of the order of 43 % of the watts the utility bills.

What the hardware exposes

NVML: power

nvmlDeviceGetPowerUsage(device, unsigned int* power) returns milliwatts for the GPU and its associated circuitry, for example the memory. It has been supported since Fermi. The note in the official documentation is the part almost nobody quotes (NVML Device Queries):

On Fermi and Kepler the reading is accurate to ±5 % of current consumption. On Ampere (except GA100) or later, the API returns power averaged over a 1 s interval. On GA100 and earlier architectures instantaneous power is returned.

In other words, on an H100 or an L40S the usual call already delivers a one-second moving average. Anyone sampling it at 1 Hz and integrating is applying a second filter on top of an already filtered signal, without declaring it. To disambiguate, NVML exposes two separate fields accessible via nvmlDeviceGetFieldValues: NVML_FI_DEV_POWER_AVERAGE and NVML_FI_DEV_POWER_INSTANT, with command-line equivalents (nvidia-smi --query-gpu=power.draw.average and power.draw.instant).

NVML: the energy counter

nvmlDeviceGetTotalEnergyConsumption returns accumulated energy in millijoules since the last driver reload, supported on Volta or later. It is a 64-bit integer, so overflow is not a practical problem: an H100 at 700 W consumes of the order of 6.1·10¹⁰ J per year against a counter range of 1.8·10¹⁶ J. The only event that resets it is a driver reload or a system restart, and that does need watching: a modprobe -r nvidia halfway through a campaign produces a negative delta.

NVIDIA does not document the internal time resolution of the counter, its accumulation mechanism or its accuracy, and I have found no peer-reviewed work validating it against an external meter. The recommendation to use it, which the rest of this sheet rests on, is a structural argument (it removes quadrature error and aliasing error), not a published empirical validation.

NVML: power limits

Function	Unit	Semantics
`nvmlDeviceGetPowerManagementLimit`	mW	Currently configured limit
`nvmlDeviceGetPowerManagementLimitConstraints`	mW	Legal minimum and maximum range
`nvmlDeviceGetPowerManagementDefaultLimit`	mW	Limit the card boots with
`nvmlDeviceGetEnforcedPowerLimit`	mW	Effective limit after considering all limiters, including the out-of-band interface

For an auditable measurement you have to record EnforcedPowerLimit, not PowerManagementLimit: the BMC may be imposing an out-of-band ceiling that the latter does not reflect, and two apparently identical nodes may be operating at different ceilings.

DCGM: the equivalent fields

Field	ID	Unit	Type in the exporter
`DCGM_FI_DEV_POWER_USAGE`	155	watts in floating point	gauge
`DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION`	156	accumulated millijoules	counter
`DCGM_FI_DEV_POWER_MGMT_LIMIT`	160	mW	gauge
`DCGM_FI_DEV_ENFORCED_POWER_LIMIT`	164	mW	gauge

Both energy fields are active in the default counter file of dcgm-exporter. The audit conclusion is direct: a standard Prometheus panel integrating DCGM_FI_DEV_POWER_USAGE is integrating a gauge that is already a one-second average of the sensor, sampled at the exporter’s collect-interval. For energy per token the correct metric is counter 156 with increase().

The real accuracy of the sensor

The reference is the work of Yang, Adámek and Armour (University of Oxford), published at SC24 and available as arXiv:2312.02741: more than 70 GPUs across 25 models, including 10 H100s and 10 A100s, checked against an external meter with a 1 mΩ shunt, a 12-bit ADC and internal sampling at 34 kHz.

Update period and averaging window

GPU or architecture	Update period	Averaging window
Kepler and Maxwell	10-50 Hz	logarithmic growth, ~200 ms
Volta (V100)	20 ms	10 ms
Turing	100 ms	100 ms
A100 (GA100)	101 ms	25 ms
Non-GA100 Ampere and Ada, driver earlier than March 2023	100 ms	1 s
Non-GA100 Ampere and Ada, driver 530	100 ms	100 ms
H100	100 ms	25 ms with `.instant`; 1 s with `.average` and with the default field
GH200	100 ms	20 ms on the GPU, 10 ms on the Grace CPU

Two readings of this table. The first: the driver is an experimental variable, not an infrastructure detail; the averaging window changed between drivers earlier than 530, driver 530, and the later ones. Any measurement meant to be compared over time has to record the driver version.

The second is the central finding of the paper. On the A100 and H100 the 25 ms window sits inside a 101 ms period, which gives a sensor duty cycle close to 25 %. The sensor does not average the full interval: it averages a quarter of it and the system presents the result as if it described the whole interval. In the words of the work itself, during the other 75 % of the time the GPU may be consuming radically different power and nvidia-smi never notices.

The steady-state error

The paper also quantifies the claim in the nvidia-smi documentation, which spoke of an accuracy of ±5 watts: the real error is proportional, ±5 %, not absolute. On a GPU capable of consuming 700 W that is ±35 W of over- or underestimation. The errors measured against the external meter, after correcting the time offset, were −4.70 % and −4.53 % on the RTX 3090 and −5.43 % on the A100, and the residue is attributed to the tolerance of the card’s own physical shunt, which no software can correct.

The response to transients

Four distinct behaviours were observed in the 10 % to 90 % rise time: an almost instantaneous real rise with the reading following on the next tick and a delay of 0 to 100 ms; a real rise of several hundred milliseconds with the reading still updating on the next tick; linear growth over a second, which corresponds to power.draw.average; and logarithmic growth over 200 ms, only on Kepler and Maxwell. Out of this comes the most uncomfortable warning in the work: when running a short program, the measured power probably corresponds to the activity before the program.

The two ways of obtaining energy

The first is to integrate samples, which is what any poller does:

$$E \approx \sum_i P(t_i) \cdot \Delta t$$

It accumulates five sources of error: the ±5 % of the sensor, quadrature error from the finite step, aliasing if the load has spectral content above half the sampling frequency, the fact that each sample is already a moving average, and the jitter of 0 to 100 ms between the host clock and the sensor tick.

The second is to subtract the counter:

$$E = \text{counter}(t_1) - \text{counter}(t_0)$$

Two calls and a subtraction. It removes quadrature, aliasing and time misalignment; the ±5 % of the sensor remains.

Why 1 Hz destroys a 200 ms inference

With sampling at 1 Hz, the sampling theorem only allows components below 0.5 Hz to be reconstructed, while a 200 ms pulse has content in the region of 5 Hz. The expected number of samples inside the event is 0.2, so with probability 0.8 none falls inside it: the result is not noisy, it is undefined.

Even sampling at the maximum useful rate, which on the A100 and H100 is about 10 Hz given the 101 ms period, a 200 ms inference produces one or two distinct sensor updates, and each has observed only 25 ms of real activity: effective coverage is around 25 % of the event. And because of the fourth transient behaviour in the previous section, the reading obtained during those 200 ms may describe what was happening before the kernel was launched.

Aliasing with periodic loads

The paper observes the phenomenon directly on the A100: with a square wave of period slightly different from 100 ms, the reading fluctuates between high and low values with a clear beat. An inference server with periodic arrivals, health checks, fixed-size batches or a scheduler tick is exactly that pathological case.

When only instantaneous power is available, the measured good practice from the Oxford work consists of running 32 consecutive iterations or a minimum of 5 s, inserting 8 evenly spaced controlled delays if the window is smaller than the period, repeating over 4 separate trials with a random delay between them, and shifting the series in post-processing to synchronise it with the real activity. Over nine real benchmarks that took the mean error from 39.27 % to 4.89 %.

What the GPU counter does not measure

The scope declared by NVML is the GPU and its associated circuitry: the SXM module or the PCIe card, with its HBM and its regulators. Left out are the CPU, host DRAM, NVSwitch, network cards, storage, fans and power supply losses.

The measured node figures come from the empirical calibration work on 8× H100 SXM5 nodes (arXiv:2506.14551):

Quantity	Value
Nominal node TDP declared by the manufacturer	10.2 kW
Node power at idle	1.8 kW
Maximum measured under saturating load	8.4 kW
Peak on real production loads	never above 76 % of TDP

Two conversion factors come out of that, both my own estimates from those figures: 8 × 700 W is 5.6 kW of GPU against 8.4 kW of node, that is, the GPU is of the order of 67 % of node power and the GPU-to-node factor is around 1.50×. Chaining that with an industry average PUE of 1.54 (Uptime Institute, 2025), the total factor from the NVML counter to the utility feed is around 2.3×; with a PUE of 1.1 it drops to 1.65×. Put in the form that matters to whoever signs the bill: the watts NVML reports are roughly 43 % of the watts the utility bills in an average datacenter.

As a complementary reference, an 80 PLUS Titanium supply at 230 V is 96 % efficient at 50 % load and 91 % at 100 %, so supply losses add between 4 % and 9 % that is already included if you measure at the socket.

The tax of keeping a context open

A GPU serving a loaded model with no traffic does not consume its nominal idle. Measurements over 335,267 production samples from 14 H100s across 18 days, plus controlled experiments, give this table:

GPU	Idle without a CUDA context	Idle with a CUDA context	Extra cost	Percentage of TDP
H100	71.8 W	121.7 W	+49.9 W	7.1 %
A100	53.7 W	80.0 W	+26.3 W	8.8 %
L40S	35.6 W	102.1 W	+66.4 W	19.0 %

The result that changes the mental model: more than 98 % of that extra cost is produced by the open CUDA context, independently of the memory occupied. Varying allocated VRAM between 0 and 72 GB moves power by less than 1 W. The size of the loaded model does not cost watts; keeping the context open does.

That cost is not marginal on a real platform. Over 11,791 long-running jobs, the measured split was 24 % of the time and 7 % of the energy in deep idle, 15 % of the time and 10 % of the energy idle with an active context, and 61 % of the time with 83 % of the energy in execution. On serving loads, GPUs can spend 48 % of their energy in low-activity periods.

CPU and DRAM: RAPL

To close the node you have to measure what is not GPU, and there the interface is RAPL through the kernel powercap framework, at /sys/devices/virtual/powercap/intel-rapl/. The hierarchy exposes intel-rapl:N as the socket and intel-rapl:N:M as subzones (core, uncore, dram), plus the psys domain for the full SoC since Skylake.

File	Content
`energy_uj`	Energy counter in microjoules; writing “0” resets it if the counter supports it
`max_energy_range_uj`	Counter range, that is, the overflow point
`power_uw`	Current power in microwatts
`name`	Zone name

The figures that condition its use come from the characterisation published in ACM TOMPECS: an update roughly every 1 ms, with jitter; an energy quantum of 61 µJ on Haswell and Skylake, 15.3 µJ on Sandy Bridge; and counter overflow in 52 minutes on a Haswell at 84 W, which forces polling at a far shorter period and detecting the wrap with max_energy_range_uj. There is also a thermal drift that is rarely declared: package power for the same load grows between 10 % and 12 % between 37 °C and 74 °C on Haswell. Warming up beforehand is not optional.

Access, on the other hand, is no longer free. The PLATYPUS attack (IEEE S&P 2021) demonstrated a purely software power side channel through RAPL, with CVE-2020-8694 and CVE-2020-8695 attached. The Linux mitigation arrived in commit 949dd0104c49, included in 5.10.0-rc4, which changed the default permissions so that only root can read energy_uj. The microcode mitigation associated with SGX goes further and introduces random noise into the reported energy and changes the reporting frequency: on machines with that mitigation active, RAPL readings are deliberately degraded. Measuring CPU and DRAM from a container therefore requires root, a udev rule relaxing the permissions, or a privileged daemon.

The software stack

Tool	Source of the GPU figure	Counter or power	Default interval	Overhead
Zeus	NVML	Counter if the architecture is Volta or later; polling otherwise	Delimited by windows	Less than 10 ms per call
dcgm-exporter	NVML via DCGM	Both fields available	`collect-interval` = 30,000 ms	5-10 W extra on the server’s IPMI reading
CodeCarbon	NVML	Counter	`measure_power_secs` = 15 s	5.38 % to 46.75 % of time at 1 kHz
Scaphandre	No GPU support; RAPL	`energy_uj` counter	Not documented	3.81 % to 28.38 % at 1 kHz
Kepler	RAPL and IPMI; GPU only as an experimental option	Hardware counters	—	—

Four details that change the choice:

Zeus decides in code with nvmlDeviceGetArchitecture and uses the accumulated counter on Volta or later, with a nestable window API (begin_window / end_window). It is the reference implementation of what this sheet recommends. Its NSDI'23 paper also documents energy reductions of 23.8 % to 75.7 % from choosing batch size and power limit well, and of 3.0 % to 31.5 % from touching only the limit, with five seconds of profiling per point enough for stable results.
Kepler has removed eBPF. The CNCF announcement of June 2026 describes the redesign towards reading /proc and /sys, motivated by the CAP_BPF and CAP_SYSADMIN permissions that blocked deployments, and by the loss of short-lived processes that underestimated the footprint. The new node metric closely follows the IPMI pattern and removes the spurious multi-kilowatt spikes of the previous version. For GPUs it remains an experimental option: today it is not the tool for measuring Wh/token on an accelerator.
CodeCarbon uses the NVML counter, which is correct, but its RAM estimate is a heuristic of 5 W per module derived from the total gigabytes, and the code itself acknowledges it.
DCGM delivers profiling metrics at 1 Hz by default, and warns that collecting at higher frequencies returns zeros because it groups metrics internally.

On the cost of measuring, the empirical overhead study of RAPL-based tools, all forced to 1 kHz, leaves a clear ordering:

Tool	Time overhead
CodeCarbon	5.38 % to 46.75 %
Scaphandre	3.81 % to 28.38 %
Turbostat	2.73 % to 14.37 %
PowerJoular	1.67 % to 8.88 %
perf	−1.00 % to 4.26 %
RAPL read in user space	−0.70 % to 2.93 %
RAPL read in kernel space	−0.17 % to 0.99 %

The underlying difference is that a system call costs of the order of 1.36·10⁻³ ms and an rdmsr instruction between 2.3·10⁻⁴ and 5.6·10⁻⁴ ms, an order of magnitude less. The authors recommend matching the granularity to the phenomenon being measured instead of raising the default frequency, and limiting polling to the necessary domains.

Procedure for making a Wh/token auditable

Fix and record the electrical state. Note EnforcedPowerLimit, the default limit and the driver version; lock clocks or the power ceiling if the experiment requires it.
Warm up to a stable temperature. RAPL thermal drift is 10 % to 12 % between 37 °C and 74 °C, and the power.draw.average transient takes up to a second to settle.
Measure the idle baseline twice: without a CUDA context and with the model loaded. They are two different values and both are needed.
Use windows of at least 5 s or 32 consecutive iterations.
Repeat over 4 separate trials with a random delay, and add 8 evenly spaced delays if you are integrating instantaneous power with a window smaller than the period.
Prefer the accumulated counter whenever the architecture is Volta or later, watching for driver reloads.
Report dispersion. The Oxford paper itself publishes the mean and standard deviation of the error; without an interval, a Wh/token figure is not auditable.
Declare the measurement boundary: GPU, node or utility feed. Without that label, two figures from the same facility differ by a factor of 2.3 with nobody knowing why.

Subtracting idle, or not

There is no consensus, and the honest approach is to treat it as a declared decision. In production the GPU is parked with the model loaded, and that consumption shows up on the bill: subtracting it erases a real cost. To compare the marginal efficiency of two models or two kernels, by contrast, subtracting it isolates the variable. The workable way out is to report both figures with labels: gross Wh/token, which includes idle and corresponds to the bill, and marginal Wh/token, comparable between models. What does not work is mixing them in the same table without distinguishing them.

One estimation trap remains alive in many spreadsheets: multiplying TDP by time overestimates real energy by up to 4.1 times, according to the ML.ENERGY Benchmark.

Power capping: what it costs and what it saves

Configuration	Energy saving	Performance loss
V100 at 200 W against 300 W (67 % of TDP), BERT training	~15 %	barely any degradation
V100 at 200 W, set of workloads	10-20 %	less than 5 %
V100 at 100 W (33 % of TDP)	40-60 %	30-40 %
A100 at 175 W against 400 W (44 % of TDP), LLaMA 65B inference	22-24 %	5-8 %

The asymmetry between phases explains the result. The Microsoft measurement at ASPLOS'24 documents that the prompt phase is short and reaches or exceeds TDP, while the token generation phase is longer and consumes less: the power ceiling penalises TTFT far more than TPOT. In their production cluster, inference uses 79 % of peak power against 97 % for training, and their measurements show that up to 20 % of power can be recovered with less than a 7 % performance loss.

The published sweep from 200 W to 700 W on the H100 and H200 adds the nuance of where the knee is: between 500 W and 700 W each 100 W step only contributes around 10 % of performance on compute-bound loads, while the H100 holds peak memory bandwidth even at 200 W. The most stable operating zones identified are 400 W on the H100 and 500 W on the H200. That the memory subsystem is protected against the trim is the physical reason why the decode phase of an LLM, which is memory-bound, loses so little performance under a power ceiling.

Two warnings about those figures: the H100/H200 sweep was measured with nvidia-smi polling every 10 s, exactly the problem described above, although on long steady loads the bias averages out; and there is no publication isolating the effect of memory clock locking on Wh/token in LLM inference, so any concrete figure on that point has to be your own measurement.

Reference figures

GPU	TDP
H100 SXM	up to 700 W, configurable
H100 NVL	350-400 W, configurable
A100 80GB SXM	400 W
A100 80GB PCIe	300 W
RTX 5090	575 W, with a recommended 1000 W supply
HGX node of 8× H100 SXM	10.2 kW nominal

Average power measured in inference: on an HPC cluster with vLLM on A100, the aggregate power of the GPU set ranged between 999 W and 2,983 W depending on model and parallelism, with Nemotron 70B on 8× A100 at 2,983 W, about 373 W per GPU, around 93 % of TDP.

Energy per token, with the warning that published figures measure different boundaries and cannot be mixed without normalising by GPU count and output tokens:

Source	Figure
TokenPowerBench, H100 94 GB	40 J per token on a standard load, more than 60 J per token in high-throughput configurations
TokenPowerBench, batch effect	−25 % of energy per token between batch 32 and 256 on a 70B model
TokenPowerBench, context effect	from 2K to 10K tokens of context on Llama 3 70B: energy ×3
TokenPowerBench, quantisation effect	Llama 3 405B in FP8 against FP16: −30 % of energy per token
TokenPowerBench, engine effect	TensorRT-LLM and vLLM against the Transformers engine: −25 % to −40 %
Per-prompt measurement in batch	0.0074-0.0289 Wh on 7B to 14B models; 0.0835-0.6912 Wh on 70B to 405B; single unbatched requests, between 10 and 100 times more expensive

Cross-links from the energy track

C2 — Energy per token: methodology and the electricity market: what to do with the joule once it is measured properly, and how the electricity price multiplies it.
C3 — Measuring energy in production: Kepler, DCGM and the practical stack: the continuous deployment of the stack whose metrology this sheet documents.
C3 — Energy benchmarking: frameworks, metrics and the state of the art: the full catalogue of tools and their metrics.
C4 — MLPerf Power: the certified alternative, measuring at the wall with an external analyser instead of the card’s sensor.
C6 — From watt to carbon: PUE, grid and grams of CO₂: the chain that turns the measured watt into emissions and into euros.
C7 — Energy efficiency levers: the quantified catalogue where the power ceiling of this sheet is just one more lever.

Sources

NVIDIA, NVML API Reference, Device Queries — https://docs.nvidia.com/deploy/nvml-api/group__nvmlDeviceQueries.html
NVIDIA, DCGM Library API, Field Identifiers — https://docs.nvidia.com/datacenter/dcgm/3.1/dcgm-api/dcgm-api-field-ids.html
NVIDIA, DCGM Feature Overview — https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html
NVIDIA, dcgm-exporter — https://github.com/NVIDIA/dcgm-exporter
NVIDIA, H100 Tensor Core GPU, especificaciones — https://www.nvidia.com/en-us/data-center/h100/
NVIDIA, A100 Tensor Core GPU, especificaciones — https://www.nvidia.com/en-us/data-center/a100/
NVIDIA, GeForce RTX 5090, especificaciones — https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/
Yang, Adámek y Armour (Oxford), Part-time Power Measurements: nvidia-smi’s Lack of Attention / SC24 — https://arxiv.org/html/2312.02741v3
Yang, GPU_Power_Benchmark, microbenchmark del trabajo anterior — https://github.com/JimZeyuYang/GPU_Power_Benchmark
ML.ENERGY, Measuring GPU Energy: Best Practices — https://ml.energy/blog/energy/measurement/measuring-gpu-energy-best-practices/
You et al., Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training, NSDI'23 — https://www.usenix.org/system/files/nsdi23-you.pdf
Zeus, Measuring Energy — https://ml.energy/zeus/measure/
ML.ENERGY, The ML.ENERGY Benchmark — https://arxiv.org/html/2505.06371v1
CNCF, Kepler, re-architected: improved power accuracy (junio 2026) — https://www.cncf.io/blog/2026/06/30/kepler-re-architected-improved-power-accuracy-and-a-community-call-to-action/
CodeCarbon, How Power Estimation Works — https://docs.codecarbon.io/latest/explanation/power-estimation/
Hubblo, Scaphandre: cálculo del consumo por proceso — https://github.com/hubblo-org/scaphandre/blob/main/docs_src/explanations/how-scaph-computes-per-process-power-consumption.md
Linux Kernel, Power Capping Framework (powercap sysfs) — https://docs.kernel.org/power/powercap/powercap.html
Intel, Running Average Power Limit Energy Reporting, INTEL-SA-00389 — https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/advisory-guidance/running-average-power-limit-energy-reporting.html
Lipp et al., PLATYPUS: With Great Power comes Great Leakage, IEEE S&P 2021 — https://platypusattack.com/
Khan et al., RAPL in Action: Experiences in Using RAPL for Power Measurements, ACM TOMPECS — https://www.devsustainability.com/p/paper-notes-rapl-in-action
What Is the Cost of Energy Monitoring? An Empirical Study on the Overhead of RAPL-Based Tools — https://arxiv.org/html/2604.26815v1
Patel et al. (Microsoft), Characterizing Power Management Opportunities for LLMs in the Cloud, ASPLOS'24 — https://www.microsoft.com/en-us/research/wp-content/uploads/2024/03/GPU_Power_ASPLOS_24.pdf
MIT Lincoln Laboratory, Sustainable Supercomputing for AI: GPU Power Capping at HPC Scale — https://arxiv.org/html/2402.18593
Architectural Trade-offs in the Energy-Efficient Era: power-capping NVIDIA H100 and H200 — https://arxiv.org/html/2604.11391v1
Empirically-Calibrated H100 Node Power Models for Datacenter Energy Analysis — https://arxiv.org/pdf/2506.14551
The Model Parking Tax: Quantifying the Hidden Energy Cost of Always-On GPU Model Deployment — https://arxiv.org/html/2605.23918
The Energy Cost of Execution-Idle in GPU Clusters — https://arxiv.org/html/2604.04745
TokenPowerBench: Benchmarking the Power Consumption of LLM Inference — https://arxiv.org/pdf/2512.03024
From Prompts to Power: Measuring the Energy Footprint of LLM Inference — https://arxiv.org/html/2511.05597
Serving LLMs in HPC Clusters — https://arxiv.org/abs/2507.00418
Uptime Institute, Global Data Center Survey 2025 — https://datacenter.uptimeinstitute.com/rs/711-RIA-145/images/2025.Annual.Survey.Report.pdf

The reproducible harness: measuring cost, performance and energy in a single auditable experiment

Tue, 16 Jun 2026 12:00:00 +0200

Notation: amounts in euros (N €), decimals with a point, thousands with a thin space. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

The “data” series has produced three independent measurement axes: cost per million tokens (OpenCost + LiteLLM), performance under an SLO (GuideLLM + AIPerf) and energy per token (DCGM + Kepler). The problem is that the three have been measured in different articles, with different loads and at different times: they are not comparable with each other. This closing article describes the integrated harness that runs the three axes in the same experiment, on the same node (4×H100 SXM, a generic reference), with all the metadata fixed, the output in versioned JSON/CSV and an idempotent Kubernetes Job. The result is the 3-axis scorecard (€/1M tok, Wh/token, TTFT/ITL P99) that allows configurations to be compared on a multi-objective Pareto frontier and any figure to be audited with the bench to reproduce it.

Why the three axes must be measured together

The series’ opening post (The three axes) established the identity:

$$\text{CPM} = \frac{\text{cost/h}}{\text{throughput (tok/s)} \times 3.6 \times 10^{-3}}$$ $$\text{energy/token (Wh)} = \frac{\text{mean power (W)}}{\text{throughput (tok/s)} \times 3\,600}$$

Throughput is the common denominator. If it is measured in different experiments, different time, different load, different GPU temperature, then CPM and energy/token do not share a denominator: they are three anecdotes, not a scorecard. The harness captures them in the same time window, over the same load, with Prometheus windows aligned to the second. Only then is the scorecard row coherent by construction.

The second reason is reproducibility. The post Bias and reproducibility in benchmarking listed twelve biases that invalidate comparisons: an unpinned engine, an undeclared tokeniser, unfixed input/output lengths, missing warmup, a client outside the cluster. The harness removes all of them because the metadata are part of the Job, not of the documentation.

Architecture of the integrated bench

The harness has four layers. Each one is OSS, exports to Prometheus and lives in the same Kubernetes namespace:

Layer	Tool(s)	Primary metric	Export protocol
Performance	GuideLLM (SLO sweep) + AIPerf	TTFT P99, ITL P99, goodput (tok/s)	native JSON/CSV + `/metrics` OpenMetrics
Cost	OpenCost + LiteLLM proxy	CPM (€/1M tok), cost/request	REST API + Prometheus scrape
Energy (GPU)	DCGM Exporter	`DCGM_FI_DEV_POWER_USAGE` (W), `DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION` (mJ)	Prometheus DaemonSet
Energy (pod)	Kepler	`kepler_container_joules_total`, energy per pod	Prometheus DaemonSet

MLPerf Power is used as an external comparability reference: its published results, with hardware documented in detail, allow the harness figures to be calibrated for plausibility. The in-house bench does not aim to be an MLPerf submission, but to be reproducible on your own cluster.

The Kubernetes Job: the full YAML

The experiment runs as a versioned Kubernetes Job. All the relevant metadata are environment variables declared in the manifest: not in ad-hoc scripts, not in external documentation. The Job is idempotent (same name = same run) and leaves traces in the pod log and in the output volume.

apiVersion: batch/v1
kind: Job
metadata:
 name: bench-llama3-70b-fp8-h100x4-20260616
 namespace: benchmark
 labels:
 bench/model: llama3-70b
 bench/precision: fp8
 bench/engine: vllm-0.9.1
 bench/hardware: h100x4-sxm
 bench/isl: "1024"
 bench/osl: "256"
 bench/concurrency: "32"
 bench/tokenizer: meta-llama-3-tokenizer-v3
 bench/run-id: "20260616T1200"
spec:
 backoffLimit: 0
 template:
 spec:
 restartPolicy: Never
 serviceAccountName: bench-runner
 volumes:
 - name: results
 persistentVolumeClaim:
 claimName: bench-results-pvc
 initContainers:
 # Warmup: 60 s of traffic before the experiment
 - name: warmup
 image: ghcr.io/vllm-project/guidellm:0.4.2
 command:
 - guidellm
 - benchmark
 - --target
 - http://vllm-svc.inference.svc.cluster.local:8000
 - --rate-type
 - concurrent
 - --rate
 - "4"
 - --max-seconds
 - "60"
 - --data
 - prompt_tokens=1024,output_tokens=256
 env:
 - name: GUIDELLM_ENV
 value: production
 containers:
 - name: guidellm-sweep
 image: ghcr.io/vllm-project/guidellm:0.4.2
 command:
 - guidellm
 - benchmark
 - --target
 - http://vllm-svc.inference.svc.cluster.local:8000
 - --rate-type
 - sweep
 - --max-seconds
 - "120"
 - --data
 - prompt_tokens=1024,output_tokens=256
 - --output-path
 - /results/guidellm-$(BENCH_RUN_ID).json
 env:
 - name: BENCH_RUN_ID
 valueFrom:
 fieldRef:
 fieldPath: metadata.labels['bench/run-id']
 volumeMounts:
 - name: results
 mountPath: /results

 - name: aiperf-concurrent
 image: nvcr.io/nvidia/aiperf:0.2.0
 command:
 - aiperf
 - profile
 - --url
 - http://vllm-svc.inference.svc.cluster.local:8000/v1
 - --model
 - meta-llama/Meta-Llama-3-70B-Instruct
 - --concurrency
 - "4,8,16,32"
 - --input-tokens
 - "1024"
 - --output-tokens
 - "256"
 - --num-requests
 - "200"
 - --output
 - /results/aiperf-$(BENCH_RUN_ID).json
 volumeMounts:
 - name: results
 mountPath: /results

 - name: scorecard-exporter
 image: python:3.12-slim
 command:
 - python
 - /scripts/export_scorecard.py
 - --run-id
 - "$(BENCH_RUN_ID)"
 - --prometheus
 - http://prometheus.monitoring.svc.cluster.local:9090
 - --output
 - /results/scorecard-$(BENCH_RUN_ID).json
 env:
 - name: BENCH_RUN_ID
 valueFrom:
 fieldRef:
 fieldPath: metadata.labels['bench/run-id']
 volumeMounts:
 - name: results
 mountPath: /results

The Job name (bench-llama3-70b-fp8-h100x4-20260616) is the unique identifier of the run. Changing it is enough to register a variant. The labels are the metadata that the scorecard-exporter reads to enrich the output JSON.

Performance layer: GuideLLM and AIPerf

GuideLLM — the SLO-driven sweep

GuideLLM (a vLLM project) generates realistic traffic patterns, synchronous, concurrent, poisson, throughput, sweep, and captures full TTFT and ITL distributions (Red Hat Developer). The sweep mode ramps from idle to saturation over 10 rounds and identifies the knee: the maximum load where goodput ≈ throughput under the declared SLO. The output is JSON/CSV with all the percentiles per round, ready to be versioned.

For the harness, the bench’s reference SLO is:

Metric	Threshold	Percentile
TTFT	500 ms	P99
ITL (TPOT)	50 ms/tok	P95
Error rate	0.5 %	—

The Job command already appears in the YAML above. The JSON output includes, per round: rate (req/s), TTFT (P50/P95/P99), ITL (P50/P95/P99), throughput (tok/s) and goodput (tok/s). The goodput under the SLO at the knee is the value that enters the scorecard.

For the cross-verification layer of the data and comparability with published results, see the post GuideLLM in depth.

AIPerf — NVIDIA’s concurrency profiler

AIPerf (the successor to GenAI-Perf, repository ai-dynamo/aiperf) measures TTFT, ITL, throughput and latency distributions at fixed concurrencies (GitHub ai-dynamo/aiperf). Where GuideLLM gives the automatic sweep up to the knee, AIPerf gives the detailed profile at specific concurrencies (4, 8, 16, 32 in the example): it allows the throughput-latency curve to be characterised point by point.

The two are complementary: GuideLLM finds the knee automatically; AIPerf confirms it and characterises the behaviour in the neighbourhood of that knee. Both outputs go to the results volume with the same BENCH_RUN_ID. The deep analysis of AIPerf/GenAI-Perf is in GenAI-Perf in depth.

Metrics the harness extracts from AIPerf for the scorecard:

# TTFT P99 at concurrency 16 (the closest to the sweep's knee):
aiperf profile ... --concurrency 16 \
 | jq '.results[] | select(.concurrency==16) | .ttft_ms.p99'

# Goodput (tok/s) at concurrency 16:
aiperf profile ... --concurrency 16 \
 | jq '.results[] | select(.concurrency==16) | .output_token_throughput'

Cost layer: OpenCost and LiteLLM

OpenCost — the denominator in euros per GPU-hour

OpenCost (CNCF incubating, opencost.io) allocates Kubernetes cost to namespace, label, pod and container in real time (GitHub opencost/opencost). For the harness, OpenCost answers the question: how much does the vllm-svc pod cost in euros per hour during the experiment’s window?

The harness calls the OpenCost REST API when the experiment finishes:

# Cost of the inference namespace over the experiment window (1 hour)
curl -s "http://opencost.monitoring.svc.cluster.local:9003/allocation" \
 --data-urlencode 'window=2026-06-16T12:00:00Z,2026-06-16T13:00:00Z' \
 --data-urlencode 'aggregate=namespace' \
 --data-urlencode 'namespace=inference' \
 | jq '.data[0].inference.totalCost'

The returned cost (in EUR, configured with the node’s real prices) is divided by the throughput measured in that same window to obtain the CPM:

$$\text{CPM} = \frac{\text{cost}_\text{namespace/h} \times 10^6}{\text{goodput (tok/s)} \times 3\,600}$$

The post OpenCost: cost allocation in Kubernetes details the price configuration and the allocation by GPU label.

LiteLLM — the per-request token counter

LiteLLM (litellm.ai, GitHub BerriAI/litellm) acts as an OpenAI-compatible proxy with token accounting per request, model and team. In the harness, GuideLLM and AIPerf point at the LiteLLM endpoint (which in turn forwards to vLLM): every request is recorded with prompt_tokens, completion_tokens and cost (using the custom pricing configured for the on-prem node).

# litellm-config.yaml (fragment of custom on-prem pricing)
model_list:
 - model_name: llama3-70b-fp8
 litellm_params:
 model: openai/meta-llama/Meta-Llama-3-70B-Instruct
 api_base: http://vllm-svc.inference.svc.cluster.local:8000/v1
 api_key: sk-dummy
 input_cost_per_token: 0.00000109 # 1.09 EUR/1M tok on-prem
 output_cost_per_token: 0.00000109

The LiteLLM records are exported to Prometheus as litellm_request_total_tokens and litellm_spend_metric_total, which the scorecard exporter consumes to compute the real CPM per token type (input vs output).

Energy layer: DCGM and Kepler

DCGM Exporter — GPU power and energy

DCGM Exporter (GitHub NVIDIA/dcgm-exporter, docs) exposes GPU metrics on /metrics for Prometheus as a DaemonSet on the GPU nodes. The two energy metrics the harness uses:

DCGM metric	Type	Unit	Use in the harness
`DCGM_FI_DEV_POWER_USAGE`	gauge	W	instantaneous power per GPU
`DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION`	counter	mJ	energy accumulated since boot

To compute the energy consumed during the experiment’s window (without the idle baseline), the harness takes the difference of the counter before and after the sweep:

# Deployment as a DaemonSet (fragment of the official chart)
helm repo add gpu-helm-charts \
 https://nvidia.github.io/dcgm-exporter/helm-charts
helm install dcgm-exporter gpu-helm-charts/dcgm-exporter \
 --namespace monitoring \
 --set serviceMonitor.enabled=true \
 --set serviceMonitor.interval=15s

The DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION field (in mJ) allows the experiment’s energy to be computed with hardware-counter accuracy, not with an estimate:

$$\text{experiment energy (J)} = \left(\text{counter}_\text{end} - \text{counter}_\text{start}\right) \times 10^{-3}$$ $$\text{Wh/token} = \frac{\text{experiment energy (J)}}{3\,600 \times \text{generated tokens}}$$

Kepler — pod-level energy

Kepler (CNCF sandbox, GitHub sustainable-computing-io/kepler) uses eBPF to estimate energy consumption at container and pod level, exporting kepler_container_joules_total to Prometheus (Red Hat Emerging Technologies). It combines RAPL (CPU/DRAM), NVML (GPU) and regression models when no sensors are available.

In the harness, Kepler complements DCGM: DCGM gives the hardware measurement of the GPU (more precise for GPU-intensive loads such as LLM inference), while Kepler attributes the energy to the specific vLLM pod (including the node’s CPU contribution). The main metric the harness consumes:

# Energy of the vllm-svc pod during the experiment (J)
increase(
 kepler_container_joules_total{
 container_namespace="inference",
 container_name="vllm"
 }[${BENCH_DURATION}]
)

The comparative analysis of DCGM vs Kepler vs Zeus is in Energy tools: deployment, precision and overhead.

The MLPerf Power reference

MLPerf Power (MLCommons, IEEE HPCA 2025 paper) establishes the protocol for measuring the energy efficiency of ML systems with high-precision external measurement. The in-house bench is not an MLPerf submission (that requires external power meters and committee review), but its published results are the calibration reference: if the harness gives a result of the same order as the MLPerf submission for the same hardware with the same model, the measurement is plausible. If it differs by more than a factor of 2, there is a methodological problem. See the post MLPerf Power: energy efficiency for the current reference data.

Unifying the metrics: the PromQL queries

The scorecard-exporter is the Job container that, once GuideLLM and AIPerf have finished, collects all the metrics from Prometheus and builds the JSON of the scorecard row. The key queries:

# Mean GPU power during the sweep (W) — 4×H100 node
avg_over_time(
 sum(DCGM_FI_DEV_POWER_USAGE{Hostname=~"gpu-node-.*"})[${BENCH_DURATION}:15s]
)

# Total GPU energy during the sweep (mJ → convert to J)
(
 sum(DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION{Hostname=~"gpu-node-.*"}) offset 0
 -
 sum(DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION{Hostname=~"gpu-node-.*"}) offset ${BENCH_DURATION}
) * 0.001

# Tokens generated during the sweep (from LiteLLM)
increase(
 litellm_request_total_tokens{model="llama3-70b-fp8", token_type="completion"}[${BENCH_DURATION}]
)

# Cost of the inference namespace over the window (from the OpenCost API)
# → REST call at the start and at the end of the Job, difference of accrual

The ${BENCH_DURATION} variable is the real duration of the sweep (in Prometheus format, e.g. 22m), which the exporter computes as end_ts - start_ts and substitutes into every query.

The output JSON of each run has this structure:

{
 "run_id": "20260616T1200",
 "metadata": {
 "model": "meta-llama/Meta-Llama-3-70B-Instruct",
 "engine": "vllm-0.9.1",
 "precision": "fp8",
 "hardware": "4xH100-SXM-80GB",
 "isl_tokens": 1024,
 "osl_tokens": 256,
 "tokenizer": "meta-llama-3-tokenizer-v3",
 "slo_ttft_p99_ms": 500,
 "slo_itl_p95_ms": 50,
 "bench_tool_guidellm": "0.4.2",
 "bench_tool_aiperf": "0.2.0",
 "dcgm_exporter": "3.3.9",
 "kepler": "0.10.2"
 },
 "performance": {
 "goodput_tok_s": 3120,
 "ttft_p99_ms": 487,
 "itl_p95_ms": 42,
 "throughput_peak_tok_s": 3890,
 "elbow_concurrency": 28
 },
 "cost": {
 "cpm_eur_1m": 0.97,
 "gpu_cost_eur_h": 10.8,
 "cost_window_eur": 3.24
 },
 "energy": {
 "wh_per_token": 0.00044,
 "j_per_token": 1.58,
 "power_mean_w": 4924,
 "energy_total_kwh": 0.287,
 "pue": 1.4,
 "wh_per_token_pue_adjusted": 0.000616
 },
 "carbon": {
 "grid_intensity_gco2_kwh": 40,
 "co2_per_1m_tokens_g": 24.6
 }
}

All the fields are computed from the same time windows. The JSON is versioned in git alongside the harness code. Reproducing any scorecard row is: git checkout <run-id> && kubectl apply -f job.yaml.

The 3-axis scorecard: table and interpretation

The following table illustrates how five configurations of the same model (Llama 3 70B) on the same reference node (4×H100 SXM) compare with the harness. The figures are illustrative of the order of magnitude (the bench fills them in with real measurements):

Config	Engine	Precision	CPM (€/1M)	Goodput (tok/s)	Wh/tok	TTFT P99 (ms)	ITL P95 (ms)	CO2 /1M (g, FR)
A	vLLM 0.9	FP16	1.64	1,890	0.00074	498	49	29.6
B	vLLM 0.9	FP8	0.97	3,120	0.00044	487	42	17.6
C	SGLang 0.4	FP8	0.88	3,410	0.00041	421	38	16.4
D	vLLM 0.9	FP8, ISL 512	1.31	2,240	0.00056	294	31	22.4
E	vLLM 0.9	FP8, max-batch 128	0.84	3,590	0.00040	721	55	16.0

How to read the table:

B vs A: FP8 over FP16 cuts CPM by 41 % and energy per token by 41 % thanks to the same higher throughput. Both meet the SLO (TTFT P99 < 500 ms, ITL P95 < 50 ms). B dominates A on all three axes: it is Pareto-superior.
C vs B: SGLang gives 9 % more goodput and 9 % lower CPM with FP8. Both meet the SLO. C is Pareto-superior to B (if the engine is indifferent for the stack).
D vs B: a shorter ISL (512 tok) lowers latency (TTFT P99 294 ms vs 487 ms) but lowers goodput and raises CPM. If the use case demands TTFT < 300 ms, D is the candidate; if TTFT < 500 ms is enough, B is better on cost and energy.
E vs C: max-batch 128 raises goodput (+5 %) and lowers CPM, but the TTFT P99 breaks the SLO (721 ms > 500 ms) and so does the ITL P95 (55 ms > 50 ms). E has the best raw throughput but is outside the SLO: it is not a valid candidate for the interactive chat use case.

The scorecard’s Pareto frontier, under the declared SLO, includes only A, B, C and D (E falls out for breaking the SLO). Of those four, C dominates B which dominates A. D only enters the frontier if the use case demands TTFT P99 < 300 ms. The decision is not a number; it is the whole row plus the SLO.

The multi-objective Pareto frontier

With three minimisation metrics, CPM (€/1M tok), Wh/tok and TTFT P99 (ms), the Pareto frontier is defined as the set of configurations where none dominates another on all three axes simultaneously. Formally, config $ i $ dominates config $ j $ if:

$$\text{CPM}_i \leq \text{CPM}_j \;\land\; \text{Wh/tok}_i \leq \text{Wh/tok}_j \;\land\; \text{TTFT}_{i} \leq \text{TTFT}_{j}$$

with at least one strict inequality. The scorecard allows it to be computed over the set of runs with a simple vector operation. In the table’s example, the frontier under the SLO (with E excluded) is $\{C, D\}$: C dominates on cost/energy/goodput; D dominates on latency. Between C and D the choice depends on the TTFT SLO of the specific use case.

Reproducible design: the twelve mandatory metadata

The checklist that closes the bias dossier (Bias and reproducibility). For the harness JSON to be auditable, the twelve fields must be present in every run:

#	Field	Example	Why it is mandatory
1	Serving engine version	`vllm-0.9.1`	the same model can perform differently across versions
2	Model version (commit/tag)	`meta-llama/Meta-Llama-3-70B-Instruct@sha256:abc`	avoids ambiguity between variants of the same name
3	Precision	`fp8`	FP16 vs FP8 changes throughput and latency
4	Tokeniser and version	`meta-llama-3-tokenizer-v3`	the ISL/OSL in tokens depends on the tokeniser
5	ISL (input sequence length)	`1024` tok	changes the prefill time and the knee
6	OSL (output sequence length)	`256` tok	changes the decode time
7	Hardware (model and count)	`4×H100-SXM-80GB`	the basis of any comparison
8	Maximum concurrency tested	`32`	defines the sweep’s range
9	Bench tool version	`guidellm-0.4.2`, `aiperf-0.2.0`	different versions can give different metrics
10	Declared SLO	`TTFT P99 < 500 ms, ITL P95 < 50 ms`	the knee depends on the SLO; without it there is no goodput
11	Warmup duration	`60 s`	without warmup the KV cache is cold and the first rounds are biased
12	Harness version / run-id	`20260616T1200`	identifies the run so it can be reproduced with `git checkout`

A scorecard missing any of these twelve fields is an anecdote. With all twelve, it is an auditable datum. The run-id in the Job name (bench-llama3-70b-fp8-h100x4-20260616) implicitly incorporates fields 1-3 and 7, forcing the Job name to change with every variant, which makes it impossible to overlap runs in the same namespace.

Idempotence and versioning of results

The Kubernetes Job is idempotent: if the same manifest is applied again (same name), the Job already exists and Kubernetes does not relaunch it (backoffLimit: 0 policy). This avoids accidental duplicate runs. To relaunch, the Job has to be deleted (kubectl delete job <name>) or the run-id changed.

Results are versioned with the following directory convention in the harness’s git repository:

results/
20260616T1200/
guidellm-20260616T1200.json
aiperf-20260616T1200.json
scorecard-20260616T1200.json
20260617T0900/
guidellm-20260617T0900.json
aiperf-20260617T0900.json
scorecard-20260617T0900.json
scorecard-aggregate.csv # all rows, for analysis and charts

The scorecard-aggregate.csv is the scorecard table in flat format: each row is a run, each column a field of the JSON. It is generated with:

# Generates the aggregate CSV from all the JSON files in results/
python scripts/aggregate_scorecards.py results/ > scorecard-aggregate.csv

The CSV is what feeds the Pareto charts, the reports and the architecture decisions. It is versioned in git; its diff is the configuration change.

How it connects with the other posts in the series

The harness is not a standalone article: it is the synthesis of the twenty-eight articles. Each tool has its deep dive:

Performance: GuideLLM in depth — sweep, SLO, goodput, knee; GenAI-Perf / AIPerf in depth — concurrency, rate sweep, distribution metrics.
Cost: OpenCost: cost allocation in Kubernetes — allocation by namespace/label, custom on-prem prices.
Energy: Energy tools: deployment, precision and overhead — DCGM vs Kepler vs Zeus, instrumentation overhead, relative precision; MLPerf Power: energy efficiency — the external calibration reference.
Theoretical framework: The three axes: cost, performance and energy — the CPM/energy/throughput identity that justifies measuring them together.
Bias: Bias and reproducibility in benchmarking — the twelve biases the harness removes.

The harness closes the series’ dossier because it makes the opening article’s promise possible: “when someone disputes a number in the proposal, the answer is not ‘a blog says so’, but ’this is the bench, this is the methodology, reproduce it’”. With the run-id and the git repo, reproducing is one command.

Full flow of a benchmarking session

The harness’s standard operating procedure, from start to finish:

# 1. Make sure the inference endpoint is up
kubectl get svc vllm-svc -n inference

# 2. Verify that DCGM and Kepler are scraping
curl -s http://prometheus.monitoring.svc.cluster.local:9090/api/v1/query \
 --data-urlencode 'query=DCGM_FI_DEV_POWER_USAGE' | jq '.data.result | length'

# 3. Record the energy counter BEFORE the experiment
PRE_ENERGY=$(kubectl exec -n monitoring dcgm-exporter-xxx -- \
 curl -s localhost:9400/metrics \
 | grep DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION | awk '{sum+=$2} END{print sum}')

# 4. Launch the Job (unique name per run)
kubectl apply -f jobs/bench-llama3-70b-fp8-h100x4-20260616.yaml

# 5. Wait for it to finish
kubectl wait --for=condition=complete job/bench-llama3-70b-fp8-h100x4-20260616 \
 -n benchmark --timeout=3600s

# 6. Record the energy counter AFTER
POST_ENERGY=$(kubectl exec -n monitoring dcgm-exporter-xxx -- \
 curl -s localhost:9400/metrics \
 | grep DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION | awk '{sum+=$2} END{print sum}')

# 7. Collect the results from the volume
kubectl cp benchmark/bench-run-pod:/results/ ./results/20260616T1200/

# 8. Compute the experiment's energy (mJ → kWh)
python scripts/calc_energy.py \
 --pre $PRE_ENERGY --post $POST_ENERGY \
 --run-id 20260616T1200

# 9. Add a row to the aggregate CSV
python scripts/aggregate_scorecards.py results/ > scorecard-aggregate.csv

# 10. Version it
git add results/20260616T1200/ scorecard-aggregate.csv
git commit -m "bench: llama3-70b fp8 4xH100 ISL1024 OSL256 (20260616T1200)"

Steps 6-9 are candidates for automation as a second post-processing Job launched when the main one completes (with ownerReferences or a clean-up CronJob). In the harness’s basic state, the manual steps are precisely the ones that force a review of the result before versioning it.

Cost of the experiment

A GuideLLM sweep of 10 rounds at 120 seconds per round occupies the node for ~22-26 minutes. The AIPerf profile at 4 concurrencies (200 requests each) adds ~8-12 minutes. Total per run: ~35-40 minutes of a 4×H100 node.

At the reference amortised cost (~10.8 €/h for 4×H100 on-prem):

$$\text{cost per run} \approx 10.8 \times \frac{38}{60} \approx 6.84 \text{ EUR}$$

At the reference European cloud price (4 × 2.73 = 10.92 €/h):

$$\text{cost per run (cloud)} \approx 10.92 \times \frac{38}{60} \approx 6.92 \text{ EUR}$$

A continuous benchmarking programme (10 configurations × 3 models × 2 sweeps/week) adds up to ~415 EUR/week. That is why the harness includes short sweeps (5 rounds, ~15 minutes) for CI and full sweeps for releases.

What the harness does not measure (and why)

Missing dimension	Reason	Where it is covered
Output quality (MMLU, HumanEval)	requires a quality evaluation bench, not a serving one	LLM quality benchmarks
Training throughput	the harness is exclusively for inference	outside the scope of the “data” series
Network and storage cost	OpenCost can attribute it, but it requires additional configuration	OpenCost: cost allocation
Hourly carbon intensity	use the ElectricityMaps API + the measured energy	From the watt to carbon: PUE and grid mix
Client-server network latency	the Job runs inside the cluster; WAN latency requires an external client	document as additional metadata

Sources

GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
GuideLLM · PyPI — https://pypi.org/project/guidellm/
Red Hat Developer · desplegar y benchmarkear vLLM con GuideLLM en Kubernetes — https://developers.redhat.com/articles/2025/12/24/how-deploy-and-benchmark-vllm-guidellm-kubernetes
Red Hat Developer · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
AIPerf · GitHub (ai-dynamo/aiperf) — https://github.com/ai-dynamo/aiperf
GenAI-Perf · documentación NVIDIA Triton — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/README.html
NVIDIA · LLM performance benchmarking con GenAI-Perf — https://developer.nvidia.com/blog/llm-performance-benchmarking-measuring-nvidia-nim-performance-with-genai-perf
NVIDIA NIM Benchmarking · métricas (TTFT, ITL, throughput) — https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html
OpenCost · web oficial — https://opencost.io/
OpenCost · GitHub (opencost/opencost) — https://github.com/opencost/opencost
OpenCost · CNCF blog (sandbox → incubating) — https://www.cncf.io/blog/2022/12/06/opencost-a-new-cncf-sandbox-project-for-real-time-kubernetes-cost-monitoring/
LiteLLM · web oficial — https://www.litellm.ai/
LiteLLM · GitHub (BerriAI/litellm) — https://github.com/BerriAI/litellm
LiteLLM · Spend Tracking docs — https://docs.litellm.ai/docs/proxy/cost_tracking
NVIDIA DCGM Exporter · documentación oficial — https://docs.nvidia.com/datacenter/dcgm/latest/gpu-telemetry/dcgm-exporter.html
NVIDIA DCGM Exporter · GitHub — https://github.com/NVIDIA/dcgm-exporter
NVIDIA DCGM Exporter · métricas CSV — https://github.com/NVIDIA/dcgm-exporter/blob/main/etc/dcp-metrics-included.csv
Kepler (Kubernetes-based Efficient Power Level Exporter) · GitHub — https://github.com/sustainable-computing-io/kepler
Red Hat Emerging Technologies · Introducing Kepler — https://next.redhat.com/2023/08/22/introducing-kepler-efficient-power-monitoring-for-kubernetes/
MLCommons · MLPerf Inference Datacenter benchmark — https://mlcommons.org/benchmarks/inference-datacenter/
MLCommons · MLPerf Power presentado en IEEE HPCA 2025 — https://mlcommons.org/2025/03/ml-commons-power-hpca/
IEEE Xplore · MLPerf Power paper (μWatts to MWatts) — https://ieeexplore.ieee.org/document/10946778/
MLCommons · MLPerf Inference v5.1 results (septiembre 2025) — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
Red Hat · Efficient and reproducible LLM inference: MLPerf Inference v5.1 — https://www.redhat.com/en/blog/efficient-and-reproducible-llm-inference-red-hat-mlperf-inference-v51-results
BentoML · Beyond tokens-per-second: cost, speed and quality in LLM inference — https://www.bentoml.com/blog/beyond-tokens-per-second-how-to-balance-speed-cost-and-quality-in-llm-inference

GPU idle: the cost that shows up on no invoice but is paid by the whole TCO

Mon, 15 Jun 2026 05:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

The cost per token of on-premise inference is $ \text{€/GPU-hour} \div \text{throughput} $. Throughput is a direct function of the GPU’s useful occupancy. On a generic node of 4×H100 SXM with a reference cost of ~11 € per GPU-hour (depreciation plus energy plus infrastructure), the cost curve over occupancy looks like this:

Useful occupancy	Effective throughput (tok/s)	Cost per 1M tokens
20 %	~700	~43 €
40 %	~1,400	~21 €
70 %	~2,500	~12 €
85 % (practical ceiling)	~3,000	~10 €

Doubling occupancy from 20 % to 40 % halves the cost per token, without buying more iron. At 70 % the cost competes with European cloud providers (~2.2 €/GPU-hour on-demand at Scaleway). At 20 % on-prem is four times dearer than renting. The lever is not the model, nor the precision: it is how many paid-for GPU-hours produce serving tokens.

1 · The fundamental identity

The cost per million tokens (CPM) in your own inference is not a list price. It is:

$$ \text{CPM} = \frac{C_{\text{GPU}} \cdot N_{\text{GPU}}}{T_{\text{ef}} \times 3600 / 10^6} $$

where $ C_{\text{GPU}} $ is the cost per GPU-hour (€/h), $ N_{\text{GPU}} $ the number of GPUs assigned to the service and $ T_{\text{ef}} $ the effective throughput (tok/s). Making the dependence on occupancy explicit:

$$ T_{\text{ef}} = T_{\text{pico}} \times \rho $$

where $ \rho \in [0,1] $ is the useful occupancy rate (the fraction of time in which the GPU is processing real serving tokens). The resulting identity:

$$ \text{CPM} = \frac{C_{\text{GPU}} \cdot N_{\text{GPU}}}{T_{\text{pico}} \times \rho \times 3600 / 10^6} $$

The direct consequence: CPM is inversely proportional to $ \rho $. Doubling $ \rho $ halves CPM. The numerator (cost of the iron) does not change.

The posts coste-por-token-y-por-request and capacity-planning-inferencia-llm-on-premise cover how to calculate $ C_{\text{GPU}} $ and the reference peak throughput. This article deals with $ \rho $: how to measure it, what limits it and how to raise it.

2 · Why the standard metric lies: `DCGM_FI_DEV_GPU_UTIL`

The field DCGM_FI_DEV_GPU_UTIL (field ID 203) appears in nvidia-smi as “GPU-Util”. Its official definition in the DCGM documentation:

“GPU Utilization” — percentage of time during which one or more kernels were executing on the GPU.

The problem for LLM inference: the decode phase is memory-bound. The GPU runs a kernel reading weights from HBM token by token; therefore GPU_UTIL registers close to 100 % even though the tensor cores are at 15 % of their capacity. The field measures activity, not useful work.

The distinction is critical for FinOps: an operator who sees GPU_UTIL 98 % assumes “saturated GPU, no room for more load”. The reality may be “the tensor cores are at 20 % and the bottleneck is HBM”, which leaves room for continuous batching or additional bin-packing.

The metrics that measure real occupancy are those of the _FI_PROF_* subsystem, available in DCGM 3.x with the profiling module enabled:

Table of relevant DCGM fields

DCGM field	Field ID	What it measures	Unit	Operational note
`DCGM_FI_DEV_GPU_UTIL`	203	% time with ≥1 active kernel	%	Misleading in LLM decode
`DCGM_FI_PROF_SM_ACTIVE`	1002	Ratio of cycles with ≥1 active warp per SM	0–1	Compute activity, not occupancy
`DCGM_FI_PROF_SM_OCCUPANCY`	1003	Resident warps / theoretical maximum per SM	0–1	Real intra-SM parallelism
`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`	1004	% cycles with tensor cores (HMMA) active	0–1	The real compute efficiency metric
`DCGM_FI_PROF_DRAM_ACTIVE`	1005	% cycles with HBM transferring	0–1	Memory saturation
`DCGM_FI_PROF_PCIE_TX_BYTES`	—	TX bytes over PCIe	bytes/s	Useful in PCIe inference
`DCGM_FI_DEV_FB_USED`	252	HBM used	MiB	VRAM budget
`DCGM_FI_DEV_POWER_USAGE`	—	Real consumption	W	For real energy cost
`DCGM_FI_DEV_CLOCK_THROTTLE_REASONS`	—	Bitmap of throttle causes	bitmap	Detects silent degradation

The _PROF_* fields require the DCGM profiling module and the right driver permissions. They are documented exhaustively in the NVIDIA DCGM field ID reference and in the sibling post observabilidad-gpu-dcgm-llm.

Characteristic reading in LLM decode (Llama 70B FP8 on H100)

Metric	Typical decode value	Interpretation
`DCGM_FI_DEV_GPU_UTIL`	95–99 %	A lie: there is an active kernel
`DCGM_FI_PROF_SM_ACTIVE`	0.45–0.65	SMs with warps 45–65 % of the time
`DCGM_FI_PROF_SM_OCCUPANCY`	0.30–0.55	Resident warps at 30–55 % of the maximum
`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`	0.10–0.25	Tensor cores active only 10–25 %
`DCGM_FI_PROF_DRAM_ACTIVE`	0.75–0.90	HBM busy 75–90 % of the time

Reading: decode is memory-bound. The model weights are loaded from HBM for each token; HBM is saturated but the tensor cores wait. All of the “utilisation” in nvidia-smi comes from memory reads, not from compute.

3 · MFU and HFU: occupancy expressed in FLOPs

The canonical compute efficiency metric is MFU (Model FLOPs Utilization), defined in the PaLM paper (Chowdhery et al., arXiv 2204.02311, section 4):

$$ \text{MFU} = \frac{T_{\text{obs}} \times C_{\text{modelo}}}{P_{\text{pico}}} $$

where $ T_{\text{obs}} $ is the observed throughput (tok/s) and $ P_{\text{pico}} $ is the theoretical peak performance of the hardware (FLOP/s).

where $ C_{\text{modelo}} $ is the number of FLOPs per token in a complete forward pass. For a dense transformer with $ P $ parameters, the usual approximation (forward plus backward = 6P FLOPs per token; forward only = 2P):

$$ C_{\text{modelo}} \approx 2P \quad \text{(inference)} $$ $$ C_{\text{modelo}} \approx 6P \quad \text{(training, forward + backward)} $$

HFU (Hardware FLOPs Utilization) measures the FLOPs actually executed in hardware, including activation recomputation (gradient checkpointing). In training with recompute:

$$ C_{\text{hardware}} \approx 8P \quad \text{(forward × 2 + backward × 4)} $$

therefore HFU > MFU when there is recompute; they are identical without it.

Typical MFU values

Regime	Hardware	Typical MFU	Limiting regime
Training (large batch, BF16)	H100 SXM	35–50 %	compute-bound
Training PaLM 540B	TPU v4	46.2 %	compute-bound
Prefill inference (large batch)	H100 SXM	25–45 %	compute-bound
Decode inference (bs=1)	H100 SXM	3–8 %	memory-bound
Decode inference (continuous batching, bs=32–64)	H100 SXM	15–30 %	memory-bound, attenuated

Decode with batch size 1 has a single-digit MFU because the hardware spends most of its time waiting for HBM to deliver weights. Raising the batch size (continuous batching) amortises the weight read across more simultaneous tokens and raises MFU.

The Roofline model

The roofline places each operation in the space (arithmetic intensity, throughput):

Decode with batch size 1 falls in the memory-bound zone to the left of the ridge point. Raising the batch (continuous batching) moves the point to the right and up, closer to the roofline.

4 · TCO sensitivity to occupancy

Assumptions for the example: a generic node with 4×H100 SXM 80 GB, total node cost ~44 €/hour (5-year depreciation of ~220,000 €, energy ~4×700 W at ~0.12 €/kWh, plus rack/colocation infrastructure; see coste-por-token-y-por-request for the detail of the identity). Reference peak throughput with Llama 70B FP8 plus continuous batching: ~3,500 tok/s aggregate.

$$ \text{CPM}(\rho) = \frac{44\ \text{€/h}}{3500 \times \rho \times 3600 / 10^6} = \frac{44 \times 10^6}{3500 \times \rho \times 3600} = \frac{3.49}{\rho}\ \text{€/1M tok} $$

Occupancy $ \rho $	Effective throughput (tok/s)	On-prem CPM	On-demand cloud CPM (~2.2 €/GPU-h)
20 %	700	~17.5 €	~6.3 €
40 %	1,400	~8.7 €	~6.3 €
60 %	2,100	~5.8 €	~6.3 €
70 %	2,450	~5.0 €	~6.3 €
85 %	2,975	~4.1 €	~6.3 €

On-prem / cloud crossover point: with this example hardware, the on-prem cost advantage over a comparable European cloud starts at $ \rho \approx 55\text{-}60\,\% $. Below that, idle makes on-prem dearer than renting. The cloud column is flat because cloud bills per hour used, not per the throughput you extract from it.

Example figures with generic hardware. The real numbers depend on the depreciation price, the local cost of energy, the model and the precision. The structure of the curve, CPM inversely proportional to $ \rho $, is universal.

5 · Idle metrics: where occupancy is lost

Before applying levers, you need to know which kind of idle dominates. Three categories:

Type of idle	Symptom in metrics	Usual cause
Scheduling idle	`DCGM_FI_DEV_POWER_USAGE` low, `SM_ACTIVE` < 0.05	No requests in the queue; GPU waiting for work
Batching idle	`SM_ACTIVE` high, `PIPE_TENSOR_ACTIVE` low, `DRAM_ACTIVE` low	Batch too small; prefill stall between requests
Memory-bound idle	`DRAM_ACTIVE` high, `PIPE_TENSOR_ACTIVE` low	Normal decode; HBM is the bottleneck; insufficient batch size

Scheduling idle is the most expensive and the most directly attacked with bin-packing. Batching idle is attacked with continuous batching. Memory-bound idle in decode does not disappear entirely (it is the physics of the transformer), but it is attenuated with a larger batch size.

The inference engine metrics that complement the diagnosis from the service side are documented in observabilidad-gpu-dcgm-llm and anatomia-metricas-dcgm-vllm-anomalias.

6 · Levers for raising occupancy

Table of lever × effect × when it applies

Lever	Effect on $ \rho $	When it applies	Operational complexity
Continuous batching	High: removes idle between requests; raises decode MFU from ~5 % to ~20 %	Always in inference; enabled by default in vLLM	Low (engine parameter)
Scheduler bin-packing (kube-scheduler / Kueue)	High: concentrates loads on fewer nodes; frees whole nodes for shutdown or rebalancing	Clusters with load that varies over the day	Medium (scheduler policy)
MIG (Multi-Instance GPU)	Medium: fills GPUs with light loads that previously lived alone on a whole GPU	Heterogeneous loads: embeddings + reranker + guardrail + large model	High (hot repartitioning not available)
Time-slicing	Low to medium: raises occupancy in dev/bursts; no isolation	Consumer GPUs (RTX 5090/4090); low-risk multi-tenant dev	Low
MPS	Medium: concurrent execution of multiple small processes; reduces context-switch overhead	Many small concurrent kernels on a datacenter GPU; trust between loads	Medium
Chunked prefill	Medium: interleaves prefill and decode; reduces the TTFT spike and raises throughput	Loads with a mix of short and long prompts	Low (vLLM flag)
Quantisation (FP16→FP8→INT4)	Indirect: raises peak throughput, which lowers CPM for the same $ \rho $	Models with quantised kernel support (native Hopper FP8)	Medium
Autoscaling (KEDA)	Keeps $ \rho $ high by scaling replicas according to the queue	Variable and predictable load; cluster with spare capacity	Medium

The post compartir-gpu-time-slicing-mps-mig details the three sharing mechanisms (time-slicing, MPS, MIG) with VRAM budgets.

Bin-packing with Kueue

Kueue (sigs.k8s.io/kueue) is the Kubernetes-native queue manager for GPU jobs. Its model of cohorts and nominal quotas allows active bin-packing: jobs accumulate in a queue and are launched only when there is a node that can take them whole, instead of fragmenting the load across partially occupied nodes.

BestFit packing in the ClusterQueue is configured with:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
 name: gpu-prod
spec:
 preemption:
 reclaimWithinCohort: Any
 withinClusterQueue: LowerPriority
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 16

The priority-based preemption policy ensures that production jobs displace experimentation ones when there is scarcity, keeping occupancy on the production nodes high.

MIG as a bin-packing lever inside the GPU

MIG makes it possible to fill an H100 with light loads that would otherwise live alone on a whole GPU. A profile of 3×2g.20gb + 1×1g.10gb on an H100 can host simultaneously a 7B FP8 model (~14 GB of weights), two embedding services and an INT4 guardrail, all with hardware isolation. Without MIG, each of those loads would take a whole GPU with an individual $ \rho $ below 10 %.

The profiles available on the H100 80 GB (SXM5) according to NVIDIA’s MIG User Guide:

MIG profile	Compute slices	Memory	Max. instances
`1g.10gb`	1/7 SMs	10 GB	7
`1g.20gb`	1/7 SMs	20 GB	4
`2g.20gb`	2/7 SMs	20 GB	3
`3g.40gb`	3/7 SMs	40 GB	2
`4g.40gb`	4/7 SMs	40 GB	1
`7g.80gb`	7/7 SMs	80 GB	1 (whole GPU)

MIG is not available on consumer GPUs (RTX 5090, RTX 4090). Time-slicing is the only sharing option on that hardware.

7 · Continuous batching: the effect on MFU

Continuous batching (also called iteration-level scheduling or in-flight batching) is the mechanism with the greatest impact on occupancy in inference. The idea: instead of waiting for a complete batch to finish before launching the next, the engine evaluates the pipeline after each token and replaces completed sequences with new requests from the waiting queue.

Quantified effect in vLLM:

Without batching (bs=1, static): decode MFU ~3–8 %; GPU idle between requests.
With continuous batching (dynamic bs 16–64): decode MFU ~15–30 %; the GPU almost never waits.
On pure prefill loads with a large batch (bs=128+): prefill MFU 25–45 %; it approaches the compute roofline.

The vLLM parameter --max-num-seqs controls the maximum number of sequences in the concurrent batch. Raising it increases occupancy until HBM becomes the bottleneck (watch for DCGM_FI_PROF_DRAM_ACTIVE > 90 % sustained).

8 · Hardware and scale: what applies to what

Hardware	MIG	MPS	Time-slicing	Continuous batching	Note
H100 SXM / H200	Yes (7 inst.)	Yes	Yes	Yes	On-prem datacenter reference
A100 SXM/PCIe	Yes (7 inst.)	Yes	Yes	Yes	Previous generation; HBM2e
L40S / L40	No	Yes	Yes	Yes	Ada Lovelace; no MIG; good price/VRAM
RTX 5090	No	Yes (limited)	Yes	Yes	Consumer; no MIG; does not scale in production
RTX 4090	No	Yes (limited)	Yes	Yes	Consumer; 24 GB VRAM; no MIG

The RTX 5090 and RTX 4090 illustrate the case of hardware that does not scale for multi-tenant with isolation: they do not support MIG, VRAM is scarce for models above 7B with a wide KV-cache, and TDP (600 W / 450 W) is high relative to throughput. For production inference at scale, the 4×H100 SXM is the reference node of this series.

9 · FinOps diagnostic flow: from high CPM to the cause

10 · Worked example: impact of continuous batching on CPM

Starting point: Llama 70B FP8 on 4×H100 SXM, a load of 8 requests/s with 512 output tokens on average, without continuous batching (static bs = 8):

Observed throughput: ~900 tok/s (decode dominant).
$ \rho_{\text{efectiva}} \approx 900 / 3500 \approx 0.26 $.
CPM: $ 44 / (900 \times 3600 / 10^6) \approx 13.6\ \text{€} $.

Same load, enabling continuous batching with --max-num-seqs 64:

Observed throughput: ~2,300 tok/s (the dynamic batch fills the gaps).
$ \rho_{\text{efectiva}} \approx 2300 / 3500 \approx 0.66 $.
CPM: $ 44 / (2300 \times 3600 / 10^6) \approx 5.3\ \text{€} $.

CPM reduction: from 13.6 € to 5.3 €, –61 %, with no hardware change and without touching the model.

The vLLM parameters relevant for raising effective throughput:

--max-num-seqs 64 # maximum concurrent batch
--max-num-batched-tokens 16384 # total tokens per iteration
--enable-chunked-prefill # interleaves prefill and decode

Sources

NVIDIA — DCGM Field Identifiers reference (v3.1), lista completa de DCGM_FI_* con field IDs y definiciones. https://docs.nvidia.com/datacenter/dcgm/3.1/dcgm-api/dcgm-api-field-ids.html
NVIDIA — GPU Profiling Metrics (Run:ai / DCGM), definiciones de DCGM_FI_PROF_SM_ACTIVE (1002), SM_OCCUPANCY (1003), PIPE_TENSOR_ACTIVE (1004). https://run-ai-docs.nvidia.com/self-hosted/platform-management/monitor-performance/gpu-profiling-metrics
Chowdhery et al. — PaLM: Scaling Language Modeling with Pathways (MFU definition, sección 4). arXiv 2204.02311. https://arxiv.org/abs/2204.02311
NVIDIA — Multi-Instance GPU (MIG) User Guide (perfiles H100 SXM5, particionado, aislamiento). https://docs.nvidia.com/datacenter/tesla/mig-user-guide/
NVIDIA — Supported MIG Profiles (catálogo completo H100 80 GB). https://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-mig-profiles.html
NVIDIA — Time-Slicing GPUs in Kubernetes (GPU Operator 24.9.0). https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/24.9.0/gpu-sharing.html
NVIDIA — Multi-Process Service (MPS) Overview (kernel concurrency, SM allocation). https://docs.nvidia.com/deploy/mps/latest/index.html
Kueue — Overview (sigs.k8s.io/kueue, bin-packing, cohorts, quotas). https://kueue.sigs.k8s.io/docs/overview/
vLLM Blog — vLLM v0.6.0: 2.7x Throughput Improvement and 5x Latency Reduction (continuous batching, chunked prefill). https://blog.vllm.ai/2024/09/05/perf-update.html
GMI Cloud — NVIDIA H100 GPU Pricing: 2026 Rent vs. Buy Cost Analysis. https://www.gmicloud.ai/en/blog/nvidia-h100-gpu-pricing-2026-rent-vs-buy-cost-analysis
Saxena et al. — LLM Inference Unveiled: Survey and Roofline Model Insights. arXiv 2402.16363. https://arxiv.org/abs/2402.16363

Measuring energy in production: Kepler, DCGM and the practical stack (accuracy and overhead)

Sun, 14 Jun 2026 05:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Third article of the energy track (C3). C1 made the conceptual inventory of the tools; here we get practical: how the measurement stack is deployed on Kubernetes, where each piece takes its figures from, how accurate it is against a power meter, how much overhead it adds, and how energy is attributed per pod and per MIG instance. It is the equivalent of B2/B3 for energy: to stop talking about what each tool measures and start building the bench that measures it. With commands; no recommendations, only the mechanics.

The measurement stack on Kubernetes

To measure energy per workload on a GPU cluster, the practical stack has four pieces, and the good news is that three of them you already have if you do observability:

The new piece compared with a normal observability stack is Kepler, which adds energy attribution per pod. DCGM already gives GPU power; Prometheus stores it; Grafana draws it. It is reused infrastructure, not a new system, in line with GPU observability via DCGM.

DCGM exporter: the base of GPU power

GPU power, the dominant source in LLM inference, comes from NVIDIA’s dcgm-exporter (part of the GPU Operator), which exposes DCGM metrics to Prometheus. The key metric is DCGM_FI_DEV_POWER_USAGE (instantaneous power in W), alongside utilisation (DCGM_FI_DEV_GPU_UTIL) and memory. With that, energy is the integral of power over time across the workload’s window. DCGM gives power at card level; to split it per pod or per MIG instance you need a layer on top, which is Kepler.

Kepler in depth: eBPF and the sources

Kepler (Kubernetes-based Efficient Power Level Exporter) is a Prometheus exporter that measures energy at container, pod and node level, using eBPF to keep overhead low and scientific methods to improve accuracy (Kepler · GitHub). Its strength is that it combines several power sources:

Source	What it covers
RAPL	CPU and DRAM
NVML	NVIDIA GPU power
ACPI	platform power management
Redfish / IPMI	platform power (BMC)
Regression models	estimation when there is no real metric

For the GPU, the dcgmEndpoint parameter points at the dcgm-exporter metrics URL, which enables power attribution per MIG instance (Kepler · configuration). That is: DCGM gives the card’s power, Kepler splits it between the pods (and MIG partitions) that use it, via eBPF and models. The result is per-pod energy metrics in Prometheus, ready for Grafana, usable for scheduling or scaling, consumption reporting and visualisation (Kepler · GitHub).

Deployment: a DaemonSet on top of Prometheus

The deployment is a DaemonSet (one Kepler per node), pointing at the existing Prometheus and at the dcgm-exporter:

# Helm, pointing at the dcgm-exporter for GPU attribution
helm install kepler kepler/kepler \
 --namespace kepler --create-namespace \
 --set dcgmEndpoint="http://dcgm-exporter:9400/metrics"

Prerequisites: a kernel with eBPF, the dcgm-exporter running (from the GPU Operator) and Prometheus scraping. Once deployed, Kepler exposes metrics such as kepler_container_joules_total per pod, queried with PromQL and drawn in Grafana. Energy per workload stops being a manual estimate and becomes a continuous time series, just like cost with OpenCost.

PromQL queries: energy per pod

With Kepler exporting, these are the queries that build the panels. The exact names vary by version; check your /metrics:

# Cumulative energy per pod (joules)
kepler_container_joules_total

# Mean power per namespace over the last hour (W)
sum by (namespace) (rate(kepler_container_joules_total[1h]))

# GPU power per node (from DCGM, W)
sum by (node) (DCGM_FI_DEV_POWER_USAGE)

# Energy per token: power ÷ throughput (with a gateway/engine metric)
sum(rate(kepler_container_joules_total[5m])) / sum(rate(vllm_generation_tokens_total[5m]))

The last one closes the circle with the cost track: it crosses Kepler’s energy with the engine/gateway tokens to give J/token in real time, per workload. With that in Grafana you have energy per pod, per namespace and per token, continuously, the energy equivalent of the OpenCost cost report.

A calibration procedure

Since Kepler estimates, it is worth calibrating once per node type. A simple procedure:

Power meter at the node input (or a PDU/BMC reading), measuring the node’s real power.
Controlled load: a stable sweep at a known power (the engine at 80 % GPU, say).
Compare: the node’s sum of kepler_container_joules_total plus idle against the meter reading over the same window.
Adjust Kepler’s factor or model if the deviation is noticeable, and document the residual error.
Repeat for each node type (calibration is not transferable between different hardware).

With that residual error documented, Kepler’s continuous figure has a known error bar, which is exactly what a CSRD auditor or a technical committee will ask for. Measuring without calibrating gives a number with no credibility; calibrating once turns the continuous estimate into a defensible figure.

Accuracy: estimation vs power meter

The data honesty of the energy track: Kepler estimates (eBPF + models + counters), it does not measure with a power meter. Accuracy depends on the source: with NVML/DCGM and RAPL available, the figure is fairly faithful (it reads hardware counters); when it falls back on regression models (with no real metric), it is an estimate with more error. Good practice: calibrate against a physical power meter at least once per node type, adjust, and use Kepler as a continuous proxy. The meter is the point-in-time ground truth; Kepler, the continuous series calibrated against it. For a CSRD report that an auditor may challenge, that initial calibration is what gives the continuous figure credibility.

Overhead: the cost of measuring

Measuring energy has its own compute cost, worth budgeting for. In a Kubernetes environment, the overhead of the monitoring system is split between the Prometheus server and the targets: the Prometheus server consumes ~150 % of CPU (1.5 cores) and ~15 GB of RAM, while the targets (node exporter and dcgm-exporter) together use ~35 % of CPU and ~300 MB of RAM (arXiv · Container-level Energy Observability). Kepler uses eBPF precisely to keep its own overhead low. The reading: the bulk of the cost of measuring is in Prometheus (which you probably already have for observability), not in the energy exporters. Adding Kepler to an existing observability stack is marginal; standing up Prometheus just for this would be expensive. That is why measuring energy is nearly free if you already observe, and a project if you do not.

Zeus: for the bench and for optimisation, not for continuous production

Zeus (ml.energy) plays a different role: it is not a continuous production exporter, it is the bench tool for measuring and optimising the energy of one specific workload (a benchmark, a training run), with power capping and frequency selection. It fits into the benchmarking flow (alongside GuideLLM/AIPerf): during a sweep, Zeus measures the energy per configuration and lets you test the effect of power capping on performance. The practical division: Kepler/DCGM in production (continuous per-pod attribution), Zeus on the bench (fine measurement + optimisation of one workload), CodeCarbon if you want carbon context integrated into an ML pipeline.

CodeCarbon and Scaphandre in practice

The other two pieces, with their specific role:

CodeCarbon integrates inside the code of an ML pipeline (a decorator or a context manager wrapping the training or the inference), and returns the energy and the CO₂eq of that block, using nvidia-smi for the GPU and estimating CPU/RAM. It fits when you want the carbon figure per experiment or per ML job, integrated into the data flow, not at infrastructure level. It is the closest to the power meter among the Python tools.
Scaphandre is a metrology agent in Rust that reads RAPL and gives process/VM power with raw hardware accuracy. For CPU it is very accurate; for GPU it has to be combined with NVML. It fits when you need the most faithful CPU metrology, or a lightweight agent independent of Kubernetes.

The division with Kepler: Kepler attributes energy per pod in the cluster (the infrastructure view); CodeCarbon gives carbon per ML job (the pipeline view); Scaphandre gives raw CPU metrology. They do not compete; they cover different layers, and a mature platform uses whichever matches the question.

Choosing the stack by objective

Objective	Tool	Why
Continuous per-pod attribution (production)	Kepler + DCGM	eBPF, per pod/MIG, into Prometheus
Fine measurement + optimisation (bench)	Zeus	measures and optimises (power capping)
Carbon in an ML pipeline	CodeCarbon	integrated CO₂eq, close to the power meter
Raw CPU accuracy (RAPL)	Scaphandre	hardware metrology
Calibration / ground truth	power meter	the standard everything is calibrated against

It is not “one tool”, it is a stack by objective: continuous production (Kepler/DCGM), bench (Zeus), reporting (CodeCarbon), calibration (power meter). Mixing them according to the phase is what gives a complete and credible energy measurement.

From figure to action

Measuring is the start; the value is in acting. With energy per pod in Prometheus:

Reporting (CSRD): the carbon footprint per workload, calibrated, ready for regulatory reporting.
Energy/carbon-aware scheduling: move deferrable load to hours or nodes of lower intensity (connects with the hourly carbon of the Spanish grid).
Power capping: with Zeus, trim peak power with little performance loss, lowering the W in the numerator of J/token.
Optimisation: identify pods with poor energy efficiency (a lot of energy per token) and attack them with quantisation or batching.

The measurement stack is not a panel to look at: it is the input to scheduling, optimisation and reporting decisions. Just as OpenCost closes the cost loop, Kepler+DCGM closes the energy one.

From energy to carbon: the final layer

The energy figure (kWh) becomes carbon by multiplying by the grid intensity, and here the practical stack connects with the hourly source. In Spain, the intensity is published by Red Eléctrica (esios) and aggregated by ElectricityMaps, hour by hour. The pattern:

# Carbon per workload: energy (kWh) × grid intensity (gCO2/kWh)
(sum by (namespace) (increase(kepler_container_joules_total[1h])) / 3.6e6)
 * on() group_left() grid_carbon_intensity_gco2_kwh

where grid_carbon_intensity_gco2_kwh is a metric you feed from esios/ElectricityMaps (an exporter or a periodic push). With that, the Grafana panel shows gCO₂ per namespace and per token, hour by hour, not an annual average. And since Spanish intensity varies a lot (from ~80 in solar hours to ~250 on gas-heavy nights), that hourly resolution is what enables carbon-aware scheduling: moving the deferrable work to the clean hours lowers the measured carbon. The measurement stack does not only report carbon; with the hourly figure, it allows it to be reduced.

Energy efficiency KPIs to watch

KPI	What it indicates	Where from
J/token per model	energy efficiency	Kepler ÷ engine tokens
Mean power per pod	consumption per workload	`rate(kepler_container_joules_total)`
gCO₂/token (hourly)	real footprint	energy × grid intensity
Idle energy	waste	power of pods with no traffic
Effective PUE	DC overhead	total energy ÷ compute energy

These five turn the energy panel into a decision tool: which model is more efficient, which pod wastes, when it pays to schedule. Without them the stack is a pretty panel; with them, it is the input to optimisation and reporting.

Example: energy attribution on a multi-tenant cluster

With Kepler splitting per pod, an energy report per team on the example node (8×H100, 5.6 kW of boards, ~7.84 kW with PUE), over one month:

Team	Energy/month (kWh)	gCO₂ (ES grid ~160)	Mean J/token
A · chat-prod	~3,100	~496 kg	~2.2 J/tok
B · batch	~1,400	~224 kg	~2.8 J/tok
C · experimentation	~900	~144 kg	~6.5 J/tok
idle	~250	~40 kg	—

What it reveals: A is efficient (FP8, high utilisation → 2.2 J/token); C is inefficient (6.5 J/token, underused GPU), just as it dragged the idle cost in the FinOps track. The carbon column takes the figure into the per-team CSRD report. And the idle row (~250 kWh, ~40 kg CO₂) is pure wasted energy. Without Kepler, that split does not exist: you would only see the node’s total consumption, with no idea whose each watt is. With it, energy is attributable and actionable, exactly like cost with OpenCost. In fact, the two panels together (OpenCost cost + Kepler energy) give the complete picture of efficiency per team.

Energy and carbon-aware scheduling

The highest-return action the stack enables: scheduling by energy and carbon. With energy per pod and hourly grid intensity in Prometheus, two policies:

By carbon: move the deferrable load (training, ingestion, overnight batch) to the hours of lowest grid intensity. In Spain, the hours of peak solar (~80 gCO₂/kWh) against gas-heavy nights (~250). The same workload emits ~3× less when scheduled at the right hour.
By efficiency: consolidate workloads to eliminate idle (a powered GPU with no work is wasted energy), and direct each workload to the hardware most efficient for it.

Both use the same data (energy per pod + hourly intensity) that the stack already exposes, and they connect with cluster scheduling (Kueue/Volcano). It is the difference between measuring carbon (and reporting it) and reducing it (by scheduling). The C3 stack enables both.

Deployment checklist

Step	Action	Verification
1	dcgm-exporter (GPU Operator)	`DCGM_FI_DEV_POWER_USAGE` in Prometheus
2	Kepler DaemonSet, kernel with eBPF	`kepler_container_joules_total` per pod
3	`dcgmEndpoint` pointing at the exporter	GPU/MIG attribution works
4	Grid intensity metric (esios)	`grid_carbon_intensity` in Prometheus
5	Calibration vs power meter (once per node type)	error bar documented
6	Grafana panels (J/token, carbon, idle)	KPIs visible

Step 5 (calibration) is the one most often skipped and the one that gives the figure credibility; step 3 (dcgmEndpoint) is the one most often forgotten, and without it the GPU is not attributed.

Energy by phase: prefill vs decode

A level of detail the stack allows and that guides optimisation: power is not constant during a request. Prefill (processing the prompt) is compute-bound and draws a lot of power in a burst; decode (generating tokens) is memory-bound and draws a lower peak but for longer. With fine DCGM sampling that bimodal profile is visible, and it has consequences:

A workload with long prompts (a lot of prefill) has a different power profile from one with long generation (a lot of decode), even if they produce the same tokens.
Power capping affects each phase differently: trimming peak power mostly touches the prefill.

For most cases the mean J/token is enough; to optimise energy efficiency thoroughly, separating prefill and decode consumption (aligning the DCGM window with each phase) reveals where the spend is. It is the energy equivalent of separating TTFT and TPOT in benchmarking, the same principle, measuring the two phases separately because they behave differently.

State of the art: the LLM resolution gap

An honest note on the limits of current tooling. The general-purpose tools (Kepler, CodeCarbon, Scaphandre) measure well at machine, process and pod level, but they lack LLM-specific resolution: out of the box they do not distinguish prefill from decode energy, nor energy per token by phase, nor the energy cost of a reasoning block against a normal response. It is an open gap in the field: per-pod attribution exists, but attribution per token and per inference phase has to be built by crossing power (DCGM/Kepler) with engine metrics (vLLM exposes tokens per phase). The 2026 state of the art is a mature stack for infrastructure and an emerging one for LLM-specific resolution, exactly the crossing that the reproducible harness (S4) has to build by hand. Acknowledging it avoids promising a per-token accuracy that the general-purpose tools do not deliver on their own.

Limits and traps (data-driven)

Uncalibrated estimation. Kepler estimates; without calibrating against a power meter at least once, the figure can carry a noticeable error, especially when it falls back on regression models.
Forgetting the dcgmEndpoint. Without pointing Kepler at the dcgm-exporter, GPU (and MIG) attribution does not work; it is the key parameter for GPU.
Accounting for overhead badly. The bulk of the cost of measuring is in Prometheus; if you stand it up just for energy, budget for it (1.5 cores, 15 GB).
Confusing bench and production. Zeus measures one workload finely; Kepler attributes continuously in production. Using one for the other gives the wrong figure.
Misaligned windows. For energy per token, the power window and the token window must coincide; if they do not, the J/token does not correspond.

With the measurement stack in place, the energy track has its instrumentation; the following articles (C6 carbon, C8 TCO and regulation) build on this data. The foundation is this: a continuous per-pod measurement, calibrated, on top of the observability infrastructure you already have.

Closing

Measuring the energy of an AI cluster in production does not require a new system: it requires adding Kepler, per-pod attribution via eBPF, to the DCGM, Prometheus and Grafana stack you already have for observability. That is the central idea of C3: the cost of measuring is almost entirely in the Prometheus you already pay for, and Kepler turns it from “card power” into “energy per pod, per team and per token”. With two cautions that separate a credible figure from a decorative panel: calibrate once against a power meter (to have an error bar that survives an audit) and point the dcgmEndpoint (without which the GPU is not attributed). Done that way, energy stops being the soft axis of sustainability and becomes an actionable time series: the J/token per model that exposes inefficiency, the hourly carbon that enables clean scheduling, the idle that gets recovered, and the CSRD report that holds up. For the sovereign platform, the Kepler+DCGM stack closes the energy loop just as OpenCost closes the cost one, and together, with GuideLLM’s goodput, they give the three faces of the dashboard measured on the same iron, in euros, joules and grams of CO₂.

Sources

Kepler · GitHub (CNCF, eBPF, por pod/nodo) — https://github.com/sustainable-computing-io/kepler
Kepler · configuración (fuentes, dcgmEndpoint para MIG) — https://github.com/sustainable-computing-io/kepler/blob/main/docs/user/configuration.md
CNCF · explorando Kepler (consumo de aplicaciones cloud) — https://www.cncf.io/blog/2023/10/11/exploring-keplers-potentials-unveiling-cloud-application-power-consumption/
Red Hat · Kepler: monitorización eficiente de energía para Kubernetes — https://next.redhat.com/2023/08/22/introducing-kepler-efficient-power-monitoring-for-kubernetes/
arXiv · Container-level Energy Observability in Kubernetes (overhead) — https://arxiv.org/pdf/2504.10702
NVIDIA · dcgm-exporter — https://github.com/NVIDIA/dcgm-exporter
Zeus Project (ml.energy) — https://ml.energy/zeus/

OpenCost in depth: how GPU cost is allocated in Kubernetes

Sun, 14 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. European references (OVH is among the supported providers). The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Second article in the FinOps track (A2), and the first tool deep dive. The FinOps introduction established that OpenCost is the CNCF standard for cost allocation; here the box gets opened: how it allocates cost internally, where it gets its data from, how on-prem prices are configured in euros, what its API exposes, and where the traps are. Understanding the mechanics matters because it defines what any tool on top (Kubecost included) can and cannot do, and because a badly set base price invalidates the whole split. No recommendations; only the mechanics and the methodology.

What OpenCost is

OpenCost is a vendor-neutral, Apache 2.0 project, originally built by Kubecost and donated to the CNCF (incubating project). It is a cost allocation layer for Kubernetes: it reads resource usage from the cluster, joins it with a price, and splits the cost across Kubernetes dimensions down to the container (OpenCost · GitHub). It supports dynamic pricing via the billing APIs of AWS, Azure and GCP, and custom prices for on-prem clusters (OpenCost · configuration).

What it is not: it is not an optimiser, nor a governance system, nor a product unit economics tool. It is the piece that answers “how much does each namespace/pod/team cost?”, and leaves the rest to the layers above. That is why it is the foundation of the FinOps track: without correct allocation, there is no cost per token and no TCO model worth anything.

The cost model: node-level allocation

The key to OpenCost is that the model works at node level. It starts from the node’s resource capacity (CPU, RAM, GPU, storage) and its total price, and splits that price across the resources. When the provider does not give explicit per-resource prices, OpenCost uses the ratio of a set of base prices (marginal rates, customisable) and normalises them so that the sum of the components equals the node’s total price (OpenCost · on-prem).

This is what makes it work on-prem: you declare the node cost (depreciated capex + opex), and OpenCost splits it without needing a cloud invoice. The normalisation guarantees that no cost is “lost” or “invented”: what you pay for the node is exactly what gets split across its resources.

Where the data comes from: Prometheus

OpenCost instruments nothing of its own: it reads from Prometheus, which is a prerequisite of the installation (OpenCost · GitHub). The sources:

Source	What it provides
`kube-state-metrics`	state of K8s objects (pods, requests/limits)
`node-exporter`	node resources and capacity
`cAdvisor`	real CPU/memory usage per container
`dcgm-exporter` (NVIDIA)	GPU usage, memory and power

With those series, OpenCost automates the join between Kubernetes resource usage and the provider price, reducing the custom PromQL you would otherwise write by hand (Grafana · manage costs). The GPU signal comes from DCGM via the dcgm-exporter (part of the NVIDIA GPU Operator), the same base as GPU observability. Without Prometheus and without DCGM exporting, OpenCost does not see the GPU.

Architecture: the pieces

OpenCost is not a monolith; it is three responsibilities worth telling apart:

Piece	Function
Cost model	resolves the node price and splits/normalises it across resources
Allocation API	serves the allocated cost per dimension (`/allocation` and endpoints)
Exporter	publishes cost metrics on `/metrics` for Prometheus/Grafana

The data flow: Prometheus holds the usage series (from kube-state-metrics, cAdvisor, dcgm-exporter); the cost model joins them with the resolved price; the Allocation API serves them aggregated by whatever dimension you ask for; and the exporter returns them to Prometheus so that Grafana can plot them. It is a cycle: metrics come in from Prometheus and cost goes back to Prometheus enriched. That symmetry is what lets you reuse your observability stack without standing up a new database: OpenCost lives on top of Prometheus, not beside it.

An operational note: historical analysis depends on Prometheus retention. For cost reports spanning months, a long-term backend behind it (Thanos, Mimir, VictoriaMetrics) is advisable, or periodically exporting the allocations to a store of your own.

On-prem prices in euros: the piece that decides everything

On-prem, the most important figure, and the most neglected, is the node price. It is configured per provider; the options are alibaba, aws, azure, gcp, oracle, ovh or default (on-prem) (OpenCost · configuration). For on-prem you use default, supplying base prices in a default.json (or by overriding the Helm values.yaml); anything you do not override uses the chart’s value (OpenCost · on-prem).

To set the node price in euros, the calculation is the TCO model one: depreciated capex + opex. An example of an 8×H100 node:

Component	Calculation	€/hour
Capex (node ~240,000 €, 36 months)	240,000 ÷ 26,280 h	~9.13
Energy (5.6 kW × PUE 1.4 × 0.058 €/kWh, France)	~0.455 €/h	~0.46
Maintenance, networking, operations	estimate	~1.5
Node total		~11.1 €/h

That ~11 €/h is the number you put into OpenCost, and everything is split from it. And here is the documented trap: with the default pricing configuration, OpenCost under-prices CPU and GPU on-prem (issue #3781). If you do not adjust the base prices to your real cost, the split will be systematically low and the cost per token you report will be unreal. Configuring the node price is not optional: it is the figure everything else hangs from.

Configuring the price: an example

The base price is given in the Helm values.yaml (or a default.json), in euros per hour and per unit of resource, derived from the node’s total cost:

# values.yaml — on-prem prices in €/hour (8xH100 node, France)
opencost:
 customPricing:
 enabled: true
 provider: custom
 costModel:
 description: "On-prem 8xH100 node"
 CPU: "0.030" # €/CPU-hour
 RAM: "0.004" # €/GB-hour
 GPU: "1.30" # €/GPU-hour ← the figure that moves cost/token the most
 storage: "0.0002" # €/GB-hour

OpenCost normalises those values so that, multiplied by the node capacity, they add up to the total price you declared (~11 €/h in the example). The number to be most careful with is the GPU one: at 1.30 €/GPU-hour, the 8 cards are ~10.4 €/h, the bulk of the node’s cost. A badly set GPU base shifts the whole cost per token. Always review the resolved prices in /allNodePricing after configuring, because the default under-pricing bug bites exactly here.

Allocation: by usage, not by presence

The behaviour that has changed most and that matters most for GPU: in recent versions, the cost allocated to a GPU is determined by its usage, not its presence. If a GPU costing 100 €/month is only half used, the allocated cost is ~50 € ([search]). This aligns allocation with reality, you pay for what you use, but it has a consequence: the other half does not disappear, it is idle that somebody keeps paying for. OpenCost exposes it, and that is where the optimisation lever comes from.

OpenCost allocates to any Kubernetes dimension: cluster, node, namespace, controller (deployment, statefulset and so on), service, pod and container, and by labels (team, product). Access is through the Allocation API and several endpoints that expose the mechanics:

Endpoint	What it returns
`/allocation`	cost allocated by the dimension you ask for
`/costDataModel`	the resolved node price
`/allNodePricing`	hourly prices per node
`/pricingSourceSummary`	summary of the pricing source
`/metrics`	cost metrics for Prometheus

The /metrics endpoint is what turns OpenCost into a Prometheus exporter: once there, you can write PromQL to calculate the cost and efficiency of any Kubernetes concept and build panels in Grafana (OpenCost · exporter).

Allocation API parameters

/allocation is controlled by a handful of parameters worth knowing, because they define which number you get:

Parameter	What it controls	Example
`window`	time range	`7d`, `today`, specific dates
`aggregate`	aggregation dimension	`namespace`, `label:team`, `pod`
`accumulate`	sum the range or break it per interval	`true` (total) / `false` (series)
`idle`	include idle cost or not	`true` / `false` / `separate`
`filter`	filter by namespace, label and so on	`namespace:"llm-prod"`

The idle parameter is the most revealing: with idle=separate, OpenCost returns the idle cost as a separate row, which lets you see at a glance how much is being paid for unused capacity. A query with aggregate=label:team and idle=separate over window=30d is, literally, the monthly chargeback report with the waste highlighted.

GPU in depth: DCGM, usage, MIG and idle

The GPU is the expensive resource, so its allocation deserves detail:

Usage signal: dcgm-exporter gives DCGM_FI_DEV_GPU_UTIL (utilisation), DCGM_FI_DEV_FB_USED (memory), and power. OpenCost uses utilisation to allocate by usage.
MIG (Multi-Instance GPU): an A100 is split into up to 7 isolated instances, which allows cutting idle from 50 % to almost 0 % by sharing one card across small workloads ([search]; see sharing the GPU: time-slicing, MPS and MIG). The cost allocation of each MIG instance follows the same usage logic.
Idle: the reliable detection pattern combines three pieces, DCGM metrics in Prometheus, an alert that fires when DCGM_FI_DEV_GPU_UTIL < 10 for more than 15 minutes, and routing of that alert to the team that owns the namespace ([search]).

An attribution nuance worth knowing: MIG creates isolated instances that Kubernetes sees as distinct resources, so OpenCost allocates them cleanly, each one to its pod. Time-slicing, by contrast, shares the same physical GPU across several pods with no compute isolation: there, per-pod utilisation is harder to separate, and cost attribution becomes approximate (the GPU is split across the pods sharing it, according to their measured usage). The practical rule: if you need exact per-team chargeback on a shared GPU, MIG gives cleaner attribution than time-slicing, at the price of rigid fixed partitions. It is a trade-off between cost accuracy and flexibility that is better decided before, not after, building the split.

Efficiency: the number that triggers action

With cost allocated by usage, OpenCost lets you calculate efficiency: how much of the allocated cost corresponds to useful work against the total. A GPU at 30 % utilisation has an efficiency of 30 %: the remaining 70 % is idle cost that somebody pays without receiving work. That figure, per team and per namespace, is what turns a dashboard into a chargeback conversation: “your namespace has 5 GPUs allocated at 25 % efficiency; either you raise utilisation or you release three cards”. It connects with FinOps and multi-tenancy: allocation locates the waste; scheduling and co-residency recover it.

Worked example: the allocation report

On the ~11 €/h node (8×H100), three teams share the cluster for a month. What a /allocation?aggregate=label:team&idle=separate&window=30d query returns:

Team	GPU-hours	Avg. util.	Allocated cost (€)	Efficiency
A · chat-prod	2,880 (4 GPU)	78 %	~3,744	high
B · batch	1,440 (2 GPU)	55 %	~1,872	medium
C · experimentation	1,440 (2 GPU)	22 %	~1,872	low
idle (separate)	—	—	~1,100	—

What the report reveals: the cost allocated to A, B and C comes out of their GPU-hours and the node price; but the idle row (~1,100 €/month) is capacity nobody used and everyone pays for in the pro rata. And the efficiency column points at C: 2 GPUs at 22 % is ~1,560 € a month of which only ~340 € is useful work. That is the actionable figure: C does not have a model problem, it has a utilisation problem. Without OpenCost, those ~1,100 € of idle and C’s inefficiency stay diluted in an aggregate invoice nobody questions. With it, they are a named row and a number in euros.

Useful PromQL queries

As a Prometheus exporter, OpenCost lets you build the panels by hand. Some reference queries (exact metric names vary by version; check in your /metrics):

# Cost per node and hour (resolved price)
node_total_hourly_cost

# GPU cost allocated per namespace (€/hour)
sum by (namespace) (
 container_gpu_allocation * on(node) group_left node_gpu_hourly_cost
)

# Average GPU utilisation per namespace (for efficiency)
avg by (namespace) (DCGM_FI_DEV_GPU_UTIL)

# Idle GPUs: utilisation < 10 % sustained
DCGM_FI_DEV_GPU_UTIL < 10

The last one, wrapped in an alert with for: 15m and routed to the namespace owner, is the standard idle detection pattern. With these series in Grafana you have cost per team, efficiency and waste on one dashboard, the “Inform report” of the FinOps phase in real time.

Dynamic cloud prices vs custom on-prem prices

The difference is worth understanding, because it changes how reliable the figure is:

Cloud (aws/azure/gcp/oracle/ovh): OpenCost obtains the price dynamically from the provider’s billing API, so the allocated cost reflects the real rate (including discounts, spot and so on) without you declaring anything.
On-prem (default/custom): you declare the price, derived from your TCO model. The advantage is total control; the risk is that a badly calculated price skews everything. That is why the TCO model (depreciated capex + opex) is not an academic exercise: it is literally OpenCost’s input.

The ovh provider among those supported is relevant for a European platform: if part of the workload goes to a European sovereign cloud, OpenCost can allocate its cost with dynamic pricing, and mix it with on-prem in the same allocation view.

From pod cost to cost per token

OpenCost gets as far as “this vLLM pod cost X €/hour”. The cost per token, the metric that compares on-prem against cloud, needs one more piece: the gateway (LiteLLM) that counts tokens per request and per team, as covered in the FinOps introduction. The full chain is: OpenCost gives the pod cost by usage → the gateway gives the tokens per team → dividing them gives the cost per token per team. OpenCost is the hardware half of that equation; without it, you would know the tokens but not their real cost. That is why A2 (allocation) is the foundation the whole cost per token of the proposal hangs from.

Deployment: Helm + Prometheus

The typical deployment is via Helm, with Prometheus as a prerequisite (for scraping and series storage). Conceptual steps:

Have Prometheus with kube-state-metrics, node-exporter, cAdvisor and dcgm-exporter.
Install OpenCost with Helm, pointing it at your Prometheus.
Configure the node price in euros (default.json / values.yaml), the step most often neglected.
Expose /metrics and build panels in Grafana with PromQL over the cost metrics.

As a Prometheus exporter, OpenCost lets you write PromQL for the cost and efficiency of any concept and create bespoke dashboards (OpenCost · exporter). It reuses infrastructure a GPU cluster already has for observability (DCGM, Prometheus, Grafana): OpenCost only adds the pricing and allocation layer.

OpenCost vs Kubecost (just the essentials)

OpenCost is the free base; Kubecost (IBM) is the commercial product built on it. Briefly, what the commercial one adds:

Capability	OpenCost	Kubecost
Cost allocation (CPU/GPU/mem/PV)	✓	✓
GPU via DCGM	✓	✓ (3.0)
Long historical retention	depends on your Prometheus	included
Automatic rightsizing	—	✓ (Turbonomic)
Governance, alerts, enterprise RBAC	basic	✓
Commercial support	community	✓ (IBM)

For most on-prem clusters with a platform team, OpenCost covers allocation; Kubecost adds value when you want automated optimisation and enterprise features without operating it yourself. The detail of the comparison, and when paying is worth it, is article A3.

Implementation checklist

For OpenCost to give a defensible per-team cost, order matters:

Step	Action	Verification
1	Prometheus with kube-state-metrics, node-exporter, cAdvisor	series present
2	NVIDIA GPU Operator + dcgm-exporter	`DCGM_FI_DEV_GPU_UTIL` in Prometheus
3	Install OpenCost (Helm) pointing at Prometheus	`/allocation` responds
4	Configure the node price in € (capex+opex)	`/allNodePricing` shows the real one
5	Label pods/namespaces by team and product	`aggregate=label:team` splits correctly
6	Grafana panels for cost, efficiency and idle	the waste is visible
7	Idle alert (`util<10` `for: 15m`) to the owner	the alert reaches the team
8	(optional) long-retention backend	months of history available

Step 4 is the one most often skipped and the one that most invalidates the result. If you only do one thing right, make it that one: without the correct node price in euros, steps 5 to 7 split a wrong cost with great precision.

Limits and traps (data-driven)

Default under-pricing. With the default pricing config, OpenCost underestimates on-prem CPU and GPU (#3781). Adjust the base prices to your real cost or the whole split will come out low.
It depends on Prometheus and DCGM. Without the right series (especially dcgm-exporter), there is no GPU allocation. Input data quality rules.
Allocation by usage ≠ total cost. A GPU at 50 % charges half; the other half is idle that still costs. Do not confuse “allocated cost” with “cost paid”.
Prometheus retention. Historical cost analysis depends on how much your Prometheus retains; for long series, plan a long-term backend.
Allocation, not per-token measurement. OpenCost reaches the pod; cost per token needs the gateway (LiteLLM) above it, as covered in the FinOps introduction.

With OpenCost’s mechanics clear, the next article (A3) compares what Kubecost and the commercial alternatives add, to decide which FinOps stack to adopt. But the foundation, correct allocation with the node price properly set in euros, is this one.

Closing

OpenCost looks like “installing a cost tool” and is, in reality, declaring what your hardware is worth and letting it split itself. That simplicity is its strength, it lives on top of the Prometheus and DCGM you already have, with no new database, and also its trap: the split is only as good as the node price you declare, and the default value under-prices on-prem GPU. Properly configured in euros, OpenCost turns the cluster’s aggregate invoice into a split per team, per product and per GPU, separates used cost from idle, and exposes the efficiency that triggers chargeback conversations. It is the hardware half of the cost per token equation, the other half, the tokens, is supplied by the gateway, and without this half there is no TCO model or on-prem vs cloud comparison that holds up. For a sovereign architecture proposal, OpenCost is the piece that makes cost stop being an intuition and become a table in euros, reproducible and auditable, with the idle GPU named and identified. The Inform phase of GPU FinOps starts here, and everything else hangs on this number being right.

Sources

OpenCost · GitHub (CNCF, Apache 2.0) — https://github.com/opencost/opencost
OpenCost · configuración (proveedores: aws/azure/gcp/oracle/ovh/default) — https://opencost.io/docs/configuration/
OpenCost · configuración on-prem (precios personalizados) — https://opencost.io/docs/configuration/on-prem/
OpenCost · exportador de Prometheus — https://opencost.io/docs/integrations/opencost-exporter/
OpenCost · issue #3781 (infra-precio CPU/GPU on-prem por defecto) — https://github.com/opencost/opencost/issues/3781
NVIDIA · dcgm-exporter — https://github.com/NVIDIA/dcgm-exporter
Grafana · manage costs (OpenCost + Kubernetes) — https://grafana.com/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/manage-costs/

GPU/LLM FinOps: frameworks, metrics and state of the art (tool by tool)

Sat, 13 Jun 2026 03:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The references are European (providers and prices from FR/DE/ES), as this is a sovereign proposal; when a source quotes dollars it is marked “USD”. The dollar sign is not used (on this site it is a formula delimiter).

What this introduction covers

Second article in the data series, and the first pillar deep dive. Here the FinOps tooling for GPU/LLM infrastructure is inventoried with the detail an architecture decision needs: what metrics each tool handles, how it allocates cost internally, what GPU support it has, under what licence and pricing model it operates, and where its limits are. It is the natural continuation of the opening article, which established that cost per token is the unit that allows on-prem and cloud to be compared. No recommendations: the final choice is settled in the synthesis article with the Pareto table; here there are only facts and methodology.

Why GPU FinOps is a different problem

Classic cloud FinOps was born to split CPU, memory and storage, cheap and elastic resources. The GPU breaks three assumptions at once, and that is why it needs its own treatment:

It is expensive and discrete. An H100 costs on the order of 2.7 to 7 €/hour in European cloud (Scaleway from 2.73 €/h; OVHcloud somewhat more), or its depreciated capex on-prem; it is not a resource that gets split in trivial fractions. A 10 % allocation error across a GPU fleet is real money.
It is easily underused. Unlike the CPU, a GPU draws its power even when idle, and average occupancy in ungoverned clusters is notoriously low. The cost of idle, the GPU powered on without working, is waste number one, and it is invisible if it is not measured.
It is hard to attribute. Who owns the cost of a GPU shared by time-slicing or MIG across several pods from several teams? Without an allocation layer, GPU spend is a black hole nobody claims and nobody optimises.

The goal of GPU FinOps is to turn that black hole into an attributed, measured and actionable invoice: knowing what each team, each model and, in the end, each token costs.

The FinOps metrics

Metric	Definition	Unit
CPM (cost/1M tokens)	cluster cost ÷ tokens produced	€ / 1M tok
Cost per request	cost charged to a complete request	€ / req
€/GPU-hour	hourly cost of a GPU (depreciated or rented)	€/h
Utilisation	fraction of the GPU actually used	% (MFU, GPU-hour util)
Cost of idle	GPU powered on without useful work	€/h wasted
Showback	reporting the cost to each team (without charging)	€ / team
Chargeback	charging the cost to each team	€ / team
Cost efficiency	real cost ÷ cost if it were at 100 %	%

The metric that closes the loop with the business is cost per token (or per request): it is the only one that can be compared across providers, across models and against the price of an external API. All GPU FinOps tooling exists to reach, one way or another, that number.

The three FinOps phases (FinOps Foundation)

The FinOps Foundation framework organises the work in a three-phase cycle. Applied to GPU:

Phase	Goal	Typical GPU action
Inform	visibility and allocation	measure cost per namespace/team/model/token
Optimize	reduce spend	rightsizing, spot, quotas, switch off the idle, quantise
Operate	continuous governance	budgets, idle alerts, automated chargeback

The common mistake is jumping to Optimize without having done Inform: GPUs get bought or replicas get adjusted without knowing where the money really goes. Allocation (the Inform phase) is the prerequisite for everything else, and it is where the tooling comes in.

How cost is allocated in Kubernetes: OpenCost’s mechanics

OpenCost is the de facto standard, so it is worth understanding how it allocates cost, because that defines what any tool on top can and cannot do. It is a vendor-neutral, Apache 2.0 project, originally built by Kubecost and donated to the CNCF (incubating project) (OpenCost · GitHub).

The model works at node level: it starts from the node’s resource capacity (CPU, RAM, GPU, storage) and its total price, and splits that price across the resources. When the provider does not give explicit CPU/GPU/RAM prices, OpenCost uses the ratio of a set of base prices (the provider’s marginal rates, customisable) and normalises them so that the sum of the components equals the node’s total price (OpenCost · on-prem). This is key on-premise: you define the node cost (depreciated capex + opex) and OpenCost splits it.

It obtains utilisation by scraping Prometheus: kube-state-metrics, node-exporter and cAdvisor give it the real consumption per pod, and with that it allocates cost per cluster, node, namespace, controller, service or pod (OpenCost · exporter). For the GPU, the signal comes from DCGM (via the NVIDIA GPU Operator) exported to Prometheus, the same base as GPU observability. A common idle detection pattern: an alert when DCGM_FI_DEV_GPU_UTIL < 10 for more than 15 minutes, routed to the team that owns the namespace.

Frameworks, tool by tool

OpenCost — the allocation standard (CNCF, Apache 2.0)

What it measures: allocated cost of in-cluster resources (CPU, GPU, memory, volumes) by any Kubernetes dimension. Method: node-level pricing model + Prometheus scraping (above). GPU: yes, via DCGM. Licence: Apache 2.0, free. It can run as a metrics exporter to Prometheus with no further dependencies. Limit: it is the allocation layer, it brings no optimisation, governance or product unit economics; for that you put something on top of it.

Kubecost — the commercial product on OpenCost (IBM)

What it measures: what OpenCost does (it is built on it) plus enterprise capabilities. Kubecost 3.0 (2025) added GPU monitoring via NVIDIA DCGM and integration with IBM Turbonomic for automatic rightsizing, and extended its scope from Kubernetes to cloud service cost. IBM acquired Kubecost and integrated Kubecost/OpenCost into its FinOps Suite alongside Cloudability and Turbonomic (CloudZero · Kubecost vs OpenCost). Differentiator: rightsizing, governance, support. Limit: it is a commercial product; the bulk of the value over free OpenCost is the optimisation and enterprise layer.

CloudZero — unit economics and cost per product

What it measures: it maps cloud cost to features, products, teams and customers, not just to resources. Method: it ingests multi-cloud billing and models it in business dimensions. Differentiator: the deepest allocation and the unit economics approach (cost per business unit). Limit: less focused on intra-Kubernetes mechanics than OpenCost/Kubecost; it is the “cost to business” layer.

Vantage — multi-cloud with many integrations

What it measures: multi-cloud cost with more than 20 native integrations (AWS, Azure, GCP, Kubernetes, Snowflake, Datadog, OpenAI and so on). Differentiator: breadth of sources, including the invoice from LLM providers (OpenAI), which brings it closer to end-to-end AI cost. Limit: the depth of intra-cluster allocation is lower than that of K8s-native tools.

Finout — virtual tagging, fast deployment

What it measures: multi-cloud cost with virtual tagging: it applies cost labels without modifying the real resources, which allows spend that was not properly tagged at source to be allocated. Differentiator: fast deployment and flexible reallocation without touching the infra. Limit: like the others in its category, it depends on the quality of the billing data it ingests.

Comparison table

Tool	Scope	GPU	Licence / model	Differentiator	Layer
OpenCost	Kubernetes	Yes (DCGM)	Apache 2.0 (CNCF), free	allocation standard	resource
Kubecost	K8s + cloud	Yes (DCGM, 3.0)	commercial (IBM)	rightsizing, enterprise	resource+optim
CloudZero	multi-cloud	indirect	commercial	unit economics per product	business
Vantage	multi-cloud (20+)	via K8s/provider	commercial	breadth (incl. OpenAI)	business
Finout	multi-cloud	via K8s/provider	commercial	virtual tagging	business

Pricing models of commercial tooling

They are worth knowing because the cost of the FinOps tool is also FinOps:

Model	How it charges	Range
Savings-based	% of the savings delivered	15–35 %
Fixed-fee	% of annual cloud spend	1–3 %

(CloudZero · FinOps Tools). The implication: at large cloud spend, a 1–3 % fixed-fee can exceed the savings-based model; at small spend with a lot of waste, savings-based aligns incentives. OpenCost, being free, changes the equation for whoever has a team to operate it.

FOCUS: the cost data standard

The cross-cutting problem of multi-source FinOps is that each provider bills in its own format. FOCUS (FinOps Open Cost and Usage Specification) is the open technical specification that defines requirements so that providers produce uniform billing datasets (FOCUS · FinOps Foundation). The committee ratified FOCUS v1.3 on 4 December 2025.

What is relevant for this series: at FinOps X 2026 the focus has been put on extending FOCUS to AI workloads, with token economics pushing the specification, and the expansion requests include AI, datacenter and SaaS/PaaS workloads (SiliconANGLE). In other words, the standard that normalises cloud cost is being stretched to cover AI cost per token. For an architecture proposal, betting on tools that emit and consume FOCUS is betting on future interoperability.

From resource to token: cost per token with a gateway

Resource allocation (OpenCost) gets as far as “this vLLM pod cost X €/hour”. To get to “this request from this team cost Y” you have to intercept the inference traffic. That is where the gateway comes in.

Tools such as LiteLLM sit between the application and the LLM provider/engine, and intercept every request to record tokens, latency and cost in real time. Layers above (for instance OpenLM’s AI Gateway) generate usage logs compatible with FOCUS (v1.0 to 1.3) and map spend to team, product, customer or feature, enabling showback or chargeback (OpenLM · token attribution). This links up with FinOps and multi-tenancy of the GPU cluster, where the gateway is the piece that splits cost between tenants.

Worked example: chargeback of a multi-tenant cluster

To see the two halves joined, a split over an example node (4×H100, depreciated cost 12 €/hour) shared by three teams via namespaces and MIG:

Team	GPU-hours allocated (OpenCost)	Hardware cost (€/h)	Tokens/day (gateway)	Cost/1M tok
A · chat product	50 %	6.00	8M	~0.75 €
B · nightly batch	30 %	3.60	3M	~1.20 €
C · experimentation	20 %	2.40	0.5M	~4.80 €

How it comes out: OpenCost splits the node’s 12 €/h according to the GPU-hours each namespace consumes (half for A → 6 €/h). The gateway supplies the tokens per team. Each one’s cost per million tokens is its hardware cost divided by its output, and it reveals something neither half would see alone: team C pays 6× more per token than A, not because its model is worse, but because its GPU is underused (a lot of allocated GPU-hours for few tokens). That 4.80 €/1M is the chargeback signal that triggers an optimisation conversation: either C raises its utilisation, or it releases the GPU. Without crossing allocation and tokens, that waste stays hidden in a cluster average.

The hidden cost: utilisation

Optimisation lever number one is not changing GPU, it is stopping paying for an idle GPU. As seen in the opening article, the electricity cost per token at 80 % utilisation is a quarter of what it is at 20 %, and utilisation spreads all the fixed cost (capex + energy) over more tokens. Measuring idle is, therefore, the highest-return action of the Optimize phase:

Signal	Source	Typical threshold
Idle GPU	`DCGM_FI_DEV_GPU_UTIL`	< 10 % for > 15 min
Unused GPU memory	`DCGM_FI_DEV_FB_USED`	reserved but not used
Pods with no traffic	gateway metrics	0 requests, GPU allocated

Allocation (OpenCost) locates whose the idle GPU is; scheduling and co-residency (sharing the GPU: time-slicing, MPS and MIG) recover it. FinOps closes the loop: measure → attribute → optimise → govern.

Optimize: the saving levers (with data)

Once cost is allocated (Inform), the Optimize phase has a bounded repertoire of levers. Ordered by typical return:

Lever	Typical saving	Mechanism	Cost/risk
Recover idle	the largest	switch off/share the idle GPU	requires measuring utilisation
Quantisation (FP8/INT4)	raises throughput, lowers VRAM	more tokens per GPU-hour	possible quality cost
Reserved commitment	20–40 % over on-demand	reserve capacity	less elasticity
Spot/preemptible	the deepest discount	interruptible capacity	you have to tolerate cuts
Rightsizing	variable	match GPU type/count to the SLO	requires benchmarks
Autoscaling (HPA/KEDA)	variable	scale replicas with demand	tuning metrics

On commitment, the 2026 cloud data is conclusive: reserved plans give 20–40 % saving against on-demand, and spot the deepest discount in exchange for interruptibility; the price of the H100 fell 64–75 % between Q4 2024 and the start of 2026 (Spheron · GPU Cloud Pricing). Rightsizing and autoscaling connect with cluster scheduling: fitting the workload onto the right GPU and scaling replicas with demand are, at the same time, performance and cost levers. Quantisation appears here because it raises throughput, and, by the identity from the opening article, lowers cost and energy per token at the same time.

GPU FinOps maturity

A simple model to place where an organisation is, and what it is missing:

Level	State	What is missing to move up
0 · blind	aggregate invoice, no attribution	instrument OpenCost + DCGM
1 · visibility	cost per namespace/team	per-token measurement (gateway)
2 · unit economics	cost per token and per product	idle alerts and budgets
3 · optimisation	idle recovered, commitment, rightsizing	automated chargeback
4 · governance	chargeback + budgets + FOCUS	continuous improvement

Most organisations with GPUs are at level 0 or 1: they see a big invoice but do not know whose each euro is. The value jump is in reaching level 2, cost per token and per product, which is exactly where the tooling in this article stops being optional. The series’ FinOps track walks that ladder up to the full TCO model and the comparable cost/token that underpins the architecture proposal.

The minimum stack to reach level 2

Bringing the previous pieces together into a concrete toolchain, all open source, that takes you from blindness (level 0) to cost per token and per product (level 2):

Piece	Function	Alternative
NVIDIA GPU Operator + DCGM	exports GPU metrics (usage, memory, power)	—
Prometheus	stores the usage and cost series	VictoriaMetrics
OpenCost	allocates cost per resource and dimension	Kubecost (commercial)
Gateway (LiteLLM)	counts tokens per team/model	OpenLM AI Gateway
Grafana	cost, utilisation and idle panels	—

It is, deliberately, infrastructure many clusters already have for observability (DCGM, Prometheus, Grafana); GPU FinOps reuses that base and adds OpenCost and the gateway to it. It is not a new product, it is a layer on top of what exists.

KPIs to watch

KPI	What it indicates	Typical target
Average GPU utilisation	waste	>70–80 % sustained
Cost per 1M tokens (per model)	economic efficiency	compare against cloud rental
% of GPU-hours idle	money thrown away	minimise (<10–15 %)
Cost per team/product	attribution	fair split, no surprises
Deviation against budget	governance	alert before it is exceeded

These five KPIs are what turn GPU FinOps from a pretty dashboard into a decision tool: if you watch them, you know at all times what each thing costs and where the waste is.

State of the art 2026

Consolidation in Kubernetes: OpenCost is the CNCF allocation standard; Kubecost (IBM) the reference commercial product, already with GPU via DCGM and rightsizing through Turbonomic.
From resource to token: the strong platforms of 2026 track cost at token and GPU level, combining resource allocation (OpenCost) with gateway measurement (LiteLLM); it is the only route to a comparable cost/token for on-prem against cloud.
FOCUS v1.3 (Dec 2025) as the interoperability layer, extending to AI at FinOps X 2026: the cloud cost standard absorbs token economics.
Mature GPU FinOps: instrumentation leans on DCGM (the same base as observability), and idle detection by a utilisation threshold is common practice.

Cost per token: the bridge to the business decision

All the previous tooling exists to produce a number the business understands: cost per token (or per request). It is the one that lets you answer the three questions that underpin an architecture proposal:

Build or buy? On-prem cost/token (depreciated capex + opex, at real utilisation) against the price of an external API or of cloud rental. Below the volume threshold (~2M tokens/day) buying usually wins; above it, building.
How do you price a product? If a feature consumes N tokens per use and each million costs C, the marginal cost per use is N·C/10⁶, the basis of any margin.
Where is the waste? Cost/token per team (the chargeback example) points at whoever underuses the GPU before the aggregate invoice hides it.

The trap: comparing cost/token across scenarios without fixing the assumptions (utilisation, precision, ownership model). An on-prem cost/token calculated at 80 % utilisation is not comparable with one at 20 %, nor an FP16 with an FP8. That is why allocation (Inform) is not an end in itself: it is the raw material of a cost model with explicit assumptions, which is what the synthesis article turns into the “build vs buy” argument with defensible numbers. GPU FinOps is not accounting; it is the quantitative basis of the architecture decision.

Limits and traps (data-driven)

Allocation ≠ per-token measurement. OpenCost splits the resource cost; without a gateway you do not reach the request. They are two halves that have to be joined explicitly.
Badly set base prices = badly set allocation. On-prem, OpenCost splits the cost you declare for the node; if the depreciated capex/opex is wrong, the whole split is wrong. Input data quality rules.
Invisible idle. Without DCGM exported to Prometheus and without utilisation alerts, waste number one does not appear on any dashboard.
Cost of the tool. A 1–3 % fixed-fee on large spend is real money; compare the tooling cost with the saving it delivers (it is FinOps on top of FinOps).
Proprietary formats. Tools that do not emit/consume FOCUS tie you to their data model; in 2026 the robust bet is FOCUS interoperability.

The next article in the series (A2) goes into OpenCost in depth; this one fixes the map. With allocation settled, the rest of the FinOps track builds the TCO model and the cost/token the proposal needs.

Sources

OpenCost · GitHub (CNCF, Apache 2.0) — https://github.com/opencost/opencost
OpenCost · documentación on-prem (modelo de precio por nodo) — https://opencost.io/docs/configuration/on-prem/
OpenCost · exporter de Prometheus — https://opencost.io/docs/integrations/opencost-exporter/
CloudZero · Kubecost vs OpenCost (2026) — https://www.cloudzero.com/blog/kubecost-vs-opencost/
CloudZero · FinOps Tools: Definitive Guide (2026) — https://www.cloudzero.com/blog/finops-tools/
FOCUS · especificación (FinOps Foundation) — https://focus.finops.org/focus-specification/
SiliconANGLE · FOCUS y la economía de tokens de IA (FinOps X 2026) — https://siliconangle.com/2026/06/08/focus-specification-ai-cost-accountability-finopsx/
OpenLM · atribución de tokens en tiempo real (LiteLLM + FOCUS) — https://www.openlm.com/enable-ai-finops-with-real-time-token-attribution/
Finout · Best AI Cost Observability Tools (2026) — https://www.finout.io/blog/best-ai-cost-observability-tools-in-2026

Energy benchmarking in AI: frameworks, metrics and state of the art (tool by tool)

Sat, 13 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The energy and carbon references are European (France, Germany, Spain), since this is a sovereign proposal. The dollar sign is not used (on this site it is a formula delimiter).

What this introduction covers

Fourth article in the data series and deep dive on the energy axis. Measuring the watts of an AI workload looks like a sustainability detail, but it is at once a cost lever (electricity is 30–50 % of TCO), an axis of sovereignty (carbon per token depends on the country) and, increasingly, a regulatory obligation (emissions reporting). This article inventories the metrics, where each figure physically comes from, the data sheet for each measurement framework, and how to go from watt to carbon with European figures. No recommendations; only data and methodology, because in energy, even more than in performance, the measurement method changes the result.

Why measuring energy matters (with numbers)

Three quantified reasons, connecting with the opening article:

Reason	Figure	Implication
Cost	electricity = 30–50 % of TCO	measuring energy is measuring almost half the cost
Sovereignty/carbon	France ~9× fewer gCO₂/token than Germany	the country of the cluster is a lever
Scale/regulation	datacenters ~460 TWh in 2025 (≈1.8 % worldwide), >800 TWh projected for 2028	emissions reporting (CSRD) mandatory

At 700 W per H100, an 8-card node draws of the order of 5.6 kW from the boards alone, ~49,000 kWh a year. Applying PUE 1.4 that becomes ~68,700 kWh/year, and there the country decides the bill: at the French wholesale price (~0.058 €/kWh) that is ~3,980 €/year of electricity per node; in Germany (~0.091 €/kWh), ~6,250 €/year, 57 % more for the same iron and the same work. Multiplied across a fleet, it is a first-order cost line, and a carbon footprint that, in Europe, has to be measured and reported. Energy stopped being an engineering detail and became a business and compliance variable.

The energy metrics

Metric	Definition	Unit
Mean power	average instantaneous consumption	W
Energy per token	energy consumed per generated token	J/token
Energy per inference	energy per request	µJ – Wh
Total energy	integral of power over time	Wh, kWh
PUE (Power Usage Effectiveness)	datacenter overhead (cooling, losses)	ratio ≥ 1
Carbon	energy × grid intensity	gCO₂eq

The two base identities of the whole axis:

$$\text{energy per token (J)} = \frac{\text{mean power (W)}}{\text{throughput (tok/s)}}$$ $$\text{carbon} = \text{energy (kWh)} \times \text{PUE} \times \text{grid intensity (gCO}_2\text{/kWh)}$$

The first has the same shape as cost per token: it shares the throughput denominator, so raising tokens/s lowers energy per token. The second brings in the two multipliers external to the compute, the datacenter PUE and the country’s grid intensity, which can change carbon per token by an order of magnitude without touching the stack.

Worked example: energy per token of an 8×H100 node

To anchor the identities with numbers, the end-to-end calculation on the example node (8×H100, Llama 3.1 70B FP16, vLLM at 2,800 tok/s):

Step	Calculation	Result
Board power	8 × 700 W	5,600 W
Energy per token (boards)	5,600 ÷ 2,800	2.0 J/token
Energy per token (with PUE 1.4)	2.0 × 1.4	2.8 J/token
Energy per 1M tokens (boards)	2 × 10⁶ J	0.56 kWh
Energy per 1M tokens (with PUE)	0.56 × 1.4	0.78 kWh

From there come the electricity cost and the carbon, by country:

Country	Electricity cost / 1M tok	Carbon / 1M tok
France (0.058 €/kWh; ~40 gCO₂/kWh)	~0.045 €	~31 gCO₂
Spain (0.054 €/kWh; ~160 gCO₂/kWh)	~0.042 €	~125 gCO₂
Germany (0.091 €/kWh; ~363 gCO₂/kWh)	~0.071 €	~283 gCO₂

And the link with performance: if an optimisation (FP8, say) raised throughput to 4,200 tok/s, energy per token would drop to ~1.33 J/token (boards) and everything below it in the same proportion. Electricity cost and carbon per token fall with throughput, by the identity. Note: these are board + PUE figures; the complete node (CPU, NICs, PSUs) consumes somewhat more, and energy measured with a power meter can differ from energy estimated with nvidia-smi.

Where the figure comes from: the measurement layers

Not every energy figure is equally reliable. The figure can come from four layers, with increasing accuracy and decreasing context:

The GPU is the dominant source in LLM inference, and its power is read from a hardware counter (NVML/nvidia-smi, aggregated by DCGM, the same basis as GPU observability). RAPL gives CPU and DRAM. The eBPF + model layer (Kepler) splits the total watt between containers and pods without instrumenting the application. And the physical power meter is the ground truth against which estimates are calibrated: no software estimate is better than the meter, only more convenient.

How it is instrumented (and where the error creeps in)

Four instrumentation decisions explain most of the discrepancies between energy measurements:

Sampling rate. nvidia-smi/DCGM read power at intervals; sampling that is too coarse misses the peaks and underestimates the energy. It must be fine enough to capture the power variation between prefill (high) and decode (lower).
Board vs node. GPU (board) power does not include CPU, NICs, fans or PSU losses (typically an extra 10–20 %). Comparing a board energy figure with a complete-node one is comparing different things.
Idle baseline. A powered GPU with no work already draws power. Is that idle attributed to the workload or discounted? The decision changes energy per token, and idle matters: it is real energy that someone pays for.
Time window. Energy is the integral of power over time; the measurement window has to line up exactly with the workload (without counting warm-up or shutdown), or the number does not correspond to the work measured.

These decisions, as in performance benchmarking, mean that the measurement method matters as much as the system measured. A J/token without a stated sampling rate, board/node boundary and idle treatment is not comparable.

Frameworks, tool by tool

Zeus (ml.energy, University of Michigan)

What it does: measure and optimise the energy of deep learning workloads. Method: NVML counters. Scope: NVIDIA and AMD GPUs, CPU, DRAM, Apple Silicon and NVIDIA Jetson. Licence: OSS; academic backing (NSDI'23 paper). Differentiator: it does not only measure, it also optimises (power capping, frequency selection) to cut energy without significant loss of performance (Zeus Project, NSDI'23). It is the reference tool when the goal is to lower J/token, not just to measure them.

CodeCarbon

What it does: estimates the energy and the CO₂eq of compute processes. Method: reads the GPU with nvidia-smi and estimates CPU/RAM. Scope: CPU + GPU + RAM. Licence: OSS (Python). Differentiator: it integrates into ML flows and CI pipelines, and it is the one that comes closest to the power meter among the Python tools (comparison). It brings carbon context out of the box. Limit: a high-level estimate, less exact than direct hardware measurement.

CarbonTracker

What it does: tracks energy and carbon at GPU run time. Scope: GPU only, it measures neither CPU nor memory. Licence: OSS. Differentiator: lightweight, easy to add to a training/inference script. Limit: partial coverage (no CPU/DRAM), so it undercounts the total system energy.

Scaphandre

What it does: a power metrology agent written in Rust. Method: RAPL (reads /proc, splits joules by CPU ticks). Scope: process or VM, with the raw accuracy of the hardware layer. Licence: OSS. Differentiator: the highest direct accuracy among the general-purpose tools, at the cost of bringing no carbon context (it is pure metrology) (Scaphandre · GitHub). Limit: focused on CPU/RAPL; for GPU it has to be combined with NVML.

Kepler (CNCF)

What it does: a Prometheus exporter that measures energy at container, pod and node level in Kubernetes. Method: eBPF + ML models to estimate consumption per workload. Language: Go + C. Licence: Apache 2.0 (the sustainable-computing-io project) (Kepler · GitHub). Differentiator: it is the cloud-native option for attributing energy per pod without instrumenting the app, just as OpenCost attributes cost. Limit: attribution via eBPF + model is estimation, not direct measurement; its accuracy depends on the model.

MLPerf Power (MLCommons)

What it does: a standardised benchmark of energy efficiency, from µW to MW. Maintainer: MLCommons. Differentiator: cross-vendor comparability of efficiency, with a dataset spanning several years and workload versions (MLPerf Power, arXiv 2410.12032). It is to energy what MLPerf Inference is to performance: the comparison standard between vendors.

Comparison table

Tool	Method	Scope	Licence	Accuracy / note
Zeus	NVML / counters	NV+AMD GPU, CPU, DRAM, Apple, Jetson	OSS	measures and optimises DL energy
CodeCarbon	nvidia-smi + estimation	CPU+GPU+RAM, CO₂eq	OSS (Python)	the closest to the power meter among Python tools
CarbonTracker	GPU runtime	GPU only	OSS	lightweight; no CPU/mem
Scaphandre	RAPL (/proc)	process/VM	OSS (Rust)	raw accuracy; no carbon context
Kepler	eBPF + model	container/pod/node (K8s)	Apache 2.0 (CNCF)	cloud-native; model-based estimation
MLPerf Power	standardised measurement	µW to MW	MLCommons	cross-vendor comparability

Accuracy: who to believe

The difference between tools is not a nuance: Scaphandre offers the raw accuracy of the hardware layer (RAPL), while the Python ones such as CodeCarbon give higher-level, carbon-aware estimates, with more context but less direct exactness; among the Python tools, CodeCarbon is the one that comes closest to the power meter, followed by CarbonTracker, with variability across infrastructures (comparison). The rule: for sizing and design, a software estimate will do; to defend a figure before a committee or an auditor, calibrate against a power meter at least once, and use the software tool as a continuous proxy. The measured figure beats the estimated one; the rest is convenience.

Energy leaderboards and benchmarks

To compare efficiency between models without building your own test bench:

Resource	What it provides
MLPerf Power	standardised efficiency measurement, from µW to MW
HF AI Energy Score	comparable per-model energy efficiency ratings (HF)
ML.ENERGY Benchmark	automated measurement and optimisation of inference energy
awesome-green-ai	curated index of Green AI resources and tools (repo)

These resources give the order of magnitude of the energy per token of each model, useful for choosing a model on efficiency at the design stage. But they do not replace measurement on your hardware and your workload: a leaderboard measures on one specific bench, not on your node.

From watt to carbon: France, Germany, Spain

Here the energy axis turns into a sovereignty argument. Carbon per token comes from multiplying the energy by the country’s grid intensity, which in Europe varies by almost two orders of magnitude:

Country	Wholesale (€/MWh, Mar 2026)	Grid carbon (gCO₂/kWh)	Profile
France	~58	~20–60	nuclear: cheap and very clean
Spain	~54	~150–170	renewables + gas: cheap, medium carbon
Germany	~91	~363 (2024 average)	coal/gas + renewables: expensive and dirty

Sources: wholesale prices (TradingEconomics), Germany’s carbon ~363 gCO₂/kWh (2024 average, Wikipedia), nuclear ~17–35 gCO₂e/kWh (EEA), Spain ~167 gCO₂/kWh (2020). Hourly intensity comes from ElectricityMaps.

The calculation, on the example node (0.78 kWh per million tokens, with PUE): in France (~40 gCO₂/kWh) it is ~31 gCO₂ per million tokens; in Germany (~363 gCO₂/kWh), ~283 gCO₂, a factor of ~9×, without touching a line of the stack. Placing the cluster in France or Spain is at once a cost lever, a carbon lever and a compliance lever (EU AI Act, controls ENS × 42001 × EU AI Act).

Energy efficiency levers

Once it is measured, it can be brought down. The levers, ordered by the identity J/token = W ÷ throughput (lower the numerator or raise the denominator):

Lever	Mechanism	Effect
Quantisation (FP8/INT4)	more tokens per GPU-hour, less VRAM/KV	raises throughput → lowers J/token
Batching / continuous batching	amortises the fixed power cost	raises throughput → lowers J/token
Power capping (Zeus)	limits GPU power	lowers W with little loss of throughput
Efficiency-aware scheduling	consolidate workloads, power down idle GPUs	eliminates idle consumption
Location (country)	cleaner grid	lowers carbon per token
Avoiding over-reasoning	fewer reasoning tokens that add nothing	less compute → less energy

Quantisation (see quantisation for inference) is the highest-return lever because it moves all three axes at once: it raises throughput, lowers cost and lowers energy per token. Zeus’s power capping is energy-specific: it trims peak power with little performance loss, lowering the W in the numerator. And idle, the GPU powered but not working, is pure wasted energy that scheduling recovers.

From design to production: where energy is measured

As with the other axes, energy is measured at three moments, with different tools and increasing accuracy:

Moment	Tool	What it gives
Design	TDP × hours (datasheet)	ceiling estimate for sizing
Benchmark	Zeus, MLPerf Power, power meter	energy per token measured on a controlled bench
Production	Kepler (eBPF), DCGM	continuous per-pod consumption in the cluster

The design figure (TDP) serves to size the utility feed and the cooling, but it overestimates (the GPU is rarely at TDP 100 % of the time). The benchmark figure is the one used for the proposal’s cost and carbon per token, because it is measured and reproducible. And the production figure (Kepler/DCGM) closes the loop: it checks that real consumption matches the budget and feeds the footprint report. The three have to be traceable to each other, which is what the harness in article S4 does.

Energy in TCO and European regulation

The closing of the axis: energy is not only sustainability, it is cost and compliance.

Cost: being 30–50 % of TCO, every J/token improvement translates directly into euros. In the example, moving from Germany (~0.091 €/kWh) to France (~0.058 €/kWh) lowers the electricity share by ~37 % on top of the carbon.
Regulation (CSRD): European companies must report their carbon footprint (scope 2). A low-carbon grid (France) directly improves that reported figure, and measuring energy per token stops being optional and becomes a reporting requirement.
EU AI Act: efficiency and energy consumption form part of the documentation expected of AI systems, which makes energy measurement part of the technical file.

One reporting nuance worth knowing: the electricity footprint is scope 2 (indirect emissions from purchased energy), and it can be accounted for by location (the real intensity of the country’s grid) or by market (according to the guarantees of origin contracted). Location-based accounting is the one that genuinely rewards putting the cluster on a clean grid such as the French one; market-based accounting allows green energy to be “bought” through certificates. Knowing which one your CSRD report requires changes which lever (physical location vs contract) lowers the figure, and both start from measuring energy per token, which is what this axis provides.

In other words: measuring energy covers at once a first-order cost line, the carbon and sovereignty argument, and a regulatory obligation. For a sovereign European proposal, the energy axis is not the “nice sustainability bit”: it is one of the three columns with a number attached.

Checklist for a reproducible energy measurement

For an energy or carbon figure to be defensible before a committee or an auditor, it has to come with its context. The minimum to fix and publish:

What to fix	Why
Tool + version	each layer (RAPL, NVML, eBPF) and tool measures differently
Sampling rate	coarse sampling misses peaks
Board vs node boundary	±10–20 % depending on whether CPU/PSUs are included
PUE used	it multiplies all the compute energy
Source and time of the grid intensity	carbon varies ~9× by country and by hour
Idle treatment	attributed or discounted changes the J/token
Model, precision and workload	energy depends on what is served and how
Calibration vs power meter	at least once, to validate the estimate

The rule: if you cannot hand over this table alongside the gCO₂/token figure, the figure is not auditable. The reproducible harness of article S4 records all these parameters so that the energy figure is as traceable as the cost and performance ones, and so that the reported footprint (CSRD) holds up under audit.

State of the art 2026

LLM-specific resolution is missing: general-purpose tools measure well at machine or process level, but they lack LLM-specific resolution (energy per token, per prefill/decode phase), and the simulators lack emissions modelling, an open gap in the field.
eBPF as a trend (Kepler) for attributing energy per pod without instrumenting the app, in parallel to how OpenCost attributes cost.
Standardisation of comparison via MLPerf Power and HF AI Energy Score.
Optimisation, not only measurement (Zeus): the field is moving from “how much do I spend” to “how do I spend less without losing performance”.

Limits and traps (data-driven)

Estimation vs measurement. An nvidia-smi figure is not a power meter. Calibrate against hardware at least once before defending a number.
Partial coverage. CarbonTracker measures GPU only; Scaphandre, mostly CPU/RAPL. Add the layers or you undercount the total energy.
Forgetting PUE. Board energy does not include cooling or losses; without PUE you underestimate the real consumption and carbon.
Carbon without a country. A gCO₂/token without a stated grid (and hour) means nothing: it varies ~9× between France and Germany.
Confusing energy with sustainability. Here it is cost and compliance, with a number; treating it as a “green” extra is losing one of the three columns of the proposal.

That closes the run of introductions: cost, performance and energy, each with its frameworks and its methodology. The rest of the series goes tool by tool through to the dashboard that underpins the sovereign architecture decision, with European figures, measured and reproducible.

Closing

Energy is the axis easiest to dismiss as “the green one” and the one that hides the most: it is almost half the cost, it defines the footprint Europe requires to be reported, and its carbon per token changes ~9× depending on whether the cluster sits in France or in Germany. Measuring it properly, with the right layer, the right sampling, PUE included and the country’s grid, turns a soft sustainability argument into a hard column of the proposal, with a number and a source. And the sovereign conclusion stands on its own with the data in this article: an inference platform in France or Spain is at once cheaper, cleaner and compliant with European jurisdiction than the same platform on a US hyperscaler, and that is demonstrated with measured J/token, gCO₂/kWh by country and euros, not with intentions. The energy axis does not decorate the proposal: it closes it.

Sources

Zeus Project (ml.energy, UMich) — https://ml.energy/zeus/
Zeus · USENIX NSDI'23 — https://www.usenix.org/system/files/nsdi23-you.pdf
Kepler (CNCF, eBPF) · GitHub — https://github.com/sustainable-computing-io/kepler
Scaphandre · GitHub — https://github.com/hubblo-org/scaphandre
CarbonTracker · GitHub — https://github.com/saintslab/carbontracker
MLPerf Power (arXiv 2410.12032) — https://arxiv.org/html/2410.12032v2
HF AI Energy Score — https://huggingface.github.io/AIEnergyScore/
awesome-green-ai — https://github.com/samuelrince/awesome-green-ai
TradingEconomics · precio electricidad Francia/Alemania/España — https://tradingeconomics.com/france/electricity-price
EEA · intensidad de emisiones de la electricidad en Europa — https://www.eea.europa.eu/en/analysis/indicators/greenhouse-gas-emission-intensity-of-1
Electricity sector in Germany (carbono ~363 gCO₂/kWh) — https://en.wikipedia.org/wiki/Electricity_sector_in_Germany

Anatomy of the twelve DCGM and five vLLM metrics: analogies, documented anomalies and real cases 2024-2026

Tue, 02 Jun 2026 04:00:00 +0200

This post goes deeper into the list of metrics presented in GPU observability for LLM inference. There each metric got its G/A/R threshold and PromQL query; here each one gets its explanatory analogy and the anomaly documented in production with a referenced public case. It is the post worth having open when an alert fires and you still do not know what to do with it; the next post on runbooks translates each anomaly into concrete action.

TL;DR

The twelve DCGM metrics (compute, memory, thermal-power, health) and the five vLLM engine metrics (concurrency, KV pool, SLO latencies) covered in the previous post paint the cluster’s cockpit, but a list without context does not teach you to diagnose. Each metric has a recurring anomalous pattern documented in the public literature (academic papers, GitHub issues, OEM KBs, operator blogs) that the veteran operator recognises instantly and the junior one does not. This post develops each metric with an analogy of its own that pins down which question it answers, and with the statistically relevant anomaly backed by figures from documented incidents. Three examples of the calibre involved: Meta published that during the training of Llama 3 405B on 16,384 H100s there were 419 unplanned failures in 54 days, one every 3 hours, with GPU + HBM3 accounting for 47 % of the total; the paper Story of Two GPUs (arXiv 2503.11901) quantifies that the H100 has 3.2× worse MTBE from uncorrectable ECC than the A100, attributable to the higher density of HBM3; issue vllm#16300 documents that on a cluster of 8×A100 80 GB TP=8 delivers worse throughput than TP=4 because NVLink saturation kills the partition speedup. The KBs Dell 000220508 and Lenovo HT514380 formalise the recurring HW Power Brake case in H100 racks overcommitted at PDU level. Issue vllm#25677 showed chunked prefill 10× slower than without it on Qwen3-30B-A3B (bad calibration of max_num_batched_tokens). Issue vllm#11912 documents a TPOT regression from 15.7 ms to 25.7 ms crossing version 0.6.4. Each case includes a verifiable URL. The operational rule: when an alert arrives, look first at the anomalous pattern associated with the metric that fired, before opening the request’s trace; 80 % of degradations match one of the documented patterns.

You are here: OBSERVE — the diagnosis layer

Family 1 — Compute

`DCGM_FI_PROF_SM_OCCUPANCY` — is there parallel work in the engines?

The analogy. An industrial kitchen with 32 hobs and a single chef. The metric answers “how many hobs have a pan on them right now?”. If half are empty, the kitchen is under-used: the orders come out one after another because the chef does not open up parallelism. If all of them are occupied but the chef is standing still watching a stopwatch, the hobs are on but nothing is being cooked (a pathological kernel saturating SMs without doing useful work).

The documented anomaly. The best-known trap: high SM occupancy does not imply real throughput. The article “GPU Utilization Is a Counter, Not a Cause” (Ingero, May 2026) put it in one exact sentence: “a kernel running at 5 % of peak FLOPS for 100 ms still reads 100 % on SM_ACTIVE”. In MoE workloads the effect turns pathological: overloaded experts produce the Straggler Effect (paper arXiv 2503.05066). The SMs look busy while waiting for the saturated expert, and the utilisation dashboard paints green while latency goes through the roof.

Operational implication. Do not trust sizing or autoscaling to SM occupancy alone. Always combine it with PIPE_TENSOR_ACTIVE (is there useful compute?) and DRAM_ACTIVE (is memory the bottleneck?). The normal LLM decode regime is 30–55 %, not 99 %; seeing a sustained 99 % with high TPOT is a symptom of a kernel bug or an MoE straggler.

`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE` — are the tensor cores producing?

The analogy. A factory with two lines: the manual one (CUDA cores) and the automated one (tensor cores). The metric answers “what percentage of the time is the automated line active?”. If you buy an H100 for its tensor cores and the automated line is at 5 %, you have paid for a Ferrari to do bicycle courier work.

The documented anomaly. Issue vllm#20783 (July 2025) was titled literally “Performance Anomaly: compressed-tensors shows no speedup over BF16 on H100”. The operator expected 1.5–2× with FP8 quantisation and got parity with BF16; the PIPE_TENSOR_ACTIVE metric revealed that the FP8 path was not running on the HMMA units (the tensor unit for FP16/BF16/FP8) and was falling back to CUDA cores. Issue vllm#31475 documented the parallel case on MI300X: FP8 slower than BF16 because of a regression in the ROCm path. DCGM exposes separate counters per unit (HMMA for FP16/BF16/FP8, IMMA for INT8, DMMA for TF32/FP32); if HMMA is low even though the model is BF16, the engine is not using tensor cores.

Operational implication. Check PIPE_TENSOR_ACTIVE after every change of quantisation or engine version; a supposedly neutral change may have disabled the optimised path. For prefill expect 50–80 %; for decode 15–30 % is normal (decode is memory-bound, not compute-bound). A figure below 5 % in prefill means the engine is not using tensor cores.

`DCGM_FI_PROF_DRAM_ACTIVE` — is the HBM saturated?

The analogy. A motorway with N lanes. The metric answers “what percentage of the time are all the lanes occupied moving cars?”. When the tensor cores ask for data faster than the HBM delivers it, the motorway is at 95 % and the engines wait. In decode, this is the normal regime: you walk the model weights and the KV cache through for every token.

The documented anomaly. The paper “Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference” (arXiv 2503.08311) quantifies that at contexts ≥ 128k, reading the KV cache dominates the total decode time and saturates the HBM3 (3.35 TB/s on the H100). The distinctive pattern: DRAM_ACTIVE > 80 % with PIPE_TENSOR_ACTIVE around 10–20 %. Raising the batch no longer helps: the bottleneck is not FLOPS, it is bandwidth. The useful lever is compressing the KV, see Quantisation for --kv-cache-dtype=fp8, which cuts the KV footprint by roughly 50 %.

Operational implication. If DRAM_ACTIVE > 95 % is sustained and gpu_cache_usage_perc < 70 %, something other than your engine is asking for HBM (a leak in a library, another process sharing the GPU without MIG). Investigate immediately with nvidia-smi and fuser /dev/nvidia*.

Family 2 — Memory

`DCGM_FI_DEV_FB_USED` — how much VRAM has been consumed?

The analogy. The fuel level in an aircraft’s tank in flight: the pilot needs to know how much is left and at what rate it is being consumed, not just the instantaneous figure. An H100 at 88 % FB used and stable can operate calmly; the same figure rising 2 %/min announces OOM in 7 minutes.

The documented anomaly. Issue dcgm-exporter#512 documents a surprise relevant to MIG clusters: DCGM_FI_DEV_FB_USED and DCGM_FI_DEV_FB_FREE are absent on H100 GPU instances with MIG enabled, present on A100 and B200, but an exporter bug hides them on H100-MIG. Operators who assume the dashboard covers everything discover the blind spot on the day of the first OOM. Issue dcgm-exporter#271 documents another detail: FB_USED + FB_FREE does not always add up to a constant because there is driver-reserved overhead that shows up in the delta. The original PagedAttention/vLLM paper estimated that pre-PagedAttention serving frameworks wasted 60–80 % of the KV cache to fragmentation; PagedAttention brought it down to under 4 %.

Operational implication. On H100 MIG clusters, check that DCGM_FI_DEV_FB_USED appears per instance before trusting alerts; if it is absent, monitor through nvidia-smi --query-gpu=memory.used directly. Operational rule: alert on the delta (a sustained rise), not just an absolute threshold.

`DCGM_FI_DEV_FB_FREE` — the absolute complement

The analogy. The “range remaining” indicator in a modern car: it complements the percentage with an absolute figure that is directly actionable.

The documented anomaly. When an aggressive PagedAttention pool leaves FB_FREE at small absolute values (< 2 GiB), any normal allocation of transient buffers (the activations of a large prefill) can push you into OOM. The classic pattern: a “green” percentage (87 %) but a “red” absolute (< 4 GiB free on an 80 GB H100).

Operational implication. A complementary alert with an absolute threshold: DCGM_FI_DEV_FB_FREE < 4096 (MiB). It is the safety net for the cases where the percentage misleads because the engine is configured with a very high gpu_memory_utilization.

`DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL` — does the internal bus hold up?

The analogy. An interstate motorway between four cities. Every car that crosses to do a tensor parallel all-reduce pays a toll and consumes width. When there are more cars than the motorway supports, the latency to reach the destination shoots up, even if each individual car is fast.

The documented anomaly. Issue vllm#16300 (April 2025) was titled “Performance degradation with tp=8 compared to tp=4 on 8×A100(80G)” and documented TP=8 delivering worse throughput than TP=4 on the same cluster, same model, same quantisation. Root cause: tensor parallelism requires an all-reduce after each attention and MLP block; at TP=8, the communication cost between 8 GPUs (even over NVSwitch) grows faster than the speedup of partitioning the compute. The practical rule that emerges: TP=4 + 2 replicas usually delivers better latency/throughput than TP=8 + 1 replica, except for extremely long contexts (≥128k) where you need the aggregate VRAM. Theoretical NVLink 4.0 capacity on the H100 SXM: around 450 GB/s per GPU; typical sustained TP=4 regime: 50–150 GB/s.

Operational implication. If NVLINK_BANDWIDTH_TOTAL > 90 % of capacity is sustained, this is not a problem solvable by raising parallelism. The opposite: lower TP. The metric is orthogonal to the sizing from capacity planning: the ceiling is not only VRAM/time, it is also the bus.

Family 3 — Thermal and power

`DCGM_FI_DEV_GPU_TEMP` — is the GPU breathing?

The analogy. The body temperature of an elite athlete under full effort. 36–37 °C is normal; 38 °C is sustainable stress; above 39 °C the body activates protective mechanisms (sweating, slowing down) that degrade performance. The GPU does the same: above a thermal threshold, it reduces its clock automatically. If it did not, it would break.

The documented anomaly. The H100 SXM5 with its 700 W TDP has thermal thresholds that are not entirely public (NVIDIA does not publish them exhaustively in the datasheet), but the behaviour is well known: above roughly 85 °C edge or roughly 95 °C HBM, the bit 0x40 HW_THERMAL appears in the clock throttle reasons. Operators on the NVIDIA developer forum report that with a rack inlet temperature above 27 °C, throttling is routine. The NVIDIA HGX Platform paper states that the minimum recommended air flow is > 1000 CFM/kW; densities above 30 kW/rack at 700 W TDP demand mandatory liquid cooling because forced air does not get there.

Operational implication. If GPU_TEMP > 83 °C is sustained, look first at CLOCK_THROTTLE_REASONS (bit 0x40) and at the rack inlet temperature: this is not an engine problem, it is an air flow one. For legacy air-cooled racks, consider redistributing thermal load or installing a rear-door HX.

`DCGM_FI_DEV_POWER_USAGE` — how much is it asking of the socket?

The analogy. The instantaneous draw of an industrial appliance plugged into a three-phase outlet with a sized breaker. If the washing machine starts at 9 kW and the breaker is 10 kW, you live on the edge; if the washing machine “gets along” with the breaker it is because somebody sized it consciously.

The documented anomaly. A published real measurement: an H100 SXM5 with vLLM running Llama 3.1 405B at batch=4 draws around 697 W at-wall sustained (NVIDIA TDP 700 W). Now the interesting operational lever: lowering nvidia-smi -pl from 700 W to 500 W delivers around 30 % energy savings with only around 20 % loss of throughput. A cluster of 4 nodes × 8 H100 at 700 W equals around 22 kW of GPU alone; at 500 W, around 16 kW. The difference pays an entire quarter’s electricity bill in clusters operated around the clock. A three-phase 415 VAC PDU branch at 60–80 A supports around 32 kW, roughly 4 DGX H100. Legacy 208 V does not support H100 density; reference: NVIDIA DGX SuperPOD Electrical Specifications.

Operational implication. A useful metric for three things: (1) detecting anomalously low workloads (unexpected idle), (2) computing per-tenant showback of real energy cost (not estimates), (3) alerting if the draw approaches the PDU branch limit. Keep GPU → PDU branch → breaker mapped in the CMDB.

`DCGM_FI_DEV_CLOCK_THROTTLE_REASONS` — who is stepping on the brake?

The analogy. The “limited mode” warning light on the dashboard of a modern car. When it lights up, the car reduces its performance automatically, but it does not tell you why unless you know how to read the combination of letters. The bits of the bitmap are those letters.

The documented anomaly. A public case formally acknowledged by two different OEMs: Dell KB 000220508 and Lenovo HT514380 address the same phenomenon: HW Power Brake Slowdown active (bit 0x80) on the H100 SXM. The cause is not the GPU: it is the chassis PDU sending an electrical power-brake signal because the rack branch is close to the breaker limit. The operator sees throughput down 30–50 % with no XID and no ECC, and the inference engine “is healthy”; the problem is in the electrics. The NVIDIA developer forum thread on “HW Power Brake Slowdown” corroborates the pattern. Bit 0x40 HW_THERMAL appears in badly ventilated racks; bit 0x04 SW_POWER_CAP appears if somebody left nvidia-smi -pl 500 set and nobody reverts it.

Operational implication. Any bit other than 0 or Idle (bit 0x01) sustained is an immediate alert. The recommended decoding: record the complete bitmap value in the log plus the attribute throttle.reasons.decoded=["HW_THERMAL", "HW_POWER_BRAKE"] on the OTel span. Without that, incident response does not know what to do.

Family 4 — Health (the catastrophic reports)

`DCGM_FI_DEV_XID_ERRORS` — the driver’s red codes

The analogy. The critical alarm lights in a nuclear control room. They do not climb gradually; they either appear or they do not. Each XID is a predefined code (XID 13 = graphics engine exception; XID 31 = MMU fault; XID 43 = stopped channel; XID 79 = GPU fallen off the bus; XID 95 = uncontained ECC), and each one has its documented procedure.

The documented anomaly. The most studied public case: Meta published that during the training of Llama 3 405B on 16,384 H100s over 54 days there were 419 unplanned failures, one every 3 hours at cluster scale. GPUs accounted for 148 (35 %) plus HBM3 for 72 (17 %), almost half of all failures. The paper “Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100” (arXiv 2503.11901) quantifies with a different dataset (2.1M GPU-hours) that the H100 has 3.2× worse MTBE for uncorrectable ECC than the A100. The ByteDance MegaScale paper reports that XID 79 (“GPU fallen off the bus”) co-occurs with PCIe errors in 43 % of cases. The NVIDIA developer forum documents persistent cases of XID 31 (MMU fault) that follow the GPU when it is moved to a different PCIe slot, a hardware bug in the module, not in the backplane.

Operational implication. Any increment of the counter is an immediate alert: many XIDs demand a node reset or an RMA of the GPU. The XID-by-XID distinction matters: XID 13/43 is usually a software bug if it coincides with a recent change; XID 31/48/79/94/95 is usually hardware. Keep a canonical xid → procedure table. See the runbooks for the translation into concrete action.

`DCGM_FI_DEV_ECC_DBE_VOL_TOTAL` — the errors that corrupt data

The analogy. An accounting ledger where sometimes someone erases an entry and rewrites it (corrected single-bit ECC: note a change in the margin and carry on) and sometimes someone burns two pages at once (double-bit: the information is lost, the audit has to stop).

The documented anomaly. The paper “Characterizing GPU Resilience” quantifies for the H100 that when XID 48 (DBE) appears, the job in flight dies with 100 % probability (5 out of 5 in the dataset studied). The documented recovery: drain the node, reset, complete the row remap, equalling around 19 hours of node downtime. HBM3 density explains the worse MTBE against HBM2e: there are more cells per unit area, and therefore a higher statistical probability of degradation. In Llama 3, HBM3 caused 72 of the 419 interruptions (17 %).

Operational implication. Any value > 0 is a critical alert. The GPU must be drained immediately, removed from the scheduler, fully reset, validated for row remap with nvidia-smi -q -d ROW_REMAPPER (Pending: No), and given an extensive smoke test before returning to the pool. If the row remap uses more than 4–8 spare pages on one GPU, plan a replacement in the next window: the degradation is progressive.

`DCGM_FI_DEV_RETIRED_DBE` — the pages marked for retirement

The analogy. The tiles a museum restorer marks with yellow tape because they are damaged. They pose no immediate danger (the room stays open), but the accumulation says the floor is degrading structurally and the full replacement has to be planned.

The documented anomaly. NVIDIA documents up to 512 spare pages per HBM bank on the H100; the RETIRED_DBE counter indicates how many have been used. Operators on NVIDIA forums report that above 4–8 retired pages on a specific GPU, the frequency of XID 48 rises. The pattern: a GPU with 6 retired pages today becomes 12 in a month, a first XID 48 two months later, and a forced drain.

Operational implication. A trend metric, not an immediate alert. Record the value per GPU and review it monthly; GPUs with rising values go into the proactive replacement plan before the catastrophic failure.

The five vLLM engine metrics

`vllm:num_requests_running` — how many requests fit in the batch?

The analogy. The number of cars a toll booth lets through at once. If the barrier opens for N at a time, car N+1 waits in the queue. Saturation shows up because the line does not get shorter.

The documented anomaly. Reaching the configured --max-num-seqs and staying there is the typical symptom of a cluster below its sizing; the engine admits up to the ceiling and no further. The query vllm:num_requests_running == max_num_seqs for more than 5 minutes indicates firm saturation.

Operational implication. Combine it with num_requests_waiting: if running is at the ceiling AND waiting > 0, you have to scale. If running is at the ceiling and waiting is 0, you are in the optimal regime (cluster used to the maximum with no queue).

`vllm:num_requests_waiting` — the primary saturation indicator

The analogy. The visible queue in front of the toll booth. As long as it is empty, the system flows; the moment a sustained queue forms, drivers start arriving late at their destination, and TTFT goes through the roof.

The documented anomaly. A public case in “11-Second Time to First Token on a Healthy vLLM Server” (Medium, Ingero, 2026): a server with no XIDs, no preemption, DCGM metrics all green, but num_requests_waiting sustained above 0 and a TTFT of 11 seconds. Issue vllm#16985 documents progressive degradation in long sessions: the queue grows slowly over hours without any other indicator moving. The root cause is not hardware, it is admission control: the arrival rate exceeds the completion rate and the system does not enqueue any more, it leaves requests in WAITING. Red Hat designates it as the primary saturation metric in its tutorial “5 steps to triage vLLM performance”.

Operational implication. The primary HPA metric in KEDA, see Autoscaling LLM on Kubernetes. Typical threshold: alert if avg_over_time(vllm:num_requests_waiting[5m]) > 5. For canaries: if the queue only forms in the v2 pool, it is a regression of the new model, not cluster load.

`vllm:gpu_cache_usage_perc` — the KV cache pool

The analogy. The capacity of an events hall where each guest takes up a variable amount of space. The maître d’ admits guests up to capacity; when a new guest arrives and there is no room, he throws out the guest who has been there longest to make space for the newcomer. That is vLLM’s preempt-on-OOM.

The documented anomaly. Issue vllm#5051 “Add num_requests_preempted metric” was born exactly out of operators observing degradation with no direct metric telling them how many requests were being thrown out. The official vLLM documentation confirms: “sustained gpu_cache_usage_perc above 90 % indicates the server is approaching its KV cache limit and will begin preempting sequences” (oldest-first). The distinctive visual pattern: a sawtooth near 100 % with preemption spikes. In swap mode, the latency of the preempted request explodes because there is a host↔device PCIe copy; in recompute mode (the default in V1), the preempted request redoes its prefill from scratch, which pushes its TTFT to double or triple.

Operational implication. If gpu_cache_usage_perc > 92 % is sustained, there are two levers: lower max_num_seqs (admit less concurrency, but none of it gets preempted) or raise gpu_memory_utilization (more pool, less VRAM for transient activations, a different risk). The choice depends on the workload. The metric that is directly missing, a preempted counter, is exported from vLLM v1.0 onwards as vllm:num_preemptions_total (see issue #5051).

`vllm:time_to_first_token_seconds` — the latency the client sees

The analogy. The time from a customer walking into a restaurant to receiving the first piece of bread at the table. Too long and the customer thinks they have been forgotten, even if the main course is going to arrive perfect.

The documented anomaly. Three documented patterns of recurring TTFT spikes:

Badly calibrated chunked prefill. Issue vllm#25677 (Qwen3-30B-A3B) reported prefill 10–11× slower with chunked prefill enabled than without it. Cause: a very low max_num_batched_tokens forces small chunks that do not fill the kernels. Issue vllm#7604 documents an equivalent regression on Llama-3-70B v0.5.4. The lever: raise max_num_batched_tokens to 4096–8192 for typical prompts above 2k.
Regression between engine versions. Issue vllm#8819 documents a regression of vllm:time_to_first_token_seconds_sum between minor versions. Issue vllm#11912 reports that with a prompt of around 8000 tokens, TPOT rose from 15.7 ms → 25.7 ms from v0.6.4.post1 onwards with no config change, a regression confirmed and trackable only with the metric.
Long-context prefill blocking decodes. The “11s TTFT on healthy server” case cited above: a 30k-token prefill monopolises the GPU for several seconds and the active decodes freeze. Fix: well-calibrated chunked prefill, or disaggregated serving (see Disaggregated serving).

Operational implication. Do not alert only on the absolute P95; alert also on the v2/v1 ratio when there is a canary (histogram_quantile(0.95, ..., version="v2") / histogram_quantile(0.95, ..., version="v1") > 1.10). If TTFT grows and the queue is stable, the bottleneck is prefill: not solvable by adding replicas, but there are levers in quantisation or chunked prefill.

`vllm:time_per_output_token_seconds` — streaming smoothness

The analogy. The speed at which the waiter brings the courses one after another after the first. If the next one takes a while, the diner notices something is wrong even though the first course arrived on time.

The documented anomaly. The distinctive pattern is the abrupt step when gpu_cache_usage_perc crosses roughly 85 %: TPOT jumps from 35 ms to 80 ms in a few seconds because the engine starts competing for the HBM with its own evictions. Issue vllm#35387 documents another anomalous case: MTP (speculative decoding) causing a 76 % latency regression on Qwen3-Next-80B-A3B-Instruct-FP8. The TPOT metric caught it before any client complaints were filed.

Operational implication. The difference from TTFT: if TTFT grows and Queue Time is stable, you are prefill bound; if TPOT grows at a steady rate, there is pressure on the HBM (KV cache pool or swap enabled). A secondary alert on the TPOT SLO, but also watch the derivative: TPOT rising 1 ms every 10 minutes is a latent regression that does not break the SLO yet but will.

The operational rule: read metrics by family, not in isolation

Three anti-patterns of the novice operator

Anti-pattern 1 — alerting only on absolute thresholds. An H100 at 87 % FB is not necessarily an alarm; an H100 at 87 % rising 2 %/min is. Alerts that fire on a fixed threshold without looking at the derivative produce twice the noise and half the value. Rule: for metrics with known dynamics (KV cache, FB, queue), alert on a sustained delta, not just on the level.

Anti-pattern 2 — confusing SBE with DBE. The DCGM_FI_DEV_ECC_SBE_VOL_TOTAL counter (single-bit, correctable) grows continuously on any HBM under load; it is not an alarm, it is physics. The one that matters is DCGM_FI_DEV_ECC_DBE_VOL_TOTAL (double-bit, uncorrectable). Confusing them yields false negatives (not alerting on a real DBE) or false positives (alerting on a harmless SBE).

Anti-pattern 3 — treating SM_OCCUPANCY at 99 % as “saturated”. The LLM decode regime is memory-bound, not compute-bound; high SM occupancy with low TENSOR_ACTIVE and high DRAM_ACTIVE is normal. Sizing for “GPU at 60 %” and asking for more hardware when the cluster is saturated in HBM (not in SMs) means buying twice the GPUs without gaining throughput. Rule: always read SM_OCCUPANCY together with TENSOR_ACTIVE and DRAM_ACTIVE; on its own it means nothing.

Applied to typical on-premise hardware

For a generic cluster of 4 nodes × 4×H100 SXM 80 GB with intra-node NVLink:

DCGM Exporter per node (a GPU Operator DaemonSet) emitting every 15 s; cardinality per GPU is around 80 series. A 16-GPU cluster gives roughly 1.3k base series, around 85k samples/min with a 15 s scrape.
vLLM /metrics per inference pod; each replica emits around 50 base series. For 16 replicas, around 800 additional series, around 3k samples/min.
Prometheus retention: 30 days at high resolution plus 1 year downsampled through a Thanos sidecar or Mimir. Estimated volume: 25–35 GB/day.
Alertmanager: the 6 critical alerts from the previous post plus derived alerts (delta, v2/v1 ratio, decoded throttle bitmap).

Each metric is worth exposing also as an OTel attribute on the spans of the GenAI tracing: gpu.fb_used_pct, gpu.dram_active, gpu.throttle_reasons.decoded. That lets you correlate a slow request with the state of the GPU at that instant, without jumping between dashboards.

What we have not covered (upcoming posts)

Per-alert runbooks — the translation of each anomalous metric into concrete action (drain, reset, RMA, scaling, rollback) in the next post: Incident response runbooks.
Tail sampling for metric ↔ trace correlation — what is preserved when an alert fires, for post-mortem investigation.
Per-tenant showback combining vllm:request_success_total × gen_ai.usage.* × DCGM_FI_DEV_POWER_USAGE to bill real energy cost.
Multi-tenant fairness metrics — when a tenant hogs the KV cache pool and how to detect it.

References

Meta — Faulty Nvidia H100 GPUs and HBM3 memory caused half of failures during Llama 3 training (Tom’s Hardware, 2024). tomshardware.com
Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100. arXiv 2503.11901. https://arxiv.org/html/2503.11901v3
ByteDance — Robust LLM Training Infrastructure at ByteDance. arXiv 2509.16293. https://arxiv.org/pdf/2509.16293
Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference. arXiv 2503.08311.
Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts. arXiv 2503.05066.
NVIDIA — Analyzing Xid Errors with the Xid Catalog and Memory Error Management (docs.nvidia.com/deploy).
Dell — PowerEdge XE8640 with H100 - GPU Performance Issue HW Power Brake Slowdown - Active (KB 000220508).
Lenovo — Power brake reporting on H100 GPU (HT514380).
vLLM project — issues #5051 (preempted metric), #7604 and #25677 (chunked prefill regression), #11912 (long-prompt regression), #16300 (TP=8 worse than TP=4), #16985 (long-running degradation), #20783 (compressed-tensors no speedup), #35387 (MTP regression).
Red Hat — 5 steps to triage vLLM performance. https://developers.redhat.com/articles/2026/03/09/5-steps-triage-vllm-performance
AI21 — Go big or go OOM: the art of scaling vLLM. https://www.ai21.com/blog/scaling-vllm-without-oom/
11-Second Time to First Token on a Healthy vLLM Server (Medium, Ingero, 2026).
NVIDIA — DGX SuperPOD Electrical Specifications (docs.nvidia.com/dgx-superpod).

Sources: the full URLs are linked inline on each reference.

GPU observability for LLM inference: the twelve DCGM and vLLM metrics that dictate the health of your production

Mon, 01 Jun 2026 15:30:00 +0200

This post complements those on LLM tracing with OpenTelemetry GenAI (the tracing layer above the metrics), Capacity planning (what was sized and what must be watched) and Continuous batching (the mechanism that explains several of the engine metrics).

TL;DR

Observability for an LLM inference cluster is built on two complementary sources: the GPU hardware metrics exposed by the DCGM (Data Center GPU Manager) Exporter, part of the NVIDIA GPU Operator, and the metrics from the inference engine (vLLM, SGLang, TensorRT-LLM) exposed on a Prometheus-compatible /metrics. Neither is enough on its own. The classic nvidia-smi metric called GPU utilization is misleading for LLMs: it reads high whenever any kernel is running, without distinguishing tensor cores burning from SMs waiting on HBM. The full cockpit has twelve DCGM metrics in four families (compute: DCGM_FI_PROF_SM_OCCUPANCY, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, DCGM_FI_PROF_DRAM_ACTIVE; memory: DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_FB_FREE, DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL; thermal-power: DCGM_FI_DEV_GPU_TEMP, DCGM_FI_DEV_POWER_USAGE, DCGM_FI_DEV_CLOCK_THROTTLE_REASONS; health: DCGM_FI_DEV_XID_ERRORS, DCGM_FI_DEV_ECC_DBE_VOL_TOTAL, DCGM_FI_DEV_RETIRED_DBE) and five vLLM engine metrics (vllm:num_requests_running, vllm:num_requests_waiting, vllm:gpu_cache_usage_perc, vllm:time_to_first_token_seconds, vllm:time_per_output_token_seconds). Each one has a defensible green/amber/red threshold, a PromQL query for alerting, and at least one common false reading that confuses the junior operator. The six critical alerts that any production cluster must fire are: HBM > 92 %, thermal or power throttle, XID error, ECC double-bit, KV cache pool > 95 %, and TTFT P95 outside SLO for 5 minutes. The point of having this panel: that the operator on shift diagnoses the origin of a degradation in under five minutes, without opening an SSH console to the GPUs. When that holds, the cluster has moved to professional operation; until then, it is operated on intuition.

You are here: OBSERVE (the other half of tracing)

Tracing, already covered in LLM tracing with OpenTelemetry GenAI, answers what happened in this specific request. Metrics answer what is happening across the cluster in aggregate. They are complementary: an alert on the metrics side tells you “the cluster is degrading”, tracing tells you “and here is the specific trace that proves it”. A cluster without tracing but with metrics operates; a cluster without metrics but with tracing does not operate, it debugs.

The analogy: the cockpit of a modern aircraft

In a modern commercial aircraft, the pilot’s instrument panel has more than 70 active indicators. If there were only one, the altimeter, say, the aircraft would fly into the ground at the first moment of low visibility. You need the altimeter and the attitude indicator, and the airspeed indicator, and the turn indicator, and the fuel gauge, and the oil pressure gauges for each engine, and the turbine exit temperatures. Each answers a different question. And together they cover the operational question: is the aircraft healthy, is it where it should be, and is it going where we want?

Observability for an LLM inference cluster works the same way. A single metric, “GPU utilization 99 %”, answers nothing useful. It is like looking only at the car’s odometer to diagnose why the engine is making a noise. The full cockpit is twelve instruments on the hardware side plus five on the inference engine side, organised into families that answer different questions:

Compute and efficiency: are the tensor cores doing the work we expect, or are they waiting?
Memory: is there VRAM left for new requests, or are we on the edge of OOM?
Thermal and power: is the hardware healthy, or is it silently limiting throughput?
Health and errors: is there hardware degradation under way (ECC, XID, NVLink)?
Inference engine: is the queue growing, is the KV pool saturated, is the SLO being met?

The first four answer “is the GPU fine?”. The fifth answers “is it delivering the service we promised?”. The two questions are different and both must have an answer at a glance.

Why `nvidia-smi` `GPU-Util` misleads with LLMs

The classic metric that appears in nvidia-smi as GPU-Util corresponds to DCGM_FI_DEV_GPU_UTIL. Its official definition: “percentage of time during which one or more kernels were executing on the GPU”. The problem with LLMs: the decode phase is memory-bound, not compute-bound. When the inference engine decodes token by token, the GPU spends 90 % of the time waiting for the HBM to finish delivering the model weights and the KV cache. There is a kernel running (the HBM read), so GPU-Util reports values close to 100 %. But the tensor cores are idle: the bottleneck is memory, not compute.

Practical result: the operator sees “GPU-Util 99 %” in Grafana and assumes “GPU saturated, no more load can go in”. But the reality may be “compute at 25 %, HBM saturated at 95 %”, which changes the operational decisions (quantisation, batch size, parallelism). The classic metric lies by oversimplifying.

The right move is to look at the three DCGM profiling metrics of the _FI_PROF_* subsystem:

DCGM_FI_PROF_SM_OCCUPANCY: ratio of active warps over the maximum per SM. Is there parallel work?
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE: % of cycles with tensor cores effectively active. Is compute working?
DCGM_FI_PROF_DRAM_ACTIVE: % of cycles with HBM transferring. Is memory saturated?

A typical decode-bound GPU running Llama 70B on an H100 shows: SM occupancy 35–55 %, tensor active 15–30 %, DRAM active 80–95 %. That is the real “saturated GPU” for LLMs. The three together tell the regimes apart; none of them alone says anything actionable.

How they are wired up in production

The platform side is covered in Five maturity levels (level 4, the GPU plane) and Seven deployment phases (phase F5). For the observer, the key pieces are:

NVIDIA GPU Operator. Helm manifests that deploy on each GPU node: drivers, container toolkit, MIG manager and DCGM Exporter. The last one exposes /metrics in Prometheus format with all the DCGM_FI_* listed above. It is scraped from the cluster’s internal Prometheus.

Inference engine. vLLM exposes /metrics on port 8000 (default) with vllm:* metrics. SGLang exposes it too, with the sglang: prefix. TensorRT-LLM exposes it through Triton Inference Server with the nv_inference: prefix. The basic naming convention is similar across the three engines; the thresholds and queries in this post assume vLLM, but they translate.

ServiceMonitor / PodMonitor. The Prometheus operator resource that says what to scrape. Minimal example:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
 name: vllm-inference
spec:
 selector:
 matchLabels: { app: vllm }
 podMetricsEndpoints:
 - port: metrics
 interval: 15s

Dashboards. The NVIDIA operator publishes reference Grafana dashboards for DCGM in nvidia/dcgm-exporter (the official repo). vLLM publishes one in vllm-project/vllm (the examples/ folder). Both work as a base; each team adds the panels specific to its own SLO.

The twelve DCGM metrics organised by family

Family 1 — Compute

DCGM_FI_PROF_SM_OCCUPANCY: ratio of active warps per SM over the maximum possible. Value between 0 and 1.

Green: 0.30–0.70 (typical LLM decode regime).
Amber: < 0.20 sustained (batch too small, GPU under-used in parallelism).
Red: 0.95 sustained with low DRAM_ACTIVE (pathological kernel saturating the SMs).

DCGM_FI_PROF_PIPE_TENSOR_ACTIVE: % of cycles with tensor cores executing. The key metric for “is compute producing?”.

Green in prefill: 50–80 %.
Green in decode: 15–30 % (decode is memory-bound, this is not a symptom of a problem).
Red: < 5 % sustained in prefill, or the engine is not using the tensor cores at all (bad config, incompatible format).

DCGM_FI_PROF_DRAM_ACTIVE: % of cycles with HBM transferring data. The key metric for detecting memory saturation.

Green in decode: 60–85 %.
Amber: > 90 % sustained (HBM is a firm bottleneck, which explains high TPOT).
Red: > 95 % sustained with KV cache pool < 70 % (something other than the engine is asking for HBM; investigate leaks).

Family 2 — Memory

DCGM_FI_DEV_FB_USED: Frame Buffer (HBM) used, in MiB.

Green: 70–85 % of total.
Amber: 86–92 %.
Red: > 92 % (risk of OOM on the next paged-attention allocation).

PromQL for the cluster-wide percentage: 100 * sum(DCGM_FI_DEV_FB_USED) / sum(DCGM_FI_DEV_FB_TOTAL).

DCGM_FI_DEV_FB_FREE: free Frame Buffer. Complementary to the previous one; useful for absolute alerts (< 4096 MiB free).

DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL: aggregate NVLink bandwidth in MB/s. For TP (tensor parallel) topologies that cross GPUs over NVLink, this metric reveals whether the parallelism split is saturating the bus.

Green: varies with topology. On 4×H100 SXM with NVLink 4.0, theoretical capacity is 450 GB/s per GPU. A typical TP=4 regime: 50–150 GB/s sustained.
Red: > 90 % of capacity sustained (check whether the model would fit with lower TP or with pipeline parallel).

Family 3 — Thermal and power

DCGM_FI_DEV_GPU_TEMP: die temperature in °C.

Green: < 75 °C.
Amber: 75–82 °C.
Red: > 83 °C (close to the H100’s automatic thermal throttle; check ventilation, air flow, rack inlet temperature).

DCGM_FI_DEV_POWER_USAGE: draw in watts. For the H100 SXM, nominal TDP is 700 W. Useful for three things: spotting an unusually low workload (suspect idle or a stall), billing real energy cost, and firing alerts if the draw approaches the PDU limit.

DCGM_FI_DEV_CLOCK_THROTTLE_REASONS: encoded bitmap with the active throttle reasons. This is the metric that silently explains TPOT degradations.

Relevant bits:

0x0000000000000001: Idle (not a problem).
0x0000000000000002: App clocks setting.
0x0000000000000004: SW Power Cap (software limit, e.g. from nvidia-smi -pl).
0x0000000000000008: HW Slowdown.
0x0000000000000010: Sync Boost (NVIDIA Sync).
0x0000000000000020: SW Thermal Slowdown (software thermal limit).
0x0000000000000040: HW Thermal Slowdown (hardware thermal limit, an emergency).
0x0000000000000080: HW Power Brake Slowdown (PSU voltage drop).
0x0000000000000100: Display Clock Setting.

Any throttle other than Idle with a sustained value > 0 is an alert. TPOT degradation with DRAM_ACTIVE already high and thermal throttle active is the classic “the rack is badly ventilated, it is not the engine’s fault”.

Family 4 — Health

DCGM_FI_DEV_XID_ERRORS: cumulative counter of driver XID errors. XIDs are critical event codes that NVIDIA documents exhaustively (XID 13: graphics engine exception; XID 31: GPU memory page fault; XID 43: reset channel verif error; XID 79: GPU has fallen off the bus; XID 95: uncontained ECC error; and so on). Any increment is an immediate alert: many XIDs require a node reset or an RMA of the GPU.

DCGM_FI_DEV_ECC_DBE_VOL_TOTAL: volatile double-bit ECC errors (uncorrectable). Unlike single-bit ones (which ECC corrects silently and which are counted in DCGM_FI_DEV_ECC_SBE_*), double-bit errors corrupt data. Any value > 0 is a critical alert: the GPU must be drained and inspected.

DCGM_FI_DEV_RETIRED_DBE: physical HBM pages retired because of accumulated double-bit errors. NVIDIA retires faulty pages automatically to prevent future corruption. More than 4–8 retired pages on one GPU suggests silicon degradation: document it and plan a replacement in the next maintenance window.

The five inference engine metrics (vLLM)

The DCGM metrics answer “is the GPU healthy?”. The engine metrics answer “is the service meeting the SLO?”. Without them, you know the hardware works but you do not know whether the clients are happy.

vllm:num_requests_running: requests currently in the batch. If it reaches the configured --max-num-seqs and does not come down, the engine is saturated on concurrency (check VRAM and rebalance through the autoscaler, see Autoscaling LLM on Kubernetes).

vllm:num_requests_waiting: requests queued, not yet in the batch. Any value > 0 sustained over minutes means the cluster is not scaling with the load. This is the primary metric for HPA.

vllm:gpu_cache_usage_perc: % of the KV cache pool in use.

Green: 50–80 %.
Amber: 80–92 %.
Red: > 92 % (risk of preempt-on-OOM: vLLM will drop requests to free memory, which raises TTFT visibly).

vllm:time_to_first_token_seconds: histogram of TTFT per request. Consumed as histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket[5m]))). Compared against the TTFT P95 SLO, it fires the primary service alert.

vllm:time_per_output_token_seconds: histogram of TPOT. Equivalent to the previous one but for streaming smoothness. Compared against the TPOT P95 SLO, it fires the secondary alert.

The six alerts that must page in production

Any serious production cluster fires these six alerts to a channel with an on-call rotation. Without them, the SLO is met by luck, not by process.

groups:
 - name: gpu-llm-critical
 rules:
 - alert: GpuHbmNearOom
 expr: 100 * (DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL) > 92
 for: 2m
 labels: { severity: critical }
 annotations:
 summary: "HBM on {{ $labels.gpu }} at {{ $value }}% — OOM risk"

 - alert: GpuThermalOrPowerThrottle
 expr: (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS != 0) and ignoring(reason) (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS != 1)
 for: 1m
 labels: { severity: warning }
 annotations:
 summary: "GPU {{ $labels.gpu }} throttling (reasons={{ $value }})"

 - alert: GpuXidErrorDetected
 expr: increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0
 labels: { severity: critical }
 annotations:
 summary: "XID error on GPU {{ $labels.gpu }} — investigate immediately"

 - alert: GpuEccDoubleBit
 expr: DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0
 labels: { severity: critical }
 annotations:
 summary: "ECC double-bit on GPU {{ $labels.gpu }} — drain node"

 - alert: VllmKvCachePoolNearFull
 expr: vllm:gpu_cache_usage_perc > 0.95
 for: 3m
 labels: { severity: warning }
 annotations:
 summary: "KV cache pool > 95% on {{ $labels.instance }}"

 - alert: VllmTtftP95OutOfSlo
 expr: histogram_quantile(0.95, sum by(le, instance)(rate(vllm:time_to_first_token_seconds_bucket[5m]))) > 1.5
 for: 5m
 labels: { severity: warning }
 annotations:
 summary: "TTFT P95 over SLO ({{ $value }}s > 1.5s)"

These six cover 80 % of the incidents that affect the SLO. The remaining 20 % demand investigation with tracing (see LLM tracing with OpenTelemetry GenAI).

Master table: thresholds and queries

Metric	Green	Amber	Red	Base query (PromQL)
SM occupancy	0.30–0.70	0.15–0.30	< 0.10 sustained	`DCGM_FI_PROF_SM_OCCUPANCY`
Tensor active (decode)	15–30 %	< 10 %	< 3 %	`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`
DRAM active	60–85 %	85–95 %	> 95 % with low KV	`DCGM_FI_PROF_DRAM_ACTIVE`
FB used	70–85 %	86–92 %	> 92 %	`100 * DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL`
NVLink BW	< 70 % cap	70–90 % cap	> 90 % cap	`DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL`
GPU temp	< 75 °C	75–82 °C	> 83 °C	`DCGM_FI_DEV_GPU_TEMP`
Power usage	< 90 % TDP	90–98 % TDP	> 98 % TDP	`DCGM_FI_DEV_POWER_USAGE`
Throttle reasons	0 or Idle	App/SW	HW Therm/Power	`DCGM_FI_DEV_CLOCK_THROTTLE_REASONS`
XID errors	no change	—	any delta	`increase(DCGM_FI_DEV_XID_ERRORS[5m])`
ECC DBE	0	—	> 0	`DCGM_FI_DEV_ECC_DBE_VOL_TOTAL`
Retired pages	< 4	4–8	> 8	`DCGM_FI_DEV_RETIRED_DBE`
KV cache used	50–80 %	80–92 %	> 92 %	`vllm:gpu_cache_usage_perc`
Requests waiting	0	1–5 sustained	> 10 sustained	`vllm:num_requests_waiting`
TTFT P95	< SLO	80–100 % SLO	> SLO	see alert query above
TPOT P95	< SLO	80–100 % SLO	> SLO	`histogram_quantile(0.95, sum by(le)(rate(vllm:time_per_output_token_seconds_bucket[5m])))`

Three pitfalls that confuse the junior operator

Pitfall 1 — “GPU-Util at 99 % = saturated”. As explained at the start, DCGM_FI_DEV_GPU_UTIL lights up with any kernel. The right move is to look at the three _PROF_* metrics (SM occupancy, tensor active, DRAM active) together. GPU util 99 % + tensor active 8 % + DRAM active 92 % means “saturated by memory, not compute”; GPU util 99 % + tensor active 75 % + DRAM active 50 % means “saturated by compute, prefill heavy”. The two situations call for different levers.

Pitfall 2 — confusing single-bit ECC (SBE) with double-bit (DBE). Single-bit errors are corrected silently and are unavoidable in any HBM under load (cosmic radiation, voltage fluctuations). An SBE counter growing slowly is not an alert, it is physics. DBE is: it corrupts data. Telling the two metrics apart avoids false alarms and false negatives in equal measure.

Pitfall 3 — alerting on num_requests_waiting > 0 without context. An instantaneous value of 1 or 2 during a spike is normal. What matters is a sustained queue: use for: 5m with a threshold of 3–5. Without that window, the system floods the alert channel with noise.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB with intra-node NVLink:

DCGM Exporter deployed through the NVIDIA GPU Operator, one DaemonSet per GPU node.
Internal Prometheus with 30-day retention for high-frequency metrics, 1 year for downsampled ones (Thanos/Mimir if the volume justifies it).
Grafana with three standard dashboards: GPU hardware (DCGM), engine (vLLM), SLO (TTFT/TPOT/RPS against written targets).
Alertmanager with an on-call rotation and rate limiting through silences grouped by node.
Cardinality kept under control: gpu (local id), node, pod, model. Do not add request_id or other high-cardinality labels to metrics; that is tracing’s job.

Estimated volume for a 16-GPU cluster scraped every 15 s: around 2 million samples/min, around 25 GB/day of raw Prometheus. Manageable with one Prometheus per cluster plus retention; if the team scales beyond 64 GPUs, consider a Thanos sidecar or VictoriaMetrics. See OSS LLMOps tooling catalogue for equivalent alternatives.

What we have not covered (upcoming articles)

Tracing of LLM workloads: already covered in LLM tracing with OpenTelemetry GenAI.
Autoscaling based on these metrics: see Autoscaling LLM on Kubernetes.
Incident response runbooks: how each of these alerts translates into concrete action (drain, restart, RMA, scaling, rollback).
Cost accounting: using DCGM_FI_DEV_POWER_USAGE and vllm:request_success_total for per-tenant cost showback.
Multi-tenant fairness monitoring: when several tenants share a cluster, which metrics detect that one of them is hogging the KV cache.

References

NVIDIA — DCGM Exporter (repo nvidia/dcgm-exporter, documented metrics and units).
NVIDIA — DCGM Field Identifiers reference (complete list of DCGM_FI_*).
NVIDIA — XID Errors documentation (catalogue of XID codes and remediation procedures).
NVIDIA — NVIDIA GPU Operator (official Helm chart).
vLLM project — examples/production_monitoring/ (reference PromQL and Grafana dashboards).
Prometheus — Histogram and summary best practices (for building defensible percentile queries).
NVIDIA — H100 Tensor Core GPU datasheet (TDP, HBM bandwidth, NVLink capacities).

Dcgm on lo0 — Tech Blog

Measuring the power of a GPU: NVML, DCGM and the sampling errors that invalidate your watts

TL;DR

What the hardware exposes

NVML: power

NVML: the energy counter

NVML: power limits

DCGM: the equivalent fields

The real accuracy of the sensor

Update period and averaging window

The steady-state error

The response to transients

The two ways of obtaining energy

Why 1 Hz destroys a 200 ms inference

Aliasing with periodic loads

What the GPU counter does not measure

The tax of keeping a context open

CPU and DRAM: RAPL

The software stack

Procedure for making a Wh/token auditable

Subtracting idle, or not

Power capping: what it costs and what it saves

Reference figures

Cross-links from the energy track

See also

Sources

The reproducible harness: measuring cost, performance and energy in a single auditable experiment

TL;DR

Why the three axes must be measured together

Architecture of the integrated bench

The Kubernetes Job: the full YAML

Performance layer: GuideLLM and AIPerf

GuideLLM — the SLO-driven sweep

AIPerf — NVIDIA’s concurrency profiler

Cost layer: OpenCost and LiteLLM

OpenCost — the denominator in euros per GPU-hour

LiteLLM — the per-request token counter

Energy layer: DCGM and Kepler

DCGM Exporter — GPU power and energy

Kepler — pod-level energy

The MLPerf Power reference

Unifying the metrics: the PromQL queries

The 3-axis scorecard: table and interpretation

The multi-objective Pareto frontier

Reproducible design: the twelve mandatory metadata

Idempotence and versioning of results

How it connects with the other posts in the series

Full flow of a benchmarking session

Cost of the experiment

What the harness does not measure (and why)

Sources

GPU idle: the cost that shows up on no invoice but is paid by the whole TCO

TL;DR

1 · The fundamental identity

2 · Why the standard metric lies: DCGM_FI_DEV_GPU_UTIL

Table of relevant DCGM fields

Characteristic reading in LLM decode (Llama 70B FP8 on H100)

3 · MFU and HFU: occupancy expressed in FLOPs

Typical MFU values

The Roofline model

4 · TCO sensitivity to occupancy

5 · Idle metrics: where occupancy is lost

6 · Levers for raising occupancy

Table of lever × effect × when it applies

Bin-packing with Kueue

MIG as a bin-packing lever inside the GPU

7 · Continuous batching: the effect on MFU

8 · Hardware and scale: what applies to what

9 · FinOps diagnostic flow: from high CPM to the cause

10 · Worked example: impact of continuous batching on CPM

See also

Sources

Measuring energy in production: Kepler, DCGM and the practical stack (accuracy and overhead)

What this article covers

The measurement stack on Kubernetes

DCGM exporter: the base of GPU power

Kepler in depth: eBPF and the sources

Deployment: a DaemonSet on top of Prometheus

PromQL queries: energy per pod

A calibration procedure

2 · Why the standard metric lies: `DCGM_FI_DEV_GPU_UTIL`