GPU idle: the cost that shows up on no invoice but is paid by the whole TCO

Mon, 15 Jun 2026 05:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

The cost per token of on-premise inference is $ \text{€/GPU-hour} \div \text{throughput} $. Throughput is a direct function of the GPU’s useful occupancy. On a generic node of 4×H100 SXM with a reference cost of ~11 € per GPU-hour (depreciation plus energy plus infrastructure), the cost curve over occupancy looks like this:

Useful occupancy	Effective throughput (tok/s)	Cost per 1M tokens
20 %	~700	~43 €
40 %	~1,400	~21 €
70 %	~2,500	~12 €
85 % (practical ceiling)	~3,000	~10 €

Doubling occupancy from 20 % to 40 % halves the cost per token, without buying more iron. At 70 % the cost competes with European cloud providers (~2.2 €/GPU-hour on-demand at Scaleway). At 20 % on-prem is four times dearer than renting. The lever is not the model, nor the precision: it is how many paid-for GPU-hours produce serving tokens.

1 · The fundamental identity

The cost per million tokens (CPM) in your own inference is not a list price. It is:

$$ \text{CPM} = \frac{C_{\text{GPU}} \cdot N_{\text{GPU}}}{T_{\text{ef}} \times 3600 / 10^6} $$

where $ C_{\text{GPU}} $ is the cost per GPU-hour (€/h), $ N_{\text{GPU}} $ the number of GPUs assigned to the service and $ T_{\text{ef}} $ the effective throughput (tok/s). Making the dependence on occupancy explicit:

$$ T_{\text{ef}} = T_{\text{pico}} \times \rho $$

where $ \rho \in [0,1] $ is the useful occupancy rate (the fraction of time in which the GPU is processing real serving tokens). The resulting identity:

$$ \text{CPM} = \frac{C_{\text{GPU}} \cdot N_{\text{GPU}}}{T_{\text{pico}} \times \rho \times 3600 / 10^6} $$

The direct consequence: CPM is inversely proportional to $ \rho $. Doubling $ \rho $ halves CPM. The numerator (cost of the iron) does not change.

The posts coste-por-token-y-por-request and capacity-planning-inferencia-llm-on-premise cover how to calculate $ C_{\text{GPU}} $ and the reference peak throughput. This article deals with $ \rho $: how to measure it, what limits it and how to raise it.

2 · Why the standard metric lies: `DCGM_FI_DEV_GPU_UTIL`

The field DCGM_FI_DEV_GPU_UTIL (field ID 203) appears in nvidia-smi as “GPU-Util”. Its official definition in the DCGM documentation:

“GPU Utilization” — percentage of time during which one or more kernels were executing on the GPU.

The problem for LLM inference: the decode phase is memory-bound. The GPU runs a kernel reading weights from HBM token by token; therefore GPU_UTIL registers close to 100 % even though the tensor cores are at 15 % of their capacity. The field measures activity, not useful work.

The distinction is critical for FinOps: an operator who sees GPU_UTIL 98 % assumes “saturated GPU, no room for more load”. The reality may be “the tensor cores are at 20 % and the bottleneck is HBM”, which leaves room for continuous batching or additional bin-packing.

The metrics that measure real occupancy are those of the _FI_PROF_* subsystem, available in DCGM 3.x with the profiling module enabled:

Table of relevant DCGM fields

DCGM field	Field ID	What it measures	Unit	Operational note
`DCGM_FI_DEV_GPU_UTIL`	203	% time with ≥1 active kernel	%	Misleading in LLM decode
`DCGM_FI_PROF_SM_ACTIVE`	1002	Ratio of cycles with ≥1 active warp per SM	0–1	Compute activity, not occupancy
`DCGM_FI_PROF_SM_OCCUPANCY`	1003	Resident warps / theoretical maximum per SM	0–1	Real intra-SM parallelism
`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`	1004	% cycles with tensor cores (HMMA) active	0–1	The real compute efficiency metric
`DCGM_FI_PROF_DRAM_ACTIVE`	1005	% cycles with HBM transferring	0–1	Memory saturation
`DCGM_FI_PROF_PCIE_TX_BYTES`	—	TX bytes over PCIe	bytes/s	Useful in PCIe inference
`DCGM_FI_DEV_FB_USED`	252	HBM used	MiB	VRAM budget
`DCGM_FI_DEV_POWER_USAGE`	—	Real consumption	W	For real energy cost
`DCGM_FI_DEV_CLOCK_THROTTLE_REASONS`	—	Bitmap of throttle causes	bitmap	Detects silent degradation

The _PROF_* fields require the DCGM profiling module and the right driver permissions. They are documented exhaustively in the NVIDIA DCGM field ID reference and in the sibling post observabilidad-gpu-dcgm-llm.

Characteristic reading in LLM decode (Llama 70B FP8 on H100)

Metric	Typical decode value	Interpretation
`DCGM_FI_DEV_GPU_UTIL`	95–99 %	A lie: there is an active kernel
`DCGM_FI_PROF_SM_ACTIVE`	0.45–0.65	SMs with warps 45–65 % of the time
`DCGM_FI_PROF_SM_OCCUPANCY`	0.30–0.55	Resident warps at 30–55 % of the maximum
`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`	0.10–0.25	Tensor cores active only 10–25 %
`DCGM_FI_PROF_DRAM_ACTIVE`	0.75–0.90	HBM busy 75–90 % of the time

Reading: decode is memory-bound. The model weights are loaded from HBM for each token; HBM is saturated but the tensor cores wait. All of the “utilisation” in nvidia-smi comes from memory reads, not from compute.

3 · MFU and HFU: occupancy expressed in FLOPs

The canonical compute efficiency metric is MFU (Model FLOPs Utilization), defined in the PaLM paper (Chowdhery et al., arXiv 2204.02311, section 4):

$$ \text{MFU} = \frac{T_{\text{obs}} \times C_{\text{modelo}}}{P_{\text{pico}}} $$

where $ T_{\text{obs}} $ is the observed throughput (tok/s) and $ P_{\text{pico}} $ is the theoretical peak performance of the hardware (FLOP/s).

where $ C_{\text{modelo}} $ is the number of FLOPs per token in a complete forward pass. For a dense transformer with $ P $ parameters, the usual approximation (forward plus backward = 6P FLOPs per token; forward only = 2P):

$$ C_{\text{modelo}} \approx 2P \quad \text{(inference)} $$ $$ C_{\text{modelo}} \approx 6P \quad \text{(training, forward + backward)} $$

HFU (Hardware FLOPs Utilization) measures the FLOPs actually executed in hardware, including activation recomputation (gradient checkpointing). In training with recompute:

$$ C_{\text{hardware}} \approx 8P \quad \text{(forward × 2 + backward × 4)} $$

therefore HFU > MFU when there is recompute; they are identical without it.

Typical MFU values

Regime	Hardware	Typical MFU	Limiting regime
Training (large batch, BF16)	H100 SXM	35–50 %	compute-bound
Training PaLM 540B	TPU v4	46.2 %	compute-bound
Prefill inference (large batch)	H100 SXM	25–45 %	compute-bound
Decode inference (bs=1)	H100 SXM	3–8 %	memory-bound
Decode inference (continuous batching, bs=32–64)	H100 SXM	15–30 %	memory-bound, attenuated

Decode with batch size 1 has a single-digit MFU because the hardware spends most of its time waiting for HBM to deliver weights. Raising the batch size (continuous batching) amortises the weight read across more simultaneous tokens and raises MFU.

The Roofline model

The roofline places each operation in the space (arithmetic intensity, throughput):

Decode with batch size 1 falls in the memory-bound zone to the left of the ridge point. Raising the batch (continuous batching) moves the point to the right and up, closer to the roofline.

4 · TCO sensitivity to occupancy

Assumptions for the example: a generic node with 4×H100 SXM 80 GB, total node cost ~44 €/hour (5-year depreciation of ~220,000 €, energy ~4×700 W at ~0.12 €/kWh, plus rack/colocation infrastructure; see coste-por-token-y-por-request for the detail of the identity). Reference peak throughput with Llama 70B FP8 plus continuous batching: ~3,500 tok/s aggregate.

$$ \text{CPM}(\rho) = \frac{44\ \text{€/h}}{3500 \times \rho \times 3600 / 10^6} = \frac{44 \times 10^6}{3500 \times \rho \times 3600} = \frac{3.49}{\rho}\ \text{€/1M tok} $$

Occupancy $ \rho $	Effective throughput (tok/s)	On-prem CPM	On-demand cloud CPM (~2.2 €/GPU-h)
20 %	700	~17.5 €	~6.3 €
40 %	1,400	~8.7 €	~6.3 €
60 %	2,100	~5.8 €	~6.3 €
70 %	2,450	~5.0 €	~6.3 €
85 %	2,975	~4.1 €	~6.3 €

On-prem / cloud crossover point: with this example hardware, the on-prem cost advantage over a comparable European cloud starts at $ \rho \approx 55\text{-}60\,\% $. Below that, idle makes on-prem dearer than renting. The cloud column is flat because cloud bills per hour used, not per the throughput you extract from it.

Example figures with generic hardware. The real numbers depend on the depreciation price, the local cost of energy, the model and the precision. The structure of the curve, CPM inversely proportional to $ \rho $, is universal.

5 · Idle metrics: where occupancy is lost

Before applying levers, you need to know which kind of idle dominates. Three categories:

Type of idle	Symptom in metrics	Usual cause
Scheduling idle	`DCGM_FI_DEV_POWER_USAGE` low, `SM_ACTIVE` < 0.05	No requests in the queue; GPU waiting for work
Batching idle	`SM_ACTIVE` high, `PIPE_TENSOR_ACTIVE` low, `DRAM_ACTIVE` low	Batch too small; prefill stall between requests
Memory-bound idle	`DRAM_ACTIVE` high, `PIPE_TENSOR_ACTIVE` low	Normal decode; HBM is the bottleneck; insufficient batch size

Scheduling idle is the most expensive and the most directly attacked with bin-packing. Batching idle is attacked with continuous batching. Memory-bound idle in decode does not disappear entirely (it is the physics of the transformer), but it is attenuated with a larger batch size.

The inference engine metrics that complement the diagnosis from the service side are documented in observabilidad-gpu-dcgm-llm and anatomia-metricas-dcgm-vllm-anomalias.

6 · Levers for raising occupancy

Table of lever × effect × when it applies

Lever	Effect on $ \rho $	When it applies	Operational complexity
Continuous batching	High: removes idle between requests; raises decode MFU from ~5 % to ~20 %	Always in inference; enabled by default in vLLM	Low (engine parameter)
Scheduler bin-packing (kube-scheduler / Kueue)	High: concentrates loads on fewer nodes; frees whole nodes for shutdown or rebalancing	Clusters with load that varies over the day	Medium (scheduler policy)
MIG (Multi-Instance GPU)	Medium: fills GPUs with light loads that previously lived alone on a whole GPU	Heterogeneous loads: embeddings + reranker + guardrail + large model	High (hot repartitioning not available)
Time-slicing	Low to medium: raises occupancy in dev/bursts; no isolation	Consumer GPUs (RTX 5090/4090); low-risk multi-tenant dev	Low
MPS	Medium: concurrent execution of multiple small processes; reduces context-switch overhead	Many small concurrent kernels on a datacenter GPU; trust between loads	Medium
Chunked prefill	Medium: interleaves prefill and decode; reduces the TTFT spike and raises throughput	Loads with a mix of short and long prompts	Low (vLLM flag)
Quantisation (FP16→FP8→INT4)	Indirect: raises peak throughput, which lowers CPM for the same $ \rho $	Models with quantised kernel support (native Hopper FP8)	Medium
Autoscaling (KEDA)	Keeps $ \rho $ high by scaling replicas according to the queue	Variable and predictable load; cluster with spare capacity	Medium

The post compartir-gpu-time-slicing-mps-mig details the three sharing mechanisms (time-slicing, MPS, MIG) with VRAM budgets.

Bin-packing with Kueue

Kueue (sigs.k8s.io/kueue) is the Kubernetes-native queue manager for GPU jobs. Its model of cohorts and nominal quotas allows active bin-packing: jobs accumulate in a queue and are launched only when there is a node that can take them whole, instead of fragmenting the load across partially occupied nodes.

BestFit packing in the ClusterQueue is configured with:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
 name: gpu-prod
spec:
 preemption:
 reclaimWithinCohort: Any
 withinClusterQueue: LowerPriority
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 16

The priority-based preemption policy ensures that production jobs displace experimentation ones when there is scarcity, keeping occupancy on the production nodes high.

MIG as a bin-packing lever inside the GPU

MIG makes it possible to fill an H100 with light loads that would otherwise live alone on a whole GPU. A profile of 3×2g.20gb + 1×1g.10gb on an H100 can host simultaneously a 7B FP8 model (~14 GB of weights), two embedding services and an INT4 guardrail, all with hardware isolation. Without MIG, each of those loads would take a whole GPU with an individual $ \rho $ below 10 %.

The profiles available on the H100 80 GB (SXM5) according to NVIDIA’s MIG User Guide:

MIG profile	Compute slices	Memory	Max. instances
`1g.10gb`	1/7 SMs	10 GB	7
`1g.20gb`	1/7 SMs	20 GB	4
`2g.20gb`	2/7 SMs	20 GB	3
`3g.40gb`	3/7 SMs	40 GB	2
`4g.40gb`	4/7 SMs	40 GB	1
`7g.80gb`	7/7 SMs	80 GB	1 (whole GPU)

MIG is not available on consumer GPUs (RTX 5090, RTX 4090). Time-slicing is the only sharing option on that hardware.

7 · Continuous batching: the effect on MFU

Continuous batching (also called iteration-level scheduling or in-flight batching) is the mechanism with the greatest impact on occupancy in inference. The idea: instead of waiting for a complete batch to finish before launching the next, the engine evaluates the pipeline after each token and replaces completed sequences with new requests from the waiting queue.

Quantified effect in vLLM:

Without batching (bs=1, static): decode MFU ~3–8 %; GPU idle between requests.
With continuous batching (dynamic bs 16–64): decode MFU ~15–30 %; the GPU almost never waits.
On pure prefill loads with a large batch (bs=128+): prefill MFU 25–45 %; it approaches the compute roofline.

The vLLM parameter --max-num-seqs controls the maximum number of sequences in the concurrent batch. Raising it increases occupancy until HBM becomes the bottleneck (watch for DCGM_FI_PROF_DRAM_ACTIVE > 90 % sustained).

8 · Hardware and scale: what applies to what

Hardware	MIG	MPS	Time-slicing	Continuous batching	Note
H100 SXM / H200	Yes (7 inst.)	Yes	Yes	Yes	On-prem datacenter reference
A100 SXM/PCIe	Yes (7 inst.)	Yes	Yes	Yes	Previous generation; HBM2e
L40S / L40	No	Yes	Yes	Yes	Ada Lovelace; no MIG; good price/VRAM
RTX 5090	No	Yes (limited)	Yes	Yes	Consumer; no MIG; does not scale in production
RTX 4090	No	Yes (limited)	Yes	Yes	Consumer; 24 GB VRAM; no MIG

The RTX 5090 and RTX 4090 illustrate the case of hardware that does not scale for multi-tenant with isolation: they do not support MIG, VRAM is scarce for models above 7B with a wide KV-cache, and TDP (600 W / 450 W) is high relative to throughput. For production inference at scale, the 4×H100 SXM is the reference node of this series.

9 · FinOps diagnostic flow: from high CPM to the cause

10 · Worked example: impact of continuous batching on CPM

Starting point: Llama 70B FP8 on 4×H100 SXM, a load of 8 requests/s with 512 output tokens on average, without continuous batching (static bs = 8):

Observed throughput: ~900 tok/s (decode dominant).
$ \rho_{\text{efectiva}} \approx 900 / 3500 \approx 0.26 $.
CPM: $ 44 / (900 \times 3600 / 10^6) \approx 13.6\ \text{€} $.

Same load, enabling continuous batching with --max-num-seqs 64:

Observed throughput: ~2,300 tok/s (the dynamic batch fills the gaps).
$ \rho_{\text{efectiva}} \approx 2300 / 3500 \approx 0.66 $.
CPM: $ 44 / (2300 \times 3600 / 10^6) \approx 5.3\ \text{€} $.

CPM reduction: from 13.6 € to 5.3 €, –61 %, with no hardware change and without touching the model.

The vLLM parameters relevant for raising effective throughput:

--max-num-seqs 64 # maximum concurrent batch
--max-num-batched-tokens 16384 # total tokens per iteration
--enable-chunked-prefill # interleaves prefill and decode

Sources

NVIDIA — DCGM Field Identifiers reference (v3.1), lista completa de DCGM_FI_* con field IDs y definiciones. https://docs.nvidia.com/datacenter/dcgm/3.1/dcgm-api/dcgm-api-field-ids.html
NVIDIA — GPU Profiling Metrics (Run:ai / DCGM), definiciones de DCGM_FI_PROF_SM_ACTIVE (1002), SM_OCCUPANCY (1003), PIPE_TENSOR_ACTIVE (1004). https://run-ai-docs.nvidia.com/self-hosted/platform-management/monitor-performance/gpu-profiling-metrics
Chowdhery et al. — PaLM: Scaling Language Modeling with Pathways (MFU definition, sección 4). arXiv 2204.02311. https://arxiv.org/abs/2204.02311
NVIDIA — Multi-Instance GPU (MIG) User Guide (perfiles H100 SXM5, particionado, aislamiento). https://docs.nvidia.com/datacenter/tesla/mig-user-guide/
NVIDIA — Supported MIG Profiles (catálogo completo H100 80 GB). https://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-mig-profiles.html
NVIDIA — Time-Slicing GPUs in Kubernetes (GPU Operator 24.9.0). https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/24.9.0/gpu-sharing.html
NVIDIA — Multi-Process Service (MPS) Overview (kernel concurrency, SM allocation). https://docs.nvidia.com/deploy/mps/latest/index.html
Kueue — Overview (sigs.k8s.io/kueue, bin-packing, cohorts, quotas). https://kueue.sigs.k8s.io/docs/overview/
vLLM Blog — vLLM v0.6.0: 2.7x Throughput Improvement and 5x Latency Reduction (continuous batching, chunked prefill). https://blog.vllm.ai/2024/09/05/perf-update.html
GMI Cloud — NVIDIA H100 GPU Pricing: 2026 Rent vs. Buy Cost Analysis. https://www.gmicloud.ai/en/blog/nvidia-h100-gpu-pricing-2026-rent-vs-buy-cost-analysis
Saxena et al. — LLM Inference Unveiled: Survey and Roofline Model Insights. arXiv 2402.16363. https://arxiv.org/abs/2402.16363

Idle on lo0 — Tech Blog

GPU idle: the cost that shows up on no invoice but is paid by the whole TCO

TL;DR

1 · The fundamental identity

2 · Why the standard metric lies: `DCGM_FI_DEV_GPU_UTIL`

Table of relevant DCGM fields

Characteristic reading in LLM decode (Llama 70B FP8 on H100)

3 · MFU and HFU: occupancy expressed in FLOPs

Typical MFU values

The Roofline model

4 · TCO sensitivity to occupancy

5 · Idle metrics: where occupancy is lost

6 · Levers for raising occupancy

Table of lever × effect × when it applies

Bin-packing with Kueue

MIG as a bin-packing lever inside the GPU

7 · Continuous batching: the effect on MFU

8 · Hardware and scale: what applies to what

9 · FinOps diagnostic flow: from high CPM to the cause

10 · Worked example: impact of continuous batching on CPM

See also

Sources

Idle on lo0 — Tech Blog

GPU idle: the cost that shows up on no invoice but is paid by the whole TCO

TL;DR

1 · The fundamental identity

2 · Why the standard metric lies: DCGM_FI_DEV_GPU_UTIL

Table of relevant DCGM fields

Characteristic reading in LLM decode (Llama 70B FP8 on H100)

3 · MFU and HFU: occupancy expressed in FLOPs

Typical MFU values

The Roofline model

4 · TCO sensitivity to occupancy

5 · Idle metrics: where occupancy is lost

6 · Levers for raising occupancy

Table of lever × effect × when it applies

Bin-packing with Kueue

MIG as a bin-packing lever inside the GPU

7 · Continuous batching: the effect on MFU

8 · Hardware and scale: what applies to what

9 · FinOps diagnostic flow: from high CPM to the cause

10 · Worked example: impact of continuous batching on CPM

See also

Sources

2 · Why the standard metric lies: `DCGM_FI_DEV_GPU_UTIL`