GPU idle: the cost that shows up on no invoice but is paid by the whole TCO

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

The cost per token of on-premise inference is \( \text{€/GPU-hour} \div \text{throughput} \). Throughput is a direct function of the GPU’s useful occupancy. On a generic node of 4×H100 SXM with a reference cost of ~11 € per GPU-hour (depreciation plus energy plus infrastructure), the cost curve over occupancy looks like this:

Doubling occupancy from 20 % to 40 % halves the cost per token, without buying more iron. At 70 % the cost competes with European cloud providers (~2.2 €/GPU-hour on-demand at Scaleway). At 20 % on-prem is four times dearer than renting. The lever is not the model, nor the precision: it is how many paid-for GPU-hours produce serving tokens.

1 · The fundamental identity

The cost per million tokens (CPM) in your own inference is not a list price. It is:

where \( C_{\text{GPU}} \) is the cost per GPU-hour (€/h), \( N_{\text{GPU}} \) the number of GPUs assigned to the service and \( T_{\text{ef}} \) the effective throughput (tok/s). Making the dependence on occupancy explicit:

where \( \rho \in [0,1] \) is the useful occupancy rate (the fraction of time in which the GPU is processing real serving tokens). The resulting identity:

The direct consequence: CPM is inversely proportional to \( \rho \). Doubling \( \rho \) halves CPM. The numerator (cost of the iron) does not change.

The posts coste-por-token-y-por-request and capacity-planning-inferencia-llm-on-premise cover how to calculate \( C_{\text{GPU}} \) and the reference peak throughput. This article deals with \( \rho \): how to measure it, what limits it and how to raise it.

2 · Why the standard metric lies: DCGM_FI_DEV_GPU_UTIL

The field DCGM_FI_DEV_GPU_UTIL (field ID 203) appears in nvidia-smi as “GPU-Util”. Its official definition in the DCGM documentation:

“GPU Utilization” — percentage of time during which one or more kernels were executing on the GPU.

The problem for LLM inference: the decode phase is memory-bound. The GPU runs a kernel reading weights from HBM token by token; therefore GPU_UTIL registers close to 100 % even though the tensor cores are at 15 % of their capacity. The field measures activity, not useful work.

The distinction is critical for FinOps: an operator who sees GPU_UTIL 98 % assumes “saturated GPU, no room for more load”. The reality may be “the tensor cores are at 20 % and the bottleneck is HBM”, which leaves room for continuous batching or additional bin-packing.

The metrics that measure real occupancy are those of the _FI_PROF_* subsystem, available in DCGM 3.x with the profiling module enabled:

Table of relevant DCGM fields

The _PROF_* fields require the DCGM profiling module and the right driver permissions. They are documented exhaustively in the NVIDIA DCGM field ID reference and in the sibling post observabilidad-gpu-dcgm-llm.

Characteristic reading in LLM decode (Llama 70B FP8 on H100)

Reading: decode is memory-bound. The model weights are loaded from HBM for each token; HBM is saturated but the tensor cores wait. All of the “utilisation” in nvidia-smi comes from memory reads, not from compute.

3 · MFU and HFU: occupancy expressed in FLOPs

The canonical compute efficiency metric is MFU (Model FLOPs Utilization), defined in the PaLM paper (Chowdhery et al., arXiv 2204.02311, section 4):

where \( T_{\text{obs}} \) is the observed throughput (tok/s) and \( P_{\text{pico}} \) is the theoretical peak performance of the hardware (FLOP/s).

where \( C_{\text{modelo}} \) is the number of FLOPs per token in a complete forward pass. For a dense transformer with \( P \) parameters, the usual approximation (forward plus backward = 6P FLOPs per token; forward only = 2P):

HFU (Hardware FLOPs Utilization) measures the FLOPs actually executed in hardware, including activation recomputation (gradient checkpointing). In training with recompute:

therefore HFU > MFU when there is recompute; they are identical without it.

Typical MFU values

Decode with batch size 1 has a single-digit MFU because the hardware spends most of its time waiting for HBM to deliver weights. Raising the batch size (continuous batching) amortises the weight read across more simultaneous tokens and raises MFU.

The Roofline model

The roofline places each operation in the space (arithmetic intensity, throughput):

Decode with batch size 1 falls in the memory-bound zone to the left of the ridge point. Raising the batch (continuous batching) moves the point to the right and up, closer to the roofline.

4 · TCO sensitivity to occupancy

Assumptions for the example: a generic node with 4×H100 SXM 80 GB, total node cost ~44 €/hour (5-year depreciation of ~220,000 €, energy ~4×700 W at ~0.12 €/kWh, plus rack/colocation infrastructure; see coste-por-token-y-por-request for the detail of the identity). Reference peak throughput with Llama 70B FP8 plus continuous batching: ~3,500 tok/s aggregate.

On-prem / cloud crossover point: with this example hardware, the on-prem cost advantage over a comparable European cloud starts at \( \rho \approx 55\text{-}60\,\% \). Below that, idle makes on-prem dearer than renting. The cloud column is flat because cloud bills per hour used, not per the throughput you extract from it.

Example figures with generic hardware. The real numbers depend on the depreciation price, the local cost of energy, the model and the precision. The structure of the curve, CPM inversely proportional to \( \rho \), is universal.

5 · Idle metrics: where occupancy is lost

Before applying levers, you need to know which kind of idle dominates. Three categories:

Scheduling idle is the most expensive and the most directly attacked with bin-packing. Batching idle is attacked with continuous batching. Memory-bound idle in decode does not disappear entirely (it is the physics of the transformer), but it is attenuated with a larger batch size.

The inference engine metrics that complement the diagnosis from the service side are documented in observabilidad-gpu-dcgm-llm and anatomia-metricas-dcgm-vllm-anomalias.

6 · Levers for raising occupancy

Table of lever × effect × when it applies

The post compartir-gpu-time-slicing-mps-mig details the three sharing mechanisms (time-slicing, MPS, MIG) with VRAM budgets.

Bin-packing with Kueue

Kueue (sigs.k8s.io/kueue) is the Kubernetes-native queue manager for GPU jobs. Its model of cohorts and nominal quotas allows active bin-packing: jobs accumulate in a queue and are launched only when there is a node that can take them whole, instead of fragmenting the load across partially occupied nodes.

BestFit packing in the ClusterQueue is configured with:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: gpu-prod
spec:
  preemption:
    reclaimWithinCohort: Any
    withinClusterQueue: LowerPriority
  resourceGroups:
    - coveredResources: ["nvidia.com/gpu"]
      flavors:
        - name: h100-sxm
          resources:
            - name: "nvidia.com/gpu"
              nominalQuota: 16

The priority-based preemption policy ensures that production jobs displace experimentation ones when there is scarcity, keeping occupancy on the production nodes high.

MIG as a bin-packing lever inside the GPU

MIG makes it possible to fill an H100 with light loads that would otherwise live alone on a whole GPU. A profile of 3×2g.20gb + 1×1g.10gb on an H100 can host simultaneously a 7B FP8 model (~14 GB of weights), two embedding services and an INT4 guardrail, all with hardware isolation. Without MIG, each of those loads would take a whole GPU with an individual \( \rho \) below 10 %.

The profiles available on the H100 80 GB (SXM5) according to NVIDIA’s MIG User Guide:

MIG is not available on consumer GPUs (RTX 5090, RTX 4090). Time-slicing is the only sharing option on that hardware.

7 · Continuous batching: the effect on MFU

Continuous batching (also called iteration-level scheduling or in-flight batching) is the mechanism with the greatest impact on occupancy in inference. The idea: instead of waiting for a complete batch to finish before launching the next, the engine evaluates the pipeline after each token and replaces completed sequences with new requests from the waiting queue.

Quantified effect in vLLM:

• Without batching (bs=1, static): decode MFU ~3–8 %; GPU idle between requests.
• With continuous batching (dynamic bs 16–64): decode MFU ~15–30 %; the GPU almost never waits.
• On pure prefill loads with a large batch (bs=128+): prefill MFU 25–45 %; it approaches the compute roofline.

The vLLM parameter --max-num-seqs controls the maximum number of sequences in the concurrent batch. Raising it increases occupancy until HBM becomes the bottleneck (watch for DCGM_FI_PROF_DRAM_ACTIVE > 90 % sustained).

8 · Hardware and scale: what applies to what

The RTX 5090 and RTX 4090 illustrate the case of hardware that does not scale for multi-tenant with isolation: they do not support MIG, VRAM is scarce for models above 7B with a wide KV-cache, and TDP (600 W / 450 W) is high relative to throughput. For production inference at scale, the 4×H100 SXM is the reference node of this series.

9 · FinOps diagnostic flow: from high CPM to the cause

10 · Worked example: impact of continuous batching on CPM

Starting point: Llama 70B FP8 on 4×H100 SXM, a load of 8 requests/s with 512 output tokens on average, without continuous batching (static bs = 8):

• Observed throughput: ~900 tok/s (decode dominant).
• \( \rho_{\text{efectiva}} \approx 900 / 3500 \approx 0.26 \).
• CPM: \( 44 / (900 \times 3600 / 10^6) \approx 13.6\ \text{€} \).

Same load, enabling continuous batching with --max-num-seqs 64:

• Observed throughput: ~2,300 tok/s (the dynamic batch fills the gaps).
• \( \rho_{\text{efectiva}} \approx 2300 / 3500 \approx 0.66 \).
• CPM: \( 44 / (2300 \times 3600 / 10^6) \approx 5.3\ \text{€} \).

CPM reduction: from 13.6 € to 5.3 €, –61 %, with no hardware change and without touching the model.

The vLLM parameters relevant for raising effective throughput:

--max-num-seqs 64            # maximum concurrent batch
--max-num-batched-tokens 16384  # total tokens per iteration
--enable-chunked-prefill      # interleaves prefill and decode

See also

• coste-por-token-y-por-request — the identity CPM = €/GPU-hour ÷ throughput with data from LiteLLM and OpenCost.
• observabilidad-gpu-dcgm-llm — the twelve DCGM metrics and five vLLM ones that make up the cockpit.
• anatomia-metricas-dcgm-vllm-anomalias — a deep dive into the DCGM metrics with anomalies documented in production.
• compartir-gpu-time-slicing-mps-mig — time-slicing, MPS and MIG with worked VRAM budgets.
• capacity-planning-inferencia-llm-on-premise — sizing the cluster from peak throughput and target occupancy.
• opencost-cost-allocation-kubernetes — how OpenCost calculates the \( C_{\text{GPU}} \) that goes into the CPM identity.
• Cloud GPU: a comparison of prices, commitment and sovereign neoclouds — the alternative €/GPU-hour when on-premise occupancy does not justify the CAPEX: spot and reserved prices from European neoclouds.
• TCO of an on-premise GPU cluster: depreciation, energy and infrastructure — where the \( C_{\text{GPU}} \) of the identity comes from when the iron is your own: CAPEX, depreciation, energy and operations.

Sources

• NVIDIA — DCGM Field Identifiers reference (v3.1), lista completa de DCGM_FI_* con field IDs y definiciones. https://docs.nvidia.com/datacenter/dcgm/3.1/dcgm-api/dcgm-api-field-ids.html
• NVIDIA — GPU Profiling Metrics (Run:ai / DCGM), definiciones de DCGM_FI_PROF_SM_ACTIVE (1002), SM_OCCUPANCY (1003), PIPE_TENSOR_ACTIVE (1004). https://run-ai-docs.nvidia.com/self-hosted/platform-management/monitor-performance/gpu-profiling-metrics
• Chowdhery et al. — PaLM: Scaling Language Modeling with Pathways (MFU definition, sección 4). arXiv 2204.02311. https://arxiv.org/abs/2204.02311
• NVIDIA — Multi-Instance GPU (MIG) User Guide (perfiles H100 SXM5, particionado, aislamiento). https://docs.nvidia.com/datacenter/tesla/mig-user-guide/
• NVIDIA — Supported MIG Profiles (catálogo completo H100 80 GB). https://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-mig-profiles.html
• NVIDIA — Time-Slicing GPUs in Kubernetes (GPU Operator 24.9.0). https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/24.9.0/gpu-sharing.html
• NVIDIA — Multi-Process Service (MPS) Overview (kernel concurrency, SM allocation). https://docs.nvidia.com/deploy/mps/latest/index.html
• Kueue — Overview (sigs.k8s.io/kueue, bin-packing, cohorts, quotas). https://kueue.sigs.k8s.io/docs/overview/
• vLLM Blog — vLLM v0.6.0: 2.7x Throughput Improvement and 5x Latency Reduction (continuous batching, chunked prefill). https://blog.vllm.ai/2024/09/05/perf-update.html
• GMI Cloud — NVIDIA H100 GPU Pricing: 2026 Rent vs. Buy Cost Analysis. https://www.gmicloud.ai/en/blog/nvidia-h100-gpu-pricing-2026-rent-vs-buy-cost-analysis
• Saxena et al. — LLM Inference Unveiled: Survey and Roofline Model Insights. arXiv 2402.16363. https://arxiv.org/abs/2402.16363