Mig on lo0 — Tech Blog

GPU idle: the cost that shows up on no invoice but is paid by the whole TCO

Mon, 15 Jun 2026 05:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

The cost per token of on-premise inference is $ \text{€/GPU-hour} \div \text{throughput} $. Throughput is a direct function of the GPU’s useful occupancy. On a generic node of 4×H100 SXM with a reference cost of ~11 € per GPU-hour (depreciation plus energy plus infrastructure), the cost curve over occupancy looks like this:

Useful occupancy	Effective throughput (tok/s)	Cost per 1M tokens
20 %	~700	~43 €
40 %	~1,400	~21 €
70 %	~2,500	~12 €
85 % (practical ceiling)	~3,000	~10 €

Doubling occupancy from 20 % to 40 % halves the cost per token, without buying more iron. At 70 % the cost competes with European cloud providers (~2.2 €/GPU-hour on-demand at Scaleway). At 20 % on-prem is four times dearer than renting. The lever is not the model, nor the precision: it is how many paid-for GPU-hours produce serving tokens.

1 · The fundamental identity

The cost per million tokens (CPM) in your own inference is not a list price. It is:

$$ \text{CPM} = \frac{C_{\text{GPU}} \cdot N_{\text{GPU}}}{T_{\text{ef}} \times 3600 / 10^6} $$

where $ C_{\text{GPU}} $ is the cost per GPU-hour (€/h), $ N_{\text{GPU}} $ the number of GPUs assigned to the service and $ T_{\text{ef}} $ the effective throughput (tok/s). Making the dependence on occupancy explicit:

$$ T_{\text{ef}} = T_{\text{pico}} \times \rho $$

where $ \rho \in [0,1] $ is the useful occupancy rate (the fraction of time in which the GPU is processing real serving tokens). The resulting identity:

$$ \text{CPM} = \frac{C_{\text{GPU}} \cdot N_{\text{GPU}}}{T_{\text{pico}} \times \rho \times 3600 / 10^6} $$

The direct consequence: CPM is inversely proportional to $ \rho $. Doubling $ \rho $ halves CPM. The numerator (cost of the iron) does not change.

The posts coste-por-token-y-por-request and capacity-planning-inferencia-llm-on-premise cover how to calculate $ C_{\text{GPU}} $ and the reference peak throughput. This article deals with $ \rho $: how to measure it, what limits it and how to raise it.

2 · Why the standard metric lies: `DCGM_FI_DEV_GPU_UTIL`

The field DCGM_FI_DEV_GPU_UTIL (field ID 203) appears in nvidia-smi as “GPU-Util”. Its official definition in the DCGM documentation:

“GPU Utilization” — percentage of time during which one or more kernels were executing on the GPU.

The problem for LLM inference: the decode phase is memory-bound. The GPU runs a kernel reading weights from HBM token by token; therefore GPU_UTIL registers close to 100 % even though the tensor cores are at 15 % of their capacity. The field measures activity, not useful work.

The distinction is critical for FinOps: an operator who sees GPU_UTIL 98 % assumes “saturated GPU, no room for more load”. The reality may be “the tensor cores are at 20 % and the bottleneck is HBM”, which leaves room for continuous batching or additional bin-packing.

The metrics that measure real occupancy are those of the _FI_PROF_* subsystem, available in DCGM 3.x with the profiling module enabled:

Table of relevant DCGM fields

DCGM field	Field ID	What it measures	Unit	Operational note
`DCGM_FI_DEV_GPU_UTIL`	203	% time with ≥1 active kernel	%	Misleading in LLM decode
`DCGM_FI_PROF_SM_ACTIVE`	1002	Ratio of cycles with ≥1 active warp per SM	0–1	Compute activity, not occupancy
`DCGM_FI_PROF_SM_OCCUPANCY`	1003	Resident warps / theoretical maximum per SM	0–1	Real intra-SM parallelism
`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`	1004	% cycles with tensor cores (HMMA) active	0–1	The real compute efficiency metric
`DCGM_FI_PROF_DRAM_ACTIVE`	1005	% cycles with HBM transferring	0–1	Memory saturation
`DCGM_FI_PROF_PCIE_TX_BYTES`	—	TX bytes over PCIe	bytes/s	Useful in PCIe inference
`DCGM_FI_DEV_FB_USED`	252	HBM used	MiB	VRAM budget
`DCGM_FI_DEV_POWER_USAGE`	—	Real consumption	W	For real energy cost
`DCGM_FI_DEV_CLOCK_THROTTLE_REASONS`	—	Bitmap of throttle causes	bitmap	Detects silent degradation

The _PROF_* fields require the DCGM profiling module and the right driver permissions. They are documented exhaustively in the NVIDIA DCGM field ID reference and in the sibling post observabilidad-gpu-dcgm-llm.

Characteristic reading in LLM decode (Llama 70B FP8 on H100)

Metric	Typical decode value	Interpretation
`DCGM_FI_DEV_GPU_UTIL`	95–99 %	A lie: there is an active kernel
`DCGM_FI_PROF_SM_ACTIVE`	0.45–0.65	SMs with warps 45–65 % of the time
`DCGM_FI_PROF_SM_OCCUPANCY`	0.30–0.55	Resident warps at 30–55 % of the maximum
`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`	0.10–0.25	Tensor cores active only 10–25 %
`DCGM_FI_PROF_DRAM_ACTIVE`	0.75–0.90	HBM busy 75–90 % of the time

Reading: decode is memory-bound. The model weights are loaded from HBM for each token; HBM is saturated but the tensor cores wait. All of the “utilisation” in nvidia-smi comes from memory reads, not from compute.

3 · MFU and HFU: occupancy expressed in FLOPs

The canonical compute efficiency metric is MFU (Model FLOPs Utilization), defined in the PaLM paper (Chowdhery et al., arXiv 2204.02311, section 4):

$$ \text{MFU} = \frac{T_{\text{obs}} \times C_{\text{modelo}}}{P_{\text{pico}}} $$

where $ T_{\text{obs}} $ is the observed throughput (tok/s) and $ P_{\text{pico}} $ is the theoretical peak performance of the hardware (FLOP/s).

where $ C_{\text{modelo}} $ is the number of FLOPs per token in a complete forward pass. For a dense transformer with $ P $ parameters, the usual approximation (forward plus backward = 6P FLOPs per token; forward only = 2P):

$$ C_{\text{modelo}} \approx 2P \quad \text{(inference)} $$ $$ C_{\text{modelo}} \approx 6P \quad \text{(training, forward + backward)} $$

HFU (Hardware FLOPs Utilization) measures the FLOPs actually executed in hardware, including activation recomputation (gradient checkpointing). In training with recompute:

$$ C_{\text{hardware}} \approx 8P \quad \text{(forward × 2 + backward × 4)} $$

therefore HFU > MFU when there is recompute; they are identical without it.

Typical MFU values

Regime	Hardware	Typical MFU	Limiting regime
Training (large batch, BF16)	H100 SXM	35–50 %	compute-bound
Training PaLM 540B	TPU v4	46.2 %	compute-bound
Prefill inference (large batch)	H100 SXM	25–45 %	compute-bound
Decode inference (bs=1)	H100 SXM	3–8 %	memory-bound
Decode inference (continuous batching, bs=32–64)	H100 SXM	15–30 %	memory-bound, attenuated

Decode with batch size 1 has a single-digit MFU because the hardware spends most of its time waiting for HBM to deliver weights. Raising the batch size (continuous batching) amortises the weight read across more simultaneous tokens and raises MFU.

The Roofline model

The roofline places each operation in the space (arithmetic intensity, throughput):

Decode with batch size 1 falls in the memory-bound zone to the left of the ridge point. Raising the batch (continuous batching) moves the point to the right and up, closer to the roofline.

4 · TCO sensitivity to occupancy

Assumptions for the example: a generic node with 4×H100 SXM 80 GB, total node cost ~44 €/hour (5-year depreciation of ~220,000 €, energy ~4×700 W at ~0.12 €/kWh, plus rack/colocation infrastructure; see coste-por-token-y-por-request for the detail of the identity). Reference peak throughput with Llama 70B FP8 plus continuous batching: ~3,500 tok/s aggregate.

$$ \text{CPM}(\rho) = \frac{44\ \text{€/h}}{3500 \times \rho \times 3600 / 10^6} = \frac{44 \times 10^6}{3500 \times \rho \times 3600} = \frac{3.49}{\rho}\ \text{€/1M tok} $$

Occupancy $ \rho $	Effective throughput (tok/s)	On-prem CPM	On-demand cloud CPM (~2.2 €/GPU-h)
20 %	700	~17.5 €	~6.3 €
40 %	1,400	~8.7 €	~6.3 €
60 %	2,100	~5.8 €	~6.3 €
70 %	2,450	~5.0 €	~6.3 €
85 %	2,975	~4.1 €	~6.3 €

On-prem / cloud crossover point: with this example hardware, the on-prem cost advantage over a comparable European cloud starts at $ \rho \approx 55\text{-}60\,\% $. Below that, idle makes on-prem dearer than renting. The cloud column is flat because cloud bills per hour used, not per the throughput you extract from it.

Example figures with generic hardware. The real numbers depend on the depreciation price, the local cost of energy, the model and the precision. The structure of the curve, CPM inversely proportional to $ \rho $, is universal.

5 · Idle metrics: where occupancy is lost

Before applying levers, you need to know which kind of idle dominates. Three categories:

Type of idle	Symptom in metrics	Usual cause
Scheduling idle	`DCGM_FI_DEV_POWER_USAGE` low, `SM_ACTIVE` < 0.05	No requests in the queue; GPU waiting for work
Batching idle	`SM_ACTIVE` high, `PIPE_TENSOR_ACTIVE` low, `DRAM_ACTIVE` low	Batch too small; prefill stall between requests
Memory-bound idle	`DRAM_ACTIVE` high, `PIPE_TENSOR_ACTIVE` low	Normal decode; HBM is the bottleneck; insufficient batch size

Scheduling idle is the most expensive and the most directly attacked with bin-packing. Batching idle is attacked with continuous batching. Memory-bound idle in decode does not disappear entirely (it is the physics of the transformer), but it is attenuated with a larger batch size.

The inference engine metrics that complement the diagnosis from the service side are documented in observabilidad-gpu-dcgm-llm and anatomia-metricas-dcgm-vllm-anomalias.

6 · Levers for raising occupancy

Table of lever × effect × when it applies

Lever	Effect on $ \rho $	When it applies	Operational complexity
Continuous batching	High: removes idle between requests; raises decode MFU from ~5 % to ~20 %	Always in inference; enabled by default in vLLM	Low (engine parameter)
Scheduler bin-packing (kube-scheduler / Kueue)	High: concentrates loads on fewer nodes; frees whole nodes for shutdown or rebalancing	Clusters with load that varies over the day	Medium (scheduler policy)
MIG (Multi-Instance GPU)	Medium: fills GPUs with light loads that previously lived alone on a whole GPU	Heterogeneous loads: embeddings + reranker + guardrail + large model	High (hot repartitioning not available)
Time-slicing	Low to medium: raises occupancy in dev/bursts; no isolation	Consumer GPUs (RTX 5090/4090); low-risk multi-tenant dev	Low
MPS	Medium: concurrent execution of multiple small processes; reduces context-switch overhead	Many small concurrent kernels on a datacenter GPU; trust between loads	Medium
Chunked prefill	Medium: interleaves prefill and decode; reduces the TTFT spike and raises throughput	Loads with a mix of short and long prompts	Low (vLLM flag)
Quantisation (FP16→FP8→INT4)	Indirect: raises peak throughput, which lowers CPM for the same $ \rho $	Models with quantised kernel support (native Hopper FP8)	Medium
Autoscaling (KEDA)	Keeps $ \rho $ high by scaling replicas according to the queue	Variable and predictable load; cluster with spare capacity	Medium

The post compartir-gpu-time-slicing-mps-mig details the three sharing mechanisms (time-slicing, MPS, MIG) with VRAM budgets.

Bin-packing with Kueue

Kueue (sigs.k8s.io/kueue) is the Kubernetes-native queue manager for GPU jobs. Its model of cohorts and nominal quotas allows active bin-packing: jobs accumulate in a queue and are launched only when there is a node that can take them whole, instead of fragmenting the load across partially occupied nodes.

BestFit packing in the ClusterQueue is configured with:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
 name: gpu-prod
spec:
 preemption:
 reclaimWithinCohort: Any
 withinClusterQueue: LowerPriority
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 16

The priority-based preemption policy ensures that production jobs displace experimentation ones when there is scarcity, keeping occupancy on the production nodes high.

MIG as a bin-packing lever inside the GPU

MIG makes it possible to fill an H100 with light loads that would otherwise live alone on a whole GPU. A profile of 3×2g.20gb + 1×1g.10gb on an H100 can host simultaneously a 7B FP8 model (~14 GB of weights), two embedding services and an INT4 guardrail, all with hardware isolation. Without MIG, each of those loads would take a whole GPU with an individual $ \rho $ below 10 %.

The profiles available on the H100 80 GB (SXM5) according to NVIDIA’s MIG User Guide:

MIG profile	Compute slices	Memory	Max. instances
`1g.10gb`	1/7 SMs	10 GB	7
`1g.20gb`	1/7 SMs	20 GB	4
`2g.20gb`	2/7 SMs	20 GB	3
`3g.40gb`	3/7 SMs	40 GB	2
`4g.40gb`	4/7 SMs	40 GB	1
`7g.80gb`	7/7 SMs	80 GB	1 (whole GPU)

MIG is not available on consumer GPUs (RTX 5090, RTX 4090). Time-slicing is the only sharing option on that hardware.

7 · Continuous batching: the effect on MFU

Continuous batching (also called iteration-level scheduling or in-flight batching) is the mechanism with the greatest impact on occupancy in inference. The idea: instead of waiting for a complete batch to finish before launching the next, the engine evaluates the pipeline after each token and replaces completed sequences with new requests from the waiting queue.

Quantified effect in vLLM:

Without batching (bs=1, static): decode MFU ~3–8 %; GPU idle between requests.
With continuous batching (dynamic bs 16–64): decode MFU ~15–30 %; the GPU almost never waits.
On pure prefill loads with a large batch (bs=128+): prefill MFU 25–45 %; it approaches the compute roofline.

The vLLM parameter --max-num-seqs controls the maximum number of sequences in the concurrent batch. Raising it increases occupancy until HBM becomes the bottleneck (watch for DCGM_FI_PROF_DRAM_ACTIVE > 90 % sustained).

8 · Hardware and scale: what applies to what

Hardware	MIG	MPS	Time-slicing	Continuous batching	Note
H100 SXM / H200	Yes (7 inst.)	Yes	Yes	Yes	On-prem datacenter reference
A100 SXM/PCIe	Yes (7 inst.)	Yes	Yes	Yes	Previous generation; HBM2e
L40S / L40	No	Yes	Yes	Yes	Ada Lovelace; no MIG; good price/VRAM
RTX 5090	No	Yes (limited)	Yes	Yes	Consumer; no MIG; does not scale in production
RTX 4090	No	Yes (limited)	Yes	Yes	Consumer; 24 GB VRAM; no MIG

The RTX 5090 and RTX 4090 illustrate the case of hardware that does not scale for multi-tenant with isolation: they do not support MIG, VRAM is scarce for models above 7B with a wide KV-cache, and TDP (600 W / 450 W) is high relative to throughput. For production inference at scale, the 4×H100 SXM is the reference node of this series.

9 · FinOps diagnostic flow: from high CPM to the cause

10 · Worked example: impact of continuous batching on CPM

Starting point: Llama 70B FP8 on 4×H100 SXM, a load of 8 requests/s with 512 output tokens on average, without continuous batching (static bs = 8):

Observed throughput: ~900 tok/s (decode dominant).
$ \rho_{\text{efectiva}} \approx 900 / 3500 \approx 0.26 $.
CPM: $ 44 / (900 \times 3600 / 10^6) \approx 13.6\ \text{€} $.

Same load, enabling continuous batching with --max-num-seqs 64:

Observed throughput: ~2,300 tok/s (the dynamic batch fills the gaps).
$ \rho_{\text{efectiva}} \approx 2300 / 3500 \approx 0.66 $.
CPM: $ 44 / (2300 \times 3600 / 10^6) \approx 5.3\ \text{€} $.

CPM reduction: from 13.6 € to 5.3 €, –61 %, with no hardware change and without touching the model.

The vLLM parameters relevant for raising effective throughput:

--max-num-seqs 64 # maximum concurrent batch
--max-num-batched-tokens 16384 # total tokens per iteration
--enable-chunked-prefill # interleaves prefill and decode

Sources

NVIDIA — DCGM Field Identifiers reference (v3.1), lista completa de DCGM_FI_* con field IDs y definiciones. https://docs.nvidia.com/datacenter/dcgm/3.1/dcgm-api/dcgm-api-field-ids.html
NVIDIA — GPU Profiling Metrics (Run:ai / DCGM), definiciones de DCGM_FI_PROF_SM_ACTIVE (1002), SM_OCCUPANCY (1003), PIPE_TENSOR_ACTIVE (1004). https://run-ai-docs.nvidia.com/self-hosted/platform-management/monitor-performance/gpu-profiling-metrics
Chowdhery et al. — PaLM: Scaling Language Modeling with Pathways (MFU definition, sección 4). arXiv 2204.02311. https://arxiv.org/abs/2204.02311
NVIDIA — Multi-Instance GPU (MIG) User Guide (perfiles H100 SXM5, particionado, aislamiento). https://docs.nvidia.com/datacenter/tesla/mig-user-guide/
NVIDIA — Supported MIG Profiles (catálogo completo H100 80 GB). https://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-mig-profiles.html
NVIDIA — Time-Slicing GPUs in Kubernetes (GPU Operator 24.9.0). https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/24.9.0/gpu-sharing.html
NVIDIA — Multi-Process Service (MPS) Overview (kernel concurrency, SM allocation). https://docs.nvidia.com/deploy/mps/latest/index.html
Kueue — Overview (sigs.k8s.io/kueue, bin-packing, cohorts, quotas). https://kueue.sigs.k8s.io/docs/overview/
vLLM Blog — vLLM v0.6.0: 2.7x Throughput Improvement and 5x Latency Reduction (continuous batching, chunked prefill). https://blog.vllm.ai/2024/09/05/perf-update.html
GMI Cloud — NVIDIA H100 GPU Pricing: 2026 Rent vs. Buy Cost Analysis. https://www.gmicloud.ai/en/blog/nvidia-h100-gpu-pricing-2026-rent-vs-buy-cost-analysis
Saxena et al. — LLM Inference Unveiled: Survey and Roofline Model Insights. arXiv 2402.16363. https://arxiv.org/abs/2402.16363

Sharing one GPU between several workloads: time-slicing, MPS and MIG

Thu, 11 Jun 2026 02:00:00 +0000

This post opens an operational series on how to squeeze a generic on-premise 4×H100 SXM LLM cluster. The sibling pieces: Serving several models on one GPU: swap and sleep (what to do when the models do not fit at the same time and have to take turns in memory), RAG on CPU: separating the data plane from generation (moving retrieval off the GPU to free it up) and Sovereign end-to-end assistant with LibreChat, LiteLLM and RAG, the final assembly, a fourth instalment in preparation. Here we start with the most basic question: you have one GPU and you want to put several workloads on it. How do you split it?

TL;DR

You have one GPU, or a few, and several workloads that want to run on it: a chat model, an embeddings service, a reranker, a queue of dev jobs. The GPU is underused if it only runs one thing, but piling several on carelessly causes contention, OOM or cascading failures. There are three mechanisms and they share out different things. Time-slicing (replicas from the NVIDIA k8s device-plugin) multiplexes in time: it advertises the GPU as “N GPUs” and the processes take turns at the compute, but they share the full physical VRAM, with no memory isolation, no fault isolation and no QoS. Its trap is an OOM that does not show up in the Kubernetes scheduler but at run time, when the sum of VRAM allocations exceeds the real memory. MPS (Multi-Process Service) multiplexes in space: it shares out the SMs between processes that execute kernels concurrently, reduces context-switch overhead and allows SM and memory limits per process. It raises throughput when there are many small kernels, but fault isolation is still weak. MIG (Multi-Instance GPU) partitions in hardware: it cuts the Hopper GPU into up to seven instances with dedicated SMs, L2, memory and bandwidth, with real memory, fault and performance isolation; datacenter only (A100/H100/H200/B200), never on an RTX 5090. The rule: real isolation / multi-tenant / compliance → MIG (if it is Hopper); many small concurrent kernels and trust between workloads → MPS; dev, bursts, consumer GPU or no need to isolate → time-slicing. This post works it through with numbers: the VRAM budget of four vLLM instances on an H100 advertised as four replicas, and what fits into a 10 GB MIG instance.

The analogy: a shared hob, one kitchen with several cooks, several kitchens

Imagine you have a single professional hob and three orders to cook at the same time. There are three ways to organise it, and they are exactly the three mechanisms.

Time-slicing is one hob taken in turns, with no larder of your own. Each cook comes in, cooks their dish, leaves, and the next one comes in. The split is temporal: nobody cooks at the same time, they take turns. The problem is not the hob, which gets shared out fine, but the common larder: the ingredients sit in a single shared cupboard and nobody has their own. If the three cooks reserve more flour than there is in total, it is not that they wait their turn: it is that there is no flour. The service goes down for everyone. And if one cook leaves a pan burning and starts a fire, it burns the whole kitchen, not just their corner.

MPS is several cooks coordinated on the same worktop. Now they do cook at the same time, sharing out the space on the worktop (the SMs). A head chef (the MPS daemon) coordinates so they do not collide and so the worktop does not sit empty while one of them waits for water to boil. You can assign each cook a percentage of the worktop and a larder limit. They work faster as a group because the worktop does not sit idle between small tasks. But they still share the kitchen: if one starts a serious fire, the others notice.

MIG is several independent kitchens in the same building. A concrete wall separates each kitchen: its own hob, its own larder, its own door and its own electrical panel. What happens in kitchen 3, a fire, an empty larder, a slow cook, does not touch kitchen 1. It is the only split with genuine isolation. The price: you have to decide in advance how many kitchens and of what size, the walls are fixed, and only expensive buildings (datacenter) come ready to put them up.

The rest of the post is, essentially, when you want cheap turns, when you want coordinated cooks and when you need concrete walls.

An H100 SXM 80 GB does not fill up with just any workload. A bge-reranker-v2-m3 reranker takes a few hundred MB and saturates a handful of SMs; a bge-m3 embeddings service is just as small; a 1B guardrail model in INT4 fits in a couple of GB. Dedicating 80 GB of HBM3 and 132 SMs to serving embeddings is using a hydraulic press to hammer in a drawing pin, the same argument as in mixed environments, but now inside the GPU instead of moving the workload to different silicon.

The aim of sharing is to raise the useful utilisation of fixed capital. But sharing badly introduces three pathologies:

Compute contention: two workloads fight over the same SMs and both run slowly, with unpredictable latency jitter.
Memory contention: the sum of requested VRAM exceeds the physical amount and something dies with a CUDA out of memory.
Cascading failure: a workload that blows up (an illegal kernel, an OOM) can drag its neighbours down with it if they share a context.

The three mechanisms attack these pathologies to different depths. None of them solves all three except MIG, and MIG costs specific hardware. Let us look at them one by one.

Time-slicing: compute turns, shared larder

Time-slicing is temporal multiplexing in software. On Kubernetes, the NVIDIA GPU Operator configures the device-plugin to advertise N replicas of each physical GPU. An H100 declared with replicas: 4 appears to the scheduler as four nvidia.com/gpu resources, and Kubernetes can place four pods on it. Internally, the GPU scheduler hands out compute turns to each process: it runs a bit of process A, switches to B, to C, to D, then back to A. It is the same time-sharing an operating system does with the CPU.

The key idea, and the one that causes most confusion, is this: a replica is NOT a fraction of the GPU. It is a compute turn. NVIDIA’s documentation is explicit: unlike MIG, there is no memory or fault isolation between replicas. The four replicas of the H100 see the full 80 GB of VRAM, unpartitioned. There is no 20 GB per replica. There are 80 GB for all four, handed out in cudaMalloc arrival order.

This has three consequences you need to internalise:

It does not isolate memory. If the sum of what the four processes reserve exceeds 80 GB, the fourth cudaMalloc fails with OOM. The Kubernetes scheduler does not see it coming: it counted four available nvidia.com/gpu resources and happily placed four pods. The OOM appears at run time, not at scheduling time. This is trap number one of time-slicing.
It does not isolate faults. A process that triggers an unrecoverable CUDA error can leave the GPU context in a state that affects its neighbours. They share the same device with no barriers.
It gives no compute QoS. Under contention, the turn-taking does not guarantee a minimum fraction to anyone. Each workload’s latency suffers jitter proportional to how many active replicas are fighting over the GPU at that instant. A latency-sensitive workload (an interactive chat) can see its TTFT dance around depending on what its neighbours are doing.

What is it good for, then? For dev, bursts and low utilisation. If you have four developers who touch the GPU sporadically, advertising four replicas lets all four have access without fighting almost ever (they rarely overlap while active). For batch workloads that tolerate jitter. And, a decisive advantage, it works on consumer GPUs: an RTX 5090 32 GB does not support MIG, but it does support time-slicing. It is the only “Kubernetes-native” way to share a 5090 between several pods.

The VRAM budget in time-slicing (the calculation that avoids the OOM)

Here is the maths you have to do before deploying, because Kubernetes will not do it for you. Suppose an H100 80 GB advertised as 4 replicas and we want to run four vLLM instances on it, one per replica.

vLLM reserves memory with the --gpu-memory-utilization parameter, which is the fraction of the total physical VRAM that each instance keeps (for model weights plus KV-cache). The detail that kills: that fraction is computed over the 80 physical GB, not over a supposed “20 GB of my replica”, because the replica does not have 20 GB. Remember, there is no memory partitioning. Each vLLM sees the 80 GB and reserves its fraction of them.

The no-OOM constraint is therefore that the sum of fractions be less than 1:

$$\sum_{i=1}^{N} g_i < 1 \quad\Longleftrightarrow\quad \sum_{i=1}^{N} g_i \cdot V_{\text{HBM}} < V_{\text{HBM}}$$

where $g_i$ is the --gpu-memory-utilization of instance $i$ and $V_{\text{HBM}} = 80$ GB. It is wise to leave headroom (runtime overhead, fragmentation, CUDA context), so in practice you aim for the sum to stay comfortably below 1, say $\le 0.9$.

A case that works. Four vLLM instances at $g_i = 0.20$:

$$\sum_{i=1}^{4} 0.20 = 0.80 \quad\Rightarrow\quad 0.80 \times 80\ \text{GB} = 64\ \text{GB} < 80\ \text{GB} \quad\checkmark$$

Each instance reserves $0.20 \times 80 = 16$ GB. Four instances add up to 64 GB, leaving a 16 GB cushion. There is no OOM. Each vLLM has 16 GB for weights plus KV-cache: enough for a 7B–8B model in FP8/INT4 with a modest KV-cache.

A case that blows up. The same four instances, but somebody raises $g_i = 0.30$ thinking “I have four replicas, I can give each one more”:

$$\sum_{i=1}^{4} 0.30 = 1.20 \quad\Rightarrow\quad 1.20 \times 80\ \text{GB} = 96\ \text{GB} > 80\ \text{GB} \quad\times$$

The first instances start up and reserve $0.30 \times 80 = 24$ GB each. Three instances are already at $72$ GB. The fourth tries to reserve another 24 GB, there is nothing left, and it dies with CUDA out of memory. And worse: Kubernetes will reschedule it on the same GPU (it still sees four replicas), where it will die again, in a CrashLoopBackOff that makes no sense if you only look at the pod manifest.

The operational rule is brutally simple: in time-slicing, you manage the VRAM budget by hand, adding up the --gpu-memory-utilization values. The replica count controls how many pods fit in compute turns, but it does not reserve a single byte of memory. Confusing the two is the recurring mistake.

MPS: coordinated cooks on the same worktop

The Multi-Process Service (MPS) attacks a different problem. By default, when several processes use the same GPU without MPS, each one has its own CUDA context, and the GPU alternates between contexts (time-slicing at driver level): they do not execute kernels at the same time, they take turns, with context-switch overhead. If your kernels are small and do not fill the GPU on their own, this leaves SMs idle: process A uses 30 % of the SMs during its turn and the other 70 % is wasted.

MPS introduces a daemon that shares a single CUDA context between processes, so that their kernels can execute concurrently occupying different SMs at the same time. It is spatial sharing of compute: instead of taking turns at the whole worktop, each cook occupies a part and they work in parallel. This reduces context-switch overhead and raises throughput when there are many small concurrent kernels that individually do not saturate the GPU.

And, unlike pure time-slicing, MPS allows per-process limits, which gives you a form of QoS:

CUDA_MPS_ACTIVE_THREAD_PERCENTAGE limits the percentage of SMs an MPS client can use. By default each client gets $100 / \text{MaxSharedClients}$. Pinning it to, say, 40 % caps that process’s compute ceiling (MPS docs).
CUDA_MPS_PINNED_DEVICE_MEM_LIMIT imposes a memory cap per client (valid from CUDA 11.5). This is what time-slicing does not have: a per-process VRAM limit that the runtime enforces.

These two limits turn MPS into a resource provisioning mechanism that mitigates the noisy neighbour: you can guarantee that a process does not eat more than X % of the SMs or more than Y GB. The combination gives reasonable QoS, not perfect but real.

The limitation MPS does not solve: fault isolation is weak. Since clients share the daemon’s CUDA context, a fatal error in one client can affect the daemon and therefore the other clients (historically, a client dying dirtily could require restarting the daemon). It is better than time-slicing in this respect, but a long way from hardware isolation. That is why MPS fits when there is trust between the workloads, processes from your own team, not third-party tenants.

The canonical use case: many small concurrent inference requests that individually leave the GPU half empty. MPS overlaps them and raises aggregate throughput. Serving several small models, or several light replicas of the same model, on a datacenter GPU where you trust all the workloads, is MPS territory.

MIG: concrete walls

The Multi-Instance GPU (MIG) is the only one of the three that gives genuine isolation, because it cuts the GPU in hardware. Available on modern datacenter GPUs, A100 (Ampere), H100/H200 (Hopper), B200 (Blackwell), and never on consumer ones: an RTX 5090 (consumer Blackwell) does not support MIG, nor do GeForce cards in general.

MIG divides the GPU into up to seven instances (GPU Instances), and each instance gets a dedicated portion of:

SMs (compute slices): compute is split into 7 slices, each one roughly 1/7 of the SMs.
L2 cache and memory: each instance has its slab of HBM and its portion of L2 cache.
Memory bandwidth: dedicated, not shared.
Data paths and engines: with fault barriers between instances.

The result is that a MIG instance behaves like a smaller, independent GPU: whatever happens in one, an OOM, a kernel blowing up, a workload saturating its compute, does not affect its neighbours. Memory, fault and performance (QoS) isolation, the three things time-slicing does not give and MPS only half gives.

The H100 80GB profiles

MIG does not allow arbitrary sizes: it has fixed profiles. On the H100 80GB, the profile catalogue (notation <compute>g.<memory>gb) is:

Profile	Compute (slices)	Memory	Max. instances
`1g.10gb`	1/7	10 GB	7
`1g.20gb`	1/7	20 GB	4
`2g.20gb`	2/7	20 GB	3
`3g.40gb`	3/7	40 GB	2
`4g.40gb`	4/7	40 GB	1
`7g.80gb`	7/7	80 GB	1 (whole GPU)

(There is also 1g.10gb+me, a variant with media engines for video encoding.) The base memory unit on the H100 80GB is 10 GB per slice (80 GB / 8, with one slice reserved), and the compute unit is 1/7 of the SMs. The profiles combine these units. Note 1g.20gb: the same compute fraction as 1g.10gb (1/7 of the SMs) but twice the memory, useful when a workload needs more VRAM than compute.

One important detail: MIG partitions do not mix freely. The GPU is divided following a valid geometry (the profiles fit together like pieces in a grid), and the profiles are fixed when you configure the GPU; changing them requires draining and repartitioning. They are concrete walls: solid, but they do not move while hot.

The calculation: 7×1g.10gb against 1×7g.80gb

Let us compare the two extremes. On the left, seven 1g.10gb instances: seven isolated 10 GB GPUs. On the right, a single 7g.80gb: the whole H100, unpartitioned.

The operational question is what fits in 10 GB. An instance’s VRAM budget is split between model weights and KV-cache:

$$V_{\text{inst}} = V_{\text{weights}} + V_{\text{KV}} + V_{\text{overhead}}$$

Take a model of 7B parameters in FP8 (1 byte/parameter):

$$V_{\text{weights}} \approx 7 \times 10^9 \times 1\ \text{byte} = 7\ \text{GB}$$

On a 1g.10gb instance (10 GB), after the 7 GB of weights and subtracting roughly 0.5–1 GB of runtime overhead, about 2 GB are left for KV-cache. That is enough for a modest context window and low concurrency, fine for a guardrail service, a classifier or an extraction model that handles short prompts one at a time. A 7B in INT4 (~3.5 GB of weights) leaves ~5.5 GB of KV-cache, much roomier. But a 13B model in FP8 (~13 GB of weights) does not fit in a 10 GB instance: not even the weights get in. For that you need 1g.20gb, 2g.20gb or larger.

Against this, the 7g.80gb (whole GPU) gives you the 80 GB for one large model: a 70B in FP8 (~70 GB of weights) fits with a tight KV-cache, or a 70B with more room spread across several whole GPUs in tensor-parallel (see TP against replicas: one big one versus N small ones).

The reading is clear: fine partitioning (7×1g.10gb) maximises the number of small isolated workloads; not partitioning (1×7g.80gb) maximises the model size that fits. The KV-cache available per instance shrinks proportionally as you partition, so fine-grained MIG serves many light isolated services, not one large model chopped up. If your workload is a single large model, MIG is not for you: use the whole GPU or several in TP.

The decision tree

The three questions, in order:

Do you need REAL isolation?
(multi-tenant, compliance, one workload failing must not touch another)
│
┌────┴────┐
YES NO
│ │
Is it Hopper/ Many SMALL concurrent kernels
Ampere/ AND you trust every workload?
Blackwell? │
│ ┌────┴────┐
┌──┴──┐ YES NO
YES NO │ │
│ │ MPS Dev / bursts / consumer GPU
MIG │ (spatial, / no need to isolate?
│ │ per-process │
│ no real QoS) YES
│ isolation: │
│ rethink TIME-SLICING
│ (move to (temporal, cheap,
│ CPU, another works on a 5090)
│ GPU, or accept
│ the risk with
│ time-slicing)

And each branch in one sentence:

MIG when isolation is a requirement (compliance, multi-tenant, hard SLA) and you have datacenter hardware that supports it. Concrete walls cost money, but if you need them there is no substitute.
MPS when you have many small concurrent workloads that leave the GPU half empty and you trust all of them (same team, not third-party tenants). You raise throughput with reasonable QoS, accepting imperfect fault isolation.
Time-slicing when it is dev, bursts, low utilisation, a consumer GPU, or you simply do not need to isolate anything. Cheap and universal, but manage the VRAM budget by hand.

One nuance the recent documentation records: they can be combined. You can do time-slicing on top of a MIG instance (hardware isolation at the instance boundary, software turns inside), or use MPS inside a MIG instance. The layers are not mutually exclusive; the tree picks the primary strategy.

Applied to the generic 4×H100 cluster

Let us get down to numbers with a generic on-premise cluster of 4×H100 SXM 80 GB with NVLink. It is common to have a heterogeneous menu of workloads: a large chat model, light services (embeddings, reranker, guardrails) and a dev/experimentation queue. Each type calls for a different mechanism. A reasoned split:

GPU 0 and GPU 1, large model in tensor-parallel (not shared). A 70B model in FP8 takes ~70 GB of weights; served comfortably with a generous KV-cache it needs more than one H100. We spread it in tensor-parallel over two whole H100s joined by NVLink (intra-node bandwidth is what makes TP viable; the detail is in TP against replicas). Here we do not share: these two GPUs belong to the large model, full stop. Total isolation by dedication.

$$V_{\text{available}} = 2 \times 80 = 160\ \text{GB};\quad V_{\text{weights}} \approx 70\ \text{GB};\quad V_{\text{KV}} \approx 80\ \text{GB of KV-cache}$$

There is memory to spare for a long request queue and high concurrency.

GPU 2, split with MIG into small instances for light services. Embeddings (bge-m3), reranker (bge-reranker-v2-m3) and a couple of guardrail models (1B–3B) are different workloads, potentially from different teams, and you want a failure or a spike in one not to touch the others. Light multi-tenancy with isolation → MIG. A reasonable partitioning of the H100:

$$\underbrace{3 \times \texttt{1g.10gb}}_{\text{30 GB, 3/7 SMs}} \;+\; \underbrace{1 \times \texttt{4g.40gb}}_{\text{40 GB, 4/7 SMs}}$$

The three 1g.10gb instances (10 GB, 1/7 of the SMs each) host embeddings, reranker and a 1B INT4 guardrail, each isolated, with no noisy neighbour. The 4g.40gb (40 GB, 4/7 of the SMs) hosts an intermediate 7B–13B model with a decent KV-cache for a support service. Each service has its larder and its wall; if the reranker blows up, the chat never finds out.

GPU 3, time-slicing for dev and bursts. The developers touch the GPU sporadically: experiments, short fine-tunes, model trials. They do not need isolation (it is the same team) and they rarely overlap while active. We advertise it as 4 replicas via the device-plugin. Four dev pods fit in turns. VRAM budget with the formula above: if each dev brings up a vLLM at --gpu-memory-utilization 0.20, they add up to $4 \times 16 = 64$ GB < 80 GB, with no OOM. If somebody needs more, lower the replica count or coordinate with the team. The cost of flexibility is manual discipline.

Summary of the split:

Resource	Mechanism	Workload	Isolation
GPU 0 + GPU 1	Dedication (TP)	70B chat in tensor-parallel	total (dedicated)
GPU 2	MIG (3×1g.10gb + 1×4g.40gb)	embeddings, reranker, guardrails, 7B–13B	real hardware
GPU 3	Time-slicing (4 replicas)	dev, bursts, experiments	none (trust)

The logic is always the same: spend isolation (MIG) where you need it, spend cheap concurrency (time-slicing) where you do not, and reserve whole GPUs for what genuinely fills them. An H100 serving embeddings on a 7g.80gb would be as absurd as an RTX 5090 trying MIG: the tool does not match the workload.

What we have not covered

What happens when they do not even fit at the same time: if you have more models than VRAM and they have to take turns in memory (loading/unloading weights, not just taking turns at compute), you enter swap and sleep territory, the sibling piece Serving several models on one GPU.
NUMA-aware scheduling: on multi-socket nodes, which GPU goes with which CPU/memory matters for latency; see Kubelet resource managers on RKE2.
Autoscaling the replicas: how many instances to bring up according to real load, with KEDA and queue metrics; see Autoscaling LLMs on Kubernetes with KEDA.
A jitter benchmark under contention: how much TTFT really wobbles in time-slicing with 4 active replicas compared with MIG, material that deserves its own measurement, not an estimate.

References

NVIDIA — Time-Slicing GPUs in Kubernetes (GPU Operator). https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html
NVIDIA — Multi-Process Service (MPS) Overview. https://docs.nvidia.com/deploy/mps/index.html
NVIDIA — MPS: Tools and Interface Reference (CUDA_MPS_ACTIVE_THREAD_PERCENTAGE, CUDA_MPS_PINNED_DEVICE_MEM_LIMIT). https://docs.nvidia.com/deploy/mps/appendix-tools-and-interface-reference.html
NVIDIA — Multi-Instance GPU (MIG) User Guide. https://docs.nvidia.com/datacenter/tesla/mig-user-guide/
NVIDIA — Supported MIG Profiles (H100 80GB catalogue). https://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-mig-profiles.html
NVIDIA — k8s-device-plugin (time-slicing replicas). https://github.com/NVIDIA/k8s-device-plugin
vLLM — Engine Arguments (--gpu-memory-utilization). https://docs.vllm.ai/en/latest/serving/engine_args.html

Mig on lo0 — Tech Blog

GPU idle: the cost that shows up on no invoice but is paid by the whole TCO

TL;DR

1 · The fundamental identity

2 · Why the standard metric lies: `DCGM_FI_DEV_GPU_UTIL`

Table of relevant DCGM fields

Characteristic reading in LLM decode (Llama 70B FP8 on H100)

3 · MFU and HFU: occupancy expressed in FLOPs

Typical MFU values

The Roofline model

4 · TCO sensitivity to occupancy

5 · Idle metrics: where occupancy is lost

6 · Levers for raising occupancy

Table of lever × effect × when it applies

Bin-packing with Kueue

MIG as a bin-packing lever inside the GPU

7 · Continuous batching: the effect on MFU

8 · Hardware and scale: what applies to what

9 · FinOps diagnostic flow: from high CPM to the cause

10 · Worked example: impact of continuous batching on CPM

See also

Sources

Sharing one GPU between several workloads: time-slicing, MPS and MIG

TL;DR

The analogy: a shared hob, one kitchen with several cooks, several kitchens

Time-slicing: compute turns, shared larder

The VRAM budget in time-slicing (the calculation that avoids the OOM)

MPS: coordinated cooks on the same worktop

MIG: concrete walls

The H100 80GB profiles

The calculation: 7×1g.10gb against 1×7g.80gb

The decision tree

Applied to the generic 4×H100 cluster

What we have not covered

See also

References

Mig on lo0 — Tech Blog

GPU idle: the cost that shows up on no invoice but is paid by the whole TCO

TL;DR

1 · The fundamental identity

2 · Why the standard metric lies: DCGM_FI_DEV_GPU_UTIL

Table of relevant DCGM fields

Characteristic reading in LLM decode (Llama 70B FP8 on H100)

3 · MFU and HFU: occupancy expressed in FLOPs

Typical MFU values

The Roofline model

4 · TCO sensitivity to occupancy

5 · Idle metrics: where occupancy is lost

6 · Levers for raising occupancy

Table of lever × effect × when it applies

Bin-packing with Kueue

MIG as a bin-packing lever inside the GPU

7 · Continuous batching: the effect on MFU

8 · Hardware and scale: what applies to what

9 · FinOps diagnostic flow: from high CPM to the cause

10 · Worked example: impact of continuous batching on CPM

See also

Sources

Sharing one GPU between several workloads: time-slicing, MPS and MIG

TL;DR

The analogy: a shared hob, one kitchen with several cooks, several kitchens

Why share: the operational problem

Time-slicing: compute turns, shared larder

The VRAM budget in time-slicing (the calculation that avoids the OOM)

MPS: coordinated cooks on the same worktop

MIG: concrete walls

The H100 80GB profiles

The calculation: 7×1g.10gb against 1×7g.80gb

The decision tree

Applied to the generic 4×H100 cluster

What we have not covered

See also

References

2 · Why the standard metric lies: `DCGM_FI_DEV_GPU_UTIL`