Sharing one GPU between several workloads: time-slicing, MPS and MIG

· By José Manuel Román
Contents

This post opens an operational series on how to squeeze a generic on-premise 4×H100 SXM LLM cluster. The sibling pieces: Serving several models on one GPU: swap and sleep (what to do when the models do not fit at the same time and have to take turns in memory), RAG on CPU: separating the data plane from generation (moving retrieval off the GPU to free it up) and Sovereign end-to-end assistant with LibreChat, LiteLLM and RAG, the final assembly, a fourth instalment in preparation. Here we start with the most basic question: you have one GPU and you want to put several workloads on it. How do you split it?

TL;DR

You have one GPU, or a few, and several workloads that want to run on it: a chat model, an embeddings service, a reranker, a queue of dev jobs. The GPU is underused if it only runs one thing, but piling several on carelessly causes contention, OOM or cascading failures. There are three mechanisms and they share out different things. Time-slicing (replicas from the NVIDIA k8s device-plugin) multiplexes in time: it advertises the GPU as “N GPUs” and the processes take turns at the compute, but they share the full physical VRAM, with no memory isolation, no fault isolation and no QoS. Its trap is an OOM that does not show up in the Kubernetes scheduler but at run time, when the sum of VRAM allocations exceeds the real memory. MPS (Multi-Process Service) multiplexes in space: it shares out the SMs between processes that execute kernels concurrently, reduces context-switch overhead and allows SM and memory limits per process. It raises throughput when there are many small kernels, but fault isolation is still weak. MIG (Multi-Instance GPU) partitions in hardware: it cuts the Hopper GPU into up to seven instances with dedicated SMs, L2, memory and bandwidth, with real memory, fault and performance isolation; datacenter only (A100/H100/H200/B200), never on an RTX 5090. The rule: real isolation / multi-tenant / compliance → MIG (if it is Hopper); many small concurrent kernels and trust between workloads → MPS; dev, bursts, consumer GPU or no need to isolate → time-slicing. This post works it through with numbers: the VRAM budget of four vLLM instances on an H100 advertised as four replicas, and what fits into a 10 GB MIG instance.

The analogy: a shared hob, one kitchen with several cooks, several kitchens

Imagine you have a single professional hob and three orders to cook at the same time. There are three ways to organise it, and they are exactly the three mechanisms.

Time-slicing is one hob taken in turns, with no larder of your own. Each cook comes in, cooks their dish, leaves, and the next one comes in. The split is temporal: nobody cooks at the same time, they take turns. The problem is not the hob, which gets shared out fine, but the common larder: the ingredients sit in a single shared cupboard and nobody has their own. If the three cooks reserve more flour than there is in total, it is not that they wait their turn: it is that there is no flour. The service goes down for everyone. And if one cook leaves a pan burning and starts a fire, it burns the whole kitchen, not just their corner.

MPS is several cooks coordinated on the same worktop. Now they do cook at the same time, sharing out the space on the worktop (the SMs). A head chef (the MPS daemon) coordinates so they do not collide and so the worktop does not sit empty while one of them waits for water to boil. You can assign each cook a percentage of the worktop and a larder limit. They work faster as a group because the worktop does not sit idle between small tasks. But they still share the kitchen: if one starts a serious fire, the others notice.

MIG is several independent kitchens in the same building. A concrete wall separates each kitchen: its own hob, its own larder, its own door and its own electrical panel. What happens in kitchen 3, a fire, an empty larder, a slow cook, does not touch kitchen 1. It is the only split with genuine isolation. The price: you have to decide in advance how many kitchens and of what size, the walls are fixed, and only expensive buildings (datacenter) come ready to put them up.

The rest of the post is, essentially, when you want cheap turns, when you want coordinated cooks and when you need concrete walls.

Why share: the operational problem

An H100 SXM 80 GB does not fill up with just any workload. A bge-reranker-v2-m3 reranker takes a few hundred MB and saturates a handful of SMs; a bge-m3 embeddings service is just as small; a 1B guardrail model in INT4 fits in a couple of GB. Dedicating 80 GB of HBM3 and 132 SMs to serving embeddings is using a hydraulic press to hammer in a drawing pin, the same argument as in mixed environments, but now inside the GPU instead of moving the workload to different silicon.

The aim of sharing is to raise the useful utilisation of fixed capital. But sharing badly introduces three pathologies:

  • Compute contention: two workloads fight over the same SMs and both run slowly, with unpredictable latency jitter.
  • Memory contention: the sum of requested VRAM exceeds the physical amount and something dies with a CUDA out of memory.
  • Cascading failure: a workload that blows up (an illegal kernel, an OOM) can drag its neighbours down with it if they share a context.

The three mechanisms attack these pathologies to different depths. None of them solves all three except MIG, and MIG costs specific hardware. Let us look at them one by one.

Time-slicing: compute turns, shared larder

Time-slicing is temporal multiplexing in software. On Kubernetes, the NVIDIA GPU Operator configures the device-plugin to advertise N replicas of each physical GPU. An H100 declared with replicas: 4 appears to the scheduler as four nvidia.com/gpu resources, and Kubernetes can place four pods on it. Internally, the GPU scheduler hands out compute turns to each process: it runs a bit of process A, switches to B, to C, to D, then back to A. It is the same time-sharing an operating system does with the CPU.

The key idea, and the one that causes most confusion, is this: a replica is NOT a fraction of the GPU. It is a compute turn. NVIDIA’s documentation is explicit: unlike MIG, there is no memory or fault isolation between replicas. The four replicas of the H100 see the full 80 GB of VRAM, unpartitioned. There is no 20 GB per replica. There are 80 GB for all four, handed out in cudaMalloc arrival order.

This has three consequences you need to internalise:

  1. It does not isolate memory. If the sum of what the four processes reserve exceeds 80 GB, the fourth cudaMalloc fails with OOM. The Kubernetes scheduler does not see it coming: it counted four available nvidia.com/gpu resources and happily placed four pods. The OOM appears at run time, not at scheduling time. This is trap number one of time-slicing.
  2. It does not isolate faults. A process that triggers an unrecoverable CUDA error can leave the GPU context in a state that affects its neighbours. They share the same device with no barriers.
  3. It gives no compute QoS. Under contention, the turn-taking does not guarantee a minimum fraction to anyone. Each workload’s latency suffers jitter proportional to how many active replicas are fighting over the GPU at that instant. A latency-sensitive workload (an interactive chat) can see its TTFT dance around depending on what its neighbours are doing.

What is it good for, then? For dev, bursts and low utilisation. If you have four developers who touch the GPU sporadically, advertising four replicas lets all four have access without fighting almost ever (they rarely overlap while active). For batch workloads that tolerate jitter. And, a decisive advantage, it works on consumer GPUs: an RTX 5090 32 GB does not support MIG, but it does support time-slicing. It is the only “Kubernetes-native” way to share a 5090 between several pods.

How each mechanism shares outTime-slicingsplit in TIMEturn Aturn Bturn Cturn Dshared VRAM: 80 GB for all 4no isolation · OOM if sum > VRAMMPSsplit in SPACE (SMs)A 40%B 30%C 20%D 10%concurrent kernels on different SMsSM and mem limit per process · weak faultMIGHARDWARE partitioning1g.10gb1g.10gb3g.40gb…dedicated SMs, L2, VRAM and BWreal isolation · concrete wallsOne hob in turns · several cooks on the worktop · several kitchens with their wallconsumer + datacenter · datacenter (CUDA) · datacenter Hopper/Ampere/Blackwell onlysoftware · software (CUDA context) · hardware (physical fuses)

The VRAM budget in time-slicing (the calculation that avoids the OOM)

Here is the maths you have to do before deploying, because Kubernetes will not do it for you. Suppose an H100 80 GB advertised as 4 replicas and we want to run four vLLM instances on it, one per replica.

vLLM reserves memory with the --gpu-memory-utilization parameter, which is the fraction of the total physical VRAM that each instance keeps (for model weights plus KV-cache). The detail that kills: that fraction is computed over the 80 physical GB, not over a supposed “20 GB of my replica”, because the replica does not have 20 GB. Remember, there is no memory partitioning. Each vLLM sees the 80 GB and reserves its fraction of them.

The no-OOM constraint is therefore that the sum of fractions be less than 1:

$$\sum_{i=1}^{N} g_i < 1 \quad\Longleftrightarrow\quad \sum_{i=1}^{N} g_i \cdot V_{\text{HBM}} < V_{\text{HBM}}$$

where $g_i$ is the --gpu-memory-utilization of instance $i$ and $V_{\text{HBM}} = 80$ GB. It is wise to leave headroom (runtime overhead, fragmentation, CUDA context), so in practice you aim for the sum to stay comfortably below 1, say $\le 0.9$.

A case that works. Four vLLM instances at $g_i = 0.20$:

$$\sum_{i=1}^{4} 0.20 = 0.80 \quad\Rightarrow\quad 0.80 \times 80\ \text{GB} = 64\ \text{GB} < 80\ \text{GB} \quad\checkmark$$

Each instance reserves $0.20 \times 80 = 16$ GB. Four instances add up to 64 GB, leaving a 16 GB cushion. There is no OOM. Each vLLM has 16 GB for weights plus KV-cache: enough for a 7B–8B model in FP8/INT4 with a modest KV-cache.

A case that blows up. The same four instances, but somebody raises $g_i = 0.30$ thinking “I have four replicas, I can give each one more”:

$$\sum_{i=1}^{4} 0.30 = 1.20 \quad\Rightarrow\quad 1.20 \times 80\ \text{GB} = 96\ \text{GB} > 80\ \text{GB} \quad\times$$

The first instances start up and reserve $0.30 \times 80 = 24$ GB each. Three instances are already at $72$ GB. The fourth tries to reserve another 24 GB, there is nothing left, and it dies with CUDA out of memory. And worse: Kubernetes will reschedule it on the same GPU (it still sees four replicas), where it will die again, in a CrashLoopBackOff that makes no sense if you only look at the pod manifest.

The operational rule is brutally simple: in time-slicing, you manage the VRAM budget by hand, adding up the --gpu-memory-utilization values. The replica count controls how many pods fit in compute turns, but it does not reserve a single byte of memory. Confusing the two is the recurring mistake.

MPS: coordinated cooks on the same worktop

The Multi-Process Service (MPS) attacks a different problem. By default, when several processes use the same GPU without MPS, each one has its own CUDA context, and the GPU alternates between contexts (time-slicing at driver level): they do not execute kernels at the same time, they take turns, with context-switch overhead. If your kernels are small and do not fill the GPU on their own, this leaves SMs idle: process A uses 30 % of the SMs during its turn and the other 70 % is wasted.

MPS introduces a daemon that shares a single CUDA context between processes, so that their kernels can execute concurrently occupying different SMs at the same time. It is spatial sharing of compute: instead of taking turns at the whole worktop, each cook occupies a part and they work in parallel. This reduces context-switch overhead and raises throughput when there are many small concurrent kernels that individually do not saturate the GPU.

And, unlike pure time-slicing, MPS allows per-process limits, which gives you a form of QoS:

  • CUDA_MPS_ACTIVE_THREAD_PERCENTAGE limits the percentage of SMs an MPS client can use. By default each client gets $100 / \text{MaxSharedClients}$. Pinning it to, say, 40 % caps that process’s compute ceiling (MPS docs).
  • CUDA_MPS_PINNED_DEVICE_MEM_LIMIT imposes a memory cap per client (valid from CUDA 11.5). This is what time-slicing does not have: a per-process VRAM limit that the runtime enforces.

These two limits turn MPS into a resource provisioning mechanism that mitigates the noisy neighbour: you can guarantee that a process does not eat more than X % of the SMs or more than Y GB. The combination gives reasonable QoS, not perfect but real.

The limitation MPS does not solve: fault isolation is weak. Since clients share the daemon’s CUDA context, a fatal error in one client can affect the daemon and therefore the other clients (historically, a client dying dirtily could require restarting the daemon). It is better than time-slicing in this respect, but a long way from hardware isolation. That is why MPS fits when there is trust between the workloads, processes from your own team, not third-party tenants.

The canonical use case: many small concurrent inference requests that individually leave the GPU half empty. MPS overlaps them and raises aggregate throughput. Serving several small models, or several light replicas of the same model, on a datacenter GPU where you trust all the workloads, is MPS territory.

MIG: concrete walls

The Multi-Instance GPU (MIG) is the only one of the three that gives genuine isolation, because it cuts the GPU in hardware. Available on modern datacenter GPUs, A100 (Ampere), H100/H200 (Hopper), B200 (Blackwell), and never on consumer ones: an RTX 5090 (consumer Blackwell) does not support MIG, nor do GeForce cards in general.

MIG divides the GPU into up to seven instances (GPU Instances), and each instance gets a dedicated portion of:

  • SMs (compute slices): compute is split into 7 slices, each one roughly 1/7 of the SMs.
  • L2 cache and memory: each instance has its slab of HBM and its portion of L2 cache.
  • Memory bandwidth: dedicated, not shared.
  • Data paths and engines: with fault barriers between instances.

The result is that a MIG instance behaves like a smaller, independent GPU: whatever happens in one, an OOM, a kernel blowing up, a workload saturating its compute, does not affect its neighbours. Memory, fault and performance (QoS) isolation, the three things time-slicing does not give and MPS only half gives.

The H100 80GB profiles

MIG does not allow arbitrary sizes: it has fixed profiles. On the H100 80GB, the profile catalogue (notation <compute>g.<memory>gb) is:

ProfileCompute (slices)MemoryMax. instances
1g.10gb1/710 GB7
1g.20gb1/720 GB4
2g.20gb2/720 GB3
3g.40gb3/740 GB2
4g.40gb4/740 GB1
7g.80gb7/780 GB1 (whole GPU)

(There is also 1g.10gb+me, a variant with media engines for video encoding.) The base memory unit on the H100 80GB is 10 GB per slice (80 GB / 8, with one slice reserved), and the compute unit is 1/7 of the SMs. The profiles combine these units. Note 1g.20gb: the same compute fraction as 1g.10gb (1/7 of the SMs) but twice the memory, useful when a workload needs more VRAM than compute.

One important detail: MIG partitions do not mix freely. The GPU is divided following a valid geometry (the profiles fit together like pieces in a grid), and the profiles are fixed when you configure the GPU; changing them requires draining and repartitioning. They are concrete walls: solid, but they do not move while hot.

The calculation: 7×1g.10gb against 1×7g.80gb

Let us compare the two extremes. On the left, seven 1g.10gb instances: seven isolated 10 GB GPUs. On the right, a single 7g.80gb: the whole H100, unpartitioned.

The operational question is what fits in 10 GB. An instance’s VRAM budget is split between model weights and KV-cache:

$$V_{\text{inst}} = V_{\text{weights}} + V_{\text{KV}} + V_{\text{overhead}}$$

Take a model of 7B parameters in FP8 (1 byte/parameter):

$$V_{\text{weights}} \approx 7 \times 10^9 \times 1\ \text{byte} = 7\ \text{GB}$$

On a 1g.10gb instance (10 GB), after the 7 GB of weights and subtracting roughly 0.5–1 GB of runtime overhead, about 2 GB are left for KV-cache. That is enough for a modest context window and low concurrency, fine for a guardrail service, a classifier or an extraction model that handles short prompts one at a time. A 7B in INT4 (~3.5 GB of weights) leaves ~5.5 GB of KV-cache, much roomier. But a 13B model in FP8 (~13 GB of weights) does not fit in a 10 GB instance: not even the weights get in. For that you need 1g.20gb, 2g.20gb or larger.

Against this, the 7g.80gb (whole GPU) gives you the 80 GB for one large model: a 70B in FP8 (~70 GB of weights) fits with a tight KV-cache, or a 70B with more room spread across several whole GPUs in tensor-parallel (see TP against replicas: one big one versus N small ones).

The reading is clear: fine partitioning (7×1g.10gb) maximises the number of small isolated workloads; not partitioning (1×7g.80gb) maximises the model size that fits. The KV-cache available per instance shrinks proportionally as you partition, so fine-grained MIG serves many light isolated services, not one large model chopped up. If your workload is a single large model, MIG is not for you: use the whole GPU or several in TP.

The decision tree

The three questions, in order:

Do you need REAL isolation?
(multi-tenant, compliance, one workload failing must not touch another)
        │
   ┌────┴────┐
  YES       NO
   │         │
Is it Hopper/ Many SMALL concurrent kernels
 Ampere/      AND you trust every workload?
 Blackwell?         │
   │           ┌────┴────┐
┌──┴──┐       YES       NO
YES  NO        │         │
│     │       MPS     Dev / bursts / consumer GPU
MIG   │     (spatial,  / no need to isolate?
│     │      per-process     │
│   no real   QoS)          YES
│   isolation:               │
│   rethink              TIME-SLICING
│   (move to            (temporal, cheap,
│   CPU, another         works on a 5090)
│   GPU, or accept
│   the risk with
│   time-slicing)

And each branch in one sentence:

  • MIG when isolation is a requirement (compliance, multi-tenant, hard SLA) and you have datacenter hardware that supports it. Concrete walls cost money, but if you need them there is no substitute.
  • MPS when you have many small concurrent workloads that leave the GPU half empty and you trust all of them (same team, not third-party tenants). You raise throughput with reasonable QoS, accepting imperfect fault isolation.
  • Time-slicing when it is dev, bursts, low utilisation, a consumer GPU, or you simply do not need to isolate anything. Cheap and universal, but manage the VRAM budget by hand.

One nuance the recent documentation records: they can be combined. You can do time-slicing on top of a MIG instance (hardware isolation at the instance boundary, software turns inside), or use MPS inside a MIG instance. The layers are not mutually exclusive; the tree picks the primary strategy.

Applied to the generic 4×H100 cluster

Let us get down to numbers with a generic on-premise cluster of 4×H100 SXM 80 GB with NVLink. It is common to have a heterogeneous menu of workloads: a large chat model, light services (embeddings, reranker, guardrails) and a dev/experimentation queue. Each type calls for a different mechanism. A reasoned split:

GPU 0 and GPU 1, large model in tensor-parallel (not shared). A 70B model in FP8 takes ~70 GB of weights; served comfortably with a generous KV-cache it needs more than one H100. We spread it in tensor-parallel over two whole H100s joined by NVLink (intra-node bandwidth is what makes TP viable; the detail is in TP against replicas). Here we do not share: these two GPUs belong to the large model, full stop. Total isolation by dedication.

$$V_{\text{available}} = 2 \times 80 = 160\ \text{GB};\quad V_{\text{weights}} \approx 70\ \text{GB};\quad V_{\text{KV}} \approx 80\ \text{GB of KV-cache}$$

There is memory to spare for a long request queue and high concurrency.

GPU 2, split with MIG into small instances for light services. Embeddings (bge-m3), reranker (bge-reranker-v2-m3) and a couple of guardrail models (1B–3B) are different workloads, potentially from different teams, and you want a failure or a spike in one not to touch the others. Light multi-tenancy with isolation → MIG. A reasonable partitioning of the H100:

$$\underbrace{3 \times \texttt{1g.10gb}}_{\text{30 GB, 3/7 SMs}} \;+\; \underbrace{1 \times \texttt{4g.40gb}}_{\text{40 GB, 4/7 SMs}}$$

The three 1g.10gb instances (10 GB, 1/7 of the SMs each) host embeddings, reranker and a 1B INT4 guardrail, each isolated, with no noisy neighbour. The 4g.40gb (40 GB, 4/7 of the SMs) hosts an intermediate 7B–13B model with a decent KV-cache for a support service. Each service has its larder and its wall; if the reranker blows up, the chat never finds out.

GPU 3, time-slicing for dev and bursts. The developers touch the GPU sporadically: experiments, short fine-tunes, model trials. They do not need isolation (it is the same team) and they rarely overlap while active. We advertise it as 4 replicas via the device-plugin. Four dev pods fit in turns. VRAM budget with the formula above: if each dev brings up a vLLM at --gpu-memory-utilization 0.20, they add up to $4 \times 16 = 64$ GB < 80 GB, with no OOM. If somebody needs more, lower the replica count or coordinate with the team. The cost of flexibility is manual discipline.

Summary of the split:

ResourceMechanismWorkloadIsolation
GPU 0 + GPU 1Dedication (TP)70B chat in tensor-paralleltotal (dedicated)
GPU 2MIG (3×1g.10gb + 1×4g.40gb)embeddings, reranker, guardrails, 7B–13Breal hardware
GPU 3Time-slicing (4 replicas)dev, bursts, experimentsnone (trust)

The logic is always the same: spend isolation (MIG) where you need it, spend cheap concurrency (time-slicing) where you do not, and reserve whole GPUs for what genuinely fills them. An H100 serving embeddings on a 7g.80gb would be as absurd as an RTX 5090 trying MIG: the tool does not match the workload.

What we have not covered

  • What happens when they do not even fit at the same time: if you have more models than VRAM and they have to take turns in memory (loading/unloading weights, not just taking turns at compute), you enter swap and sleep territory, the sibling piece Serving several models on one GPU.
  • NUMA-aware scheduling: on multi-socket nodes, which GPU goes with which CPU/memory matters for latency; see Kubelet resource managers on RKE2.
  • Autoscaling the replicas: how many instances to bring up according to real load, with KEDA and queue metrics; see Autoscaling LLMs on Kubernetes with KEDA.
  • A jitter benchmark under contention: how much TTFT really wobbles in time-slicing with 4 active replicas compared with MIG, material that deserves its own measurement, not an estimate.

See also

References