Vram on lo0 — Tech Blog

Serving several models on a single GPU: co-residency, model-swapping and sleep mode

Thu, 11 Jun 2026 02:40:00 +0000

This is the second piece of an operational series on squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB with NVLink. Its sibling, Sharing a GPU: time-slicing, MPS and MIG, splits the GPU so that several processes can use it at once; this one splits the complementary problem: you have more models than VRAM and you need them to coexist in time, not just in space. The third, The RAG data plane on CPU, moves off the GPU everything that does not need to be there; and the end-to-end sovereign assistant (fourth instalment, in preparation) assembles all of this behind LibreChat and LiteLLM. Here we assume that VRAM is the scarce resource and that keeping everything loaded at once is not an option.

TL;DR

You have an agent LLM serving almost all the traffic, one or two rerankers for RAG, an alternate model (another family, another language, a fine-tune) and, now and then, a large model for hard tasks. Added up, they do not fit in the 80 GB of an H100. There are three ways to make them coexist, and the key to not confusing them is to understand that they are three different states of where the weights live:

Co-residency: several models loaded at the same time in HBM, if the sum of their VRAM budgets fits in the physical memory. It is the ideal when they fit: zero switching latency. The catch: each engine must cap its own VRAM; if two engines both believe they own the whole 80 GB, the second one to ask for memory dies with OOM. On a GPU shared by time-slicing, sharing time does not protect memory (the processes still compete for the same HBM) and this problem gets worse, as we see in the sibling piece.
Model-swapping (with llama-swap): only one (or very few) resident at a time. The proxy looks at the request’s model field and, if another one is needed, unloads the current model and loads the requested one. Since there is only one inside, each model can use almost the whole GPU. The cost is a full cold start: seconds from local NVMe, minutes from network storage. It mixes engines: it exposes OpenAI-compatible endpoints including /rerank with llama-server on GGUF.
vLLM sleep mode (--enable-sleep-mode, endpoints /sleep and /wake_up): the model is not unloaded, it is put to sleep. The weights are parked in host RAM (level 1) or discarded while the process stays alive (level 2). The wake is 18–200× faster than a cold start because the process is still alive and keeps the CUDA allocator, the CUDA graphs and the compiled JIT kernels. The only thing rebuilt is the KV cache, which is discarded on sleep. It needs enough RAM for the sleeping weights.

The decision tree is short. Do they all fit? Co-residency. They do not fit, switching is infrequent and you mix engines? llama-swap. Everything is vLLM, there is RAM to spare and wake latency matters? Sleep mode. Sustained load on all of them at once? That is not swapping, it is more GPUs or replicas (see one big vs N small). This post puts numbers on each of them on an 80 GB H100.

The analogy: one stage, one spotlight

Picture a theatre with a single stage and a single spotlight, the GPU. You have more actors (models) in the company than fit under the light at once. There are three ways to stage the show:

Co-residency: several small actors on stage. If the cast for this scene is three slight actors, the main LLM, one reranker, another reranker, they all fit under the spotlight at the same time. Switching lines between them is instantaneous: they are already on stage. But the stage has a fixed size; add a fourth, burly actor (the large model) and they do not fit, so someone falls off the edge. That is the OOM.
Model-swapping: the actor goes home. When an actor finishes their part, they go home (the disk). If the next scene needs them again, they have to come back from home: get dressed, put the make-up on, travel to the theatre. That takes minutes. In exchange, while they are on stage they have the whole stage to themselves. That is the llama-swap model: maximum VRAM per actor, but re-entry costs a full round trip.
Sleep mode: the actor waits in the wings. Instead of going home, the actor steps out of the spotlight but stays backstage (host RAM), already dressed and made up. When their turn comes, they step in in two strides. No journey, no wardrobe: just crossing the curtain. That is the vLLM wake: the weights come back not from disk but from RAM, and everything that goes into “preparing the actor”, the allocator, the CUDA graphs, the kernels, is already done because they never left the building.

The operational moral is the same as in the theatre: the cost of an actor is not only their talent, it is how long they take to walk on when you need them. Co-residency pays for the permanent spot; swapping pays for the journey; sleep pays for the RAM of keeping them waiting nearby. The rest of this post is about choosing which, with a VRAM budget and a stopwatch in hand.

The three states: where the weights live

Co-residency: the VRAM budget

Co-residing is the default option when they fit: zero switching latency because every model is already in HBM. The whole question is do they fit?, and it is answered with a VRAM budget. The VRAM of an inference engine splits, roughly, into three items:

$$\text{VRAM}_{\text{model}} = \underbrace{P \cdot b}_{\text{weights}} + \underbrace{\text{KV}}_{\text{cache}} + \underbrace{A}_{\text{activations + overhead}}$$

where $P$ is the number of parameters, $b$ the bytes per parameter given the quantisation, KV the KV-cache budget (which scales with concurrency and context length) and $A$ the overhead of activations, CUDA buffers and fragmentation. The hard rule of co-residency is:

$$\sum_i \text{VRAM}_{\text{model } i} + \text{headroom} \;<\; \text{VRAM}_{\text{physical}}$$

If the sum overflows, no sharing scheme saves you: the first engine to ask for memory beyond the free gap dies with OOM. And here is the critical nuance that connects to the sibling piece: splitting the GPU by time does not split the memory. Under time-slicing, two processes take turns on compute but share the whole HBM with no isolation; if both assume they own the 80 GB, they collide. Only MIG gives memory partitions with a real boundary, as we see in Sharing a GPU. To co-reside properly, each engine must cap itself: in vLLM, --gpu-memory-utilization 0.45 tells it “do not use more than 45 % of the GPU”; in llama-server, you control the layers on GPU and the KV size. If you do not set those limits, vLLM claims 90 % of the GPU for itself by default, and there is no room left for anyone else.

A worked example: what fits in 80 GB

Take the realistic scenario. A 32B agent LLM in FP8 ($b = 1$ byte/param):

$$P \cdot b = 32 \times 10^9 \cdot 1 \text{ B} = 32 \text{ GB of weights}$$

On top of that comes the KV cache. To serve with decent concurrency and agent contexts (which are long: history, tools, documents), a KV budget of ~12 GB is reasonable, plus some ~2 GB of activations and overhead. Total for the main LLM: ~46 GB. That leaves ~34 GB of the 80. Let us see what fits there:

Service	Size (FP8/INT8)	KV + overhead	Total VRAM
32B agent LLM	32 GB	~14 GB	~46 GB
Reranker A (cross-encoder ~0.5B)	~0.5 GB	~0.5 GB	~1 GB
Reranker B (cross-encoder ~0.5B)	~0.5 GB	~0.5 GB	~1 GB
8B alternate model	~8 GB	~4 GB	~12 GB
Sum			~60 GB
Free headroom			~20 GB

With this they co-reside comfortably: the 32B LLM, the two rerankers (which are featherweights, a few hundred MB to 1 GB each) and even an 8B alternate, leaving 20 GB of cushion. Rerankers are the textbook case for co-residency: so small that they always fit alongside the main LLM, and rotating them would be absurd. Switching between them is instantaneous because they are all loaded.

When does it break? When the occasional large model arrives, a 70B in FP8:

$$70 \times 10^9 \cdot 1 \text{ B} = 70 \text{ GB of weights}$$

The 70B’s weights alone already eat 70 of the 80 GB. There is no way to co-reside it with the 32B, not remotely. That is where co-residency ends and rotation begins: the 70B can only come in if we evict the 32B from the GPU. That is the exact boundary: co-reside while the sum of budgets fits with headroom; rotate as soon as one model needs, on its own, more VRAM than is left over.

Model-swapping with llama-swap: the actor goes home

When they do not fit at once, the first answer is to rotate: keep one resident model and change it on demand. The canonical tool for this in the on-premise world is llama-swap, a Go proxy that sits in front of your inference servers (llama.cpp, vLLM, TabbyAPI and so on) and starts and stops them as needed.

The mechanism is elegant in its simplicity. Every OpenAI-compatible request carries a model field. llama-swap reads that field, looks at which upstream server it has configured for that model, and:

If the requested model is already loaded, it routes the request straight through.
If another one is loaded, it stops it (freeing its VRAM) and starts the right one.
When the new server reports ready, it forwards the request.

The cost of a swap is exactly a full cold start of the new model: unloading the current one (fast, just freeing memory) plus loading the new one (slow, moving the weights from disk to HBM). That second term is the painful one, and its magnitude is decided by where the weights are: from local NVMe it is seconds; from network storage (Ceph RGW, NFS) it can be minutes. The whole analysis of the load path, and why the default loader makes it slow, is in From disk to HBM; here the consequence is enough: a swap is only viable if it is infrequent, because every switch pays that toll in full.

The advantage in exchange: since there is only one resident, that model can use almost the whole GPU. The 70B that co-resides with nobody fits comfortably if it is the only one inside. And llama-swap mixes engines: you can have a vLLM configured for the big LLM, a llama-server with GGUF for the alternate, and two llama-server instances for the rerankers, which expose /rerank, /v1/rerank and /v1/reranking natively. That heterogeneity (GGUF + rerank + vLLM behind a single OpenAI endpoint) is exactly what vLLM alone does not give you, and the main reason to choose llama-swap.

Example ConfigMap: two rerankers rotating on the same port

A concrete and useful case: you have two rerankers, one multilingual and one specialised in code, that you do not need at the same time and would rather not keep both resident. With llama-swap they rotate on the same endpoint, triggered by the model field. The ConfigMap (mounted as the proxy’s config.yaml in a Kubernetes deployment) would be:

apiVersion: v1
kind: ConfigMap
metadata:
 name: llama-swap-rerankers
data:
 config.yaml: |
 # How long a model stays loaded after the last request
 # before llama-swap unloads it to free VRAM
 healthCheckTimeout: 60

 models:
 # Multilingual reranker (GGUF, via llama-server)
 "reranker-multilang":
 cmd: >
 /usr/bin/llama-server
 --model /models/bge-reranker-v2-m3.Q8_0.gguf
 --reranking
 --port ${PORT}
 --n-gpu-layers 99
 --ctx-size 8192
 # ttl: after 300 s idle, it is unloaded and the GPU is freed
 ttl: 300

 # Code reranker (GGUF, via llama-server)
 "reranker-code":
 cmd: >
 /usr/bin/llama-server
 --model /models/codereranker.Q8_0.gguf
 --reranking
 --port ${PORT}
 --n-gpu-layers 99
 --ctx-size 8192
 ttl: 300

A request to /v1/rerank with "model": "reranker-multilang" starts that server; the next one with "model": "reranker-code" stops the multilingual one and starts the code one on the same ${PORT} that llama-swap manages. Since both are small, the swap between them takes one or two seconds: quantised GGUFs weigh a few hundred MB and come from local NVMe. The ttl controls how long a model stays loaded after its last request: raising it avoids swaps if requests arrive in bursts; lowering it frees the GPU sooner for other uses.

An honest caveat: this pattern of two rerankers that rotate makes sense when they do not fit alongside the rest, or when you want to reserve the VRAM for something else. If they do fit (and a couple of 0.5 GB rerankers almost always fit, as we saw above), co-residing them is strictly better: zero swap latency. llama-swap shines when you rotate large models or mix engines that do not coexist well, not for juggling tiny models that would fit together.

vLLM sleep mode: the actor in the wings

Swapping has a problem: every switch pays the entire cold start, and a cold start is not just moving weights. As we saw in From disk to HBM, starting an inference engine includes initialising the Python process and the CUDA context, setting up the memory allocator, capturing the CUDA graphs and compiling the JIT kernels (DeepGEMM, FlashInfer, TorchInductor). Moving the weights is only one of five items, and often not the largest.

vLLM’s sleep mode (--enable-sleep-mode) attacks exactly that: instead of killing the process when a model stops being used, it puts it to sleep, leaving the process alive. There are two levels, and the difference is where the weights go:

Level 1: offloads the weights to host RAM (CPU) and discards the KV cache. The process stays alive. The wake copies the weights from RAM to VRAM, not from disk. Typical wake: ~0.1–0.8 s for small models, ~3–6 s for large ones. It needs enough RAM for the sleeping weights (on the order of the model’s size in GB).
Level 2: discards the weights entirely, keeping only small buffers (rope scaling tensors and the like). The wake does reload the weights from disk, but everything else, process, allocator, CUDA graphs, JIT kernels, is already done. Typical wake: ~0.8–2.6 s for small models. RAM use is almost nil (megabytes).

The key that explains the numbers: at both levels, keeping the process alive preserves the expensive infrastructure. That is why the benchmark in the vLLM blog (Oct 2025) reports that a wake is 18–200× faster than a full reload, and, most counter-intuitively, level 2 is still 23–45× faster than a cold start despite reloading the weights from the same disk, because it skips the other four items. In their measurements on an A100, a full cycle of 5 model switches goes from 357 s without sleep (≈48 s per switch) to 112 s with level 1 (wake of 0.26 s / 0.82 s) or 125 s with level 2 (0.85 s / 2.58 s).

The KV cache is always discarded on sleep. That is not a minor detail: it means the first response after the wake rebuilds the KV from scratch, paying a full prefill. That is why the wake is not “entirely free”: the model is available in sub-second time, but the first request is a little slower until the KV is repopulated. A one-request warm-up after the wake hides that cost.

# Start vLLM with sleep mode (admin endpoints, trusted networks only)
export VLLM_SERVER_DEV_MODE=1
vllm serve <model> --enable-sleep-mode --port 8001

# Sleep (level 1: weights to host RAM)
curl -X POST 'localhost:8001/sleep?level=1'

# Wake up
curl -X POST 'localhost:8001/wake_up'

Security warning (from vLLM’s own blog): the /sleep, /wake_up, /collective_rpc and /reset_prefix_cache endpoints require VLLM_SERVER_DEV_MODE=1 and should only be exposed on trusted networks, since they can take the service down. They are for internal orchestration (a controller that sleeps and wakes models according to the queue), not for the public plane.

The latency maths: why the wake wins

Let us put numbers on “the wake comes from RAM, not from disk”. The cost of having a model available is, in each strategy, the time to move its weights from wherever they are to HBM, plus, in the swap case, the other four cold-start costs. Take the 34 GB of weights of the 32B LLM in FP8 and compare the three paths.

Cold start from NVMe (swap). A reasonable Gen4/Gen5 NVMe gives on the order of ~5 GB/s effective per stream with the default loader (the disk’s theoretical floor is higher, but single-threaded deserialisation does not saturate it, see From disk to HBM). For 34 GB:

$$t_{\text{NVMe→HBM}} \approx \frac{34 \text{ GB}}{5 \text{ GB/s}} \approx 6.8 \text{ s just moving bytes}$$

And that is before adding CUDA graph capture and kernel compilation, which add several more seconds. The real cold start of a 32B is around 15–40 s depending on loader and storage. From the network (Ceph RGW), multiply.

Level 1 wake from RAM. The weights come not from disk but from host RAM, and travel over PCIe Gen5 x16, whose practical host→GPU bandwidth is ~50–64 GB/s. For the same 34 GB:

$$t_{\text{RAM→HBM}} \approx \frac{34 \text{ GB}}{55 \text{ GB/s}} \approx 0.62 \text{ s}$$

And there is nothing else to pay: the allocator, the graphs and the kernels are already there. The real wake of a model this size lands in the sub-second to a few seconds range that vLLM reports. The speed-up against an NVMe cold start is on the order of:

$$\frac{t_{\text{cold start}}}{t_{\text{wake}}} \approx \frac{15\text{–}40 \text{ s}}{0.6\text{–}3 \text{ s}} \approx 10\text{–}60\times$$

consistent with the 18–200× from the blog, whose measurements include smaller models, where the relative weight of the preserved CUDA graphs is even larger and the factor goes up.

Why the bandwidth difference explains almost everything. The key jump is not 5 vs 55 GB/s (a ~11× in transport). It is that the cold start also pays for rebuilding infrastructure, which the wake skips entirely. The table:

Path	Weight source	Bandwidth	34 GB bytes only	+ CUDA graphs / JIT	Realistic total
Cold start (swap) NVMe	disk	~5 GB/s	~6.8 s	yes (several s)	15–40 s
Cold start (swap) network	network	~1–2 GB/s	17–34 s	yes	30 s – min
Level 1 wake	host RAM	~50–64 GB/s	~0.6 s	no (preserved)	0.6–3 s
Level 2 wake	disk	~5 GB/s	~6.8 s	no (preserved)	7–10 s

Look at the level 2 row: it reloads the weights from the same disk as the swap (~6.8 s of bytes), but since it does not rebuild graphs or kernels, its total (~7–10 s) still beats a full cold start (15–40 s). It is the proof that moving bytes is only part of the cost, and the part sleep mode exploits.

Assumptions, honestly: the bandwidths are indicative. NVMe “5 GB/s effective” assumes the default loader; with a concurrent streamer it goes up. PCIe “55 GB/s” assumes Gen5 x16 with a pinned buffer and NUMA locality; if the buffer lands on the wrong socket, it drops. And the “15–40 s” cold-start range depends on the model, the quantisation and whether the files are in page cache (the “the second time it loaded fast” trap). The numbers are there to reason about orders of magnitude, not to size anything without measuring on your hardware.

The decision tree

The three strategies do not compete: each wins in a different regime. The tree, in order:

The node most often ignored is the one at the top right: “sustained load on all of them at once?”. If your four models get constant and simultaneous traffic, neither swapping nor sleep helps, since both assume the models take turns in time. Rotating under sustained load only adds switching latency without solving the underlying problem: there is not enough compute. The answer then is to scale horizontally (more replicas) or spread across more GPUs, a capacity decision analysed in One big vs N small. Swapping and sleep are tools for temporally unbalanced workloads: many models, but rarely active at the same time.

Applied to the generic 4×H100 cluster

Let us bring this down to the 4 H100 SXM 80 GB with NVLink. The winning strategy is not to pick one of the three, but to spread the models across GPUs according to their usage pattern and apply to each GPU the strategy it calls for. A reasonable split:

H100 #0, the workhorse (co-residency). The 32B agent LLM (main service, constant traffic) co-resides with the two rerankers and, if they fit, the 8B alternate. It is the GPU that never rotates: everything living here is used continuously and fits comfortably in 80 GB (the ~60 GB of the example above). Zero switching latency between the LLM and its rerankers, which is exactly what RAG needs, since a fast reranker is useless if you have to wait for a swap every time.

H100 #1, light services with MIG. If you have many small heterogeneous services, an embedding model, a classifier, a guardrail, an STT/TTS, splitting this GPU with MIG into isolated instances (each with its slice of HBM with a real boundary) gives co-residency with memory isolation, preventing a service that inflates its KV from taking the others down. The detail of when MIG beats time-slicing is in Sharing a GPU; the rule here: co-residing light services on one GPU makes sense when they fit and when isolating them is worthwhile, and MIG is the tool for the second part.

H100 #2, the occasional large model (sleep mode or swap). The 70B that is only invoked for hard tasks does not deserve a dedicated GPU kept awake, since it would sit idle most of the time, burning 700 W for nothing. Two options:

If this GPU also serves a medium model routinely and you only occasionally need the 70B, use sleep mode: put the medium one to sleep (level 1, weights to RAM), wake the 70B. Note that level 1 sleep frees the VRAM, the weights go to RAM, so yes, it fits even though the 70B would not fit alongside an awake medium model. Waking the medium one again is sub-second.
If the 70B comes as GGUF or you mix engines, llama-swap rotates between the medium model and the 70B by the model field. Each 70B invocation pays its cold start (seconds from local NVMe), acceptable if it is occasional.

H100 #3, replica / overflow. The fourth GPU absorbs peaks: a replica of the main LLM for when #0’s queue grows, or reserve capacity. There is no swapping here: it is pure capacity, the answer to the tree’s “sustained load” node.

The cross-cutting principle: co-reside what is used together and continuously (LLM + rerankers); isolate the light and heterogeneous with MIG; sleep or rotate the large and occasional; replicate what saturates. The four GPUs do not all do the same thing, each runs the strategy its load pattern calls for. And the NVLink between them matters for something else (tensor parallelism for the 70B if it would not fit even on one; see One big vs N small), but for this post’s problem, many models, one GPU, the lever is when each model needs to be awake.

Traps and things that are not what they seem

“Co-residing is always better if they fit.” Almost, but watch the KV cache: co-residing two models means splitting the KV budget between them. If the main LLM needs a large KV for high concurrency and long contexts, adding a flatmate cuts that KV and lowers its throughput. Sometimes it is better to give the whole GPU to the main model and rotate the secondary one. Co-residency is not free: the tenant takes room away from the cache of the one that matters.

“Sleep mode is like swapping but faster.” Not exactly. Swapping releases the process; you can have N models configured and only pay RAM/disk for the resident one. Sleep mode keeps one live process per sleeping model, so each sleeping vLLM still occupies its process slot, its RAM (level 1) and its management footprint. Sleep scales well to a few rotating models; for many (10+), level 2 (minimal RAM) or plain swapping fit better. Do not put 15 models into level 1 sleep and expect the RAM to hold.

“The wake is instantaneous, I lose nothing.” The model’s wake is sub-second, but the KV cache was discarded on sleep. The first request after the wake pays a full prefill to repopulate the KV, slower than normal. If your SLA is strict on the first response after an idle period, add an automatic warm-up after the wake. Prefix caching helps make that reprefill cheaper if there are stable prefixes.

“llama-swap with a low ttl saves me VRAM for free.” It saves you VRAM while nobody uses that model, but every time it comes back it pays the cold start. An aggressive ttl on a model with bursty traffic turns every burst into a load wait. The right ttl depends on the temporal pattern of the requests, not on how much VRAM you want to free. Measure it.

“Time-slicing lets me co-reside more models.” False and dangerous. Time-slicing shares out compute time, not memory, and all the processes still compete for the same HBM with no isolation. Co-residing via time-slicing does not give you more effective VRAM; it gives you more processes fighting over the same VRAM, and an OOM when the sum overflows. For real memory partitioning, MIG. The detail is in the sibling piece.

Conclusion

Having more models than VRAM is not a problem of insufficient hardware: it is a problem of managing a scarce resource over time. The intuition of “I need one GPU per model” is expensive and almost always wrong, because models are rarely used all at once. The three strategies are three answers to the same question, where do the weights of the models you are not currently using live: in HBM if they fit (co-residency, zero latency but they cost space), on disk if switching is rare (swapping, maximum VRAM per model but a return trip of seconds to minutes), or in RAM if switching latency matters (sleep mode, sub-second wake at the cost of tied-up RAM). Sleep mode is the most interesting addition of 2025 because it breaks the false dilemma of “everything loaded vs reload every time”: by keeping the process alive and preserving the allocator, the CUDA graphs and the kernels, it turns a 30–100 s cold start into a sub-second wake, and it does so even when it reloads the weights from the same disk (level 2), because moving bytes was never the whole cost. On the four-H100 cluster, the play is not to pick one strategy but to spread: co-reside what goes together, isolate the light stuff with MIG, sleep or rotate the large and occasional, replicate what saturates. The GPU is the stage with a single spotlight; the art is knowing which actor walks on, which goes home and which waits in the wings.

References

vLLM Blog (Embedded LLM), Zero-Reload Model Switching with vLLM Sleep Mode, 26 oct 2025: https://blog.vllm.ai/2025/10/26/sleep-mode.html
vLLM Docs, Sleep Mode: https://docs.vllm.ai/en/latest/features/sleep_mode/
mostlygeek, llama-swap (OpenAI/Anthropic-compatible model-swapping proxy): https://github.com/mostlygeek/llama-swap
llama-swap, Configuration: https://github.com/mostlygeek/llama-swap/blob/main/docs/configuration.md
NVIDIA, H100 Tensor Core GPU (HBM3 specs, 80 GB, ~3.35 TB/s): https://www.nvidia.com/en-us/data-center/h100/
NVIDIA, Reducing Cold Start Latency for LLM Inference with NVIDIA Run:ai Model Streamer: https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer/

Sharing one GPU between several workloads: time-slicing, MPS and MIG

Thu, 11 Jun 2026 02:00:00 +0000

This post opens an operational series on how to squeeze a generic on-premise 4×H100 SXM LLM cluster. The sibling pieces: Serving several models on one GPU: swap and sleep (what to do when the models do not fit at the same time and have to take turns in memory), RAG on CPU: separating the data plane from generation (moving retrieval off the GPU to free it up) and Sovereign end-to-end assistant with LibreChat, LiteLLM and RAG, the final assembly, a fourth instalment in preparation. Here we start with the most basic question: you have one GPU and you want to put several workloads on it. How do you split it?

TL;DR

You have one GPU, or a few, and several workloads that want to run on it: a chat model, an embeddings service, a reranker, a queue of dev jobs. The GPU is underused if it only runs one thing, but piling several on carelessly causes contention, OOM or cascading failures. There are three mechanisms and they share out different things. Time-slicing (replicas from the NVIDIA k8s device-plugin) multiplexes in time: it advertises the GPU as “N GPUs” and the processes take turns at the compute, but they share the full physical VRAM, with no memory isolation, no fault isolation and no QoS. Its trap is an OOM that does not show up in the Kubernetes scheduler but at run time, when the sum of VRAM allocations exceeds the real memory. MPS (Multi-Process Service) multiplexes in space: it shares out the SMs between processes that execute kernels concurrently, reduces context-switch overhead and allows SM and memory limits per process. It raises throughput when there are many small kernels, but fault isolation is still weak. MIG (Multi-Instance GPU) partitions in hardware: it cuts the Hopper GPU into up to seven instances with dedicated SMs, L2, memory and bandwidth, with real memory, fault and performance isolation; datacenter only (A100/H100/H200/B200), never on an RTX 5090. The rule: real isolation / multi-tenant / compliance → MIG (if it is Hopper); many small concurrent kernels and trust between workloads → MPS; dev, bursts, consumer GPU or no need to isolate → time-slicing. This post works it through with numbers: the VRAM budget of four vLLM instances on an H100 advertised as four replicas, and what fits into a 10 GB MIG instance.

The analogy: a shared hob, one kitchen with several cooks, several kitchens

Imagine you have a single professional hob and three orders to cook at the same time. There are three ways to organise it, and they are exactly the three mechanisms.

Time-slicing is one hob taken in turns, with no larder of your own. Each cook comes in, cooks their dish, leaves, and the next one comes in. The split is temporal: nobody cooks at the same time, they take turns. The problem is not the hob, which gets shared out fine, but the common larder: the ingredients sit in a single shared cupboard and nobody has their own. If the three cooks reserve more flour than there is in total, it is not that they wait their turn: it is that there is no flour. The service goes down for everyone. And if one cook leaves a pan burning and starts a fire, it burns the whole kitchen, not just their corner.

MPS is several cooks coordinated on the same worktop. Now they do cook at the same time, sharing out the space on the worktop (the SMs). A head chef (the MPS daemon) coordinates so they do not collide and so the worktop does not sit empty while one of them waits for water to boil. You can assign each cook a percentage of the worktop and a larder limit. They work faster as a group because the worktop does not sit idle between small tasks. But they still share the kitchen: if one starts a serious fire, the others notice.

MIG is several independent kitchens in the same building. A concrete wall separates each kitchen: its own hob, its own larder, its own door and its own electrical panel. What happens in kitchen 3, a fire, an empty larder, a slow cook, does not touch kitchen 1. It is the only split with genuine isolation. The price: you have to decide in advance how many kitchens and of what size, the walls are fixed, and only expensive buildings (datacenter) come ready to put them up.

The rest of the post is, essentially, when you want cheap turns, when you want coordinated cooks and when you need concrete walls.

An H100 SXM 80 GB does not fill up with just any workload. A bge-reranker-v2-m3 reranker takes a few hundred MB and saturates a handful of SMs; a bge-m3 embeddings service is just as small; a 1B guardrail model in INT4 fits in a couple of GB. Dedicating 80 GB of HBM3 and 132 SMs to serving embeddings is using a hydraulic press to hammer in a drawing pin, the same argument as in mixed environments, but now inside the GPU instead of moving the workload to different silicon.

The aim of sharing is to raise the useful utilisation of fixed capital. But sharing badly introduces three pathologies:

Compute contention: two workloads fight over the same SMs and both run slowly, with unpredictable latency jitter.
Memory contention: the sum of requested VRAM exceeds the physical amount and something dies with a CUDA out of memory.
Cascading failure: a workload that blows up (an illegal kernel, an OOM) can drag its neighbours down with it if they share a context.

The three mechanisms attack these pathologies to different depths. None of them solves all three except MIG, and MIG costs specific hardware. Let us look at them one by one.

Time-slicing: compute turns, shared larder

Time-slicing is temporal multiplexing in software. On Kubernetes, the NVIDIA GPU Operator configures the device-plugin to advertise N replicas of each physical GPU. An H100 declared with replicas: 4 appears to the scheduler as four nvidia.com/gpu resources, and Kubernetes can place four pods on it. Internally, the GPU scheduler hands out compute turns to each process: it runs a bit of process A, switches to B, to C, to D, then back to A. It is the same time-sharing an operating system does with the CPU.

The key idea, and the one that causes most confusion, is this: a replica is NOT a fraction of the GPU. It is a compute turn. NVIDIA’s documentation is explicit: unlike MIG, there is no memory or fault isolation between replicas. The four replicas of the H100 see the full 80 GB of VRAM, unpartitioned. There is no 20 GB per replica. There are 80 GB for all four, handed out in cudaMalloc arrival order.

This has three consequences you need to internalise:

It does not isolate memory. If the sum of what the four processes reserve exceeds 80 GB, the fourth cudaMalloc fails with OOM. The Kubernetes scheduler does not see it coming: it counted four available nvidia.com/gpu resources and happily placed four pods. The OOM appears at run time, not at scheduling time. This is trap number one of time-slicing.
It does not isolate faults. A process that triggers an unrecoverable CUDA error can leave the GPU context in a state that affects its neighbours. They share the same device with no barriers.
It gives no compute QoS. Under contention, the turn-taking does not guarantee a minimum fraction to anyone. Each workload’s latency suffers jitter proportional to how many active replicas are fighting over the GPU at that instant. A latency-sensitive workload (an interactive chat) can see its TTFT dance around depending on what its neighbours are doing.

What is it good for, then? For dev, bursts and low utilisation. If you have four developers who touch the GPU sporadically, advertising four replicas lets all four have access without fighting almost ever (they rarely overlap while active). For batch workloads that tolerate jitter. And, a decisive advantage, it works on consumer GPUs: an RTX 5090 32 GB does not support MIG, but it does support time-slicing. It is the only “Kubernetes-native” way to share a 5090 between several pods.

The VRAM budget in time-slicing (the calculation that avoids the OOM)

Here is the maths you have to do before deploying, because Kubernetes will not do it for you. Suppose an H100 80 GB advertised as 4 replicas and we want to run four vLLM instances on it, one per replica.

vLLM reserves memory with the --gpu-memory-utilization parameter, which is the fraction of the total physical VRAM that each instance keeps (for model weights plus KV-cache). The detail that kills: that fraction is computed over the 80 physical GB, not over a supposed “20 GB of my replica”, because the replica does not have 20 GB. Remember, there is no memory partitioning. Each vLLM sees the 80 GB and reserves its fraction of them.

The no-OOM constraint is therefore that the sum of fractions be less than 1:

$$\sum_{i=1}^{N} g_i < 1 \quad\Longleftrightarrow\quad \sum_{i=1}^{N} g_i \cdot V_{\text{HBM}} < V_{\text{HBM}}$$

where $g_i$ is the --gpu-memory-utilization of instance $i$ and $V_{\text{HBM}} = 80$ GB. It is wise to leave headroom (runtime overhead, fragmentation, CUDA context), so in practice you aim for the sum to stay comfortably below 1, say $\le 0.9$.

A case that works. Four vLLM instances at $g_i = 0.20$:

$$\sum_{i=1}^{4} 0.20 = 0.80 \quad\Rightarrow\quad 0.80 \times 80\ \text{GB} = 64\ \text{GB} < 80\ \text{GB} \quad\checkmark$$

Each instance reserves $0.20 \times 80 = 16$ GB. Four instances add up to 64 GB, leaving a 16 GB cushion. There is no OOM. Each vLLM has 16 GB for weights plus KV-cache: enough for a 7B–8B model in FP8/INT4 with a modest KV-cache.

A case that blows up. The same four instances, but somebody raises $g_i = 0.30$ thinking “I have four replicas, I can give each one more”:

$$\sum_{i=1}^{4} 0.30 = 1.20 \quad\Rightarrow\quad 1.20 \times 80\ \text{GB} = 96\ \text{GB} > 80\ \text{GB} \quad\times$$

The first instances start up and reserve $0.30 \times 80 = 24$ GB each. Three instances are already at $72$ GB. The fourth tries to reserve another 24 GB, there is nothing left, and it dies with CUDA out of memory. And worse: Kubernetes will reschedule it on the same GPU (it still sees four replicas), where it will die again, in a CrashLoopBackOff that makes no sense if you only look at the pod manifest.

The operational rule is brutally simple: in time-slicing, you manage the VRAM budget by hand, adding up the --gpu-memory-utilization values. The replica count controls how many pods fit in compute turns, but it does not reserve a single byte of memory. Confusing the two is the recurring mistake.

MPS: coordinated cooks on the same worktop

The Multi-Process Service (MPS) attacks a different problem. By default, when several processes use the same GPU without MPS, each one has its own CUDA context, and the GPU alternates between contexts (time-slicing at driver level): they do not execute kernels at the same time, they take turns, with context-switch overhead. If your kernels are small and do not fill the GPU on their own, this leaves SMs idle: process A uses 30 % of the SMs during its turn and the other 70 % is wasted.

MPS introduces a daemon that shares a single CUDA context between processes, so that their kernels can execute concurrently occupying different SMs at the same time. It is spatial sharing of compute: instead of taking turns at the whole worktop, each cook occupies a part and they work in parallel. This reduces context-switch overhead and raises throughput when there are many small concurrent kernels that individually do not saturate the GPU.

And, unlike pure time-slicing, MPS allows per-process limits, which gives you a form of QoS:

CUDA_MPS_ACTIVE_THREAD_PERCENTAGE limits the percentage of SMs an MPS client can use. By default each client gets $100 / \text{MaxSharedClients}$. Pinning it to, say, 40 % caps that process’s compute ceiling (MPS docs).
CUDA_MPS_PINNED_DEVICE_MEM_LIMIT imposes a memory cap per client (valid from CUDA 11.5). This is what time-slicing does not have: a per-process VRAM limit that the runtime enforces.

These two limits turn MPS into a resource provisioning mechanism that mitigates the noisy neighbour: you can guarantee that a process does not eat more than X % of the SMs or more than Y GB. The combination gives reasonable QoS, not perfect but real.

The limitation MPS does not solve: fault isolation is weak. Since clients share the daemon’s CUDA context, a fatal error in one client can affect the daemon and therefore the other clients (historically, a client dying dirtily could require restarting the daemon). It is better than time-slicing in this respect, but a long way from hardware isolation. That is why MPS fits when there is trust between the workloads, processes from your own team, not third-party tenants.

The canonical use case: many small concurrent inference requests that individually leave the GPU half empty. MPS overlaps them and raises aggregate throughput. Serving several small models, or several light replicas of the same model, on a datacenter GPU where you trust all the workloads, is MPS territory.

MIG: concrete walls

The Multi-Instance GPU (MIG) is the only one of the three that gives genuine isolation, because it cuts the GPU in hardware. Available on modern datacenter GPUs, A100 (Ampere), H100/H200 (Hopper), B200 (Blackwell), and never on consumer ones: an RTX 5090 (consumer Blackwell) does not support MIG, nor do GeForce cards in general.

MIG divides the GPU into up to seven instances (GPU Instances), and each instance gets a dedicated portion of:

SMs (compute slices): compute is split into 7 slices, each one roughly 1/7 of the SMs.
L2 cache and memory: each instance has its slab of HBM and its portion of L2 cache.
Memory bandwidth: dedicated, not shared.
Data paths and engines: with fault barriers between instances.

The result is that a MIG instance behaves like a smaller, independent GPU: whatever happens in one, an OOM, a kernel blowing up, a workload saturating its compute, does not affect its neighbours. Memory, fault and performance (QoS) isolation, the three things time-slicing does not give and MPS only half gives.

The H100 80GB profiles

MIG does not allow arbitrary sizes: it has fixed profiles. On the H100 80GB, the profile catalogue (notation <compute>g.<memory>gb) is:

Profile	Compute (slices)	Memory	Max. instances
`1g.10gb`	1/7	10 GB	7
`1g.20gb`	1/7	20 GB	4
`2g.20gb`	2/7	20 GB	3
`3g.40gb`	3/7	40 GB	2
`4g.40gb`	4/7	40 GB	1
`7g.80gb`	7/7	80 GB	1 (whole GPU)

(There is also 1g.10gb+me, a variant with media engines for video encoding.) The base memory unit on the H100 80GB is 10 GB per slice (80 GB / 8, with one slice reserved), and the compute unit is 1/7 of the SMs. The profiles combine these units. Note 1g.20gb: the same compute fraction as 1g.10gb (1/7 of the SMs) but twice the memory, useful when a workload needs more VRAM than compute.

One important detail: MIG partitions do not mix freely. The GPU is divided following a valid geometry (the profiles fit together like pieces in a grid), and the profiles are fixed when you configure the GPU; changing them requires draining and repartitioning. They are concrete walls: solid, but they do not move while hot.

The calculation: 7×1g.10gb against 1×7g.80gb

Let us compare the two extremes. On the left, seven 1g.10gb instances: seven isolated 10 GB GPUs. On the right, a single 7g.80gb: the whole H100, unpartitioned.

The operational question is what fits in 10 GB. An instance’s VRAM budget is split between model weights and KV-cache:

$$V_{\text{inst}} = V_{\text{weights}} + V_{\text{KV}} + V_{\text{overhead}}$$

Take a model of 7B parameters in FP8 (1 byte/parameter):

$$V_{\text{weights}} \approx 7 \times 10^9 \times 1\ \text{byte} = 7\ \text{GB}$$

On a 1g.10gb instance (10 GB), after the 7 GB of weights and subtracting roughly 0.5–1 GB of runtime overhead, about 2 GB are left for KV-cache. That is enough for a modest context window and low concurrency, fine for a guardrail service, a classifier or an extraction model that handles short prompts one at a time. A 7B in INT4 (~3.5 GB of weights) leaves ~5.5 GB of KV-cache, much roomier. But a 13B model in FP8 (~13 GB of weights) does not fit in a 10 GB instance: not even the weights get in. For that you need 1g.20gb, 2g.20gb or larger.

Against this, the 7g.80gb (whole GPU) gives you the 80 GB for one large model: a 70B in FP8 (~70 GB of weights) fits with a tight KV-cache, or a 70B with more room spread across several whole GPUs in tensor-parallel (see TP against replicas: one big one versus N small ones).

The reading is clear: fine partitioning (7×1g.10gb) maximises the number of small isolated workloads; not partitioning (1×7g.80gb) maximises the model size that fits. The KV-cache available per instance shrinks proportionally as you partition, so fine-grained MIG serves many light isolated services, not one large model chopped up. If your workload is a single large model, MIG is not for you: use the whole GPU or several in TP.

The decision tree

The three questions, in order:

Do you need REAL isolation?
(multi-tenant, compliance, one workload failing must not touch another)
│
┌────┴────┐
YES NO
│ │
Is it Hopper/ Many SMALL concurrent kernels
Ampere/ AND you trust every workload?
Blackwell? │
│ ┌────┴────┐
┌──┴──┐ YES NO
YES NO │ │
│ │ MPS Dev / bursts / consumer GPU
MIG │ (spatial, / no need to isolate?
│ │ per-process │
│ no real QoS) YES
│ isolation: │
│ rethink TIME-SLICING
│ (move to (temporal, cheap,
│ CPU, another works on a 5090)
│ GPU, or accept
│ the risk with
│ time-slicing)

And each branch in one sentence:

MIG when isolation is a requirement (compliance, multi-tenant, hard SLA) and you have datacenter hardware that supports it. Concrete walls cost money, but if you need them there is no substitute.
MPS when you have many small concurrent workloads that leave the GPU half empty and you trust all of them (same team, not third-party tenants). You raise throughput with reasonable QoS, accepting imperfect fault isolation.
Time-slicing when it is dev, bursts, low utilisation, a consumer GPU, or you simply do not need to isolate anything. Cheap and universal, but manage the VRAM budget by hand.

One nuance the recent documentation records: they can be combined. You can do time-slicing on top of a MIG instance (hardware isolation at the instance boundary, software turns inside), or use MPS inside a MIG instance. The layers are not mutually exclusive; the tree picks the primary strategy.

Applied to the generic 4×H100 cluster

Let us get down to numbers with a generic on-premise cluster of 4×H100 SXM 80 GB with NVLink. It is common to have a heterogeneous menu of workloads: a large chat model, light services (embeddings, reranker, guardrails) and a dev/experimentation queue. Each type calls for a different mechanism. A reasoned split:

GPU 0 and GPU 1, large model in tensor-parallel (not shared). A 70B model in FP8 takes ~70 GB of weights; served comfortably with a generous KV-cache it needs more than one H100. We spread it in tensor-parallel over two whole H100s joined by NVLink (intra-node bandwidth is what makes TP viable; the detail is in TP against replicas). Here we do not share: these two GPUs belong to the large model, full stop. Total isolation by dedication.

$$V_{\text{available}} = 2 \times 80 = 160\ \text{GB};\quad V_{\text{weights}} \approx 70\ \text{GB};\quad V_{\text{KV}} \approx 80\ \text{GB of KV-cache}$$

There is memory to spare for a long request queue and high concurrency.

GPU 2, split with MIG into small instances for light services. Embeddings (bge-m3), reranker (bge-reranker-v2-m3) and a couple of guardrail models (1B–3B) are different workloads, potentially from different teams, and you want a failure or a spike in one not to touch the others. Light multi-tenancy with isolation → MIG. A reasonable partitioning of the H100:

$$\underbrace{3 \times \texttt{1g.10gb}}_{\text{30 GB, 3/7 SMs}} \;+\; \underbrace{1 \times \texttt{4g.40gb}}_{\text{40 GB, 4/7 SMs}}$$

The three 1g.10gb instances (10 GB, 1/7 of the SMs each) host embeddings, reranker and a 1B INT4 guardrail, each isolated, with no noisy neighbour. The 4g.40gb (40 GB, 4/7 of the SMs) hosts an intermediate 7B–13B model with a decent KV-cache for a support service. Each service has its larder and its wall; if the reranker blows up, the chat never finds out.

GPU 3, time-slicing for dev and bursts. The developers touch the GPU sporadically: experiments, short fine-tunes, model trials. They do not need isolation (it is the same team) and they rarely overlap while active. We advertise it as 4 replicas via the device-plugin. Four dev pods fit in turns. VRAM budget with the formula above: if each dev brings up a vLLM at --gpu-memory-utilization 0.20, they add up to $4 \times 16 = 64$ GB < 80 GB, with no OOM. If somebody needs more, lower the replica count or coordinate with the team. The cost of flexibility is manual discipline.

Summary of the split:

Resource	Mechanism	Workload	Isolation
GPU 0 + GPU 1	Dedication (TP)	70B chat in tensor-parallel	total (dedicated)
GPU 2	MIG (3×1g.10gb + 1×4g.40gb)	embeddings, reranker, guardrails, 7B–13B	real hardware
GPU 3	Time-slicing (4 replicas)	dev, bursts, experiments	none (trust)

The logic is always the same: spend isolation (MIG) where you need it, spend cheap concurrency (time-slicing) where you do not, and reserve whole GPUs for what genuinely fills them. An H100 serving embeddings on a 7g.80gb would be as absurd as an RTX 5090 trying MIG: the tool does not match the workload.

What we have not covered

What happens when they do not even fit at the same time: if you have more models than VRAM and they have to take turns in memory (loading/unloading weights, not just taking turns at compute), you enter swap and sleep territory, the sibling piece Serving several models on one GPU.
NUMA-aware scheduling: on multi-socket nodes, which GPU goes with which CPU/memory matters for latency; see Kubelet resource managers on RKE2.
Autoscaling the replicas: how many instances to bring up according to real load, with KEDA and queue metrics; see Autoscaling LLMs on Kubernetes with KEDA.
A jitter benchmark under contention: how much TTFT really wobbles in time-slicing with 4 active replicas compared with MIG, material that deserves its own measurement, not an estimate.

References

NVIDIA — Time-Slicing GPUs in Kubernetes (GPU Operator). https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html
NVIDIA — Multi-Process Service (MPS) Overview. https://docs.nvidia.com/deploy/mps/index.html
NVIDIA — MPS: Tools and Interface Reference (CUDA_MPS_ACTIVE_THREAD_PERCENTAGE, CUDA_MPS_PINNED_DEVICE_MEM_LIMIT). https://docs.nvidia.com/deploy/mps/appendix-tools-and-interface-reference.html
NVIDIA — Multi-Instance GPU (MIG) User Guide. https://docs.nvidia.com/datacenter/tesla/mig-user-guide/
NVIDIA — Supported MIG Profiles (H100 80GB catalogue). https://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-mig-profiles.html
NVIDIA — k8s-device-plugin (time-slicing replicas). https://github.com/NVIDIA/k8s-device-plugin
vLLM — Engine Arguments (--gpu-memory-utilization). https://docs.vllm.ai/en/latest/serving/engine_args.html

Vram on lo0 — Tech Blog

Serving several models on a single GPU: co-residency, model-swapping and sleep mode

TL;DR

The analogy: one stage, one spotlight

The three states: where the weights live

Co-residency: the VRAM budget

A worked example: what fits in 80 GB

Model-swapping with llama-swap: the actor goes home

Example ConfigMap: two rerankers rotating on the same port

vLLM sleep mode: the actor in the wings

The latency maths: why the wake wins

The decision tree

Applied to the generic 4×H100 cluster

Traps and things that are not what they seem

Conclusion

See also

References

Sharing one GPU between several workloads: time-slicing, MPS and MIG

TL;DR

The analogy: a shared hob, one kitchen with several cooks, several kitchens

Time-slicing: compute turns, shared larder

The VRAM budget in time-slicing (the calculation that avoids the OOM)

MPS: coordinated cooks on the same worktop

MIG: concrete walls

The H100 80GB profiles

The calculation: 7×1g.10gb against 1×7g.80gb

The decision tree

Applied to the generic 4×H100 cluster

What we have not covered

See also

References

Vram on lo0 — Tech Blog

Serving several models on a single GPU: co-residency, model-swapping and sleep mode

TL;DR

The analogy: one stage, one spotlight

The three states: where the weights live

Co-residency: the VRAM budget

A worked example: what fits in 80 GB

Model-swapping with llama-swap: the actor goes home

Example ConfigMap: two rerankers rotating on the same port

vLLM sleep mode: the actor in the wings

The latency maths: why the wake wins

The decision tree

Applied to the generic 4×H100 cluster

Traps and things that are not what they seem

Conclusion

See also

References

Sharing one GPU between several workloads: time-slicing, MPS and MIG

TL;DR

The analogy: a shared hob, one kitchen with several cooks, several kitchens

Why share: the operational problem

Time-slicing: compute turns, shared larder

The VRAM budget in time-slicing (the calculation that avoids the OOM)

MPS: coordinated cooks on the same worktop

MIG: concrete walls

The H100 80GB profiles

The calculation: 7×1g.10gb against 1×7g.80gb

The decision tree

Applied to the generic 4×H100 cluster

What we have not covered

See also

References