Serving several models on a single GPU: co-residency, model-swapping and sleep mode

This is the second piece of an operational series on squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB with NVLink. Its sibling, Sharing a GPU: time-slicing, MPS and MIG, splits the GPU so that several processes can use it at once; this one splits the complementary problem: you have more models than VRAM and you need them to coexist in time, not just in space. The third, The RAG data plane on CPU, moves off the GPU everything that does not need to be there; and the end-to-end sovereign assistant (fourth instalment, in preparation) assembles all of this behind LibreChat and LiteLLM. Here we assume that VRAM is the scarce resource and that keeping everything loaded at once is not an option.

TL;DR

You have an agent LLM serving almost all the traffic, one or two rerankers for RAG, an alternate model (another family, another language, a fine-tune) and, now and then, a large model for hard tasks. Added up, they do not fit in the 80 GB of an H100. There are three ways to make them coexist, and the key to not confusing them is to understand that they are three different states of where the weights live:

• Co-residency: several models loaded at the same time in HBM, if the sum of their VRAM budgets fits in the physical memory. It is the ideal when they fit: zero switching latency. The catch: each engine must cap its own VRAM; if two engines both believe they own the whole 80 GB, the second one to ask for memory dies with OOM. On a GPU shared by time-slicing, sharing time does not protect memory (the processes still compete for the same HBM) and this problem gets worse, as we see in the sibling piece.
• Model-swapping (with llama-swap): only one (or very few) resident at a time. The proxy looks at the request’s model field and, if another one is needed, unloads the current model and loads the requested one. Since there is only one inside, each model can use almost the whole GPU. The cost is a full cold start: seconds from local NVMe, minutes from network storage. It mixes engines: it exposes OpenAI-compatible endpoints including /rerank with llama-server on GGUF.
• vLLM sleep mode (--enable-sleep-mode, endpoints /sleep and /wake_up): the model is not unloaded, it is put to sleep. The weights are parked in host RAM (level 1) or discarded while the process stays alive (level 2). The wake is 18–200× faster than a cold start because the process is still alive and keeps the CUDA allocator, the CUDA graphs and the compiled JIT kernels. The only thing rebuilt is the KV cache, which is discarded on sleep. It needs enough RAM for the sleeping weights.

The decision tree is short. Do they all fit? Co-residency. They do not fit, switching is infrequent and you mix engines? llama-swap. Everything is vLLM, there is RAM to spare and wake latency matters? Sleep mode. Sustained load on all of them at once? That is not swapping, it is more GPUs or replicas (see one big vs N small). This post puts numbers on each of them on an 80 GB H100.

The analogy: one stage, one spotlight

Picture a theatre with a single stage and a single spotlight, the GPU. You have more actors (models) in the company than fit under the light at once. There are three ways to stage the show:

• Co-residency: several small actors on stage. If the cast for this scene is three slight actors, the main LLM, one reranker, another reranker, they all fit under the spotlight at the same time. Switching lines between them is instantaneous: they are already on stage. But the stage has a fixed size; add a fourth, burly actor (the large model) and they do not fit, so someone falls off the edge. That is the OOM.

• Model-swapping: the actor goes home. When an actor finishes their part, they go home (the disk). If the next scene needs them again, they have to come back from home: get dressed, put the make-up on, travel to the theatre. That takes minutes. In exchange, while they are on stage they have the whole stage to themselves. That is the llama-swap model: maximum VRAM per actor, but re-entry costs a full round trip.

• Sleep mode: the actor waits in the wings. Instead of going home, the actor steps out of the spotlight but stays backstage (host RAM), already dressed and made up. When their turn comes, they step in in two strides. No journey, no wardrobe: just crossing the curtain. That is the vLLM wake: the weights come back not from disk but from RAM, and everything that goes into “preparing the actor”, the allocator, the CUDA graphs, the kernels, is already done because they never left the building.

The operational moral is the same as in the theatre: the cost of an actor is not only their talent, it is how long they take to walk on when you need them. Co-residency pays for the permanent spot; swapping pays for the journey; sleep pays for the RAM of keeping them waiting nearby. The rest of this post is about choosing which, with a VRAM budget and a stopwatch in hand.

The three states: where the weights live

Co-residency: the VRAM budget

Co-residing is the default option when they fit: zero switching latency because every model is already in HBM. The whole question is do they fit?, and it is answered with a VRAM budget. The VRAM of an inference engine splits, roughly, into three items:

where $P$ is the number of parameters, $b$ the bytes per parameter given the quantisation, KV the KV-cache budget (which scales with concurrency and context length) and $A$ the overhead of activations, CUDA buffers and fragmentation. The hard rule of co-residency is:

If the sum overflows, no sharing scheme saves you: the first engine to ask for memory beyond the free gap dies with OOM. And here is the critical nuance that connects to the sibling piece: splitting the GPU by time does not split the memory. Under time-slicing, two processes take turns on compute but share the whole HBM with no isolation; if both assume they own the 80 GB, they collide. Only MIG gives memory partitions with a real boundary, as we see in Sharing a GPU. To co-reside properly, each engine must cap itself: in vLLM, --gpu-memory-utilization 0.45 tells it “do not use more than 45 % of the GPU”; in llama-server, you control the layers on GPU and the KV size. If you do not set those limits, vLLM claims 90 % of the GPU for itself by default, and there is no room left for anyone else.

A worked example: what fits in 80 GB

Take the realistic scenario. A 32B agent LLM in FP8 ($b = 1$ byte/param):

On top of that comes the KV cache. To serve with decent concurrency and agent contexts (which are long: history, tools, documents), a KV budget of ~12 GB is reasonable, plus some ~2 GB of activations and overhead. Total for the main LLM: ~46 GB. That leaves ~34 GB of the 80. Let us see what fits there:

With this they co-reside comfortably: the 32B LLM, the two rerankers (which are featherweights, a few hundred MB to 1 GB each) and even an 8B alternate, leaving 20 GB of cushion. Rerankers are the textbook case for co-residency: so small that they always fit alongside the main LLM, and rotating them would be absurd. Switching between them is instantaneous because they are all loaded.

When does it break? When the occasional large model arrives, a 70B in FP8:

The 70B’s weights alone already eat 70 of the 80 GB. There is no way to co-reside it with the 32B, not remotely. That is where co-residency ends and rotation begins: the 70B can only come in if we evict the 32B from the GPU. That is the exact boundary: co-reside while the sum of budgets fits with headroom; rotate as soon as one model needs, on its own, more VRAM than is left over.

Model-swapping with llama-swap: the actor goes home

When they do not fit at once, the first answer is to rotate: keep one resident model and change it on demand. The canonical tool for this in the on-premise world is llama-swap, a Go proxy that sits in front of your inference servers (llama.cpp, vLLM, TabbyAPI and so on) and starts and stops them as needed.

The mechanism is elegant in its simplicity. Every OpenAI-compatible request carries a model field. llama-swap reads that field, looks at which upstream server it has configured for that model, and:

1. If the requested model is already loaded, it routes the request straight through.
2. If another one is loaded, it stops it (freeing its VRAM) and starts the right one.
3. When the new server reports ready, it forwards the request.

The cost of a swap is exactly a full cold start of the new model: unloading the current one (fast, just freeing memory) plus loading the new one (slow, moving the weights from disk to HBM). That second term is the painful one, and its magnitude is decided by where the weights are: from local NVMe it is seconds; from network storage (Ceph RGW, NFS) it can be minutes. The whole analysis of the load path, and why the default loader makes it slow, is in From disk to HBM; here the consequence is enough: a swap is only viable if it is infrequent, because every switch pays that toll in full.

The advantage in exchange: since there is only one resident, that model can use almost the whole GPU. The 70B that co-resides with nobody fits comfortably if it is the only one inside. And llama-swap mixes engines: you can have a vLLM configured for the big LLM, a llama-server with GGUF for the alternate, and two llama-server instances for the rerankers, which expose /rerank, /v1/rerank and /v1/reranking natively. That heterogeneity (GGUF + rerank + vLLM behind a single OpenAI endpoint) is exactly what vLLM alone does not give you, and the main reason to choose llama-swap.

Example ConfigMap: two rerankers rotating on the same port

A concrete and useful case: you have two rerankers, one multilingual and one specialised in code, that you do not need at the same time and would rather not keep both resident. With llama-swap they rotate on the same endpoint, triggered by the model field. The ConfigMap (mounted as the proxy’s config.yaml in a Kubernetes deployment) would be:

apiVersion: v1
kind: ConfigMap
metadata:
  name: llama-swap-rerankers
data:
  config.yaml: |
    # How long a model stays loaded after the last request
    # before llama-swap unloads it to free VRAM
    healthCheckTimeout: 60

    models:
      # Multilingual reranker (GGUF, via llama-server)
      "reranker-multilang":
        cmd: >
          /usr/bin/llama-server
          --model /models/bge-reranker-v2-m3.Q8_0.gguf
          --reranking
          --port ${PORT}
          --n-gpu-layers 99
          --ctx-size 8192
        # ttl: after 300 s idle, it is unloaded and the GPU is freed
        ttl: 300

      # Code reranker (GGUF, via llama-server)
      "reranker-code":
        cmd: >
          /usr/bin/llama-server
          --model /models/codereranker.Q8_0.gguf
          --reranking
          --port ${PORT}
          --n-gpu-layers 99
          --ctx-size 8192
        ttl: 300    

A request to /v1/rerank with "model": "reranker-multilang" starts that server; the next one with "model": "reranker-code" stops the multilingual one and starts the code one on the same ${PORT} that llama-swap manages. Since both are small, the swap between them takes one or two seconds: quantised GGUFs weigh a few hundred MB and come from local NVMe. The ttl controls how long a model stays loaded after its last request: raising it avoids swaps if requests arrive in bursts; lowering it frees the GPU sooner for other uses.

An honest caveat: this pattern of two rerankers that rotate makes sense when they do not fit alongside the rest, or when you want to reserve the VRAM for something else. If they do fit (and a couple of 0.5 GB rerankers almost always fit, as we saw above), co-residing them is strictly better: zero swap latency. llama-swap shines when you rotate large models or mix engines that do not coexist well, not for juggling tiny models that would fit together.

vLLM sleep mode: the actor in the wings

Swapping has a problem: every switch pays the entire cold start, and a cold start is not just moving weights. As we saw in From disk to HBM, starting an inference engine includes initialising the Python process and the CUDA context, setting up the memory allocator, capturing the CUDA graphs and compiling the JIT kernels (DeepGEMM, FlashInfer, TorchInductor). Moving the weights is only one of five items, and often not the largest.

vLLM’s sleep mode (--enable-sleep-mode) attacks exactly that: instead of killing the process when a model stops being used, it puts it to sleep, leaving the process alive. There are two levels, and the difference is where the weights go:

• Level 1: offloads the weights to host RAM (CPU) and discards the KV cache. The process stays alive. The wake copies the weights from RAM to VRAM, not from disk. Typical wake: ~0.1–0.8 s for small models, ~3–6 s for large ones. It needs enough RAM for the sleeping weights (on the order of the model’s size in GB).
• Level 2: discards the weights entirely, keeping only small buffers (rope scaling tensors and the like). The wake does reload the weights from disk, but everything else, process, allocator, CUDA graphs, JIT kernels, is already done. Typical wake: ~0.8–2.6 s for small models. RAM use is almost nil (megabytes).

The key that explains the numbers: at both levels, keeping the process alive preserves the expensive infrastructure. That is why the benchmark in the vLLM blog (Oct 2025) reports that a wake is 18–200× faster than a full reload, and, most counter-intuitively, level 2 is still 23–45× faster than a cold start despite reloading the weights from the same disk, because it skips the other four items. In their measurements on an A100, a full cycle of 5 model switches goes from 357 s without sleep (≈48 s per switch) to 112 s with level 1 (wake of 0.26 s / 0.82 s) or 125 s with level 2 (0.85 s / 2.58 s).

The KV cache is always discarded on sleep. That is not a minor detail: it means the first response after the wake rebuilds the KV from scratch, paying a full prefill. That is why the wake is not “entirely free”: the model is available in sub-second time, but the first request is a little slower until the KV is repopulated. A one-request warm-up after the wake hides that cost.

# Start vLLM with sleep mode (admin endpoints, trusted networks only)
export VLLM_SERVER_DEV_MODE=1
vllm serve <model> --enable-sleep-mode --port 8001

# Sleep (level 1: weights to host RAM)
curl -X POST 'localhost:8001/sleep?level=1'

# Wake up
curl -X POST 'localhost:8001/wake_up'

Security warning (from vLLM’s own blog): the /sleep, /wake_up, /collective_rpc and /reset_prefix_cache endpoints require VLLM_SERVER_DEV_MODE=1 and should only be exposed on trusted networks, since they can take the service down. They are for internal orchestration (a controller that sleeps and wakes models according to the queue), not for the public plane.

The latency maths: why the wake wins

Let us put numbers on “the wake comes from RAM, not from disk”. The cost of having a model available is, in each strategy, the time to move its weights from wherever they are to HBM, plus, in the swap case, the other four cold-start costs. Take the 34 GB of weights of the 32B LLM in FP8 and compare the three paths.

Cold start from NVMe (swap). A reasonable Gen4/Gen5 NVMe gives on the order of ~5 GB/s effective per stream with the default loader (the disk’s theoretical floor is higher, but single-threaded deserialisation does not saturate it, see From disk to HBM). For 34 GB:

And that is before adding CUDA graph capture and kernel compilation, which add several more seconds. The real cold start of a 32B is around 15–40 s depending on loader and storage. From the network (Ceph RGW), multiply.

Level 1 wake from RAM. The weights come not from disk but from host RAM, and travel over PCIe Gen5 x16, whose practical host→GPU bandwidth is ~50–64 GB/s. For the same 34 GB:

And there is nothing else to pay: the allocator, the graphs and the kernels are already there. The real wake of a model this size lands in the sub-second to a few seconds range that vLLM reports. The speed-up against an NVMe cold start is on the order of:

consistent with the 18–200× from the blog, whose measurements include smaller models, where the relative weight of the preserved CUDA graphs is even larger and the factor goes up.

Why the bandwidth difference explains almost everything. The key jump is not 5 vs 55 GB/s (a ~11× in transport). It is that the cold start also pays for rebuilding infrastructure, which the wake skips entirely. The table:

Look at the level 2 row: it reloads the weights from the same disk as the swap (~6.8 s of bytes), but since it does not rebuild graphs or kernels, its total (~7–10 s) still beats a full cold start (15–40 s). It is the proof that moving bytes is only part of the cost, and the part sleep mode exploits.

Assumptions, honestly: the bandwidths are indicative. NVMe “5 GB/s effective” assumes the default loader; with a concurrent streamer it goes up. PCIe “55 GB/s” assumes Gen5 x16 with a pinned buffer and NUMA locality; if the buffer lands on the wrong socket, it drops. And the “15–40 s” cold-start range depends on the model, the quantisation and whether the files are in page cache (the “the second time it loaded fast” trap). The numbers are there to reason about orders of magnitude, not to size anything without measuring on your hardware.

The decision tree

The three strategies do not compete: each wins in a different regime. The tree, in order:

The node most often ignored is the one at the top right: “sustained load on all of them at once?”. If your four models get constant and simultaneous traffic, neither swapping nor sleep helps, since both assume the models take turns in time. Rotating under sustained load only adds switching latency without solving the underlying problem: there is not enough compute. The answer then is to scale horizontally (more replicas) or spread across more GPUs, a capacity decision analysed in One big vs N small. Swapping and sleep are tools for temporally unbalanced workloads: many models, but rarely active at the same time.

Applied to the generic 4×H100 cluster

Let us bring this down to the 4 H100 SXM 80 GB with NVLink. The winning strategy is not to pick one of the three, but to spread the models across GPUs according to their usage pattern and apply to each GPU the strategy it calls for. A reasonable split:

H100 #0, the workhorse (co-residency). The 32B agent LLM (main service, constant traffic) co-resides with the two rerankers and, if they fit, the 8B alternate. It is the GPU that never rotates: everything living here is used continuously and fits comfortably in 80 GB (the ~60 GB of the example above). Zero switching latency between the LLM and its rerankers, which is exactly what RAG needs, since a fast reranker is useless if you have to wait for a swap every time.

H100 #1, light services with MIG. If you have many small heterogeneous services, an embedding model, a classifier, a guardrail, an STT/TTS, splitting this GPU with MIG into isolated instances (each with its slice of HBM with a real boundary) gives co-residency with memory isolation, preventing a service that inflates its KV from taking the others down. The detail of when MIG beats time-slicing is in Sharing a GPU; the rule here: co-residing light services on one GPU makes sense when they fit and when isolating them is worthwhile, and MIG is the tool for the second part.

H100 #2, the occasional large model (sleep mode or swap). The 70B that is only invoked for hard tasks does not deserve a dedicated GPU kept awake, since it would sit idle most of the time, burning 700 W for nothing. Two options:

• If this GPU also serves a medium model routinely and you only occasionally need the 70B, use sleep mode: put the medium one to sleep (level 1, weights to RAM), wake the 70B. Note that level 1 sleep frees the VRAM, the weights go to RAM, so yes, it fits even though the 70B would not fit alongside an awake medium model. Waking the medium one again is sub-second.
• If the 70B comes as GGUF or you mix engines, llama-swap rotates between the medium model and the 70B by the model field. Each 70B invocation pays its cold start (seconds from local NVMe), acceptable if it is occasional.

H100 #3, replica / overflow. The fourth GPU absorbs peaks: a replica of the main LLM for when #0’s queue grows, or reserve capacity. There is no swapping here: it is pure capacity, the answer to the tree’s “sustained load” node.

The cross-cutting principle: co-reside what is used together and continuously (LLM + rerankers); isolate the light and heterogeneous with MIG; sleep or rotate the large and occasional; replicate what saturates. The four GPUs do not all do the same thing, each runs the strategy its load pattern calls for. And the NVLink between them matters for something else (tensor parallelism for the 70B if it would not fit even on one; see One big vs N small), but for this post’s problem, many models, one GPU, the lever is when each model needs to be awake.

Traps and things that are not what they seem

“Co-residing is always better if they fit.” Almost, but watch the KV cache: co-residing two models means splitting the KV budget between them. If the main LLM needs a large KV for high concurrency and long contexts, adding a flatmate cuts that KV and lowers its throughput. Sometimes it is better to give the whole GPU to the main model and rotate the secondary one. Co-residency is not free: the tenant takes room away from the cache of the one that matters.

“Sleep mode is like swapping but faster.” Not exactly. Swapping releases the process; you can have N models configured and only pay RAM/disk for the resident one. Sleep mode keeps one live process per sleeping model, so each sleeping vLLM still occupies its process slot, its RAM (level 1) and its management footprint. Sleep scales well to a few rotating models; for many (10+), level 2 (minimal RAM) or plain swapping fit better. Do not put 15 models into level 1 sleep and expect the RAM to hold.

“The wake is instantaneous, I lose nothing.” The model’s wake is sub-second, but the KV cache was discarded on sleep. The first request after the wake pays a full prefill to repopulate the KV, slower than normal. If your SLA is strict on the first response after an idle period, add an automatic warm-up after the wake. Prefix caching helps make that reprefill cheaper if there are stable prefixes.

“llama-swap with a low ttl saves me VRAM for free.” It saves you VRAM while nobody uses that model, but every time it comes back it pays the cold start. An aggressive ttl on a model with bursty traffic turns every burst into a load wait. The right ttl depends on the temporal pattern of the requests, not on how much VRAM you want to free. Measure it.

“Time-slicing lets me co-reside more models.” False and dangerous. Time-slicing shares out compute time, not memory, and all the processes still compete for the same HBM with no isolation. Co-residing via time-slicing does not give you more effective VRAM; it gives you more processes fighting over the same VRAM, and an OOM when the sum overflows. For real memory partitioning, MIG. The detail is in the sibling piece.

Conclusion

Having more models than VRAM is not a problem of insufficient hardware: it is a problem of managing a scarce resource over time. The intuition of “I need one GPU per model” is expensive and almost always wrong, because models are rarely used all at once. The three strategies are three answers to the same question, where do the weights of the models you are not currently using live: in HBM if they fit (co-residency, zero latency but they cost space), on disk if switching is rare (swapping, maximum VRAM per model but a return trip of seconds to minutes), or in RAM if switching latency matters (sleep mode, sub-second wake at the cost of tied-up RAM). Sleep mode is the most interesting addition of 2025 because it breaks the false dilemma of “everything loaded vs reload every time”: by keeping the process alive and preserving the allocator, the CUDA graphs and the kernels, it turns a 30–100 s cold start into a sub-second wake, and it does so even when it reloads the weights from the same disk (level 2), because moving bytes was never the whole cost. On the four-H100 cluster, the play is not to pick one strategy but to spread: co-reside what goes together, isolate the light stuff with MIG, sleep or rotate the large and occasional, replicate what saturates. The GPU is the stage with a single spotlight; the art is knowing which actor walks on, which goes home and which waits in the wings.

See also

• Speeding up model cold start: from minutes to seconds — how to lower the cold-start cost that every swap pays.

• Sharing a GPU: time-slicing, MPS and MIG — the sibling piece: how to split one GPU among several processes in space (not in time). Key here to understand why time-slicing does not protect memory and why MIG is what gives co-residency with real HBM isolation.

• From disk to HBM: cold start and model loading — the load path that swapping pays in full on every switch and that sleep mode sidesteps; why moving bytes is only one of five start-up items.

• Multi-LoRA serving: fundamentals — the alternative when the “several models” are adapters of the same base: instead of rotating whole models, you serve many LoRAs on a co-resident base with no swap cost.

• Engineering the prefix cache hit rate — the KV cache is discarded on sleep; a good hit rate on stable prefixes makes the reprefill of the first request after the wake cheaper.

• One big vs N small: tensor parallelism and replicas — the “sustained load on all of them at once” branch of the tree: when rotating is not enough and you have to spread across more GPUs or replicas.

• Mixed NVIDIA / Intel environments: servers and NUCs — where to place the light services (rerankers, embeddings) that do not need an H100: sometimes co-residing is not on the big GPU but on more modest hardware.

References

• vLLM Blog (Embedded LLM), Zero-Reload Model Switching with vLLM Sleep Mode, 26 oct 2025: https://blog.vllm.ai/2025/10/26/sleep-mode.html
• vLLM Docs, Sleep Mode: https://docs.vllm.ai/en/latest/features/sleep_mode/
• mostlygeek, llama-swap (OpenAI/Anthropic-compatible model-swapping proxy): https://github.com/mostlygeek/llama-swap
• llama-swap, Configuration: https://github.com/mostlygeek/llama-swap/blob/main/docs/configuration.md
• NVIDIA, H100 Tensor Core GPU (HBM3 specs, 80 GB, ~3.35 TB/s): https://www.nvidia.com/en-us/data-center/h100/
• NVIDIA, Reducing Cold Start Latency for LLM Inference with NVIDIA Run:ai Model Streamer: https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer/