Speeding up model cold start: from minutes to seconds

· By José Manuel Román
Contents

This is the third batch of an operational series on squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB with NVLink. The sister pieces in this batch are Multimodal: serving a VLM on-premise with vLLM, adding vision to the same engine, and FinOps and GPU multi-tenancy with LiteLLM, sharing out and charging for the GPU across teams. This post takes the problem that From disk to HBM laid out conceptually and turns it into a runbook: how you actually bring the cold start down, knob by knob, so that elasticity is usable.

TL;DR

The post From disk to HBM left an uncomfortable idea behind: moving the weights from disk to HBM is only one of the five line items of cold start. The other four, starting the process, creating the CUDA context, setting up the allocator and, the most expensive one that hardly anyone looks at, capturing the CUDA graphs and compiling the JIT kernels, are untouched by any fast loader. That is why a server that loads the weights in 6 s can still take 90 s to be ready. This runbook attacks all five. For the weights: safetensors (mmap, zero-copy, no bounce through a host FP32 buffer) versus the pickle of torch.load; Tensorizer (CoreWeave), which serialises the model into one file and streams it tensor by tensor straight to the GPU from object storage/HTTP/S3; and the Run:ai Model Streamer (from NVIDIA), which reads concurrently and overlaps the read with the H2D copy. All three are switched on with one flag, --load-format, in vLLM. For the non-weight part: torch.compile/Inductor compilation cache (persisted and reused across start-ups), bounding the CUDA graph capture to the batch sizes you really use (the default capture can eat ~54 s), and sleep mode as a shortcut that skips all five line items. The maths: $t = W/B$ for the weights, the saving from overlapping read and H2D instead of doing them in series, and a 70B cold start broken down item by item. And the economics: when scale-to-zero with pre-warm beats keeping a warm replica. On the generic 4×H100 SXM 80 GB cluster.

The analogy: opening the kitchen in the morning

A restaurant does not open by flicking a switch. Anyone who has worked in one knows it: opening time is decided by everything that happens beforehand, and almost none of it is “bringing in the food”.

The first cook arrives at a cold, dark place. They have to: turn on the lights and start the systems (the process init); fire up the ovens and let them come up to temperature (the CUDA context, which takes its time to become operational); get the boards out, sort the cold room, organise the work space (the memory allocator, which reserves and structures the HBM); go down to the store and bring up the produce, the crates of vegetables, the meat, the fish, and put it in its place (loading the weights from disk to HBM, the entire subject of From disk to HBM); and, the most forgotten of all, sharpen every knife, set up the mise en place, make the stocks, pre-heat the mother sauces, all the preparation that is not an ingredient but without which not a single dish goes out (CUDA graph capture and JIT compilation of the kernels).

Here is the trap that From disk to HBM already anticipated: if you only optimise the produce, you still open late. You can hire the best delivery service in the world, a blazingly fast service lift, crates that put themselves away. But if the cook is still sharpening knives and waiting for the ovens to come up to temperature, the kitchen does not serve the first dish. The produce arrived early and the restaurant is still shut.

Today’s runbook is, literally, the list of everything that has to be done for the kitchen to open on time, and which of those tasks can be sped up, which can be left done from the night before (cached), and when it pays better never to switch the kitchen off (warm replica) than to light it again every morning (scale-to-zero).

The five line items of cold start

From the moment an inference pod is born until it returns its first token, the clock runs in five different places. It is worth naming them, because each optimisation attacks one or two, never all of them:

  1. Process init. Starting the Python interpreter, importing the engine (vLLM and its dependencies), parsing the configuration. Seconds, and it grows with the size of the environment and the CUDA/PyTorch imports.
  2. CUDA context. The first cudaSetDevice / cudaFree(0) initialises the CUDA runtime against the driver: it loads the context and maps the GPU. It is not instant, on the order of one to several seconds per GPU depending on driver and number of devices.
  3. Allocator. vLLM sets up its memory manager over the HBM (PyTorch’s caching allocator plus the KV-cache layout). Reserving and structuring tens of GB has its cost.
  4. Weight loading disk→HBM. The journey that From disk to HBM dissected: disk → page cache → host buffer → PCIe → HBM. What almost everyone believes cold start is, and it is only one of the five.
  5. CUDA graph capture + kernel JIT. vLLM captures CUDA graphs for the various batch shapes and compiles kernels with torch.compile/Inductor (and backends such as DeepGEMM or FlashInfer). This item is the great ignored one: the default graph capture in vLLM takes on the order of 54 s because it covers a wide range of batch sizes (vLLM, torch.compile integration; Red Hat, vLLM with torch.compile).
The five line items of a cold start (example 70B, default loader)width proportional to time · scale 0–110 s0 s55 s110 s1 · process init — ~4 s2 · CUDA context — ~6 s3 · allocator — ~4 s4 · weight loading disk→HBM — ~40 s5 · CUDA graphs + JIT — ~50 stotal ≈ 104 s · optimising only item 4 leaves item 5 intact — and 5 is the largest here

The lesson the diagram is shouting: on a modern server with torch.compile and CUDA graphs, item 5 can be as big as or bigger than item 4. Speeding up weight loading from 40 s to 6 s is a huge advance, but if you leave graph capture at 50 s, the cold start falls from 104 s to ~70 s, not to 12 s. You have to attack both halves.

Speeding up item 4: loading the weights

safetensors versus pickle: the format rules

The first knob is the on-disk format. The old torch.load uses pickle, which has two problems. The security one is well known: deserialising a pickle executes arbitrary Python code, since the __reduce__ protocol lets the file invoke any callable on load, so every model download was a remote execution vector (HuggingFace, Safetensors). The performance one is what concerns us: loading a pickle rebuilds Python objects and usually passes through an intermediate host buffer before reaching the GPU.

safetensors, now under the PyTorch Foundation (HuggingFace, Safetensors joins PyTorch Foundation), solves both. The file is just a JSON header + raw tensor bytes: loading it cannot do anything except populate tensor buffers. And the key property for us is physical: the data region is aligned to a page boundary (the header is padded so that the first tensor starts at a multiple of the OS page size). That is what makes zero-copy possible: a loader can mmap the file, cudaHostRegister over the mapped region and DMA straight from the page cache into VRAM, with no torch.load deserialisation and no temporary host FP32 buffer (HuggingFace, Safetensors). It is the base format; everything else is built on top of it.

The mmap trap (already flagged in From disk to HBM): mmap reads nothing immediately, it defers the cost to the first access to each page. If you do not force the read, the cold start looks short and the first token pays the page faults. And the “blazingly fast second load” is the page cache lying to you: in production pods are born cold, on nodes where those files are not cached.

Tensorizer: streaming tensor by tensor straight to the GPU

Tensorizer, from CoreWeave, serialises the model’s weights and its tensors into a single file and, instead of loading the whole model into RAM before moving it to the GPU, streams the data tensor by tensor from disk, an HTTP/HTTPS endpoint or an S3 bucket, deserialising on the fly straight onto the GPU (vLLM, Loading Models with CoreWeave’s Tensorizer). The operational advantage: near-instant loading and low host RAM usage during initialisation, which matters especially in serverless and autoscaling scenarios, exactly the scale-to-zero case. And it decouples the weights from the container image: the model lives in object storage, not inflating the image pull.

In vLLM it is enabled with --load-format tensorizer (or load_format="tensorizer" via the API). It requires serialising the model once into Tensorizer’s format; from then on, any pod streams it.

Run:ai Model Streamer: reading and copying at the same time

The Run:ai Model Streamer (from NVIDIA) attacks the bottleneck from another angle: it reads the tensors concurrently, N OS threads reading from storage into the CPU buffer, while streaming them to VRAM, so that the storage read and the H2D copy overlap instead of happening in series (vLLM, Loading models with Run:ai Model Streamer; NVIDIA, Reducing Cold Start Latency). It reads safetensors directly, with no reconversion.

The numbers published by NVIDIA give the order of magnitude: the streamer reaches 4.88 s reading from S3 at concurrency 32 and 7.53 s from an SSD IO2 at concurrency 8; integrated into vLLM, the total time to ready drops to 23.18 s from S3, 28.28 s from SSD IO2 and 35.08 s from GP3 (NVIDIA, Reducing Cold Start Latency). Note the gap between the weight loading time (~5–8 s) and the total to ready (~23–35 s): those ~18–27 s of difference are the other four line items, above all item 5. The streamer fixed item 4 and the rest is still there, exactly what this post warns about.

Concurrency is the tuning parameter: it controls how many OS threads read tensors into the CPU buffer (and, for S3, how many client connections the host opens) (vLLM, Run:ai Model Streamer). 16 is usually enough for local NVMe; 32 for high-throughput object storage. One thread does not saturate a Gen5 NVMe; overlapping does.

# default safetensors (the slowest of the fast ones)
vllm serve <model> --load-format safetensors

# Run:ai Model Streamer (concurrent reads, overlaps read + H2D)
vllm serve <model> --load-format runai_streamer \
  --model-loader-extra-config '{"concurrency": 32}'

# Tensorizer (tensor-by-tensor stream straight to GPU from object storage/S3)
vllm serve <model> --load-format tensorizer

The maths of the weights: $t = W/B$ and the saving from overlapping

The floor of item 4 is simple: moving $W$ gigabytes at a bandwidth $B$ takes

$$t = \frac{W}{B}$$

Take a large model, a 70B. In BF16 ($b=2$ bytes/param) that is $W = 70 \times 10^9 \cdot 2 = 140$ GB; in FP8 ($b=1$), $W = 70$ GB. Bandwidth depends on the tier (see the table below): local Gen5 NVMe reads on the order of ~14 GB/s per disk; PCIe Gen5 x16 copies host→GPU at ~50 GB/s; network storage, ~1–3 GB/s. The theoretical floor for reading 140 GB from an NVMe is $140/14 = 10$ s; from the network at 2 GB/s, 70 s. The bottleneck rules.

The overlapping trick. The default loader does the two steps, reading from disk and copying H2D, in series: first it fills a buffer, then it copies, then the next one. The time is the sum:

$$t_{\text{series}} = t_{\text{read}} + t_{\text{H2D}}$$

Tensorizer and the Run:ai Model Streamer overlap them: while one thread copies a chunk to VRAM, another is already reading the next one from disk. With enough concurrency, the total time tends towards the maximum of the two, not the sum:

$$t_{\text{overlapped}} \approx \max(t_{\text{read}},\, t_{\text{H2D}})$$

With the 70 GB of the 70B in FP8, from NVMe at 14 GB/s and PCIe at 50 GB/s: $t_{\text{read}} = 70/14 = 5.0$ s, $t_{\text{H2D}} = 70/50 = 1.4$ s. In series, $5.0 + 1.4 = 6.4$ s; overlapped, $\max(5.0,\,1.4) = 5.0$ s. The saving here is modest (~22%) because the disk dominates comfortably. Overlapping shines when the two steps are comparable: from network storage at 5 GB/s, $t_{\text{read}} = 70/5 = 14$ s and $t_{\text{H2D}} = 1.4$ s, series 15.4 s vs overlapped 14 s; but with many concurrent streams saturating a fast disk and a matched H2D, going from the sum to the maximum can almost double the effective throughput. The flip side: overlapping only gives you what the physical bottleneck allows. If the disk only delivers 14 GB/s, no streamer brings you below 5 s for 70 GB, and that is what the other lever is for, moving fewer bytes (FP8 versus BF16 halves $W$).

The storage tier

TierTypical bandwidth140 GB (70B BF16)70 GB (70B FP8)Notes
Local Gen5 NVMe~14 GB/s per disk~10 s~5 sthe short path; one disk
Local Gen5 NVMe (several, RAID0)~28–50 GB/s~3–5 s~1.5–2.5 sif the loader saturates several streams
Object storage (S3/RGW)~1–3 GB/s per stream~47–140 s~23–70 sstreamer concurrency helps a great deal
NFS / shared network~1–2 GB/s~70–140 s~35–70 sputs the network and its contention in the path

The operational conclusion is the same as in From disk to HBM: the weights that serve the cold start live on the node’s local NVMe, not on the network. Network storage is the repository; the inference node has a hot local copy (pre-pull with an initContainer or a per-node cache DaemonSet). Tensorizer is the interesting exception: it streams from object storage so efficiently that it sometimes makes serving from S3/RGW viable with no local copy, at the price of depending on the bandwidth of the storage network.

Speeding up item 5: everything that is not weights

Here is the half of the cold start that fast loaders do not touch. Three fronts.

Caching the torch.compile/Inductor compilation

torch.compile has a built-in cache system: the compiled artefacts are saved after the first start-up and reused across start-ups, even across machines if it is configured properly. In vLLM, the result of the Dynamo compilation is stored in ~/.cache/vllm/torch_compile_cache/ (vLLM Blog, Introduction to torch.compile; vLLM, torch.compile integration). The consequence for the runbook: if that directory is a persistent volume shared between pods (a PVC, or a hostPath on the node’s NVMe populated by the first start-up), subsequent pods skip the compilation and start faster. The first pod pays for the compilation; the rest inherit the cache.

It is exactly “sharpening the knives the night before”: the expensive preparation is done once and reused. The nuance: the cache is sensitive to the vLLM/PyTorch version, the GPU, the quantisation and the configuration, and changing any of them invalidates the cache and you pay for the compilation once again.

Bounding the CUDA graph capture

vLLM captures CUDA graphs for a wide range of batch sizes by default, and that is expensive: the default capture is around 54 s, and on large configurations up to 294 s has been measured. By limiting the capture to the batch sizes your workload really uses (for example 1, 2, 4, 8, 16, 24, 32, 64) the cold start is cut by more than 70%, from 294 s to ~82 s in the measured case (Red Hat, vLLM with torch.compile). The knob is the engine’s compilation/CUDA graphs configuration; the principle is not to capture graphs for batch sizes you will never see.

A real trade-off: fewer captured batch sizes = faster start-up but less coverage. A batch outside the captured set falls back to the eager path (slower per request). You bound to the frequent sizes of your traffic, not of any traffic. And there is always the extreme of disabling CUDA graphs (enforce_eager): near-instant start-up on item 5 at the cost of throughput in the hot state, valid for debugging or for very low-traffic services where cold start weighs more than steady state.

Warm pools: not paying for item 5 on the critical path

The definitive shortcut for all five items is not to run them while the user waits. A warm pool, a replica already started, with weights loaded, graphs captured and kernels compiled, waiting for traffic, turns the cold start into zero for that request. It is GPU cost (partly or wholly idle) in exchange for start-up latency; we analyse it in the economics section below.

Sleep mode: the shortcut that skips all five items

And there is a shortcut halfway between “start from scratch” and “keep a whole replica warm”: vLLM’s sleep mode, the subject of Serving several models on a single GPU. Instead of killing the process, it puts it to sleep: it parks the weights in host RAM (level 1) or discards them (level 2), but keeps the process alive, and with it the CUDA context, the allocator, the CUDA graphs and the already compiled JIT kernels. The wake pays for neither item 1, 2, 3 nor 5; it only puts the weights back into VRAM (item 4, and from RAM, not from disk). That is why a wake is 18–200× faster than a full cold start, and even level 2, which reloads the weights from the same disk, is still 23–45× faster, because it skips the other four items (vLLM Blog, Sleep Mode). Sleep mode is living proof of this post’s thesis: if preserving items 1, 2, 3 and 5 gives you 18–200×, then moving bytes was never the whole cost.

An example cold start, item by item

Let us put numbers on a cold start of 70B in FP8 (70 GB) from local NVMe and see which optimisation attacks each item:

#Line itemDefaultOptimisationOptimised
1Process init~4 s(little room; minimal environment)~3 s
2CUDA context~6 s(driver; fixed)~6 s
3Allocator~4 s(fixed)~4 s
4Weight loading~25 sRun:ai streamer / Tensorizer (concurrency + overlap)~5 s
5CUDA graphs + JIT~54 storch.compile cache + bounded capture~10 s
Total~93 s~28 s

Two readings. First: optimising only item 4 (a fast loader and nothing else) brings the total down from 93 to ~73 s, a real but disappointing improvement, because item 5 is still intact. Optimising only item 5 (cache + bounded capture) brings it down to ~49 s. Attacking both brings it down to ~28 s, and that is what turns theoretical scale-to-zero into usable scale-to-zero. Second: items 2 and 3 are practically irreducible, they depend on the driver and the hardware; they constitute the floor of the cold start, on the order of 10 s, which no software optimisation crosses. Below that floor, the only answer is not to start from scratch: sleep mode (wake from sub-second to a few seconds) or a warm replica (zero).

The economics of scale-to-zero

The capacity question all of this serves is: switch off or keep warm? Scale-to-zero saves GPU while there is no traffic, but pays the cold start when it comes back. The decision is a balance between the cost of an idle GPU and the latency SLA.

The basic criterion: scale-to-zero is only viable if the cold start fits inside the start-up SLA your users tolerate, or if you have a pre-warming mechanism that starts the replica before the request arrives (load prediction, an early signal from the queue). Without pre-warm, a 90 s cold start means the first user after an idle period waits 90 s, unacceptable for almost any interactive SLO. With the cold start brought down to ~28 s it is still a lot for an interactive request, but it is already tolerable for batch workloads or for a pre-warm triggered by autoscaling with KEDA on queue depth.

The decision rule, in one sentence: keep a warm replica when the cost of the idle GPU during the troughs is lower than the cost of missing the SLA on every start-up; do scale-to-zero (with pre-warm) when the troughs are long and deep and the optimised cold start fits in your tolerance. For a 24×7 interactive service with irregular but continuous traffic, a floor of warm replicas almost always wins. For a large model invoked a few times a day (a 70B for hard tasks), scale-to-zero with an optimised cold start, or sleep mode if it shares a GPU with another model, is the sensible option. It is the same capacity planning boundary: cold start is a parameter of the replica cushion, not a start-up detail.

Applied to the generic 4×H100 cluster

Let us bring the runbook down to the 4 H100 SXM 80 GB with NVLink. The concrete decisions:

Loader and format. Weights in safetensors on disk (never pickle). For the short path from local NVMe, the Run:ai Model Streamer (--load-format runai_streamer, concurrency 16–32) is the default option: concurrent reads, overlapping read and H2D, and you only change one flag. Keep Tensorizer for the case where you serve from object storage (RGW/S3) and want to decouple the weights from the image with no local copy, since its tensor-by-tensor streaming straight to the GPU is what makes that pattern viable. In both cases, FP8 versus BF16 halves item 4 almost for free (measure the quality, do not assume it).

Storage tier. Model repository on the network (Ceph RGW), hot copy on the node’s local NVMe populated by pre-pull. The cold start that counts is the cold one, on a node where the files are not in page cache; serving that start-up from the network puts its latency and contention into the critical path.

Caching item 5. The torch_compile_cache directory on a persistent volume or a hostPath NVMe shared between pods of the same model: the first pod compiles, the rest inherit. And bound the CUDA graph capture to the real batch sizes of each service’s traffic.

When scale-to-zero with pre-warm vs a warm replica. The agent LLM (main service, continuous traffic) never does scale-to-zero: a floor of warm replicas on one or two GPUs. The occasional 70B does: scale-to-zero with an optimised cold start, or, if it shares a GPU with a mid-sized model, sleep mode so the wake is sub-second instead of a 28 s start-up (exactly the play in Serving several models on a single GPU). The optimised cold start in this post is what makes swap and failover between replicas tolerable: a replica that falls over and is replaced in 28 s degrades less than one that takes 90 s. And it connects directly with multi-tenancy: a short cold start lets you share GPUs between teams with per-tenant scale-to-zero without the first user of each tenant paying for an endless start-up.

The cross-cutting principle: cold start is the ceiling on elasticity. The five line items, with both halves attacked, a fast loader for the weights and cache plus bounded capture for the graphs, bring it down from minutes to seconds. And below the irreducible floor of ~10 s (CUDA context + allocator), the only way out is not to start from scratch: sleep mode or a warm replica.

Traps and things that are not what they seem

“I switched to a fast loader and start-up barely improved.” Your bottleneck probably was not item 4. If CUDA graph capture eats 54 s, bringing weight loading down from 25 to 5 s takes 20 s off 93, which is barely noticeable. Measure where the time goes before optimising: vLLM’s start-up logs break down weight loading versus graph capture.

“The second time it started in a flash.” The page cache (for the weights) and the torch.compile cache (for the graphs) lying to you at the same time. The start-up that matters is the cold one, on a clean node. Benchmarking the second start-up measures a situation that almost never occurs in the peak that triggers the autoscaler.

“Tensorizer/streamer will give me the 18–200× of sleep mode.” No. Those loaders speed up only item 4. The 18–200× of sleep mode comes from preserving the other four by keeping the process alive. They are tools for different problems: a fast loader is for a real cold start (a new process); sleep mode is for alternating models without killing the process.

"enforce_eager fixes the cold start." It fixes item 5 (no graph capture, near-instant start-up) but degrades throughput in the hot state, since CUDA graphs exist because they speed up the steady state. It is a trade: it wins for very low traffic services where start-up weighs more than the hot path; it loses for a service with sustained load.

“I bound the CUDA graphs to a single batch size and that is it.” You bound to the sizes your traffic uses, not to one. A batch outside the captured set falls back to the eager path and runs slower. Too aggressive and you penalise the steady state to gain a few seconds of start-up.

“Moving the weights to FP8 also speeds up item 5.” Not directly. FP8 halves the bytes (item 4) and doubles inference throughput, but graph capture and kernel compilation do not depend on the size of the weights, they depend on the shape of the graph and the batch sizes. Item 5 is attacked with caching and bounded capture, not with quantisation.

Conclusion

From disk to HBM said it conceptually: moving the weights is one of the five line items of cold start. This runbook turns it into action. For the weights, item 4, there is a clear menu: safetensors as the base (mmap, zero-copy, no pickle), Run:ai Model Streamer for the short path from NVMe (concurrent reads overlapping read and H2D), Tensorizer for streaming from object storage with no local copy, and the cross-cutting lever of moving fewer bytes with FP8. But the forgotten half of the start-up is item 5, CUDA graph capture and JIT compilation, which on a modern server can be as big as the weight loading: you attack it by caching the compilation across pods and bounding the graph capture to the real batch sizes. Attacking only one half leaves the cold start half done; attacking both brings it from ~90 s to ~28 s. And below the irreducible floor of ~10 s imposed by the CUDA context and the allocator, the only way out is not to start from scratch: sleep mode (sub-second wake) or a warm replica (zero). The kitchen opens on time when somebody took care of the produce and of sharpening the knives the night before. Optimising only the delivery from the store leaves the ovens cold and the restaurant shut.

See also

References