--- title: "The pantry lift: from disk to HBM, or why the kitchen opens late" url: https://blog.lo0.es/en/posts/pantry-lift-from-disk-to-hbm-why-the-kitchen-opens-late/ markdown: https://blog.lo0.es/en/posts/pantry-lift-from-disk-to-hbm-why-the-kitchen-opens-late/index.md lang: en date: 2026-06-07 lastmod: 2026-09-13 author: "José Manuel Román" description: "Before the first token you have to push 140 GB of weights from disk up to HBM. Hardly anyone times that journey, disk → page cache → host buffer → PCIe → HBM, and it is the one that decides how long an inference pod takes to open: 10 s or 60 s. This post opens up the model loading path: why HBM is never the bottleneck, why the default loader deserialises slowly, what GPUDirect Storage, the Run:ai Model Streamer and fastsafetensors do, the 10 knobs worth touching, and why cold start is the hidden tax on autoscaling." tags: ["cold-start","model-loading","safetensors","gpudirect-storage","nvme","vllm","runai-streamer","fastsafetensors","on-premise","fundamentals"] translations: es: https://blog.lo0.es/posts/del-disco-a-la-hbm-cold-start-carga-modelo/index.md --- # The pantry lift: from disk to HBM, or why the kitchen opens late > This is a trip down to the basement. The [under the engine](https://blog.lo0.es/en/posts/nvlink-nvswitch-nccl-cable-every-token-crosses/index.md) series optimised the **hot path**, what happens with every token once you are already serving. This post looks at the journey that comes **before** serving: how the weights climb from disk to HBM. It is the first of a pair on the two things that happen outside the API and that almost nobody times: **model loading** (this one) and [execution on the silicon](https://blog.lo0.es/en/posts/sms-cuda-streams-cuda-graphs-inference/index.md) (the next one). ## TL;DR Before an inference pod generates its first token, it has to **push the whole model into HBM**. A Llama-70B in FP16 is **140 GB** travelling along a path nobody draws: **disk → page cache → host buffer → PCIe → HBM**. Intuition fails here: HBM is not the bottleneck, it moves 3.35 TB/s and swallows 140 GB in **42 ms**; the bottleneck is the **supply chain**. The Gen5 NVMe disk reads at ~14 GB/s (10 s for 140 GB); PCIe Gen5 copies host→GPU at ~50 GB/s (2.8 s); and the **default safetensors loader**, which deserialises tensor by tensor and bounces every byte through a CPU buffer, inflates all of that up to **30-60 s**. That time is the **cold start**, and it is the hidden tax paid by [autoscaling](https://blog.lo0.es/en/posts/llm-inference-autoscaling-kubernetes-hpa-keda/index.md) (scale-from-zero), by [canary/blue-green](https://blog.lo0.es/en/posts/canary-blue-green-shadow-llm-models/index.md) and by [disaggregated serving](https://blog.lo0.es/en/posts/disaggregated-serving-prefill-decode-specialised-pods/index.md) every time a pod is born. There are three families of solution, **GPUDirect Storage** (direct DMA disk→HBM, no bounce through the CPU), **fastsafetensors** (4.8-7.5× over the default loader) and the **Run:ai Model Streamer** (concurrent reads that saturate the disk), plus the simplest lever of all: **move fewer bytes** (FP8 is half of FP16). This post explains the path, the maths, the 10 knobs, and the cruellest trap: *"the second time it loaded fast"* is not your loader being good, it is the **page cache** lying to you. On the generic 4×H100 SXM cluster. ## Where you are: the basement, before opening > **Diagram:** The loading path: from disk to HBM, before the hot path of the token — [see the HTML version](https://blog.lo0.es/en/posts/pantry-lift-from-disk-to-hbm-why-the-kitchen-opens-late/) ## The analogy: the pantry lift Let us stay in the restaurant of the series. The [shared table](https://blog.lo0.es/en/posts/nvlink-nvswitch-nccl-cable-every-token-crosses/index.md) was the NVSwitch, the [floor next door](https://blog.lo0.es/en/posts/numa-hugepages-cpu-isolation-gpu-waits-kernel/index.md) was NUMA, the [maître](https://blog.lo0.es/en/posts/cpu-memory-topology-manager-rke2/index.md) was the kubelet. All of that describes the restaurant **running**, with diners seated. But there is a moment no post looked at: **before opening**, somebody has to bring the entire pantry up from the basement store to the kitchen. The model weights are the ingredients. They live in the **basement store** (the disk). The kitchen, the **hot line** where tokens are plated, is the GPU's HBM. And between one and the other there is a **service lift**: the path disk → host → PCIe → HBM. The kitchen cannot serve the first dish until the pantry is upstairs and in place. That restocking time is the **cold start**. The intuition trap: the kitchen (HBM) is enormous and blazingly fast, it places ingredients at 3.35 TB/s. So we blame the kitchen when the restaurant opens late. But the kitchen is standing idle **waiting for the lift**. The bottleneck is never the hot line: it is the **lift and the store**. And worse still, there is a porter (the default loader) who, instead of carrying whole crates, **takes the ingredients out one by one, writes each one down in a notebook and packs them again** before sending them up. That porter, not the lift, is half the problem. ## The mechanism: what really happens when a model loads When vLLM starts up with a model in `safetensors`, the 140 GB of the Llama-70B FP16 make this journey: 1. **Disk → page cache.** The kernel reads the `.safetensors` files from NVMe into the **page cache** (host RAM). If this is the first time after a reboot, it is a physical read from disk (~14 GB/s on Gen5). If the files are already in page cache from an earlier start-up, this is almost free, and here is where the trap we will see is born. 2. **Deserialise.** The default Hugging Face loader `mmap`s the file and builds the tensors **one by one**, copying them into a CPU tensor before moving them. This is single-threaded CPU work that saturates neither the disk nor PCIe: most of the "slow" loading time goes here, not on moving bytes. 3. **Host → HBM (H2D).** Each tensor is copied from the host buffer into HBM over **PCIe Gen5 x16** (~50 GB/s in practice). For the DMA to be efficient, the host buffer should be **pinned**, which connects directly with the [hugepages and pinned memory](https://blog.lo0.es/en/posts/numa-hugepages-cpu-isolation-gpu-waits-kernel/index.md) of the NUMA post. 4. **Place in HBM.** HBM receives the 140 GB. At 3.35 TB/s, **this takes 42 ms**. It is never the bottleneck. The path has a **shortcut**: **GPUDirect Storage (GDS)**. Instead of bouncing through the CPU buffer (steps 2-3), a DMA engine close to the NVMe controller writes **straight from disk to HBM**, without involving the CPU. It is the same principle as the [GPUDirect RDMA of the network](https://blog.lo0.es/en/posts/network-numa-cilium-ebpf-dranet-fourth-leg-pinning/index.md): get the CPU out of the way. `fastsafetensors` uses GDS and reaches **26.4 GB/s** reading a Llama-70B from NVMe across 4 GPUs. > **Diagram:** Two paths: default loader bouncing through the CPU vs direct GPUDirect Storage — [see the HTML version](https://blog.lo0.es/en/posts/pantry-lift-from-disk-to-hbm-why-the-kitchen-opens-late/) ## Why the problem exists: the economics of bytes The size of the model in bytes is decided by quantisation, and that sets the floor of the cold start, because **all** those bytes have to move before the first token. For a 70B parameter model: | Format | Bytes/param | Size 70B | Read 1 NVMe @14 GB/s | H2D PCIe @50 GB/s | HBM @3.35 TB/s | |---|---|---|---|---|---| | FP16 / BF16 | 2 | 140 GB | 10.0 s | 2.8 s | 42 ms | | FP8 | 1 | 70 GB | 5.0 s | 1.4 s | 21 ms | | INT4 (GPTQ/AWQ) | 0.5 | ~35 GB | 2.5 s | 0.7 s | 10 ms | Three readings of this table: **HBM never shows up as a problem.** The last column is always milliseconds. Anyone saying "the GPU is slow to load" is blaming the wrong place. **Quantising is the most underrated cold start lever.** Going from FP16 to FP8 does not only double inference throughput (less HBM bandwidth per token, as we saw in [quantization](https://blog.lo0.es/en/posts/quantization-llm-inference-fp8-int4-gguf/index.md)): **it also halves the cold start**, because there are half as many bytes to push up. It is a two-for-one that sizing usually ignores. **The disk is the bottleneck on bytes; the loader is the bottleneck on time.** The figures in the table are the **theoretical floor**, just moving bytes. The default loader adds single-threaded deserialisation on top, which is the difference between the theoretical 10 s and the real 30-60 s. That is why the solutions attack on two fronts: **fewer bytes** (quantisation) and **a better porter** (concurrent loaders / GDS). ## The maths that matter: cold start as a tax on autoscaling Cold start is not paid once. It is paid **every time a pod is born**. And on an elastic platform, pods are born continuously. Take an [autoscaling setup with KEDA](https://blog.lo0.es/en/posts/llm-inference-autoscaling-kubernetes-hpa-keda/index.md) that scales from 2 to 6 replicas when the queue grows. The 4 new replicas take this long to be ready: $$ T_{\text{ready}} = T_{\text{schedule}} + T_{\text{pull-image}} + T_{\text{load-weights}} + T_{\text{cuda-graphs}} $$ With a Llama-70B FP16 and the default loader, $T_{\text{load-weights}}$ dominates: it can be **40 s** of the ~60 s total. During those 40 s, the queue that triggered the autoscaling **keeps growing**, since the new replicas absorb no traffic until they have loaded. The real number in the formula is not "how many replicas", it is **how long each one takes to start serving**, and that number is written by the loading path. This has a harsh operational consequence: **scale-to-zero is unworkable for workloads with a latency SLO if the cold start is 40 s.** Nobody waits 40 s for the first token. The real elasticity of an inference platform is not limited by GPU availability, it is limited by **how long that GPU takes to have the model inside it**. Bringing the cold start down from 40 s to 8 s (with streamer + FP8) is what turns "theoretical scale-to-zero" into "usable scale-to-zero". The published numbers give the order of magnitude of the improvement: `fastsafetensors` cuts start-up from **12.39 s to 4.74 s** on a Llama-2-13B over 4×L40S, and from **16.04 s to 6.88 s** on 1×A100, **4.8-7.5×** over the default deserialiser. The **Run:ai Model Streamer** loads in **4.88 s from S3 at concurrency 32** and **7.53 s from SSD IO2 at concurrency 8**; integrated into vLLM, total time to *ready* drops to ~23 s from S3. There is no magic here: it is **getting the CPU out of the loop** (GDS) and **reading in parallel** (concurrency) to saturate the disk instead of leaving it half idle while one thread deserialises. ## The 10 knobs worth touching ### Knob 1 — Measure where the time goes (read vs deserialise vs H2D) Before touching anything: time it. Is the time going on reading from disk, on deserialising, or on the H2D? `iostat -x 1` during the load tells you whether the NVMe is saturated (disk bottleneck) or nearly idle (loader/CPU bottleneck). If the disk is at 20%, the problem is **not** the disk: it is the porter. Changing disk would fix nothing; changing loader would. ### Knob 2 — `--load-format`: choosing the porter vLLM exposes several loaders via `--load-format`: `safetensors` (default), `runai_streamer`, `fastsafetensors`, `tensorizer`. The default is the slowest. Changing one flag can be the cheapest 4-7× there is. ```bash # Run:ai Model Streamer (concurrent reads, saturates the disk) vllm serve meta-llama/Llama-3.1-70B --load-format runai_streamer # fastsafetensors (GPUDirect Storage, direct DMA disk→HBM) vllm serve meta-llama/Llama-3.1-70B --load-format fastsafetensors ``` ### Knob 3 — Streamer concurrency The Run:ai Model Streamer splits the read across N threads according to the size of each tensor in order to saturate the storage bandwidth. Concurrency is the key parameter: **16 is usually enough for local NVMe; 32 (sometimes 64)** for high-throughput network storage. One thread does not saturate a Gen5 NVMe; 32 do. ```bash vllm serve --load-format runai_streamer \ --model-loader-extra-config '{"concurrency": 32}' ``` ### Knob 4 — GPUDirect Storage (GDS) + fastsafetensors If there is an `nvidia-fs` driver, a supported filesystem and local NVMe, GDS writes straight disk→HBM with no bounce through the CPU. It is the difference between the two paths in the diagram. **But**: it requires the stack to be in place, and it only wins if the bottleneck is the CPU bounce, not the disk itself. Check with `gdscheck`. ### Knob 5 — Local NVMe for the weights, not network Serving the weights from Ceph RGW / NFS is convenient (one shared place) but it puts the **network** into the loading path. For cold start, **weights on the node's local NVMe** (or a local cache). Network storage is for the model repository; the inference node should have a hot local copy. ### Knob 6 — Pre-pull / local model cache An `initContainer` that downloads the model onto a `local` or `hostPath` NVMe volume before starting vLLM turns a cold start "from the network" into one "from local NVMe". Combined with a per-node cache DaemonSet, new pods on an already warm node read from local disk, not from the network. ### Knob 7 — Quantisation: move fewer bytes Weights already in FP8 or INT4 on disk = half or a quarter of the cold start. It is the knob from the table above. And it composes with all the others: FP8 + streamer + GDS is multiplicative. ### Knob 8 — Parallel loading across GPUs (vLLM V1) vLLM's V1 engine (the default since 0.19) loads the weight shards **in parallel** across the GPUs of a TP group, instead of sequentially. At TP=4, each GPU loads its quarter at the same time. Check it is active; in old versions loading was serial and the cold start of TP=4 was almost 4× that of TP=1. ### Knob 9 — NUMA locality of the NVMe The NVMe hangs off a **PCIe root under a specific socket**, exactly the same NUMA map as the [host post](https://blog.lo0.es/en/posts/numa-hugepages-cpu-isolation-gpu-waits-kernel/index.md). If the load's host buffer lands on the wrong socket, the H2D crosses the UPI. The fifth list to align, alongside `isolcpus`, `reserved-cpus` and the NIC IRQs: **which socket is local to the NVMe and to the destination GPU**. `nvidia-smi topo -m` shows it. ### Knob 10 — Do not pay the cold start: keep pods warm Sometimes the answer is not to load faster, but **not to unload**. A floor of always-live replicas (no scale-to-zero), or a pool of preloaded warm standbys, swaps "wait 40 s" for "0 s". It is the cost of idle GPU in exchange for start-up latency: a [capacity planning](https://blog.lo0.es/en/posts/capacity-planning-on-premise-llm-inference/index.md) decision, not a technical one. ### Summary table | # | Knob | What it attacks | Risk / cost | |---|---|---|---| | 1 | `iostat` while loading | knowing whether the bottleneck is disk or loader | none; always do it first | | 2 | `--load-format` | single-threaded deserialisation | format compatibility | | 3 | streamer concurrency | underused disk | host RAM for buffers | | 4 | GPUDirect Storage | bounce through the CPU | requires nvidia-fs + supported FS | | 5 | local NVMe vs network | the network in the path | duplicating weights per node | | 6 | pre-pull / node cache | network on every start-up | local disk space | | 7 | FP8/INT4 quantisation | bytes to move | quality (measure, do not assume) | | 8 | V1 parallel loading | serial loading across GPUs | none if V1 is active | | 9 | NUMA of the NVMe | H2D crossing the UPI | align with the rest of the NUMA lists | | 10 | warm pods / no zero | the whole cold start | idle GPU paid for | ## How it connects with the rest of the stack **With autoscaling.** The whole of [scale-from-zero with KEDA](https://blog.lo0.es/en/posts/llm-inference-autoscaling-kubernetes-hpa-keda/index.md) rests on this: real elasticity is limited by the cold start, not by GPU availability. An autoscaler with 40 s of loading reacts late to every peak. **With disaggregated serving.** In [disaggregated prefill/decode](https://blog.lo0.es/en/posts/disaggregated-serving-prefill-decode-specialised-pods/index.md), bringing up a decode pool on demand pays the cold start of loading the model into each new pod. The elasticity of the pattern depends on how fast those pods start. **With canary/blue-green.** Every [canary deployment](https://blog.lo0.es/en/posts/canary-blue-green-shadow-llm-models/index.md) loads a new version of the model alongside the old one. The validation time of a canary includes its cold start; large models make deployments slower and more expensive. **With NUMA and hugepages.** The load's host buffer wants to be **pinned** and **NUMA-local**, the same thing the [host post](https://blog.lo0.es/en/posts/numa-hugepages-cpu-isolation-gpu-waits-kernel/index.md) asked for on the hot path. The loading path is another client of the same NUMA map. **With quantisation.** [FP8/INT4](https://blog.lo0.es/en/posts/quantization-llm-inference-fp8-int4-gguf/index.md) is not only inference throughput: it is the direct lever on the bytes of the cold start. **With capacity planning.** [Sizing](https://blog.lo0.es/en/posts/capacity-planning-on-premise-llm-inference/index.md) that ignores cold start underestimates how many replicas are needed to absorb a peak: if they take 40 s to start, you need a bigger permanent cushion. ## Traps and things that are not what they seem **"The second time it loaded really fast."** This is the star trap. The first load filled the **page cache** (host RAM); the second reads from RAM, not from disk, and flies. But in production pods are ephemeral and are born on different nodes: the start-up that counts is the **cold** one, on a node where those files are not in page cache. Benchmarking the second load is measuring a situation that almost never happens at the moment that matters (the peak that triggers the autoscaler). **"GDS always speeds things up."** No. GDS removes the bounce through the CPU; if your bottleneck is the disk itself (saturated NVMe) or the deserialisation, GDS does not touch that part. Measure first (knob 1). It also demands `nvidia-fs`, a supported filesystem, and sometimes it does not work over whatever network storage you have. **"mmap makes loading instant."** `mmap` maps the file but **reads nothing yet**: the cost is deferred to the first access to each page. The time does not disappear, it moves, and the **first token** pays the page faults that start-up did not. You have moved the cold start into the latency of the first request, which is probably a worse place to have it. **Weights on network storage "because it is cleaner".** Sharing a model repository on Ceph RGW is fine for storing them; serving the cold start from there puts the network (and its latency and its contention) into the critical path. Local NVMe cache on the inference node. **Loading FP16 and quantising at start-up.** Quantising on the fly during loading (for example FP16→FP8 on the GPU) can be **slower** than having the weights already quantised on disk: you move twice the bytes and on top of that you do conversion work. If you are going to serve in FP8, store the weights in FP8. **Optimising the load and ignoring `T_cuda-graphs`.** Bringing weight loading down to 8 s and forgetting that CUDA graph capture adds several more seconds leaves the cold start half done. That second half of start-up is the subject of the [next post](https://blog.lo0.es/en/posts/sms-cuda-streams-cuda-graphs-inference/index.md). ## Conclusion The whole series optimised what happens **with every token**: the cable, the host, the network, the silicon. But before the first token there is a journey almost nobody times and that decides whether an inference pod opens in 10 s or in 60 s: pushing the model from disk to HBM. Intuition blames the GPU, and the GPU is innocent: HBM swallows 140 GB in 42 ms. The bottleneck is the **supply chain**: a disk reading at 14 GB/s, a PCIe copying at 50, and above all a default loader that deserialises tensor by tensor with a single thread and turns 10 s of bytes into 60 s of waiting. The solutions attack the two right fronts, **fewer bytes** (quantisation) and **better transport** (GDS, concurrent streamers, local NVMe), and give 4-7× almost for free. And above the technique, an idea that reorders the priorities: on an elastic platform, cold start **is not a start-up detail, it is the ceiling on elasticity**. The fastest GPU in the world does not scale if it takes 40 s to have the model inside. The pantry lift, the one nobody timed, is what decides what time the kitchen really opens. ## See also - [Speeding up model cold start: from minutes to seconds](https://blog.lo0.es/en/posts/speeding-up-model-cold-start-minutes-to-seconds/index.md) — the runbook that attacks each of the five line items of start-up. - [Serving several models on a single GPU: co-residency, model-swapping and sleep mode](https://blog.lo0.es/en/posts/serving-several-models-single-gpu-co-residency-swapping-sleep-mode/index.md) — sleep mode dodges this cold start by parking the weights in RAM instead of on disk. - [The corridors and the guard: PCIe, GPUDirect P2P and ACS](https://blog.lo0.es/en/posts/pcie-topology-gpudirect-p2p-acs-security-guard/index.md) — the bus GDS uses to push the weights up from NVMe; ACS enabled throttles that direct path just as it throttles P2P between GPUs. - [The pass: vLLM's scheduler step](https://blog.lo0.es/en/posts/vllm-scheduler-step-maitre-d-each-round/index.md) — cold start is the ceiling on the elasticity the scheduler exercises: preemption and scale-from-zero pay this load in every new pod. - [The shared table: NVLink, NVSwitch and NCCL](https://blog.lo0.es/en/posts/nvlink-nvswitch-nccl-cable-every-token-crosses/index.md) — the first of "under the engine"; this post goes down to the basement that series took for granted as full. - [The floor next door: NUMA, hugepages and CPU isolation](https://blog.lo0.es/en/posts/numa-hugepages-cpu-isolation-gpu-waits-kernel/index.md) — the pinned, NUMA-local memory the loading path needs for an efficient H2D. - [The kitchen door: network NUMA, Cilium eBPF and DRANET](https://blog.lo0.es/en/posts/network-numa-cilium-ebpf-dranet-fourth-leg-pinning/index.md) — the same "get the CPU out of the way" principle (GPUDirect) that GDS applies here to the disk. - [SM, CUDA streams and CUDA graphs](https://blog.lo0.es/en/posts/sms-cuda-streams-cuda-graphs-inference/index.md) — the second half of the cold start (graph capture) and what happens on the silicon once the weights are inside. - [Quantization for inference](https://blog.lo0.es/en/posts/quantization-llm-inference-fp8-int4-gguf/index.md) — the lever that halves the bytes of the cold start, not just HBM bandwidth per token. - [LLM autoscaling on Kubernetes with KEDA](https://blog.lo0.es/en/posts/llm-inference-autoscaling-kubernetes-hpa-keda/index.md) — why cold start is the real ceiling on elasticity and kills scale-to-zero. - [Disaggregated serving: prefill and decode separated](https://blog.lo0.es/en/posts/disaggregated-serving-prefill-decode-specialised-pods/index.md) — bringing up pools on demand pays the cold start in every new pod. - [Canary, blue-green and shadow for models](https://blog.lo0.es/en/posts/canary-blue-green-shadow-llm-models/index.md) — the validation time of a canary includes its model loading. - [Capacity planning for on-premise inference](https://blog.lo0.es/en/posts/capacity-planning-on-premise-llm-inference/index.md) — cold start as a parameter of the replica cushion and of the warm-vs-zero decision. ## References - NVIDIA, *Reducing Cold Start Latency for LLM Inference with NVIDIA Run:ai Model Streamer*: . - vLLM, *Loading models with Run:ai Model Streamer*: . - vLLM, *Loading model weights with fastsafetensors*: . - foundation-model-stack, *fastsafetensors* (high-performance loader, GDS): . - *Speeding up Model Loading with fastsafetensors* (arXiv 2505.23072): . - NVIDIA, *GPUDirect Storage: A Direct Path Between Storage and GPU Memory*: . - NVIDIA, *Magnum IO GPUDirect Storage* (benchmarking and configuration): .