Cold-Start on lo0 — Tech Blog

Knative and scale-to-zero for LLM inference: when switching off the GPU saves money and when it costs you the SLO

Mon, 31 Aug 2026 10:00:00 +0200

Closing the platform and self-service run, after Backstage and Kubeflow. The question here is purely economic with a technical trap inside: switching off the GPU when nobody is using it looks like free money, until the first user of the day waits two minutes.

TL;DR

A GPU costs the same serving as it does idle. An H100 on dedicated cloud runs at 2 to 4 euros an hour whether it is used or not, so switching the service off when there is no traffic looks like money given away. Knative is the piece that does it: its autoscaler scales down to zero replicas and brings them back when a request arrives, measuring concurrency instead of CPU, which is what the Kubernetes autoscaler cannot do.

The trap is in the cold start. Knative brings up a pod in seconds, but that is only the beginning: an LLM has to load its weights from disk into HBM and compile its graphs, and there you go from tens of seconds to minutes depending on model size and storage. One reference study measures vLLM startup at around 20 seconds even for a 3 billion parameter model, and weight loading grows linearly with size. The first user of every traffic trough pays that whole wait.

Hence the rule. Switching off the GPU pays off with sporadic traffic, many lightly used models or development environments, and it is a trap with a strict latency SLO, sustained traffic or huge models whose startup breaks any interactive budget. Between all and nothing there are three middle grounds that almost always win: keep one replica warm, share the GPU with model-swap, or speed up weight loading so that startup fits inside the autoscaler’s cycle.

The analogy: the restaurant that switches off the kitchen

A restaurant with the kitchen running at four in the afternoon, without a single customer, burns gas for nothing. The temptation is to switch it off and light it again when somebody walks in. The problem is the time it takes to heat up. If the kitchen is an induction hob, it heats in seconds and switching it off between services is obvious. If it is a wood-fired oven that takes two hours to come up to temperature, switching it off means the first customer of the night eats cold or leaves.

Serving a small model is an induction hob. Serving a 70 billion parameter model is the wood-fired oven: the weights that have to go up into HBM are the wood you have to burn until you reach temperature, and wishing does not speed it up. The decision to switch off the kitchen, that is, to scale to zero, does not depend on whether you want to save gas, but on how long your kitchen takes to heat and on how long your first customer is willing to wait. This whole article is that calculation.

How Knative scales

Knative graduated in the CNCF in October 2025 and has three components; the one that concerns us is Serving, the serverless application component with autoscaling. Its object model has four pieces: the high-level Service, the Configuration that describes the desired state, the immutable Revision which is what actually scales, and the Route that splits traffic between revisions.

What makes scaling to zero possible are two data plane components. The queue-proxy is a sidecar in every pod that measures concurrency and applies the simultaneous request limit. The activator is the key piece: when the service is at zero replicas, requests are routed to it, and it parks them in a queue, tells the autoscaler that capacity is needed, and forwards the parked request once a pod is ready. It is the waiter who takes your order and tells you the kitchen is warming up, instead of shutting the door on you.

KPA versus HPA

Knative’s own autoscaler, the KPA, measures concurrency and requests per second, not CPU or memory. That is the difference that matters: the Kubernetes autoscaler, the HPA, scales on CPU and does not know how to go down to zero, its operational minimum is one. To switch off completely you need the KPA.

The KPA works with two windows. The stable one, 60 seconds by default, averages normal traffic. The panic one, much shorter, reacts to sharp spikes: when demand exceeds twice the capacity of the current replicas, it enters panic mode and scales in one go. The default concurrency target per replica is 100, but for LLM inference the useful value is usually 1 to 4, because each request occupies the GPU heavily; the official KServe example uses a target of 1.

Two timers govern the shutdown. The scale-to-zero-grace-period, 30 seconds by default, is how long the system waits to have the restart machinery ready before withdrawing the last replica. The scale-to-zero-pod-retention-period, zero by default, is the minimum time the last pod survives after the decision to switch off. Raising this second value is one of the levers for not switching off as soon as there is a micro-trough in traffic.

Why an LLM cold start is not starting a pod

Here is the core of the article, and there is a nuance that most discussions skip. Knative bringing up a pod in seconds does not mean the model is ready in seconds. Starting an inference service has several phases, and the two costs that dominate are not the ones you would expect.

A recent study measuring vLLM cold start breaks it down step by step. For a 3 billion parameter model, total startup is around 20 seconds, and the counterintuitive finding is that most of it is CPU-bound, not GPU-bound: process initialisation, library imports, engine construction. Inside that there are two costs that grow, and they are the ones the acceleration techniques attack.

The first is weight loading, which scales linearly with model size. In that same study it goes from half a second for a small model to almost five for a 16 billion one. Projected onto real sizes, the on-disk weight rules: an 8 billion model takes around 16 GB in FP16 and 8 in FP8; a 70 billion one, 140 and 70; a 405 billion one, 810 and 405. Loading 140 GB from disk into HBM does not happen in a blink, and if storage is slow or sits on the network, the minute arrives on its own.

The second is graph compilation. Graph transformation and capture with torch.compile costs 11 to 21 seconds if there is no compilation cache, and drops to 3 to 6 with one. It is half a battle that has nothing to do with the weights, and it is won by keeping the compilation cache between startups.

The operational conclusion is that the enemy is not only HBM: it is weight loading plus compilation. For small models the weights are only a fraction of startup, and switching off the GPU is almost painless. For large models the weights dominate, and the cold start becomes the problem. Loading a 70 billion model with the naive loader of the usual library can run to several minutes, and that is exactly the time your first user waits.

The techniques that shorten the wait

The good news is that the cold start can be attacked on both fronts. On the weights side, streaming loaders such as NVIDIA’s transfer the weights to the GPU concurrently instead of sequentially: the published data takes an 8 billion model from around 48 seconds with the standard loader to around 14 with streaming from SSD, and to under 8 with fast storage. The most graphic case is a 122 billion model, 233 GB on disk, which goes from three and a half minutes to around 37 seconds. That number has a direct consequence: 37 seconds fit inside an autoscaler polling cycle of 30 to 60 seconds; three and a half minutes do not. It is the difference between scaling to zero being viable with large models or not being viable at all.

On the compilation side, the technique is cheaper still: keep the torch.compile cache between startups, and with it the 11 to 21 seconds become 3 to 6. Tools such as Tensorizer and OCI images with the weights packaged inside attack the same problem from another angle, as we saw in the article on speeding up cold start.

Knative and KServe: the two modes

KServe leans on Knative for its serverless mode, and this is what connects everything above to a real platform. KServe has two deployment modes.

The Serverless mode uses Knative Serving and the KPA, supports scaling to zero and measures concurrency and requests per second. It is enabled by setting minReplicas: 0 in the InferenceService. It is the natural path for serving many intermittent models.

The Standard or raw mode uses the Kubernetes HPA, does not scale to zero, and measures CPU and memory. You pick it when you need something Knative restricts, such as mounting several volumes, and you do not need to switch off completely.

There is a third path of particular interest to anyone who already has KEDA deployed: in Standard mode, KServe recommends KEDA for generative inference, and KEDA can scale to zero on Prometheus metrics such as queue length or tokens per second. It is a route to scaling to zero on your own metrics, outside Knative, and it fits better with autoscaling guided by the model’s real saturation signal rather than by generic concurrency.

For anyone serving many lightweight predictive models there is also ModelMesh, the high-density pattern that packs many models into the same pods and rotates them in memory, without leaning on Knative. It is the equivalent of model-swap: instead of switching off and on per model, it shares one GPU between many. It is aimed at classic short-response inference, not at generative LLMs.

The economics

The decision comes down to comparing what you save by switching off with what you risk at startup. The saving is easy to estimate. If a GPU costs $ C $ euros per hour and your service is idle $ H $ hours a day, switching it off saves on the order of

$$\text{daily saving} \approx C \cdot H$$

With an H100 at 3 euros an hour and 16 idle hours a day, that is 48 euros a day per GPU, on the order of 1,400 a month. Across a fleet of several lightly used models the figure multiplies and stops being negligible.

The cost is on the other side, and it is not measured in euros: it is measured in SLO. Scaling to zero only pays off if the cold start fits inside what your worst-case latency tolerates. With a small model and fast storage, tens of seconds at most, and for an internal assistant used in bursts that may be acceptable. With a 70 billion model and the naive loader, minutes, and no interactive SLO survives that.

Between switching off completely and never switching off there are three middle grounds, and almost always one of them is the right answer:

Warm replica, with minReplicas: 1. You remove the first user’s cold start in exchange for paying a continuous GPU-hour. The break-even point is direct: keeping it warm pays off when the cost of that GPU running hot is lower than the reputational or SLO cost of the first user waiting. For a customer-facing service, almost always.

Model-swap or sleep on a shared GPU, as we saw in serving several models on one GPU. The process stays alive and the model is unloaded and reloaded in seconds instead of starting from scratch. It is the middle ground for several models that do not justify a GPU each but do not tolerate a full startup either.

Speed up loading so that the cold start fits inside the autoscaler’s cycle, with weight streaming and a compilation cache. It is what makes real scaling to zero viable with large models, and what turns “three and a half minutes” into “37 seconds”.

Operational traps and honest scepticism

The pod starts in seconds, the model does not. Do not confuse Knative’s startup time with the time until the model answers. The difference is the whole article.

The default concurrency target is 100, and for an LLM that is a disaster. Each request occupies the GPU; a target of 1 to 4 is the reasonable one. Leaving the default 100 means accepting a hundred simultaneous requests on a replica that handles four.

The cold start is not just weight loading. Graph compilation costs what it costs, and it is attacked separately, with a cache. Optimising only storage leaves half the problem untouched.

Cilium and Gateway API are not a supported path yet. Knative supports Gateway API in beta and tests with Istio, Contour and Envoy Gateway; integration with Cilium does not appear as tested and has documented friction. If your ingress is Cilium, verify it before counting on it.

KEDA may be a better signal than the KPA. Scaling on generic concurrency is worse than scaling on the model’s real saturation metric. If you already have KEDA, Standard mode with scaling to zero on your own metrics deserves a test against the default Serverless.

For an inference factory

The decision to switch off the GPU is not ideological, it is arithmetic, and it depends on two numbers: how long your model takes to start and how long your worst user waits. Three rules close the matter.

First, switch off what is intermittent and cheap to start: development environments, small models used sporadically, the long tail of models almost nobody touches. There, scaling to zero is direct money with no appreciable SLO cost.

Second, do not switch off what is critical and expensive to start: the customer-facing model with a strict SLO, and the huge models whose cold start is measured in minutes. There a warm replica costs less than a user waiting.

Third, before deciding, measure your real startup and fix it. Many services are ruled out of scaling to zero over a three-minute startup that, with weight streaming and a compilation cache, drops to thirty seconds and changes the answer. The most profitable lever is not choosing well between switching off or not: it is making switching off cheap.

With this we close the platform and self-service run. The portal shows what is there, the ML toolbox gives what is missing without duplicating what you already have, and scaling to zero decides how much of all that is switched on when nobody is looking. All three answer the same question from different angles: how to put an expensive platform at the disposal of its users without going broke keeping it switched on.

Sources

CNCF, Cloud Native Computing Foundation announces Knative’s graduation — https://www.cncf.io/announcements/2025/10/08/cloud-native-computing-foundation-announces-knatives-graduation/
Knative Docs, Serving architecture — https://knative.dev/docs/serving/architecture/
Knative Docs, Request flow — https://knative.dev/docs/serving/request-flow/
Knative Docs, Configuring KPA-specific autoscaling — https://knative.dev/docs/serving/autoscaling/kpa-specific/
Knative Docs, Configuring concurrency — https://knative.dev/docs/serving/autoscaling/concurrency/
Knative Docs, Configuring scale-to-zero — https://knative.dev/docs/serving/autoscaling/scale-to-zero/
Knative Blog, Managing Gateway API ingress with the Knative Operator — https://knative.dev/blog/articles/gateway-api-ingress-with-knative-operator/
CNCF, KServe becomes a CNCF incubating project — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
KServe Docs, Autoscaling with Kubernetes HPA (Serverless vs Standard) — https://kserve.github.io/website/docs/model-serving/predictive-inference/autoscaling/hpa-autoscaler
KServe Docs, Autoscaler for generative inference (KEDA) — https://kserve.github.io/website/docs/model-serving/generative-inference/autoscaling
KServe GitHub, docs/samples/autoscaling (InferenceService minReplicas 0) — https://github.com/kserve/kserve/blob/master/docs/samples/autoscaling/README.md
Wang et al. (MLSys 2026), Breaking the ice: analyzing cold start latency in vLLM — https://arxiv.org/abs/2606.07362
NVIDIA Developer Blog, Reducing cold start latency for LLM inference with NVIDIA Run:ai Model Streamer — https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer
Microsoft Azure SDK Blog, Eliminate LLM cold starts: load models up to 6x faster — https://devblogs.microsoft.com/azure-sdk/eliminate-llm-cold-starts-load-models-up-to-6x-faster-with-azure-blob-storage-and-runai-model-streamer/
Hugging Face Blog, Llama 3.1 (VRAM and precision tables) — https://huggingface.co/blog/llama31
CloudZero, H100 GPU cost 2026 — https://www.cloudzero.com/blog/h100-gpu-cost/

Speeding up model cold start: from minutes to seconds

Thu, 11 Jun 2026 13:00:00 +0000

This is the third batch of an operational series on squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB with NVLink. The sister pieces in this batch are Multimodal: serving a VLM on-premise with vLLM, adding vision to the same engine, and FinOps and GPU multi-tenancy with LiteLLM, sharing out and charging for the GPU across teams. This post takes the problem that From disk to HBM laid out conceptually and turns it into a runbook: how you actually bring the cold start down, knob by knob, so that elasticity is usable.

TL;DR

The post From disk to HBM left an uncomfortable idea behind: moving the weights from disk to HBM is only one of the five line items of cold start. The other four, starting the process, creating the CUDA context, setting up the allocator and, the most expensive one that hardly anyone looks at, capturing the CUDA graphs and compiling the JIT kernels, are untouched by any fast loader. That is why a server that loads the weights in 6 s can still take 90 s to be ready. This runbook attacks all five. For the weights: safetensors (mmap, zero-copy, no bounce through a host FP32 buffer) versus the pickle of torch.load; Tensorizer (CoreWeave), which serialises the model into one file and streams it tensor by tensor straight to the GPU from object storage/HTTP/S3; and the Run:ai Model Streamer (from NVIDIA), which reads concurrently and overlaps the read with the H2D copy. All three are switched on with one flag, --load-format, in vLLM. For the non-weight part: torch.compile/Inductor compilation cache (persisted and reused across start-ups), bounding the CUDA graph capture to the batch sizes you really use (the default capture can eat ~54 s), and sleep mode as a shortcut that skips all five line items. The maths: $t = W/B$ for the weights, the saving from overlapping read and H2D instead of doing them in series, and a 70B cold start broken down item by item. And the economics: when scale-to-zero with pre-warm beats keeping a warm replica. On the generic 4×H100 SXM 80 GB cluster.

The analogy: opening the kitchen in the morning

A restaurant does not open by flicking a switch. Anyone who has worked in one knows it: opening time is decided by everything that happens beforehand, and almost none of it is “bringing in the food”.

The first cook arrives at a cold, dark place. They have to: turn on the lights and start the systems (the process init); fire up the ovens and let them come up to temperature (the CUDA context, which takes its time to become operational); get the boards out, sort the cold room, organise the work space (the memory allocator, which reserves and structures the HBM); go down to the store and bring up the produce, the crates of vegetables, the meat, the fish, and put it in its place (loading the weights from disk to HBM, the entire subject of From disk to HBM); and, the most forgotten of all, sharpen every knife, set up the mise en place, make the stocks, pre-heat the mother sauces, all the preparation that is not an ingredient but without which not a single dish goes out (CUDA graph capture and JIT compilation of the kernels).

Here is the trap that From disk to HBM already anticipated: if you only optimise the produce, you still open late. You can hire the best delivery service in the world, a blazingly fast service lift, crates that put themselves away. But if the cook is still sharpening knives and waiting for the ovens to come up to temperature, the kitchen does not serve the first dish. The produce arrived early and the restaurant is still shut.

Today’s runbook is, literally, the list of everything that has to be done for the kitchen to open on time, and which of those tasks can be sped up, which can be left done from the night before (cached), and when it pays better never to switch the kitchen off (warm replica) than to light it again every morning (scale-to-zero).

The five line items of cold start

From the moment an inference pod is born until it returns its first token, the clock runs in five different places. It is worth naming them, because each optimisation attacks one or two, never all of them:

Process init. Starting the Python interpreter, importing the engine (vLLM and its dependencies), parsing the configuration. Seconds, and it grows with the size of the environment and the CUDA/PyTorch imports.
CUDA context. The first cudaSetDevice / cudaFree(0) initialises the CUDA runtime against the driver: it loads the context and maps the GPU. It is not instant, on the order of one to several seconds per GPU depending on driver and number of devices.
Allocator. vLLM sets up its memory manager over the HBM (PyTorch’s caching allocator plus the KV-cache layout). Reserving and structuring tens of GB has its cost.
Weight loading disk→HBM. The journey that From disk to HBM dissected: disk → page cache → host buffer → PCIe → HBM. What almost everyone believes cold start is, and it is only one of the five.
CUDA graph capture + kernel JIT. vLLM captures CUDA graphs for the various batch shapes and compiles kernels with torch.compile/Inductor (and backends such as DeepGEMM or FlashInfer). This item is the great ignored one: the default graph capture in vLLM takes on the order of 54 s because it covers a wide range of batch sizes (vLLM, torch.compile integration; Red Hat, vLLM with torch.compile).

The lesson the diagram is shouting: on a modern server with torch.compile and CUDA graphs, item 5 can be as big as or bigger than item 4. Speeding up weight loading from 40 s to 6 s is a huge advance, but if you leave graph capture at 50 s, the cold start falls from 104 s to ~70 s, not to 12 s. You have to attack both halves.

Speeding up item 4: loading the weights

safetensors versus pickle: the format rules

The first knob is the on-disk format. The old torch.load uses pickle, which has two problems. The security one is well known: deserialising a pickle executes arbitrary Python code, since the __reduce__ protocol lets the file invoke any callable on load, so every model download was a remote execution vector (HuggingFace, Safetensors). The performance one is what concerns us: loading a pickle rebuilds Python objects and usually passes through an intermediate host buffer before reaching the GPU.

safetensors, now under the PyTorch Foundation (HuggingFace, Safetensors joins PyTorch Foundation), solves both. The file is just a JSON header + raw tensor bytes: loading it cannot do anything except populate tensor buffers. And the key property for us is physical: the data region is aligned to a page boundary (the header is padded so that the first tensor starts at a multiple of the OS page size). That is what makes zero-copy possible: a loader can mmap the file, cudaHostRegister over the mapped region and DMA straight from the page cache into VRAM, with no torch.load deserialisation and no temporary host FP32 buffer (HuggingFace, Safetensors). It is the base format; everything else is built on top of it.

The mmap trap (already flagged in From disk to HBM): mmap reads nothing immediately, it defers the cost to the first access to each page. If you do not force the read, the cold start looks short and the first token pays the page faults. And the “blazingly fast second load” is the page cache lying to you: in production pods are born cold, on nodes where those files are not cached.

Tensorizer: streaming tensor by tensor straight to the GPU

Tensorizer, from CoreWeave, serialises the model’s weights and its tensors into a single file and, instead of loading the whole model into RAM before moving it to the GPU, streams the data tensor by tensor from disk, an HTTP/HTTPS endpoint or an S3 bucket, deserialising on the fly straight onto the GPU (vLLM, Loading Models with CoreWeave’s Tensorizer). The operational advantage: near-instant loading and low host RAM usage during initialisation, which matters especially in serverless and autoscaling scenarios, exactly the scale-to-zero case. And it decouples the weights from the container image: the model lives in object storage, not inflating the image pull.

In vLLM it is enabled with --load-format tensorizer (or load_format="tensorizer" via the API). It requires serialising the model once into Tensorizer’s format; from then on, any pod streams it.

Run:ai Model Streamer: reading and copying at the same time

The Run:ai Model Streamer (from NVIDIA) attacks the bottleneck from another angle: it reads the tensors concurrently, N OS threads reading from storage into the CPU buffer, while streaming them to VRAM, so that the storage read and the H2D copy overlap instead of happening in series (vLLM, Loading models with Run:ai Model Streamer; NVIDIA, Reducing Cold Start Latency). It reads safetensors directly, with no reconversion.

The numbers published by NVIDIA give the order of magnitude: the streamer reaches 4.88 s reading from S3 at concurrency 32 and 7.53 s from an SSD IO2 at concurrency 8; integrated into vLLM, the total time to ready drops to 23.18 s from S3, 28.28 s from SSD IO2 and 35.08 s from GP3 (NVIDIA, Reducing Cold Start Latency). Note the gap between the weight loading time (~5–8 s) and the total to ready (~23–35 s): those ~18–27 s of difference are the other four line items, above all item 5. The streamer fixed item 4 and the rest is still there, exactly what this post warns about.

Concurrency is the tuning parameter: it controls how many OS threads read tensors into the CPU buffer (and, for S3, how many client connections the host opens) (vLLM, Run:ai Model Streamer). 16 is usually enough for local NVMe; 32 for high-throughput object storage. One thread does not saturate a Gen5 NVMe; overlapping does.

# default safetensors (the slowest of the fast ones)
vllm serve <model> --load-format safetensors

# Run:ai Model Streamer (concurrent reads, overlaps read + H2D)
vllm serve <model> --load-format runai_streamer \
 --model-loader-extra-config '{"concurrency": 32}'

# Tensorizer (tensor-by-tensor stream straight to GPU from object storage/S3)
vllm serve <model> --load-format tensorizer

The maths of the weights: $t = W/B$ and the saving from overlapping

The floor of item 4 is simple: moving $W$ gigabytes at a bandwidth $B$ takes

$$t = \frac{W}{B}$$

Take a large model, a 70B. In BF16 ($b=2$ bytes/param) that is $W = 70 \times 10^9 \cdot 2 = 140$ GB; in FP8 ($b=1$), $W = 70$ GB. Bandwidth depends on the tier (see the table below): local Gen5 NVMe reads on the order of ~14 GB/s per disk; PCIe Gen5 x16 copies host→GPU at ~50 GB/s; network storage, ~1–3 GB/s. The theoretical floor for reading 140 GB from an NVMe is $140/14 = 10$ s; from the network at 2 GB/s, 70 s. The bottleneck rules.

The overlapping trick. The default loader does the two steps, reading from disk and copying H2D, in series: first it fills a buffer, then it copies, then the next one. The time is the sum:

$$t_{\text{series}} = t_{\text{read}} + t_{\text{H2D}}$$

Tensorizer and the Run:ai Model Streamer overlap them: while one thread copies a chunk to VRAM, another is already reading the next one from disk. With enough concurrency, the total time tends towards the maximum of the two, not the sum:

$$t_{\text{overlapped}} \approx \max(t_{\text{read}},\, t_{\text{H2D}})$$

With the 70 GB of the 70B in FP8, from NVMe at 14 GB/s and PCIe at 50 GB/s: $t_{\text{read}} = 70/14 = 5.0$ s, $t_{\text{H2D}} = 70/50 = 1.4$ s. In series, $5.0 + 1.4 = 6.4$ s; overlapped, $\max(5.0,\,1.4) = 5.0$ s. The saving here is modest (~22%) because the disk dominates comfortably. Overlapping shines when the two steps are comparable: from network storage at 5 GB/s, $t_{\text{read}} = 70/5 = 14$ s and $t_{\text{H2D}} = 1.4$ s, series 15.4 s vs overlapped 14 s; but with many concurrent streams saturating a fast disk and a matched H2D, going from the sum to the maximum can almost double the effective throughput. The flip side: overlapping only gives you what the physical bottleneck allows. If the disk only delivers 14 GB/s, no streamer brings you below 5 s for 70 GB, and that is what the other lever is for, moving fewer bytes (FP8 versus BF16 halves $W$).

The storage tier

Tier	Typical bandwidth	140 GB (70B BF16)	70 GB (70B FP8)	Notes
Local Gen5 NVMe	~14 GB/s per disk	~10 s	~5 s	the short path; one disk
Local Gen5 NVMe (several, RAID0)	~28–50 GB/s	~3–5 s	~1.5–2.5 s	if the loader saturates several streams
Object storage (S3/RGW)	~1–3 GB/s per stream	~47–140 s	~23–70 s	streamer concurrency helps a great deal
NFS / shared network	~1–2 GB/s	~70–140 s	~35–70 s	puts the network and its contention in the path

The operational conclusion is the same as in From disk to HBM: the weights that serve the cold start live on the node’s local NVMe, not on the network. Network storage is the repository; the inference node has a hot local copy (pre-pull with an initContainer or a per-node cache DaemonSet). Tensorizer is the interesting exception: it streams from object storage so efficiently that it sometimes makes serving from S3/RGW viable with no local copy, at the price of depending on the bandwidth of the storage network.

Speeding up item 5: everything that is not weights

Here is the half of the cold start that fast loaders do not touch. Three fronts.

Caching the torch.compile/Inductor compilation

torch.compile has a built-in cache system: the compiled artefacts are saved after the first start-up and reused across start-ups, even across machines if it is configured properly. In vLLM, the result of the Dynamo compilation is stored in ~/.cache/vllm/torch_compile_cache/ (vLLM Blog, Introduction to torch.compile; vLLM, torch.compile integration). The consequence for the runbook: if that directory is a persistent volume shared between pods (a PVC, or a hostPath on the node’s NVMe populated by the first start-up), subsequent pods skip the compilation and start faster. The first pod pays for the compilation; the rest inherit the cache.

It is exactly “sharpening the knives the night before”: the expensive preparation is done once and reused. The nuance: the cache is sensitive to the vLLM/PyTorch version, the GPU, the quantisation and the configuration, and changing any of them invalidates the cache and you pay for the compilation once again.

Bounding the CUDA graph capture

vLLM captures CUDA graphs for a wide range of batch sizes by default, and that is expensive: the default capture is around 54 s, and on large configurations up to 294 s has been measured. By limiting the capture to the batch sizes your workload really uses (for example 1, 2, 4, 8, 16, 24, 32, 64) the cold start is cut by more than 70%, from 294 s to ~82 s in the measured case (Red Hat, vLLM with torch.compile). The knob is the engine’s compilation/CUDA graphs configuration; the principle is not to capture graphs for batch sizes you will never see.

A real trade-off: fewer captured batch sizes = faster start-up but less coverage. A batch outside the captured set falls back to the eager path (slower per request). You bound to the frequent sizes of your traffic, not of any traffic. And there is always the extreme of disabling CUDA graphs (enforce_eager): near-instant start-up on item 5 at the cost of throughput in the hot state, valid for debugging or for very low-traffic services where cold start weighs more than steady state.

Warm pools: not paying for item 5 on the critical path

The definitive shortcut for all five items is not to run them while the user waits. A warm pool, a replica already started, with weights loaded, graphs captured and kernels compiled, waiting for traffic, turns the cold start into zero for that request. It is GPU cost (partly or wholly idle) in exchange for start-up latency; we analyse it in the economics section below.

Sleep mode: the shortcut that skips all five items

And there is a shortcut halfway between “start from scratch” and “keep a whole replica warm”: vLLM’s sleep mode, the subject of Serving several models on a single GPU. Instead of killing the process, it puts it to sleep: it parks the weights in host RAM (level 1) or discards them (level 2), but keeps the process alive, and with it the CUDA context, the allocator, the CUDA graphs and the already compiled JIT kernels. The wake pays for neither item 1, 2, 3 nor 5; it only puts the weights back into VRAM (item 4, and from RAM, not from disk). That is why a wake is 18–200× faster than a full cold start, and even level 2, which reloads the weights from the same disk, is still 23–45× faster, because it skips the other four items (vLLM Blog, Sleep Mode). Sleep mode is living proof of this post’s thesis: if preserving items 1, 2, 3 and 5 gives you 18–200×, then moving bytes was never the whole cost.

An example cold start, item by item

Let us put numbers on a cold start of 70B in FP8 (70 GB) from local NVMe and see which optimisation attacks each item:

#	Line item	Default	Optimisation	Optimised
1	Process init	~4 s	(little room; minimal environment)	~3 s
2	CUDA context	~6 s	(driver; fixed)	~6 s
3	Allocator	~4 s	(fixed)	~4 s
4	Weight loading	~25 s	Run:ai streamer / Tensorizer (concurrency + overlap)	~5 s
5	CUDA graphs + JIT	~54 s	torch.compile cache + bounded capture	~10 s
	Total	~93 s		~28 s

Two readings. First: optimising only item 4 (a fast loader and nothing else) brings the total down from 93 to ~73 s, a real but disappointing improvement, because item 5 is still intact. Optimising only item 5 (cache + bounded capture) brings it down to ~49 s. Attacking both brings it down to ~28 s, and that is what turns theoretical scale-to-zero into usable scale-to-zero. Second: items 2 and 3 are practically irreducible, they depend on the driver and the hardware; they constitute the floor of the cold start, on the order of 10 s, which no software optimisation crosses. Below that floor, the only answer is not to start from scratch: sleep mode (wake from sub-second to a few seconds) or a warm replica (zero).

The economics of scale-to-zero

The capacity question all of this serves is: switch off or keep warm? Scale-to-zero saves GPU while there is no traffic, but pays the cold start when it comes back. The decision is a balance between the cost of an idle GPU and the latency SLA.

The basic criterion: scale-to-zero is only viable if the cold start fits inside the start-up SLA your users tolerate, or if you have a pre-warming mechanism that starts the replica before the request arrives (load prediction, an early signal from the queue). Without pre-warm, a 90 s cold start means the first user after an idle period waits 90 s, unacceptable for almost any interactive SLO. With the cold start brought down to ~28 s it is still a lot for an interactive request, but it is already tolerable for batch workloads or for a pre-warm triggered by autoscaling with KEDA on queue depth.

The decision rule, in one sentence: keep a warm replica when the cost of the idle GPU during the troughs is lower than the cost of missing the SLA on every start-up; do scale-to-zero (with pre-warm) when the troughs are long and deep and the optimised cold start fits in your tolerance. For a 24×7 interactive service with irregular but continuous traffic, a floor of warm replicas almost always wins. For a large model invoked a few times a day (a 70B for hard tasks), scale-to-zero with an optimised cold start, or sleep mode if it shares a GPU with another model, is the sensible option. It is the same capacity planning boundary: cold start is a parameter of the replica cushion, not a start-up detail.

Applied to the generic 4×H100 cluster

Let us bring the runbook down to the 4 H100 SXM 80 GB with NVLink. The concrete decisions:

Loader and format. Weights in safetensors on disk (never pickle). For the short path from local NVMe, the Run:ai Model Streamer (--load-format runai_streamer, concurrency 16–32) is the default option: concurrent reads, overlapping read and H2D, and you only change one flag. Keep Tensorizer for the case where you serve from object storage (RGW/S3) and want to decouple the weights from the image with no local copy, since its tensor-by-tensor streaming straight to the GPU is what makes that pattern viable. In both cases, FP8 versus BF16 halves item 4 almost for free (measure the quality, do not assume it).

Storage tier. Model repository on the network (Ceph RGW), hot copy on the node’s local NVMe populated by pre-pull. The cold start that counts is the cold one, on a node where the files are not in page cache; serving that start-up from the network puts its latency and contention into the critical path.

Caching item 5. The torch_compile_cache directory on a persistent volume or a hostPath NVMe shared between pods of the same model: the first pod compiles, the rest inherit. And bound the CUDA graph capture to the real batch sizes of each service’s traffic.

When scale-to-zero with pre-warm vs a warm replica. The agent LLM (main service, continuous traffic) never does scale-to-zero: a floor of warm replicas on one or two GPUs. The occasional 70B does: scale-to-zero with an optimised cold start, or, if it shares a GPU with a mid-sized model, sleep mode so the wake is sub-second instead of a 28 s start-up (exactly the play in Serving several models on a single GPU). The optimised cold start in this post is what makes swap and failover between replicas tolerable: a replica that falls over and is replaced in 28 s degrades less than one that takes 90 s. And it connects directly with multi-tenancy: a short cold start lets you share GPUs between teams with per-tenant scale-to-zero without the first user of each tenant paying for an endless start-up.

The cross-cutting principle: cold start is the ceiling on elasticity. The five line items, with both halves attacked, a fast loader for the weights and cache plus bounded capture for the graphs, bring it down from minutes to seconds. And below the irreducible floor of ~10 s (CUDA context + allocator), the only way out is not to start from scratch: sleep mode or a warm replica.

Traps and things that are not what they seem

“I switched to a fast loader and start-up barely improved.” Your bottleneck probably was not item 4. If CUDA graph capture eats 54 s, bringing weight loading down from 25 to 5 s takes 20 s off 93, which is barely noticeable. Measure where the time goes before optimising: vLLM’s start-up logs break down weight loading versus graph capture.

“The second time it started in a flash.” The page cache (for the weights) and the torch.compile cache (for the graphs) lying to you at the same time. The start-up that matters is the cold one, on a clean node. Benchmarking the second start-up measures a situation that almost never occurs in the peak that triggers the autoscaler.

“Tensorizer/streamer will give me the 18–200× of sleep mode.” No. Those loaders speed up only item 4. The 18–200× of sleep mode comes from preserving the other four by keeping the process alive. They are tools for different problems: a fast loader is for a real cold start (a new process); sleep mode is for alternating models without killing the process.

"enforce_eager fixes the cold start." It fixes item 5 (no graph capture, near-instant start-up) but degrades throughput in the hot state, since CUDA graphs exist because they speed up the steady state. It is a trade: it wins for very low traffic services where start-up weighs more than the hot path; it loses for a service with sustained load.

“I bound the CUDA graphs to a single batch size and that is it.” You bound to the sizes your traffic uses, not to one. A batch outside the captured set falls back to the eager path and runs slower. Too aggressive and you penalise the steady state to gain a few seconds of start-up.

“Moving the weights to FP8 also speeds up item 5.” Not directly. FP8 halves the bytes (item 4) and doubles inference throughput, but graph capture and kernel compilation do not depend on the size of the weights, they depend on the shape of the graph and the batch sizes. Item 5 is attacked with caching and bounded capture, not with quantisation.

Conclusion

From disk to HBM said it conceptually: moving the weights is one of the five line items of cold start. This runbook turns it into action. For the weights, item 4, there is a clear menu: safetensors as the base (mmap, zero-copy, no pickle), Run:ai Model Streamer for the short path from NVMe (concurrent reads overlapping read and H2D), Tensorizer for streaming from object storage with no local copy, and the cross-cutting lever of moving fewer bytes with FP8. But the forgotten half of the start-up is item 5, CUDA graph capture and JIT compilation, which on a modern server can be as big as the weight loading: you attack it by caching the compilation across pods and bounding the graph capture to the real batch sizes. Attacking only one half leaves the cold start half done; attacking both brings it from ~90 s to ~28 s. And below the irreducible floor of ~10 s imposed by the CUDA context and the allocator, the only way out is not to start from scratch: sleep mode (sub-second wake) or a warm replica (zero). The kitchen opens on time when somebody took care of the produce and of sharpening the knives the night before. Optimising only the delivery from the store leaves the ovens cold and the restaurant shut.

References

HuggingFace, Safetensors (format, zero-copy, security versus pickle): https://huggingface.co/docs/safetensors/index
HuggingFace, Safetensors is Joining the PyTorch Foundation: https://huggingface.co/blog/safetensors-joins-pytorch-foundation
vLLM, Loading Models with CoreWeave’s Tensorizer: https://docs.vllm.ai/en/stable/models/extensions/tensorizer/
vLLM, Loading models with Run:ai Model Streamer: https://docs.vllm.ai/en/stable/models/extensions/runai_model_streamer/
NVIDIA, Reducing Cold Start Latency for LLM Inference with NVIDIA Run:ai Model Streamer (S3/SSD benchmarks, concurrency): https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer/
vLLM Blog, Introduction to torch.compile and How It Works with vLLM (compilation cache): https://blog.vllm.ai/2025/08/20/torch-compile.html
vLLM, torch.compile integration (cache in ~/.cache/vllm/torch_compile_cache/): https://docs.vllm.ai/en/latest/design/torch_compile/
Red Hat Developer, vLLM with torch.compile: Efficient LLM inference on PyTorch (CUDA graph capture ~54 s, bounding batch sizes −70%): https://developers.redhat.com/articles/2025/09/03/vllm-torchcompile-efficient-llm-inference-pytorch
vLLM Blog, Zero-Reload Model Switching with vLLM Sleep Mode (wake 18–200×): https://blog.vllm.ai/2025/10/26/sleep-mode.html

The pantry lift: from disk to HBM, or why the kitchen opens late

Sun, 07 Jun 2026 08:30:00 +0200

This is a trip down to the basement. The under the engine series optimised the hot path, what happens with every token once you are already serving. This post looks at the journey that comes before serving: how the weights climb from disk to HBM. It is the first of a pair on the two things that happen outside the API and that almost nobody times: model loading (this one) and execution on the silicon (the next one).

TL;DR

Before an inference pod generates its first token, it has to push the whole model into HBM. A Llama-70B in FP16 is 140 GB travelling along a path nobody draws: disk → page cache → host buffer → PCIe → HBM. Intuition fails here: HBM is not the bottleneck, it moves 3.35 TB/s and swallows 140 GB in 42 ms; the bottleneck is the supply chain. The Gen5 NVMe disk reads at ~14 GB/s (10 s for 140 GB); PCIe Gen5 copies host→GPU at ~50 GB/s (2.8 s); and the default safetensors loader, which deserialises tensor by tensor and bounces every byte through a CPU buffer, inflates all of that up to 30-60 s. That time is the cold start, and it is the hidden tax paid by autoscaling (scale-from-zero), by canary/blue-green and by disaggregated serving every time a pod is born. There are three families of solution, GPUDirect Storage (direct DMA disk→HBM, no bounce through the CPU), fastsafetensors (4.8-7.5× over the default loader) and the Run:ai Model Streamer (concurrent reads that saturate the disk), plus the simplest lever of all: move fewer bytes (FP8 is half of FP16). This post explains the path, the maths, the 10 knobs, and the cruellest trap: “the second time it loaded fast” is not your loader being good, it is the page cache lying to you. On the generic 4×H100 SXM cluster.

Where you are: the basement, before opening

The analogy: the pantry lift

Let us stay in the restaurant of the series. The shared table was the NVSwitch, the floor next door was NUMA, the maître was the kubelet. All of that describes the restaurant running, with diners seated. But there is a moment no post looked at: before opening, somebody has to bring the entire pantry up from the basement store to the kitchen.

The model weights are the ingredients. They live in the basement store (the disk). The kitchen, the hot line where tokens are plated, is the GPU’s HBM. And between one and the other there is a service lift: the path disk → host → PCIe → HBM. The kitchen cannot serve the first dish until the pantry is upstairs and in place. That restocking time is the cold start.

The intuition trap: the kitchen (HBM) is enormous and blazingly fast, it places ingredients at 3.35 TB/s. So we blame the kitchen when the restaurant opens late. But the kitchen is standing idle waiting for the lift. The bottleneck is never the hot line: it is the lift and the store. And worse still, there is a porter (the default loader) who, instead of carrying whole crates, takes the ingredients out one by one, writes each one down in a notebook and packs them again before sending them up. That porter, not the lift, is half the problem.

The mechanism: what really happens when a model loads

When vLLM starts up with a model in safetensors, the 140 GB of the Llama-70B FP16 make this journey:

Disk → page cache. The kernel reads the .safetensors files from NVMe into the page cache (host RAM). If this is the first time after a reboot, it is a physical read from disk (~14 GB/s on Gen5). If the files are already in page cache from an earlier start-up, this is almost free, and here is where the trap we will see is born.
Deserialise. The default Hugging Face loader mmaps the file and builds the tensors one by one, copying them into a CPU tensor before moving them. This is single-threaded CPU work that saturates neither the disk nor PCIe: most of the “slow” loading time goes here, not on moving bytes.
Host → HBM (H2D). Each tensor is copied from the host buffer into HBM over PCIe Gen5 x16 (~50 GB/s in practice). For the DMA to be efficient, the host buffer should be pinned, which connects directly with the hugepages and pinned memory of the NUMA post.
Place in HBM. HBM receives the 140 GB. At 3.35 TB/s, this takes 42 ms. It is never the bottleneck.

The path has a shortcut: GPUDirect Storage (GDS). Instead of bouncing through the CPU buffer (steps 2-3), a DMA engine close to the NVMe controller writes straight from disk to HBM, without involving the CPU. It is the same principle as the GPUDirect RDMA of the network: get the CPU out of the way. fastsafetensors uses GDS and reaches 26.4 GB/s reading a Llama-70B from NVMe across 4 GPUs.

Why the problem exists: the economics of bytes

The size of the model in bytes is decided by quantisation, and that sets the floor of the cold start, because all those bytes have to move before the first token. For a 70B parameter model:

Format	Bytes/param	Size 70B	Read 1 NVMe @14 GB/s	H2D PCIe @50 GB/s	HBM @3.35 TB/s
FP16 / BF16	2	140 GB	10.0 s	2.8 s	42 ms
FP8	1	70 GB	5.0 s	1.4 s	21 ms
INT4 (GPTQ/AWQ)	0.5	~35 GB	2.5 s	0.7 s	10 ms

Three readings of this table:

HBM never shows up as a problem. The last column is always milliseconds. Anyone saying “the GPU is slow to load” is blaming the wrong place.

Quantising is the most underrated cold start lever. Going from FP16 to FP8 does not only double inference throughput (less HBM bandwidth per token, as we saw in quantization): it also halves the cold start, because there are half as many bytes to push up. It is a two-for-one that sizing usually ignores.

The disk is the bottleneck on bytes; the loader is the bottleneck on time. The figures in the table are the theoretical floor, just moving bytes. The default loader adds single-threaded deserialisation on top, which is the difference between the theoretical 10 s and the real 30-60 s. That is why the solutions attack on two fronts: fewer bytes (quantisation) and a better porter (concurrent loaders / GDS).

The maths that matter: cold start as a tax on autoscaling

Cold start is not paid once. It is paid every time a pod is born. And on an elastic platform, pods are born continuously.

Take an autoscaling setup with KEDA that scales from 2 to 6 replicas when the queue grows. The 4 new replicas take this long to be ready:

$$ T_{\text{ready}} = T_{\text{schedule}} + T_{\text{pull-image}} + T_{\text{load-weights}} + T_{\text{cuda-graphs}} $$

With a Llama-70B FP16 and the default loader, $T_{\text{load-weights}}$ dominates: it can be 40 s of the ~60 s total. During those 40 s, the queue that triggered the autoscaling keeps growing, since the new replicas absorb no traffic until they have loaded. The real number in the formula is not “how many replicas”, it is how long each one takes to start serving, and that number is written by the loading path.

This has a harsh operational consequence: scale-to-zero is unworkable for workloads with a latency SLO if the cold start is 40 s. Nobody waits 40 s for the first token. The real elasticity of an inference platform is not limited by GPU availability, it is limited by how long that GPU takes to have the model inside it. Bringing the cold start down from 40 s to 8 s (with streamer + FP8) is what turns “theoretical scale-to-zero” into “usable scale-to-zero”.

The published numbers give the order of magnitude of the improvement: fastsafetensors cuts start-up from 12.39 s to 4.74 s on a Llama-2-13B over 4×L40S, and from 16.04 s to 6.88 s on 1×A100, 4.8-7.5× over the default deserialiser. The Run:ai Model Streamer loads in 4.88 s from S3 at concurrency 32 and 7.53 s from SSD IO2 at concurrency 8; integrated into vLLM, total time to ready drops to ~23 s from S3. There is no magic here: it is getting the CPU out of the loop (GDS) and reading in parallel (concurrency) to saturate the disk instead of leaving it half idle while one thread deserialises.

The 10 knobs worth touching

Knob 1 — Measure where the time goes (read vs deserialise vs H2D)

Before touching anything: time it. Is the time going on reading from disk, on deserialising, or on the H2D? iostat -x 1 during the load tells you whether the NVMe is saturated (disk bottleneck) or nearly idle (loader/CPU bottleneck). If the disk is at 20%, the problem is not the disk: it is the porter. Changing disk would fix nothing; changing loader would.

Knob 2 — `--load-format`: choosing the porter

vLLM exposes several loaders via --load-format: safetensors (default), runai_streamer, fastsafetensors, tensorizer. The default is the slowest. Changing one flag can be the cheapest 4-7× there is.

# Run:ai Model Streamer (concurrent reads, saturates the disk)
vllm serve meta-llama/Llama-3.1-70B --load-format runai_streamer
# fastsafetensors (GPUDirect Storage, direct DMA disk→HBM)
vllm serve meta-llama/Llama-3.1-70B --load-format fastsafetensors

Knob 3 — Streamer concurrency

The Run:ai Model Streamer splits the read across N threads according to the size of each tensor in order to saturate the storage bandwidth. Concurrency is the key parameter: 16 is usually enough for local NVMe; 32 (sometimes 64) for high-throughput network storage. One thread does not saturate a Gen5 NVMe; 32 do.

vllm serve <model> --load-format runai_streamer \
 --model-loader-extra-config '{"concurrency": 32}'

Knob 4 — GPUDirect Storage (GDS) + fastsafetensors

If there is an nvidia-fs driver, a supported filesystem and local NVMe, GDS writes straight disk→HBM with no bounce through the CPU. It is the difference between the two paths in the diagram. But: it requires the stack to be in place, and it only wins if the bottleneck is the CPU bounce, not the disk itself. Check with gdscheck.

Knob 5 — Local NVMe for the weights, not network

Serving the weights from Ceph RGW / NFS is convenient (one shared place) but it puts the network into the loading path. For cold start, weights on the node’s local NVMe (or a local cache). Network storage is for the model repository; the inference node should have a hot local copy.

Knob 6 — Pre-pull / local model cache

An initContainer that downloads the model onto a local or hostPath NVMe volume before starting vLLM turns a cold start “from the network” into one “from local NVMe”. Combined with a per-node cache DaemonSet, new pods on an already warm node read from local disk, not from the network.

Knob 7 — Quantisation: move fewer bytes

Weights already in FP8 or INT4 on disk = half or a quarter of the cold start. It is the knob from the table above. And it composes with all the others: FP8 + streamer + GDS is multiplicative.

Knob 8 — Parallel loading across GPUs (vLLM V1)

vLLM’s V1 engine (the default since 0.19) loads the weight shards in parallel across the GPUs of a TP group, instead of sequentially. At TP=4, each GPU loads its quarter at the same time. Check it is active; in old versions loading was serial and the cold start of TP=4 was almost 4× that of TP=1.

Knob 9 — NUMA locality of the NVMe

The NVMe hangs off a PCIe root under a specific socket, exactly the same NUMA map as the host post. If the load’s host buffer lands on the wrong socket, the H2D crosses the UPI. The fifth list to align, alongside isolcpus, reserved-cpus and the NIC IRQs: which socket is local to the NVMe and to the destination GPU. nvidia-smi topo -m shows it.

Knob 10 — Do not pay the cold start: keep pods warm

Sometimes the answer is not to load faster, but not to unload. A floor of always-live replicas (no scale-to-zero), or a pool of preloaded warm standbys, swaps “wait 40 s” for “0 s”. It is the cost of idle GPU in exchange for start-up latency: a capacity planning decision, not a technical one.

Summary table

#	Knob	What it attacks	Risk / cost
1	`iostat` while loading	knowing whether the bottleneck is disk or loader	none; always do it first
2	`--load-format`	single-threaded deserialisation	format compatibility
3	streamer concurrency	underused disk	host RAM for buffers
4	GPUDirect Storage	bounce through the CPU	requires nvidia-fs + supported FS
5	local NVMe vs network	the network in the path	duplicating weights per node
6	pre-pull / node cache	network on every start-up	local disk space
7	FP8/INT4 quantisation	bytes to move	quality (measure, do not assume)
8	V1 parallel loading	serial loading across GPUs	none if V1 is active
9	NUMA of the NVMe	H2D crossing the UPI	align with the rest of the NUMA lists
10	warm pods / no zero	the whole cold start	idle GPU paid for

How it connects with the rest of the stack

With autoscaling. The whole of scale-from-zero with KEDA rests on this: real elasticity is limited by the cold start, not by GPU availability. An autoscaler with 40 s of loading reacts late to every peak.

With disaggregated serving. In disaggregated prefill/decode, bringing up a decode pool on demand pays the cold start of loading the model into each new pod. The elasticity of the pattern depends on how fast those pods start.

With canary/blue-green. Every canary deployment loads a new version of the model alongside the old one. The validation time of a canary includes its cold start; large models make deployments slower and more expensive.

With NUMA and hugepages. The load’s host buffer wants to be pinned and NUMA-local, the same thing the host post asked for on the hot path. The loading path is another client of the same NUMA map.

With quantisation. FP8/INT4 is not only inference throughput: it is the direct lever on the bytes of the cold start.

With capacity planning. Sizing that ignores cold start underestimates how many replicas are needed to absorb a peak: if they take 40 s to start, you need a bigger permanent cushion.

Traps and things that are not what they seem

“The second time it loaded really fast.” This is the star trap. The first load filled the page cache (host RAM); the second reads from RAM, not from disk, and flies. But in production pods are ephemeral and are born on different nodes: the start-up that counts is the cold one, on a node where those files are not in page cache. Benchmarking the second load is measuring a situation that almost never happens at the moment that matters (the peak that triggers the autoscaler).

“GDS always speeds things up.” No. GDS removes the bounce through the CPU; if your bottleneck is the disk itself (saturated NVMe) or the deserialisation, GDS does not touch that part. Measure first (knob 1). It also demands nvidia-fs, a supported filesystem, and sometimes it does not work over whatever network storage you have.

“mmap makes loading instant.” mmap maps the file but reads nothing yet: the cost is deferred to the first access to each page. The time does not disappear, it moves, and the first token pays the page faults that start-up did not. You have moved the cold start into the latency of the first request, which is probably a worse place to have it.

Weights on network storage “because it is cleaner”. Sharing a model repository on Ceph RGW is fine for storing them; serving the cold start from there puts the network (and its latency and its contention) into the critical path. Local NVMe cache on the inference node.

Loading FP16 and quantising at start-up. Quantising on the fly during loading (for example FP16→FP8 on the GPU) can be slower than having the weights already quantised on disk: you move twice the bytes and on top of that you do conversion work. If you are going to serve in FP8, store the weights in FP8.

Optimising the load and ignoring T_cuda-graphs. Bringing weight loading down to 8 s and forgetting that CUDA graph capture adds several more seconds leaves the cold start half done. That second half of start-up is the subject of the next post.

Conclusion

The whole series optimised what happens with every token: the cable, the host, the network, the silicon. But before the first token there is a journey almost nobody times and that decides whether an inference pod opens in 10 s or in 60 s: pushing the model from disk to HBM. Intuition blames the GPU, and the GPU is innocent: HBM swallows 140 GB in 42 ms. The bottleneck is the supply chain: a disk reading at 14 GB/s, a PCIe copying at 50, and above all a default loader that deserialises tensor by tensor with a single thread and turns 10 s of bytes into 60 s of waiting. The solutions attack the two right fronts, fewer bytes (quantisation) and better transport (GDS, concurrent streamers, local NVMe), and give 4-7× almost for free. And above the technique, an idea that reorders the priorities: on an elastic platform, cold start is not a start-up detail, it is the ceiling on elasticity. The fastest GPU in the world does not scale if it takes 40 s to have the model inside. The pantry lift, the one nobody timed, is what decides what time the kitchen really opens.

References

NVIDIA, Reducing Cold Start Latency for LLM Inference with NVIDIA Run:ai Model Streamer: https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer/.
vLLM, Loading models with Run:ai Model Streamer: https://docs.vllm.ai/en/stable/models/extensions/runai_model_streamer/.
vLLM, Loading model weights with fastsafetensors: https://docs.vllm.ai/en/stable/models/extensions/fastsafetensor/.
foundation-model-stack, fastsafetensors (high-performance loader, GDS): https://github.com/foundation-model-stack/fastsafetensors.
Speeding up Model Loading with fastsafetensors (arXiv 2505.23072): https://arxiv.org/html/2505.23072v1.
NVIDIA, GPUDirect Storage: A Direct Path Between Storage and GPU Memory: https://developer.nvidia.com/blog/gpudirect-storage/.
NVIDIA, Magnum IO GPUDirect Storage (benchmarking and configuration): https://developer.nvidia.com/gpudirect-storage.

Autoscaling LLM inference on Kubernetes: HPA with custom metrics and KEDA for vLLM

Mon, 01 Jun 2026 16:00:00 +0200

This post complements those on GPU observability for LLM inference (where the metrics that feed the HPA come from), Capacity planning (what ceiling and what head-room the autoscaler assumes) and Continuous batching (which explains why num_requests_waiting is the primary metric).

TL;DR

Classic Kubernetes autoscaling, HPA on cpu or memory, is no use for LLM inference. The reason: the vLLM pod consumes little CPU (the GPU does the work) and the process RSS is flat; both metrics can sit at 30 % while the GPU is saturated and the request queue grows unchecked. The four viable signals that do respond to the real load are: vllm:num_requests_waiting (the queue, the primary metric), vllm:gpu_cache_usage_perc (pressure on the KV cache pool), TTFT P95 via the vllm:time_to_first_token_seconds_bucket histogram (the SLO guarantee) and the batch fill ratio num_requests_running / max_num_seqs (utilisation of the concurrency ceiling). For an HPA to consume Prometheus metrics you need an adapter; as of May 2026 there are two mature options: prometheus-adapter (cluster-wide sigma, static configuration, external.metrics.k8s.io output) and KEDA (ScaledObject with a Prometheus trigger, configurable polling, optional scale to zero, cron integration). KEDA is the dominant option for LLM on a generic cluster because it solves the “warm pool + cron + engine metric” pattern in a single CRD. The dominant operational challenge is not the scaling logic but the cold start: a vLLM pod with Llama 70B BF16 (140 GB) takes between 90 seconds (model pre-cached on a local PV) and 6 minutes (image pull + model download from object store) to serve the first token. The five levers that cut it down are an image pre-pulled via DaemonSet, the model cached on a regional PV or tmpfs, a warm pool with minReplicaCount > 0, predictive scaling via KEDA cron when the traffic pattern is predictable (offices 9–18 h), and parallel model download. The three specific LLM scale-down pitfalls: cutting streaming SSE connections halfway through a response (graceful drain with terminationGracePeriodSeconds ≥ 60 s), scale-out/in oscillation from a badly calibrated stabilization window, and forgetting that the HPA only scales pods; GPU nodes are scaled with cluster-autoscaler over labelled nodepools. This post includes the minimal YAML manifests.

You are here: DEPLOY

The analogy: the bakery with wood-fired ovens

An artisan bakery has three wood-fired ovens. Each oven takes 25 minutes to reach temperature from cold. Once hot, it bakes bread continuously with a run of 18 minutes per batch. The manager wants to maximise bread sold per day without burning wood for nothing, and she knows three things: that there is a demand peak at 7:30 every morning, that Mondays sell almost nothing, and that when the bread on the counter runs out the customers go to the supermarket next door.

The cheap strategy, lighting ovens when there is a queue in the shop, does not work. By the time the queue grows and the manager lights the second oven, that oven will not be ready until 25 minutes later; the customers in that window are lost. The “queue at the counter” signal arrives late.

The smart strategy: light the second oven at 6:55, ahead of the predictable 7:30 peak, and leave it running until 10:00 even though the queue drops at 8:15. Keep the third oven off from Monday to Wednesday because demand does not arrive; light it proactively on Thursdays at 12:00 because historically it goes up. Keep a stock of pre-proved raw dough in the chiller so that when the oven is ready the bread goes in within 30 seconds and there is no two-hour proving wait.

Autoscaling an LLM inference cluster works the same way:

Lighting ovens from cold = reactive scale-out when the queue grows (slow, loses customers).
Proactive cron = predictive scaling when the pattern is known (working hours, expected peaks).
Pre-proved dough = warm pool of replicas with the model loaded but under 0 load.
Switching off ovens with no bread inside = scale-down respecting the active streams (you do not shut the oven with bread in it).

The key metric, “how many customers are in the queue”, is called num_requests_waiting. The metric that says “the oven is about to run out of dough for new loaves” is called gpu_cache_usage_perc. And the quality-of-service metric, “how long the first loaf takes to come out when a new customer walks in”, is called TTFT.

Why HPA on CPU is no use

The classic Kubernetes HPA looks at the pod’s resource.cpu. For a conventional HTTP service, Node.js or a REST API, CPU moves linearly with traffic and the HPA scales with reasonable accuracy. For a vLLM or SGLang pod on GPU, the pod’s CPU typically lives between 5 % and 15 % regardless of whether the GPU is at 30 % or 99 % load: the real work is done by the device, not the process. Result: the CPU-based HPA never triggers scale-out even though the GPU is bursting, and clients pile up in the queue until TTFT P95 crosses the SLO. The operator discovers the problem through the TTFT alert, not through the HPA.

memory is no use either: the vLLM process RSS is flat after startup (model plus buffers loaded in one go); it does not reflect the real pressure on the GPU. The only things that rise and fall with the useful inference load are metrics the engine publishes explicitly: request queue, KV cache pool, SLO latencies. Without an adapter that makes them visible to the HPA, autoscaling is blind.

The four viable signals

Signal 1, vllm:num_requests_waiting (queue). It is the most direct metric: how many requests are waiting to enter the batch. It reacts the instant the target concurrency saturates. It is robust against model changes (the number of requests is the same concept whether it is Llama 7B or 70B). It is the primary metric of the LLM HPA. Typical threshold: target = 5 requests waiting on average; if the queue grows above 5 sustained for 2 minutes, scale out.

Signal 2, vllm:gpu_cache_usage_perc (KV pool). It moves before the queue does: the KV pool fills up while the batch slots are still free, until the engine starts rejecting new requests for OOM prevention and the queue forms. It is therefore predictive: it triggers scale-out before the client notices degradation. Typical threshold: target = 0.85 (85 % of pool used).

Signal 3, TTFT P95. The contractual guarantee. If TTFT P95 leaves the SLO, scale out even though the queue and the KV pool look reasonable (there may be a spike of long prompts). It is reactive, it leaves the SLO before your HPA reacts, but it works as a final guardrail.

Signal 4, batch fill ratio. The num_requests_running / max_num_seqs quotient (the latter is engine config, not a metric). Useful for scale-down: if the ratio stays below 0.4 for 10 minutes, there is spare capacity and replicas can be reduced safely.

The recommended policy combines all four: the queue and the KV pool trigger scale-out (whichever arrives first), TTFT confirms it as a guardrail, and the batch fill ratio manages scale-down. Implementing that in a single HPA demands external metrics; KEDA makes this manageable.

The wiring: KEDA as the Prometheus adapter

KEDA introduces two main CRDs: TriggerAuthentication (how to authenticate against the source) and ScaledObject (which deployment to scale with which triggers). For a vLLM deployment with Prometheus as the source:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
 name: vllm-llama70b-scaler
 namespace: inference
spec:
 scaleTargetRef:
 name: vllm-llama70b
 minReplicaCount: 2 # warm pool
 maxReplicaCount: 20
 pollingInterval: 15
 cooldownPeriod: 300 # 5 min before scale-down
 advanced:
 horizontalPodAutoscalerConfig:
 behavior:
 scaleDown:
 stabilizationWindowSeconds: 600 # large window to avoid oscillation
 policies:
 - type: Pods
 value: 1
 periodSeconds: 120
 scaleUp:
 stabilizationWindowSeconds: 30
 policies:
 - type: Pods
 value: 2
 periodSeconds: 60
 triggers:
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_queue_depth
 threshold: "5"
 query: |
 avg(vllm:num_requests_waiting{deployment="vllm-llama70b"})
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_kv_cache
 threshold: "0.85"
 query: |
 avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama70b"})
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_ttft_p95
 threshold: "1.5"
 query: |
 histogram_quantile(0.95,
 sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{deployment="vllm-llama70b"}[5m])))

Three non-obvious operational details:

minReplicaCount: 2. This is the warm pool. Keeping at least two replicas guarantees availability if a node is lost and absorbs spikes without waiting for the cold start of the first scale-up. Dropping it to 0 saves GPU off-peak but introduces 90 s–6 min of latency for the first new client.

stabilizationWindowSeconds: 600 on scale-down. Ten minutes. Models are not nginx: if a replica closes prematurely and two minutes later there is another peak, the cold start of a new pod is what the client waits. Better to keep extra replicas around for twice as long as you would for a normal web service.

scaleUp: stabilizationWindowSeconds: 30. Thirty seconds. Scale-out has to be fast: the new pod’s cold start adds its own delay, and if on top of that the HPA waits several more minutes before triggering, the SLO is already broken.

The big operational problem: cold start

A vLLM pod loading Llama 70B goes through these phases before serving the first token:

Phase	Typical time	Can be accelerated with
Image pull (4–6 GB)	30–90 s	DaemonSet pre-pull
Model download (140 GB BF16)	60–300 s	Cached regional PV, S3 + multi-thread
Loading the model into HBM	30–90 s	tmpfs or local NVMe
CUDA graph capture	20–60 s	`--enforce-eager` (slower at runtime but fast startup)
PagedAttention warmup	5–15 s	—
Health check ready	10–30 s	probe tuning

Total without optimisation: 4–10 minutes. That is how long a new replica takes to absorb traffic. With all the levers combined: 45–90 seconds. The difference between those two numbers is the main platform work for LLM autoscaling.

The five levers

Lever 1, pre-pulled image. A trivial DaemonSet runs ctr image pull (or crictl pull) on the GPU nodes as soon as they join the cluster. The inference engine image stays on disk; new pods skip the 30–90 s of pull. Cost: around 6 GB of disk per node.

apiVersion: apps/v1
kind: DaemonSet
metadata: { name: vllm-image-warmer }
spec:
 selector: { matchLabels: { app: vllm-warmer } }
 template:
 metadata: { labels: { app: vllm-warmer } }
 spec:
 nodeSelector: { workload: gpu }
 initContainers:
 - name: pull
 image: vllm/vllm-openai:v0.10.0
 command: ["/bin/true"]
 containers:
 - name: pause
 image: registry.k8s.io/pause:3.10

Lever 2, model on a regional PV. The model download (140 GB BF16 or 35 GB FP8) from central object storage is the dominant component of the cold start. Caching the model on a zone or rack PV, Rook-Ceph RBD or local NVMe provisioned by the operator, cuts 60–300 s down to 5–15 s. The antipattern: downloading the model on every startup from external S3.

volumeMounts:
 - name: model-cache
 mountPath: /models
 readOnly: true
volumes:
 - name: model-cache
 persistentVolumeClaim:
 claimName: llama70b-fp8-pvc  # RWX shared, filled offline

Lever 3, warm pool. minReplicaCount > 0 keeps pre-loaded replicas idle. The cost is idle GPU; the benefit is 0 s of cold start for the first client of a peak. For production clusters with continuous traffic: a warm pool of 2–3 replicas. For nightly batch clusters with 0 traffic: warm pool 0 and accept the cold start, or KEDA with a cron that switches on 10 minutes earlier.

Lever 4, predictive scaling with cron. When the pattern is predictable (offices 9–18 h):

triggers:
 - type: cron
 metadata:
 timezone: Europe/Madrid
 start: "30 8 * * 1-5" # 8:30 Monday–Friday
 end: "0 19 * * 1-5" # 19:00
 desiredReplicas: "6"

Combined with reactive triggers. The HPA scales according to the maximum of the signals: if the cron asks for 6 and the queue asks for 10, the result is 10.

Lever 5, parallel download and an efficient format. For PVs that are not pre-loaded, tools such as nvidia-modelmanager, s5cmd or aria2c parallelise the model download. Going from serial download (~150 MB/s) to parallel with 8 threads (~1.2 GB/s) divides the time by 8. And formats such as safetensors load into HBM faster than the original PyTorch pickle.

When to scale nodes, not just pods

The HPA scales pods. If the cluster has no free GPU nodes, the new pod stays Pending for lack of resources. To scale nodes, you need cluster-autoscaler with a specific GPU nodepool, labelled:

# nodepool config (Karpenter or cluster-autoscaler equivalent)
labels:
 workload: gpu
 gpu-model: h100-sxm-80gb
taints:
 - key: nvidia.com/gpu
 effect: NoSchedule
limits:
 min: 2 nodes
 max: 8 nodes

Without this, the HPA can ask for 10 replicas but the cluster only delivers those that fit on nodes already up. The cold start of a new node (bare metal or cloud provisioning, PXE, OS boot, NVIDIA drivers, cluster join) is much longer than the cold start of a pod: typically 5–15 minutes on preconfigured bare metal, 30–60 minutes on real provisioning. For on-premise clusters, the nodepool must be always sized to the expected maximum, and the “scaling” is only on the pod side. The concept of reactive node scale-out only applies to clouds; on-premise you have to buy for the peak.

Three pitfalls specific to LLM scale-down

Pitfall 1, cutting streaming SSE connections. When a replica enters Terminating, Kubernetes sends SIGTERM to the pod and, by default, kills it 30 seconds later. For vLLM that means cutting streaming SSE connections halfway through the response. The client gets a 502 error with the partial output lost. Solution: terminationGracePeriodSeconds: 120 plus a preStop hook that tells the engine not to accept new requests but to finish those in flight:

spec:
 terminationGracePeriodSeconds: 120
 containers:
 - name: vllm
 lifecycle:
 preStop:
 httpGet:
 path: /shutdown
 port: 8000

This requires the engine to expose a graceful shutdown endpoint; vLLM v1 supports it via --enable-graceful-shutdown. Without it, scale-down breaks the SLO even though the metrics do not capture it (cut requests never enter the TTFT histogram).

Pitfall 2, scale-up/scale-down oscillation. If the scale-down stabilizationWindowSeconds is short (~60 s default), the next drop in the queue triggers scale-down, and two minutes later the next peak triggers scale-up. The system oscillates, pays repeated cold starts, and never reaches a stable regime. Solution: scale-down with a window of 10 minutes minimum and conservative policies (type: Pods, value: 1, periodSeconds: 120, at most one replica fewer every 2 minutes).

Pitfall 3, vllm:num_requests_waiting with avg when there is rebalancing. If two replicas are unbalanced (one with queue 20, the other with queue 0), avg gives 10 and the HPA triggers scale-out when the right move would be to rebalance via the load balancer. To detect it: add an alert on stddev(vllm:num_requests_waiting) per deployment. If the dispersion is high, the problem is not capacity but routing.

Full example manifest

For a vLLM deployment with Llama 70B FP8 on 4×H100 SXM per replica, KEDA with warm pool 2:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: vllm-llama70b
 namespace: inference
spec:
 replicas: 2 # managed by KEDA afterwards
 selector: { matchLabels: { app: vllm-llama70b } }
 template:
 metadata:
 labels: { app: vllm-llama70b, deployment: vllm-llama70b }
 spec:
 terminationGracePeriodSeconds: 120
 nodeSelector: { workload: gpu, gpu-model: h100-sxm-80gb }
 tolerations:
 - key: nvidia.com/gpu
 operator: Exists
 effect: NoSchedule
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.10.0
 args:
 - --model=/models/llama-3.3-70b-fp8
 - --tensor-parallel-size=4
 - --max-num-seqs=64
 - --enable-prefix-caching
 - --enable-graceful-shutdown
 ports:
 - { name: http, containerPort: 8000 }
 - { name: metrics, containerPort: 8000 }
 resources:
 limits:
 nvidia.com/gpu: "4"
 memory: 200Gi
 readinessProbe:
 httpGet: { path: /health, port: 8000 }
 initialDelaySeconds: 60
 periodSeconds: 10
 failureThreshold: 30 # tolerates the warmup
 lifecycle:
 preStop:
 httpGet: { path: /shutdown, port: 8000 }
 volumeMounts:
 - { name: model-cache, mountPath: /models, readOnly: true }
 - { name: dshm, mountPath: /dev/shm }
 volumes:
 - name: model-cache
 persistentVolumeClaim: { claimName: llama70b-fp8-pvc }
 - name: dshm
 emptyDir: { medium: Memory, sizeLimit: 16Gi }
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata: { name: vllm-llama70b-metrics, namespace: inference }
spec:
 selector: { matchLabels: { app: vllm-llama70b } }
 podMetricsEndpoints:
 - port: metrics
 path: /metrics
 interval: 15s
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: vllm-llama70b-scaler, namespace: inference }
spec:
 scaleTargetRef: { name: vllm-llama70b }
 minReplicaCount: 2
 maxReplicaCount: 20
 pollingInterval: 15
 cooldownPeriod: 300
 advanced:
 horizontalPodAutoscalerConfig:
 behavior:
 scaleDown:
 stabilizationWindowSeconds: 600
 policies:
 - { type: Pods, value: 1, periodSeconds: 120 }
 scaleUp:
 stabilizationWindowSeconds: 30
 policies:
 - { type: Pods, value: 2, periodSeconds: 60 }
 triggers:
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_queue
 threshold: "5"
 query: avg(vllm:num_requests_waiting{deployment="vllm-llama70b"})
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_kv
 threshold: "0.85"
 query: avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama70b"})
 - type: cron
 metadata:
 timezone: Europe/Madrid
 start: "30 8 * * 1-5"
 end: "0 19 * * 1-5"
 desiredReplicas: "6"

This set is the minimum viable one for LLM autoscaling on a generic cluster with the NVIDIA GPU Operator. Each team adapts it to its own SLO.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB per node, 4 GPU nodes:

Each node hosts one vLLM replica with TP=4 and Llama 70B FP8 (one model per node, they are not shared).
Warm pool of 2 replicas off-peak; the KEDA cron raises it to 4 during working hours.
Cluster-autoscaler does not apply (4 physical nodes bought; scaling is pods only). The number of concurrent replicas is at most the number of available nodes (if each replica uses all 4 GPUs of the whole node).
If the sizing requires more simultaneous replicas than nodes, there are two routes: (a) lower the TP of each replica so two fit per node, (b) expand the physical nodepool. The decision is dictated by capacity planning, see Capacity planning for on-premise LLM inference.

KEDA event volume: around 5 evaluations/min per ScaledObject. For 10 models served in parallel, 3,000 evaluations/h. Manageable with one KEDA operator per cluster.

What we have not covered (upcoming articles)

Cluster-autoscaler for on-premise GPU nodes: how to orchestrate bare metal provisioning (Tinkerbell, Metal³) as a function of demand.
Multi-cluster autoscaling: scaling across clusters in different DCs for geographic resilience.
Cost-aware autoscaling: prioritising nodes by hourly energy cost (on clusters with indexed tariffs).
Predictive ML-based scaling: instead of a static cron, training a model that predicts demand 30 minutes ahead.
Multi-tenant quotas and fairness: KEDA with namespace quotas so that one tenant does not monopolise the HPA.

References

KEDA project — keda.sh (documentación oficial de triggers Prometheus y cron).
Kubernetes — Horizontal Pod Autoscaler walkthrough (kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale).
NVIDIA — GPU Operator on Kubernetes (Helm chart oficial con DaemonSet de drivers y DCGM).
vLLM project — production_monitoring/ (métricas Prometheus expuestas por el servidor).
Karpenter — NodePool spec (etiquetado y taints para nodepools GPU).
Cluster Autoscaler — Scaling GPU nodes (caveats de descubrimiento de recursos GPU).
Kubernetes — Pod lifecycle and termination (preStop, terminationGracePeriodSeconds).

Cold-Start on lo0 — Tech Blog

Knative and scale-to-zero for LLM inference: when switching off the GPU saves money and when it costs you the SLO

TL;DR

The analogy: the restaurant that switches off the kitchen

How Knative scales

KPA versus HPA

Why an LLM cold start is not starting a pod

The techniques that shorten the wait

Knative and KServe: the two modes

The economics

Operational traps and honest scepticism

For an inference factory

See also

Sources

Speeding up model cold start: from minutes to seconds

TL;DR

The analogy: opening the kitchen in the morning

The five line items of cold start

Speeding up item 4: loading the weights

safetensors versus pickle: the format rules

Tensorizer: streaming tensor by tensor straight to the GPU

Run:ai Model Streamer: reading and copying at the same time

The maths of the weights: $t = W/B$ and the saving from overlapping

The storage tier

Speeding up item 5: everything that is not weights

Caching the torch.compile/Inductor compilation

Bounding the CUDA graph capture

Warm pools: not paying for item 5 on the critical path

Sleep mode: the shortcut that skips all five items

An example cold start, item by item

The economics of scale-to-zero

Applied to the generic 4×H100 cluster

Traps and things that are not what they seem

Conclusion

See also

References

The pantry lift: from disk to HBM, or why the kitchen opens late

TL;DR

Where you are: the basement, before opening

The analogy: the pantry lift

The mechanism: what really happens when a model loads

Why the problem exists: the economics of bytes

The maths that matter: cold start as a tax on autoscaling

The 10 knobs worth touching

Knob 1 — Measure where the time goes (read vs deserialise vs H2D)

Knob 2 — --load-format: choosing the porter

Knob 3 — Streamer concurrency

Knob 4 — GPUDirect Storage (GDS) + fastsafetensors

Knob 5 — Local NVMe for the weights, not network

Knob 6 — Pre-pull / local model cache

Knob 7 — Quantisation: move fewer bytes

Knob 8 — Parallel loading across GPUs (vLLM V1)

Knob 9 — NUMA locality of the NVMe

Knob 10 — Do not pay the cold start: keep pods warm

Summary table

How it connects with the rest of the stack

Traps and things that are not what they seem

Conclusion

See also

References

Autoscaling LLM inference on Kubernetes: HPA with custom metrics and KEDA for vLLM

TL;DR

You are here: DEPLOY

The analogy: the bakery with wood-fired ovens

Why HPA on CPU is no use

The four viable signals

The wiring: KEDA as the Prometheus adapter

The big operational problem: cold start

The five levers

When to scale nodes, not just pods

Three pitfalls specific to LLM scale-down

Full example manifest

Applied to typical on-premise hardware

What we have not covered (upcoming articles)

See also

References

Knob 2 — `--load-format`: choosing the porter