Cuda-Graphs on lo0 — Tech Blog

Speeding up model cold start: from minutes to seconds

Thu, 11 Jun 2026 13:00:00 +0000

This is the third batch of an operational series on squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB with NVLink. The sister pieces in this batch are Multimodal: serving a VLM on-premise with vLLM, adding vision to the same engine, and FinOps and GPU multi-tenancy with LiteLLM, sharing out and charging for the GPU across teams. This post takes the problem that From disk to HBM laid out conceptually and turns it into a runbook: how you actually bring the cold start down, knob by knob, so that elasticity is usable.

TL;DR

The post From disk to HBM left an uncomfortable idea behind: moving the weights from disk to HBM is only one of the five line items of cold start. The other four, starting the process, creating the CUDA context, setting up the allocator and, the most expensive one that hardly anyone looks at, capturing the CUDA graphs and compiling the JIT kernels, are untouched by any fast loader. That is why a server that loads the weights in 6 s can still take 90 s to be ready. This runbook attacks all five. For the weights: safetensors (mmap, zero-copy, no bounce through a host FP32 buffer) versus the pickle of torch.load; Tensorizer (CoreWeave), which serialises the model into one file and streams it tensor by tensor straight to the GPU from object storage/HTTP/S3; and the Run:ai Model Streamer (from NVIDIA), which reads concurrently and overlaps the read with the H2D copy. All three are switched on with one flag, --load-format, in vLLM. For the non-weight part: torch.compile/Inductor compilation cache (persisted and reused across start-ups), bounding the CUDA graph capture to the batch sizes you really use (the default capture can eat ~54 s), and sleep mode as a shortcut that skips all five line items. The maths: $t = W/B$ for the weights, the saving from overlapping read and H2D instead of doing them in series, and a 70B cold start broken down item by item. And the economics: when scale-to-zero with pre-warm beats keeping a warm replica. On the generic 4×H100 SXM 80 GB cluster.

The analogy: opening the kitchen in the morning

A restaurant does not open by flicking a switch. Anyone who has worked in one knows it: opening time is decided by everything that happens beforehand, and almost none of it is “bringing in the food”.

The first cook arrives at a cold, dark place. They have to: turn on the lights and start the systems (the process init); fire up the ovens and let them come up to temperature (the CUDA context, which takes its time to become operational); get the boards out, sort the cold room, organise the work space (the memory allocator, which reserves and structures the HBM); go down to the store and bring up the produce, the crates of vegetables, the meat, the fish, and put it in its place (loading the weights from disk to HBM, the entire subject of From disk to HBM); and, the most forgotten of all, sharpen every knife, set up the mise en place, make the stocks, pre-heat the mother sauces, all the preparation that is not an ingredient but without which not a single dish goes out (CUDA graph capture and JIT compilation of the kernels).

Here is the trap that From disk to HBM already anticipated: if you only optimise the produce, you still open late. You can hire the best delivery service in the world, a blazingly fast service lift, crates that put themselves away. But if the cook is still sharpening knives and waiting for the ovens to come up to temperature, the kitchen does not serve the first dish. The produce arrived early and the restaurant is still shut.

Today’s runbook is, literally, the list of everything that has to be done for the kitchen to open on time, and which of those tasks can be sped up, which can be left done from the night before (cached), and when it pays better never to switch the kitchen off (warm replica) than to light it again every morning (scale-to-zero).

The five line items of cold start

From the moment an inference pod is born until it returns its first token, the clock runs in five different places. It is worth naming them, because each optimisation attacks one or two, never all of them:

Process init. Starting the Python interpreter, importing the engine (vLLM and its dependencies), parsing the configuration. Seconds, and it grows with the size of the environment and the CUDA/PyTorch imports.
CUDA context. The first cudaSetDevice / cudaFree(0) initialises the CUDA runtime against the driver: it loads the context and maps the GPU. It is not instant, on the order of one to several seconds per GPU depending on driver and number of devices.
Allocator. vLLM sets up its memory manager over the HBM (PyTorch’s caching allocator plus the KV-cache layout). Reserving and structuring tens of GB has its cost.
Weight loading disk→HBM. The journey that From disk to HBM dissected: disk → page cache → host buffer → PCIe → HBM. What almost everyone believes cold start is, and it is only one of the five.
CUDA graph capture + kernel JIT. vLLM captures CUDA graphs for the various batch shapes and compiles kernels with torch.compile/Inductor (and backends such as DeepGEMM or FlashInfer). This item is the great ignored one: the default graph capture in vLLM takes on the order of 54 s because it covers a wide range of batch sizes (vLLM, torch.compile integration; Red Hat, vLLM with torch.compile).

The lesson the diagram is shouting: on a modern server with torch.compile and CUDA graphs, item 5 can be as big as or bigger than item 4. Speeding up weight loading from 40 s to 6 s is a huge advance, but if you leave graph capture at 50 s, the cold start falls from 104 s to ~70 s, not to 12 s. You have to attack both halves.

Speeding up item 4: loading the weights

safetensors versus pickle: the format rules

The first knob is the on-disk format. The old torch.load uses pickle, which has two problems. The security one is well known: deserialising a pickle executes arbitrary Python code, since the __reduce__ protocol lets the file invoke any callable on load, so every model download was a remote execution vector (HuggingFace, Safetensors). The performance one is what concerns us: loading a pickle rebuilds Python objects and usually passes through an intermediate host buffer before reaching the GPU.

safetensors, now under the PyTorch Foundation (HuggingFace, Safetensors joins PyTorch Foundation), solves both. The file is just a JSON header + raw tensor bytes: loading it cannot do anything except populate tensor buffers. And the key property for us is physical: the data region is aligned to a page boundary (the header is padded so that the first tensor starts at a multiple of the OS page size). That is what makes zero-copy possible: a loader can mmap the file, cudaHostRegister over the mapped region and DMA straight from the page cache into VRAM, with no torch.load deserialisation and no temporary host FP32 buffer (HuggingFace, Safetensors). It is the base format; everything else is built on top of it.

The mmap trap (already flagged in From disk to HBM): mmap reads nothing immediately, it defers the cost to the first access to each page. If you do not force the read, the cold start looks short and the first token pays the page faults. And the “blazingly fast second load” is the page cache lying to you: in production pods are born cold, on nodes where those files are not cached.

Tensorizer: streaming tensor by tensor straight to the GPU

Tensorizer, from CoreWeave, serialises the model’s weights and its tensors into a single file and, instead of loading the whole model into RAM before moving it to the GPU, streams the data tensor by tensor from disk, an HTTP/HTTPS endpoint or an S3 bucket, deserialising on the fly straight onto the GPU (vLLM, Loading Models with CoreWeave’s Tensorizer). The operational advantage: near-instant loading and low host RAM usage during initialisation, which matters especially in serverless and autoscaling scenarios, exactly the scale-to-zero case. And it decouples the weights from the container image: the model lives in object storage, not inflating the image pull.

In vLLM it is enabled with --load-format tensorizer (or load_format="tensorizer" via the API). It requires serialising the model once into Tensorizer’s format; from then on, any pod streams it.

Run:ai Model Streamer: reading and copying at the same time

The Run:ai Model Streamer (from NVIDIA) attacks the bottleneck from another angle: it reads the tensors concurrently, N OS threads reading from storage into the CPU buffer, while streaming them to VRAM, so that the storage read and the H2D copy overlap instead of happening in series (vLLM, Loading models with Run:ai Model Streamer; NVIDIA, Reducing Cold Start Latency). It reads safetensors directly, with no reconversion.

The numbers published by NVIDIA give the order of magnitude: the streamer reaches 4.88 s reading from S3 at concurrency 32 and 7.53 s from an SSD IO2 at concurrency 8; integrated into vLLM, the total time to ready drops to 23.18 s from S3, 28.28 s from SSD IO2 and 35.08 s from GP3 (NVIDIA, Reducing Cold Start Latency). Note the gap between the weight loading time (~5–8 s) and the total to ready (~23–35 s): those ~18–27 s of difference are the other four line items, above all item 5. The streamer fixed item 4 and the rest is still there, exactly what this post warns about.

Concurrency is the tuning parameter: it controls how many OS threads read tensors into the CPU buffer (and, for S3, how many client connections the host opens) (vLLM, Run:ai Model Streamer). 16 is usually enough for local NVMe; 32 for high-throughput object storage. One thread does not saturate a Gen5 NVMe; overlapping does.

# default safetensors (the slowest of the fast ones)
vllm serve <model> --load-format safetensors

# Run:ai Model Streamer (concurrent reads, overlaps read + H2D)
vllm serve <model> --load-format runai_streamer \
 --model-loader-extra-config '{"concurrency": 32}'

# Tensorizer (tensor-by-tensor stream straight to GPU from object storage/S3)
vllm serve <model> --load-format tensorizer

The maths of the weights: $t = W/B$ and the saving from overlapping

The floor of item 4 is simple: moving $W$ gigabytes at a bandwidth $B$ takes

$$t = \frac{W}{B}$$

Take a large model, a 70B. In BF16 ($b=2$ bytes/param) that is $W = 70 \times 10^9 \cdot 2 = 140$ GB; in FP8 ($b=1$), $W = 70$ GB. Bandwidth depends on the tier (see the table below): local Gen5 NVMe reads on the order of ~14 GB/s per disk; PCIe Gen5 x16 copies host→GPU at ~50 GB/s; network storage, ~1–3 GB/s. The theoretical floor for reading 140 GB from an NVMe is $140/14 = 10$ s; from the network at 2 GB/s, 70 s. The bottleneck rules.

The overlapping trick. The default loader does the two steps, reading from disk and copying H2D, in series: first it fills a buffer, then it copies, then the next one. The time is the sum:

$$t_{\text{series}} = t_{\text{read}} + t_{\text{H2D}}$$

Tensorizer and the Run:ai Model Streamer overlap them: while one thread copies a chunk to VRAM, another is already reading the next one from disk. With enough concurrency, the total time tends towards the maximum of the two, not the sum:

$$t_{\text{overlapped}} \approx \max(t_{\text{read}},\, t_{\text{H2D}})$$

With the 70 GB of the 70B in FP8, from NVMe at 14 GB/s and PCIe at 50 GB/s: $t_{\text{read}} = 70/14 = 5.0$ s, $t_{\text{H2D}} = 70/50 = 1.4$ s. In series, $5.0 + 1.4 = 6.4$ s; overlapped, $\max(5.0,\,1.4) = 5.0$ s. The saving here is modest (~22%) because the disk dominates comfortably. Overlapping shines when the two steps are comparable: from network storage at 5 GB/s, $t_{\text{read}} = 70/5 = 14$ s and $t_{\text{H2D}} = 1.4$ s, series 15.4 s vs overlapped 14 s; but with many concurrent streams saturating a fast disk and a matched H2D, going from the sum to the maximum can almost double the effective throughput. The flip side: overlapping only gives you what the physical bottleneck allows. If the disk only delivers 14 GB/s, no streamer brings you below 5 s for 70 GB, and that is what the other lever is for, moving fewer bytes (FP8 versus BF16 halves $W$).

The storage tier

Tier	Typical bandwidth	140 GB (70B BF16)	70 GB (70B FP8)	Notes
Local Gen5 NVMe	~14 GB/s per disk	~10 s	~5 s	the short path; one disk
Local Gen5 NVMe (several, RAID0)	~28–50 GB/s	~3–5 s	~1.5–2.5 s	if the loader saturates several streams
Object storage (S3/RGW)	~1–3 GB/s per stream	~47–140 s	~23–70 s	streamer concurrency helps a great deal
NFS / shared network	~1–2 GB/s	~70–140 s	~35–70 s	puts the network and its contention in the path

The operational conclusion is the same as in From disk to HBM: the weights that serve the cold start live on the node’s local NVMe, not on the network. Network storage is the repository; the inference node has a hot local copy (pre-pull with an initContainer or a per-node cache DaemonSet). Tensorizer is the interesting exception: it streams from object storage so efficiently that it sometimes makes serving from S3/RGW viable with no local copy, at the price of depending on the bandwidth of the storage network.

Speeding up item 5: everything that is not weights

Here is the half of the cold start that fast loaders do not touch. Three fronts.

Caching the torch.compile/Inductor compilation

torch.compile has a built-in cache system: the compiled artefacts are saved after the first start-up and reused across start-ups, even across machines if it is configured properly. In vLLM, the result of the Dynamo compilation is stored in ~/.cache/vllm/torch_compile_cache/ (vLLM Blog, Introduction to torch.compile; vLLM, torch.compile integration). The consequence for the runbook: if that directory is a persistent volume shared between pods (a PVC, or a hostPath on the node’s NVMe populated by the first start-up), subsequent pods skip the compilation and start faster. The first pod pays for the compilation; the rest inherit the cache.

It is exactly “sharpening the knives the night before”: the expensive preparation is done once and reused. The nuance: the cache is sensitive to the vLLM/PyTorch version, the GPU, the quantisation and the configuration, and changing any of them invalidates the cache and you pay for the compilation once again.

Bounding the CUDA graph capture

vLLM captures CUDA graphs for a wide range of batch sizes by default, and that is expensive: the default capture is around 54 s, and on large configurations up to 294 s has been measured. By limiting the capture to the batch sizes your workload really uses (for example 1, 2, 4, 8, 16, 24, 32, 64) the cold start is cut by more than 70%, from 294 s to ~82 s in the measured case (Red Hat, vLLM with torch.compile). The knob is the engine’s compilation/CUDA graphs configuration; the principle is not to capture graphs for batch sizes you will never see.

A real trade-off: fewer captured batch sizes = faster start-up but less coverage. A batch outside the captured set falls back to the eager path (slower per request). You bound to the frequent sizes of your traffic, not of any traffic. And there is always the extreme of disabling CUDA graphs (enforce_eager): near-instant start-up on item 5 at the cost of throughput in the hot state, valid for debugging or for very low-traffic services where cold start weighs more than steady state.

Warm pools: not paying for item 5 on the critical path

The definitive shortcut for all five items is not to run them while the user waits. A warm pool, a replica already started, with weights loaded, graphs captured and kernels compiled, waiting for traffic, turns the cold start into zero for that request. It is GPU cost (partly or wholly idle) in exchange for start-up latency; we analyse it in the economics section below.

Sleep mode: the shortcut that skips all five items

And there is a shortcut halfway between “start from scratch” and “keep a whole replica warm”: vLLM’s sleep mode, the subject of Serving several models on a single GPU. Instead of killing the process, it puts it to sleep: it parks the weights in host RAM (level 1) or discards them (level 2), but keeps the process alive, and with it the CUDA context, the allocator, the CUDA graphs and the already compiled JIT kernels. The wake pays for neither item 1, 2, 3 nor 5; it only puts the weights back into VRAM (item 4, and from RAM, not from disk). That is why a wake is 18–200× faster than a full cold start, and even level 2, which reloads the weights from the same disk, is still 23–45× faster, because it skips the other four items (vLLM Blog, Sleep Mode). Sleep mode is living proof of this post’s thesis: if preserving items 1, 2, 3 and 5 gives you 18–200×, then moving bytes was never the whole cost.

An example cold start, item by item

Let us put numbers on a cold start of 70B in FP8 (70 GB) from local NVMe and see which optimisation attacks each item:

#	Line item	Default	Optimisation	Optimised
1	Process init	~4 s	(little room; minimal environment)	~3 s
2	CUDA context	~6 s	(driver; fixed)	~6 s
3	Allocator	~4 s	(fixed)	~4 s
4	Weight loading	~25 s	Run:ai streamer / Tensorizer (concurrency + overlap)	~5 s
5	CUDA graphs + JIT	~54 s	torch.compile cache + bounded capture	~10 s
	Total	~93 s		~28 s

Two readings. First: optimising only item 4 (a fast loader and nothing else) brings the total down from 93 to ~73 s, a real but disappointing improvement, because item 5 is still intact. Optimising only item 5 (cache + bounded capture) brings it down to ~49 s. Attacking both brings it down to ~28 s, and that is what turns theoretical scale-to-zero into usable scale-to-zero. Second: items 2 and 3 are practically irreducible, they depend on the driver and the hardware; they constitute the floor of the cold start, on the order of 10 s, which no software optimisation crosses. Below that floor, the only answer is not to start from scratch: sleep mode (wake from sub-second to a few seconds) or a warm replica (zero).

The economics of scale-to-zero

The capacity question all of this serves is: switch off or keep warm? Scale-to-zero saves GPU while there is no traffic, but pays the cold start when it comes back. The decision is a balance between the cost of an idle GPU and the latency SLA.

The basic criterion: scale-to-zero is only viable if the cold start fits inside the start-up SLA your users tolerate, or if you have a pre-warming mechanism that starts the replica before the request arrives (load prediction, an early signal from the queue). Without pre-warm, a 90 s cold start means the first user after an idle period waits 90 s, unacceptable for almost any interactive SLO. With the cold start brought down to ~28 s it is still a lot for an interactive request, but it is already tolerable for batch workloads or for a pre-warm triggered by autoscaling with KEDA on queue depth.

The decision rule, in one sentence: keep a warm replica when the cost of the idle GPU during the troughs is lower than the cost of missing the SLA on every start-up; do scale-to-zero (with pre-warm) when the troughs are long and deep and the optimised cold start fits in your tolerance. For a 24×7 interactive service with irregular but continuous traffic, a floor of warm replicas almost always wins. For a large model invoked a few times a day (a 70B for hard tasks), scale-to-zero with an optimised cold start, or sleep mode if it shares a GPU with another model, is the sensible option. It is the same capacity planning boundary: cold start is a parameter of the replica cushion, not a start-up detail.

Applied to the generic 4×H100 cluster

Let us bring the runbook down to the 4 H100 SXM 80 GB with NVLink. The concrete decisions:

Loader and format. Weights in safetensors on disk (never pickle). For the short path from local NVMe, the Run:ai Model Streamer (--load-format runai_streamer, concurrency 16–32) is the default option: concurrent reads, overlapping read and H2D, and you only change one flag. Keep Tensorizer for the case where you serve from object storage (RGW/S3) and want to decouple the weights from the image with no local copy, since its tensor-by-tensor streaming straight to the GPU is what makes that pattern viable. In both cases, FP8 versus BF16 halves item 4 almost for free (measure the quality, do not assume it).

Storage tier. Model repository on the network (Ceph RGW), hot copy on the node’s local NVMe populated by pre-pull. The cold start that counts is the cold one, on a node where the files are not in page cache; serving that start-up from the network puts its latency and contention into the critical path.

Caching item 5. The torch_compile_cache directory on a persistent volume or a hostPath NVMe shared between pods of the same model: the first pod compiles, the rest inherit. And bound the CUDA graph capture to the real batch sizes of each service’s traffic.

When scale-to-zero with pre-warm vs a warm replica. The agent LLM (main service, continuous traffic) never does scale-to-zero: a floor of warm replicas on one or two GPUs. The occasional 70B does: scale-to-zero with an optimised cold start, or, if it shares a GPU with a mid-sized model, sleep mode so the wake is sub-second instead of a 28 s start-up (exactly the play in Serving several models on a single GPU). The optimised cold start in this post is what makes swap and failover between replicas tolerable: a replica that falls over and is replaced in 28 s degrades less than one that takes 90 s. And it connects directly with multi-tenancy: a short cold start lets you share GPUs between teams with per-tenant scale-to-zero without the first user of each tenant paying for an endless start-up.

The cross-cutting principle: cold start is the ceiling on elasticity. The five line items, with both halves attacked, a fast loader for the weights and cache plus bounded capture for the graphs, bring it down from minutes to seconds. And below the irreducible floor of ~10 s (CUDA context + allocator), the only way out is not to start from scratch: sleep mode or a warm replica.

Traps and things that are not what they seem

“I switched to a fast loader and start-up barely improved.” Your bottleneck probably was not item 4. If CUDA graph capture eats 54 s, bringing weight loading down from 25 to 5 s takes 20 s off 93, which is barely noticeable. Measure where the time goes before optimising: vLLM’s start-up logs break down weight loading versus graph capture.

“The second time it started in a flash.” The page cache (for the weights) and the torch.compile cache (for the graphs) lying to you at the same time. The start-up that matters is the cold one, on a clean node. Benchmarking the second start-up measures a situation that almost never occurs in the peak that triggers the autoscaler.

“Tensorizer/streamer will give me the 18–200× of sleep mode.” No. Those loaders speed up only item 4. The 18–200× of sleep mode comes from preserving the other four by keeping the process alive. They are tools for different problems: a fast loader is for a real cold start (a new process); sleep mode is for alternating models without killing the process.

"enforce_eager fixes the cold start." It fixes item 5 (no graph capture, near-instant start-up) but degrades throughput in the hot state, since CUDA graphs exist because they speed up the steady state. It is a trade: it wins for very low traffic services where start-up weighs more than the hot path; it loses for a service with sustained load.

“I bound the CUDA graphs to a single batch size and that is it.” You bound to the sizes your traffic uses, not to one. A batch outside the captured set falls back to the eager path and runs slower. Too aggressive and you penalise the steady state to gain a few seconds of start-up.

“Moving the weights to FP8 also speeds up item 5.” Not directly. FP8 halves the bytes (item 4) and doubles inference throughput, but graph capture and kernel compilation do not depend on the size of the weights, they depend on the shape of the graph and the batch sizes. Item 5 is attacked with caching and bounded capture, not with quantisation.

Conclusion

From disk to HBM said it conceptually: moving the weights is one of the five line items of cold start. This runbook turns it into action. For the weights, item 4, there is a clear menu: safetensors as the base (mmap, zero-copy, no pickle), Run:ai Model Streamer for the short path from NVMe (concurrent reads overlapping read and H2D), Tensorizer for streaming from object storage with no local copy, and the cross-cutting lever of moving fewer bytes with FP8. But the forgotten half of the start-up is item 5, CUDA graph capture and JIT compilation, which on a modern server can be as big as the weight loading: you attack it by caching the compilation across pods and bounding the graph capture to the real batch sizes. Attacking only one half leaves the cold start half done; attacking both brings it from ~90 s to ~28 s. And below the irreducible floor of ~10 s imposed by the CUDA context and the allocator, the only way out is not to start from scratch: sleep mode (sub-second wake) or a warm replica (zero). The kitchen opens on time when somebody took care of the produce and of sharpening the knives the night before. Optimising only the delivery from the store leaves the ovens cold and the restaurant shut.

References

HuggingFace, Safetensors (format, zero-copy, security versus pickle): https://huggingface.co/docs/safetensors/index
HuggingFace, Safetensors is Joining the PyTorch Foundation: https://huggingface.co/blog/safetensors-joins-pytorch-foundation
vLLM, Loading Models with CoreWeave’s Tensorizer: https://docs.vllm.ai/en/stable/models/extensions/tensorizer/
vLLM, Loading models with Run:ai Model Streamer: https://docs.vllm.ai/en/stable/models/extensions/runai_model_streamer/
NVIDIA, Reducing Cold Start Latency for LLM Inference with NVIDIA Run:ai Model Streamer (S3/SSD benchmarks, concurrency): https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer/
vLLM Blog, Introduction to torch.compile and How It Works with vLLM (compilation cache): https://blog.vllm.ai/2025/08/20/torch-compile.html
vLLM, torch.compile integration (cache in ~/.cache/vllm/torch_compile_cache/): https://docs.vllm.ai/en/latest/design/torch_compile/
Red Hat Developer, vLLM with torch.compile: Efficient LLM inference on PyTorch (CUDA graph capture ~54 s, bounding batch sizes −70%): https://developers.redhat.com/articles/2025/09/03/vllm-torchcompile-efficient-llm-inference-pytorch
vLLM Blog, Zero-Reload Model Switching with vLLM Sleep Mode (wake 18–200×): https://blog.vllm.ai/2025/10/26/sleep-mode.html

The roofline flips: why optimising small models is a different performance game

Tue, 09 Jun 2026 01:30:00 +0000

This post is the anchor of a mini-series on inference performance in small models (SLMs). Almost every optimisation post on the blog, KV cache, decode, quantisation, was written with a 70B in mind. Here I argue that when the model shrinks by an order of magnitude, the roofline changes regime and several of those intuitions flip. This is not a nuance: it is a different game.

TL;DR

Autoregressive decode in a large LLM is memory-bandwidth-bound: at every step you have to move all the model weights from HBM to the registers of the SMs, and that dominates over the arithmetic operations. The GPU spends its time waiting for bytes, not computing. That single sentence, that decode “waits on HBM”, is the root of half the optimisations on this blog. In a small model (SLM, say 0.5B–7B) the sentence stops being true in the simple form we used to tell it. At batch 1 you are still memory-bound with respect to the hardware, yes, but the forward pass is so cheap (moving 6 GB at 1 TB/s takes ~6 ms, not 70 ms) that the fixed per-step costs, kernel launches, Python scheduler overhead, the sampler, host↔device copies, the synchronize calls, stop being noise and start eating 20-30 % of the time. The bottleneck moves from HBM to orchestration. The concrete, quantitative consequences: (1) CUDA graphs and cutting scheduler overhead pay off more in SLMs than in large models; (2) weight quantisation gives less latency improvement at batch 1 in SLMs, because proportionally there are fewer weights to move relative to activations, KV cache and fixed overhead; (3) batching has more headroom because you cross the ridge point late; (4) the KV cache can dominate relative memory. All of this comes out of a single model, the roofline, applied with numerical honesty.

The analogy: the pantry and the waiter

A kitchen with two very different services.

Tasting menu service, one enormous slow dish (the 70B LLM). Every dish involves heavy ingredients that the assistant has to fetch from the pantry at the back, several times, carrying boxes. The chef, by contrast, assembles the dish in a moment: the slow part is fetching the ingredients, not cooking them. If you want the service to go faster, you do not hire a more skilful chef: you widen the corridor to the pantry, or make each trip bring more boxes. The pantry is HBM; the trip is memory bandwidth; cooking is compute. The big dish is bound by the pantry.

Tapas service, tiny plates (the SLM). Now each tapa has two ingredients and is assembled in a second. The trip to the pantry per tapa is very short. But a cost appears that was negligible for the big dish: the waiter. For every tapa, the waiter has to walk to the kitchen, pick up the plate, carry it to the bar, walk back, take the order, call it out. That back and forth is fixed: it costs the same for one tapa as for the enormous dish. When the tapa is assembled in a second, the waiter, not the pantry, is the bottleneck. Shortening the corridor to the pantry (widening HBM, quantising the weights) barely improves the service any more; what improves it is the waiter chaining several orders together without going back to the kitchen each time (CUDA graphs) or serving several tables in one pass (batching).

The roofline is the tool that says, with numbers, at what point the waiter dominates over the pantry. That boundary is the ridge point, and the joke in the title is that in SLMs we cross the regime much earlier than large-model intuition led us to believe.

The bare mechanism: what the roofline says

The roofline model (Williams, Waterman and Patterson, 2009) starts from a single quantity: arithmetic intensity, which is how many operations you do per byte moved from memory.

$$\text{AI} = \frac{\text{FLOPs}}{\text{bytes moved from memory}} \quad [\text{FLOP/byte}]$$

The hardware has two ceilings: compute (peak FLOPS) and memory (peak bandwidth × AI). Achievable performance is the minimum of the two:

$$\text{Perf} = \min\big(\text{peak FLOPS},\; \text{BW} \times \text{AI}\big)$$

Where the two lines cross is the ridge point, the AI above which you stop being limited by memory and start being limited by compute:

$$\text{AI}_{\text{ridge}} = \frac{\text{peak FLOPS}}{\text{peak BW}}$$

If your kernel has an AI below the ridge, you are memory-bound (the GPU waits for bytes). Above it, compute-bound (the GPU computes flat out and memory is in surplus). What matters is that the ridge point is a property of the hardware, not of the model. Let us look at the numbers, approximate, and I flag them as such because marketing figures mix dense and sparse, different dtypes and unrealistic thermal conditions.

Generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). Per GPU, ~989 TFLOPS BF16 dense (~1,979 TFLOPS FP8 dense; the figure with sparsity is double that and almost never applies to LLM inference). HBM3 ~3.35 TB/s. The ridge in BF16:

$$\text{AI}_{\text{ridge}}^{\text{H100,BF16}} \approx \frac{989 \times 10^{12}}{3.35 \times 10^{12}} \approx 295 \ \text{FLOP/byte}$$

In FP8 the ridge rises to ~590 FLOP/byte (twice the FLOPS against the same BW). Careful: these are datasheet peak figures; in practice a real kernel rarely gets past 70-80 % of either ceiling.

RTX 4090 (24 GB, Ada Lovelace). ~330 TFLOPS FP16 with FP16 accumulation via tensor cores (the “660 TOPS” figure that circulates is with sparsity), and ~1 TB/s of GDDR6X. The ridge:

$$\text{AI}_{\text{ridge}}^{\text{4090,FP16}} \approx \frac{330 \times 10^{12}}{1.0 \times 10^{12}} \approx 330 \ \text{FLOP/byte}$$

Curiously, the same order as the H100 in BF16: the 4090 has less BW but also fewer FLOPS, and the ratio comes out similar. The ridge sits around 300 FLOP/byte in both cases. Hold on to that number.

And where does decode fall? In decode at batch 1, each weight is loaded once from HBM and used for a single multiply-accumulate (one token, one row of activation). The AI of the decode GEMM at batch 1 is on the order of AI ≈ 1-2 FLOP/byte (each weight byte takes part in ~2 FLOP). With batch B, the same weight loaded once serves B rows of activation, so the AI scales roughly linearly:

$$\text{AI}_{\text{decode}}(B) \approx 2B \ \text{FLOP/byte} \quad (\text{for the weight GEMM part})$$

You cross the ridge when 2B ≈ 300, that is B ≈ 150 as an order of magnitude (in practice earlier, because of attention and overheads, but that is the frame). The clean conclusion: decode at low batch is always deeply memory-bound, nowhere near the ridge. That is why we say “decode waits on HBM”, and why quantising weights (moving fewer bytes) speeds up decode in a large model almost linearly. So far, this is the standard story from the large-model posts.

The point of the title: why it flips in SLMs

The classic roofline has a blind spot that does not matter in large models and is everything in small ones: it only models the work inside the kernel. It assumes the only time is bytes/BW or FLOPs/FLOPS. But a real decode step is not just the GEMM. It is a sequence of dozens of kernels (QKV projections, attention, the two MLP layers, normalisations, residuals, the logits head, sampling) and, around each one, there is a fixed orchestration cost:

Kernel launches (kernel launch): each cudaLaunchKernel costs on the order of 5-10 µs of CPU/driver overhead, regardless of kernel size. A decode forward with ~30-60 kernels launched sequentially drags ~0.3-0.6 ms just in launching.
Python scheduler overhead: the vLLM scheduler loop prepares metadata, decides which requests enter the step, builds the input tensors. In pure Python this is hundreds of µs to a couple of ms per step, especially at low concurrency where it is not amortised.
Sampling and post-processing: applying temperature, top-p, penalties, the argmax/multinomial, copying the token back. Another block of hundreds of µs.
Synchronisations and host↔device copies: every synchronize or small copy adds latency that is neither FLOPs nor HBM bytes.

Let us call the sum of all of that T_fixed, the per-step cost independent of model size, on the order of 1-3 ms in an unoptimised Python stack. The real time of a step is now:

$$T_{\text{step}} \approx \underbrace{\frac{\text{weight bytes}}{\text{BW}}}_{T_{\text{HBM}} \text{ (memory-bound)}} + \; T_{\text{fixed}}$$

In a 70B BF16, moving ~140 GB at 3.35 TB/s takes ~42 ms of T_HBM. Against that, a T_fixed of 1-3 ms is noise (2-7 %). The classic roofline is right: the model is memory-bound, full stop. But in a 3B BF16, T_HBM drops to a few ms, and suddenly T_fixed is the same order as T_HBM. The bottleneck stops being the pantry and becomes the waiter. This is the flip in the title, and four counterintuitive consequences follow from it:

(a) At batch 1 you are still memory-bound with respect to the hardware. The AI has not changed: it is still ~2 FLOP/byte, below the ridge. Anyone reading only the roofline will conclude “memory-bound, quantise the weights”. That is true but incomplete: the roofline does not see T_fixed.

(b) Fixed costs become a huge fraction of the step. This is the central point. In the 70B, T_fixed / T_step ≈ 5 %. In the 3B it can be 20-30 %. The effective bottleneck of the 3B is half HBM, half orchestration.

(c) That is why CUDA graphs and cutting scheduler overhead pay off MORE in SLMs. A CUDA graph captures the whole kernel sequence of the step and relaunches it with a single cudaGraphLaunch, removing almost all of the per-kernel launch overhead and much of the per-iteration Python scheduler work. In the 70B, shaving 0.5 ms off a 42 ms step is a +1 % you barely notice. In the 3B, shaving those same 0.5 ms off a ~7 ms step is +7 %, and if you take away nearly all of T_fixed you can gain 20-30 %. The same optimisation, a different prize, because the denominator changed.

(d) Weight quantisation gives LESS latency improvement at batch 1 in SLMs. This is the most counterintuitive one. In the 70B, T_HBM is almost the whole step; going from BF16 to INT4 quadruples the effective weight bandwidth and almost quadruples decode speed. In the 3B, T_HBM is only part of the step (the rest is T_fixed + attention + KV). By Amdahl’s law, if the weights are 60 % of the step and you speed them up 4×, the total step improves only 1/(0.4 + 0.6/4) = 1.8×, not 4×. And proportionally there are fewer weights to move relative to activations, KV cache and fixed overhead. Aggressive quantisation in SLMs does help, yes, but not through raw latency at batch 1, where it gives diminishing returns, rather through capacity and concurrency (we will see that at the end).

(e) The KV cache can dominate relative memory. With 6 GB of weights (3B BF16), a single long-context session can approach that order of magnitude in KV cache. In a 70B (140 GB of weights) the KV is proportionally small until high concurrency. In SLMs the VRAM balance tips towards KV much earlier (the detail is in KV cache), and that changes which memory optimisation is the lever.

The maths that matter: a 3B on a 4090

Let us do the whole calculation, which is where the flip shows up without rhetoric.

Model: 3B parameters, BF16 → 2 bytes/param → ~6 GB of weights. Hardware: RTX 4090, BW ≈ 1 TB/s.

Memory-bound ceiling for decode (batch 1). Each token requires loading the 6 GB once:

$$T_{\text{HBM}} = \frac{6 \times 10^{9} \ \text{bytes}}{1 \times 10^{12} \ \text{bytes/s}} = 6 \times 10^{-3}\ \text{s} = 6\ \text{ms/token}$$ $$\text{Ceiling} = \frac{1}{6\ \text{ms}} \approx 166\ \text{tok/s}$$

That is the theoretical memory-bound ceiling: 166 tok/s, assuming that moving the weights is the only cost. The classic roofline would stop here and say “166 tok/s, go get more BW or quantise”.

Now the fixed overhead. Let us put T_fixed ≈ 2 ms/step (a reasonable value for a Python scheduler + ~40 kernels launched + sampling, without CUDA graphs). The real step:

$$T_{\text{step}} = T_{\text{HBM}} + T_{\text{fixed}} = 6 + 2 = 8\ \text{ms} \;\Rightarrow\; \frac{1}{8\ \text{ms}} = 125\ \text{tok/s}$$

The overhead has eaten 41 tok/s out of the theoretical 166: T_fixed is 25 % of the step (2 of 8 ms). Compare with the 70B: T_HBM ≈ 42 ms, T_step ≈ 44 ms, T_fixed is 4.5 %. Same absolute overhead, relative impact 5-6× larger in the SLM.

What happens if you apply CUDA graphs and take away, say, 1.5 of the 2 ms of T_fixed:

$$T_{\text{step}}^{\text{graphs}} = 6 + 0.5 = 6.5\ \text{ms} \;\Rightarrow\; 154\ \text{tok/s}$$

From 125 to 154 tok/s: +23 % from orchestration alone, without touching the model or the memory hardware. In the 70B the same intervention would have gone from 44 to 42.5 ms, +3.5 %. There it is, in two numbers: “a different game”.

What happens if you quantise the weights to INT4 (1.5 GB instead of 6 GB), with T_fixed still at 2 ms:

$$T_{\text{HBM}}^{\text{INT4}} = \frac{1.5 \times 10^{9}}{1 \times 10^{12}} = 1.5\ \text{ms};\quad T_{\text{step}} = 1.5 + 2 = 3.5\ \text{ms} \;\Rightarrow\; 285\ \text{tok/s}$$

The 4× weight quantisation did not give 4× in latency: it went from 125 to 285 tok/s, a 2.3×, because the 2 ms of T_fixed now dominates (it is 57 % of the step). In the 70B, quantising to INT4 gives almost the full 4× because T_fixed is still noise. The same quantisation delivers twice the speedup on the large model as on the small one, at batch 1. And if you also apply CUDA graphs on top of INT4 (T_fixed → 0.5 ms): 1.5 + 0.5 = 2 ms → 500 tok/s. The order of the optimisations matters: in SLMs, attacking T_fixed first unlocks the rest.

Configuration (3B, 4090, batch 1)	T_HBM	T_fixed	T_step	tok/s	vs. baseline
BF16, no graphs (baseline)	6.0 ms	2.0 ms	8.0 ms	125	1.00×
BF16 + CUDA graphs	6.0 ms	0.5 ms	6.5 ms	154	1.23×
INT4, no graphs	1.5 ms	2.0 ms	3.5 ms	285	2.28×
INT4 + CUDA graphs	1.5 ms	0.5 ms	2.0 ms	500	4.00×

(Illustrative figures with T_fixed rounded; the point is the pattern, not the decimal. The real T_fixed depends on the stack, the PyTorch/CUDA version and whether there is tensor parallelism. Measure it in your own setup before believing any row.)

Implications per optimisation

With the model in hand, the blog’s levers reorder themselves once the regime changes.

Batching: far more headroom in SLMs. Remember that you cross the ridge at B ≈ ridge/2 ≈ 150 as an order of magnitude. In a large model, VRAM runs out long before you saturate compute (weights + KV do not let you reach batch 150). In an SLM the weights take up little space, so you can fit large batches in VRAM and stay memory-bound over a much wider range: the T_HBM of the weights is amortised across the B requests (you load them once, they serve B), so aggregate throughput per GPU rises almost linearly with B until very high values. This is exactly the opposite of the 70B’s fear of saturating compute. In SLMs, batching is the throughput lever par excellence because you saturate compute late; the batch grid search in vLLM has a much wider plateau of good behaviour. Careful: batching improves throughput, not per-request latency; for single-stream latency the prize is in T_fixed.

Speculative decoding: a different crossover point. Speculative wins when verifying γ tokens is “almost free” because you are memory-bound. In an SLM the target is already cheap, so the draft has to be tiny for c = T_draft/T_target to stay small, and the draft’s own T_fixed (launching its kernels) bites harder. The crossover to compute-bound with batching also arrives earlier in absolute terms of tok/s served. The variant that fits best here avoids a separate draft: self-speculative / early-exit reuses early layers of the model itself and saves the T_fixed of orchestrating two models.

Quantisation: it helps for capacity, not for latency at batch 1. As the table showed, INT4 in an SLM at batch 1 gives diminishing returns in latency. Its real prize in SLMs is capacity: weights 4× smaller free VRAM for more KV cache → more concurrency, and it is at high concurrency (aggregate throughput) that saving bytes pays off again. Aggressive sub-4-bit and ternary quantisation takes this to the extreme: in SLMs it makes sense above all for fitting more sessions per GPU, not for lowering the latency of a single one. And it is worth remembering (see quantisation) that at batch 1 the dequantize adds compute work which, in a regime already grazed by T_fixed, is not always free.

Architecture: fine-grained MoE changes which bytes you move. A fine-grained device-native MoE activates few parameters per token, so T_HBM drops relative to a dense model of the same total size, but the T_fixed fraction rises even further, and the router adds its own fixed overhead. It is the SLM regime taken to its limit: almost the whole game is played in orchestration.

Scheduler and CUDA graphs first. The operational conclusion, inverted with respect to the large-model posts: in SLMs, before touching the model, kill T_fixed. CUDA graphs (see SMs, streams and graphs), a vLLM scheduler with its Python part minimised or compiled, and kernel persistence are the first-order levers. In a 70B they would be marginal polish; in a 3B they are half the available speedup.

Applied to on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace). This is the scenario where the flip is most visible, because the 4090 has ~1 TB/s (a third of the H100) but T_fixed is the same in absolute terms. A 3B BF16 without CUDA graphs leaves ~125 tok/s on the table when the memory-bound ceiling is 166; enabling graphs and cleaning up the scheduler recovers most of that. The 4090 has plenty of VRAM room for an SLM, so the bottleneck is almost never total memory but orchestration and, at high concurrency, the KV cache. Rule of thumb: on a 4090 with an SLM, profile the per-step overhead first (Nsight Systems on the gap between kernels) before you quantise.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). The H100 has 3.35 TB/s, so the T_HBM of an SLM is even smaller (a 3B FP8 is ~3 GB → ~0.9 ms) and T_fixed dominates even earlier: a badly orchestrated SLM on an H100 can spend more time in the Python scheduler than moving weights. Serving a single single-stream SLM on an H100 is close to a waste; the right mode is aggressive batching (you saturate compute late, so you push large batches through and per-GPU throughput takes off) or multiplexing many SLMs/sessions per GPU via MPS/MIG. This connects with capacity planning: for SLMs the capacity calculation is governed by concurrency and KV cache, not by the weights. And with the dilemma of one big versus N small: replicating SLMs makes sense precisely because each replica saturates compute late and TP brings nothing (the model already fits; TP would only add communication T_fixed).

What we have not covered

The exact measured T_fixed, kernel by kernel, with Nsight Systems: how much is launch, how much scheduler, how much sampling. That is the content of the next post in the series.
torch.compile / partial captures: alternatives and complements to CUDA graphs when there is dynamic control flow.
The prefill regime in SLMs: prefill is compute-bound even in small models (it processes many tokens at once, high AI), so its roofline is the opposite of decode’s; see prefill.
Attention and KV as the second term of T_HBM: here we have folded them in implicitly; the fine-grained breakdown of attention (which scales with sequence length, not with the weights) deserves its own treatment.

References

Williams, S., Waterman, A., Patterson, D. Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM, 52(4), 2009. https://doi.org/10.1145/1498765.1498785
Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference. arXiv:2503.08311, 2025. https://arxiv.org/abs/2503.08311
Databricks. LLM Inference Performance Engineering: Best Practices. https://www.databricks.com/blog/llm-inference-performance-engineering-best-practices
NVIDIA. NVIDIA H100 Tensor Core GPU Datasheet. https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor-core-gpu-datasheet
NVIDIA. GeForce RTX 4090 — product specifications (Ada Lovelace tensor core figures; treat as approximate, they mix dense/sparse).
Yuan, Z. et al. LLM Inference Unveiled: Survey and Roofline Model Insights. arXiv:2402.16363, 2024 — application of the roofline specifically to LLM inference.

The chef who calls out every order: SMs, CUDA streams and CUDA graphs, or why the GPU gets bored generating tokens

Sun, 07 Jun 2026 09:00:00 +0200

This closes the “outside the API” pair. The previous post moved the weights from disk into HBM; here we look at what happens once they are inside, in the silicon that executes them. It is the floor below the kernel launch that the NUMA post mentioned without opening: who launches those kernels, how, and why in decode the GPU spends more time waiting for orders than computing.

TL;DR

An H100 has ~132 streaming multiprocessors (SMs), the “hobs” that execute the compute, and occupancy measures how many warps (groups of 32 threads) it has active to hide latency. But the decode bottleneck is rarely the power of those SMs. Each decode step launches hundreds of tiny kernels (several projections per layer × ~80 layers), and each kernel launch costs 5-10 µs of serial CPU. Since decode kernels are small (small batch, a single token), the GPU finishes them before the CPU calls out the next one: bubbles appear and the GPU sits bored waiting for orders. That regime is called launch-bound, and it is the deep reason, not power, not memory, why --enforce-eager delivers 54 tok/s where with optimisations you reach 89-140. The solution is CUDA graphs: record the entire kernel sequence once and replay it as a single submission, removing the per-launch overhead (~28% of the latency per iteration). vLLM captures ~102 graphs at startup and pads the real batch to the nearest bucket so it can replay a graph with fixed shapes. This post explains SM, occupancy, streams, launch overhead with the maths, CUDA graphs, the 10 knobs, and the trap that this capture is the second half of the cold start from the previous post. With scepticism about what actually moves the needle. On the generic 4×H100 SXM cluster.

Where you are: the silicon, below the kernel launch

The analogy: the chef who calls out every order

Last scene in the restaurant of the series. The kitchen is set up, the pantry is stocked (the previous post). Now the food has to be plated. The hobs are the SMs: 132 stations cooking in parallel. The head chef is the CPU: he calls out the orders, each kernel launch is a shout of “one matrix multiplication, coming up!”. The cooks (the SMs) execute what the chef calls out.

In prefill, processing the whole prompt, each order is a huge dish: a giant matmul over hundreds of tokens at once. The chef calls out an order and the hobs take a good while to produce it. The chef has plenty of time to call out the next one. The hobs are flat out: compute-bound.

In decode, generating one token at a time, each order is minuscule: a matmul over a single token. The hob finishes it in an instant… and sits staring at the chef waiting for the next one. But the chef can only call out one order every 5-10 µs, and there are hundreds of orders per token. The hobs, lightning fast, get bored between shouts. The restaurant is not slow because the cooks are bad: it is slow because the chef does not call out fast enough. That is the launch-bound regime.

The solution is not more hobs or faster cooks. It is to stop calling out order by order. If the chef prints the whole night’s sequence on a single sheet and hands it to the line (“do this, in this order, without waiting for me”), the hobs run without pauses. That is a CUDA graph: record the kernel sequence once and replay it in one go, without the CPU calling out each one. And --enforce-eager is exactly the opposite: forcing the chef to call out order by order, all night long.

The mechanism: SM, warps and occupancy

An H100 SXM has ~132 SMs. Each SM executes threads in groups of 32 called warps, and can have several warps “in flight” at once. Occupancy is the fraction of active warps against the maximum the SM supports. What is the point of having many active warps? To hide latency: while one warp waits for data from HBM (hundreds of cycles), the SM executes another ready warp. With few warps, the SM runs out of anyone to give a turn to and stalls.

But, and this is key, occupancy is a necessary, not sufficient condition, and it only matters if the SM has work to do. In decode, the typical problem is not low occupancy inside a kernel: it is that between kernels the SM has nothing, because the CPU has not launched the next one yet. Raising the occupancy of a kernel that lasts 8 µs does not help if the GPU spends 6 µs waiting for it to be launched.

The mechanism: streams, the order queue

A CUDA stream is a queue of operations that the GPU executes in order. Operations in the same stream are sequential; operations in different streams can overlap. That is what allows, for instance, copying data H2D in one stream while another stream computes, the compute/copy overlap. vLLM uses streams to overlap work, but a stream on its own does not remove the cost of launching each kernel: it only decides ordering and parallelism. The launch cost is still there, order by order, until the graphs come in.

The maths that matters: when the GPU is left waiting

The number that governs everything: a kernel launch costs 5-10 µs of CPU, serially. Take a Llama-70B with ~80 layers. Each layer, without fusion, launches on the order of ~10 kernels (Q/K/V projections, attention, output projection, the two or three MLP matmuls, the normalisations, RoPE and so on). That gives:

$$ N_{\text{kernels}} \approx 80 \text{ layers} \times 10 \approx 800 \text{ launches per token} $$

At 5 µs per launch, serially:

$$ T_{\text{launch}} \approx 800 \times 5\,\mu s = 4{.}0 \text{ ms per token} $$

Those 4 ms are CPU calling out orders only, not counting how long the SMs take to cook. If the GPU could compute instantaneously, the launch ceiling would be ~250 tok/s, and with synchronisation points between kernels, worse. Now compare with the memory ceiling of decode: each token reads the 140 GB of weights once from HBM:

$$ T_{\text{mem}} = \frac{140 \text{ GB}}{3{.}35 \text{ TB/s}} \approx 42 \text{ ms} \;\Rightarrow\; \approx 24 \text{ tok/s (one sequence, no batch)} $$

Here is the subtlety almost nobody has in mind. For a single sequence, decode is memory-bound at ~24 tok/s, and the 4 ms of launch fit inside the 42 ms of reading: the launch hides. But batching changes everything. When serving a batch of B sequences, the weights are read once and serve all B, so the memory cost per token amortises and falls. The GPU stops being memory-bound… and what was underneath emerges: the launch cost, which does not amortise with batch because you have to launch the same kernel sequence regardless. The result: the better you batch, the more launch-bound you become, and the more CUDA graphs pay off. That is why the raw measurement confirms it, --enforce-eager gives 54 tok/s where graphs give 89, and up to 8× in configurations where decode is very small and launch dominates completely.

vLLM’s CUDA graphs, specifically

vLLM does not capture a single graph: it captures ~102 at startup, on the order of 51 piecewise (for the mixed prefill+decode steps) and 51 full (for pure decode). Each one is recorded for a fixed batch size (a bucket: 1, 2, 4, 8… up to a maximum). In service the real batch almost never falls exactly on a bucket, so vLLM pads with zeros up to the next bucket up, replays that graph, and trims the output to the real size. That is the price of graphs: they need static shapes, and padding is what makes them static.

This has two consequences that show up in the knobs and the traps:

Capture costs time and memory. Recording 102 graphs at startup adds seconds to the cold start, the second half of the startup that the previous post left pending, and consumes HBM (each graph retains its buffers). The FULL_AND_PIECEWISE mode (the default) is the fastest in service but the one that asks for the most memory and the longest capture; FULL_DECODE_ONLY saves both in exchange for not accelerating the mixed steps.

Padding wastes some compute. Padding a batch of 33 up to the bucket of 64 computes 31 phantom sequences. It is a small cost against what removing the launch overhead saves, but it exists, and it grows if the buckets are badly chosen.

The 10 knobs worth touching

Knob 1 — Measure whether decode is launch-bound

Before touching anything: is the GPU computing or waiting? With nsys (Nsight Systems) you can see the gaps between kernels on the timeline; if there are gaps in decode, it is launch-bound and graphs will help. If the GPU is at 100% with no gaps, the bottleneck is something else (memory or compute) and graphs will not work miracles. nvidia-smi dmon showing low utilisation in decode but poor TPS is the cheap signal.

Knob 2 — Do not use `--enforce-eager` in production

--enforce-eager disables CUDA graphs. It is a debugging tool (to isolate which kernel fails), not a production one. Leaving it on “because it used to start up” throws away 26-50% of decode throughput. If it is in your production command, take it out and measure.

Knob 3 — Capture buckets (`cudagraph_capture_sizes`)

Which batch sizes to capture. Buckets too far apart make padding expensive; too many of them make capture slow and eat HBM. Tuning them to the real distribution of batch sizes you see in production is the fine adjustment, but only after measuring that distribution.

Knob 4 — CUDA graph mode

FULL_AND_PIECEWISE (default, fastest, more memory/capture), FULL_DECODE_ONLY (saves memory and capture, ideal for pure decode pods in disaggregated serving), PIECEWISE, or NONE (= eager). The right mode depends on whether the pod does pure or mixed decode.

Knob 5 — `torch.compile`

vLLM leans on torch.compile to fuse and optimise kernels before capturing them in graphs. Fewer kernels (fusion) = fewer launches = less dependence on the graph and better decode even in eager. The compilation level is a knob, with its startup time cost.

Knob 6 — Batch size: fill the hobs

Memory-bound decode amortises with batching (as we saw in continuous batching): read the weights once for B sequences. More batch = more SM occupancy and more memory amortisation. The limit is set by the HBM available for the KV cache. It is the knob that moves aggregate throughput most.

Knob 7 — Do not break stream overlap

vLLM overlaps compute and copy with streams. Patching the code to “simplify” can serialise what was overlapped. If you do not know why there are several streams, do not collapse them.

Knob 8 — Persistence mode + locked clocks

nvidia-smi -pm 1 keeps the driver resident (avoiding reinitialisations that add launch latency). Locking clocks to the boost frequency stops the GPU dropping P-state between tiny decode kernels and paying the ramp-up latency. It is the same anti-jitter spirit as the NUMA post, applied to the GPU.

Knob 9 — Fused kernels (FlashAttention, FP8 kernels)

Fewer kernels = fewer orders to call out. FlashAttention fuses attention into one kernel instead of several; fused FP8 kernels reduce the count. Fusion attacks the problem at the root: it does not speed up the launch, it removes launches.

Knob 10 — Accept the capture cost in the cold start

Graph capture adds seconds to startup. In a pod that lives for hours, it amortises easily. In one that scales up and down every minute, that cost is paid over and over; there, FULL_DECODE_ONLY (shorter capture) or accepting slightly less throughput can be worth it. It is the same warm-vs-elastic tension as in the cold start.

Summary table

#	Knob	What it attacks	Risk / cost
1	`nsys` / `dmon`	knowing whether it is launch-bound	none; do it first
2	remove `--enforce-eager`	graphs disabled	it was for debugging; reactivate if a bug returns
3	capture buckets	expensive padding / slow capture	requires measuring the real distribution
4	graph mode	memory and capture	less coverage on mixed steps
5	`torch.compile`	unfused kernels	startup time
6	batch size	occupancy + memory	HBM for KV cache
7	streams	broken overlap	do not touch if not understood
8	persistence + clocks	jitter / P-states	electrical consumption
9	fused kernels	number of launches	kernel compatibility
10	capture vs cold start	slower startup	less throughput if trimmed

How this connects with the rest of the stack

With the cold start. CUDA graph capture is the second half of the startup that the previous post opened: loading weights + capturing graphs = the complete cold start.

With continuous batching. Continuous batching is what makes decode launch-bound (it amortises memory and leaves the launch exposed), and that is why graphs and batching reinforce each other.

With the KV cache. The KV cache decides how much batch fits in HBM, and the batch decides SM occupancy and how much launch overhead matters. Everything is coupled through memory.

With the interconnect. In TP, between the compute kernels there are all-reduces (NVLink/NCCL) that also get launched and synchronised. vLLM’s custom all-reduce is integrated into the same graph so the sequence is not broken by a CPU synchronisation.

With NUMA. Who launches the kernels is the CPU from the host post; if that thread suffers jitter or lands on the wrong socket, launch overhead gets worse. Graphs reduce the dependence on that thread, which is another reason they help.

With disaggregated serving. The pure decode pods of disaggregated serving are the ideal case for FULL_DECODE_ONLY: they maximise the graph benefit exactly in the most launch-bound phase.

Traps and things that are not what they look like

“Raising occupancy will fix slow decode.” Not if the problem is launch-bound. Occupancy matters inside a kernel that has work; if the GPU is idle between kernels waiting for the CPU, more occupancy does not touch that bubble. Measure before optimising what is not the bottleneck.

“CUDA graphs always speed things up.” They speed things up when decode is launch-bound. If the GPU is already at 100% (compute-bound in prefill, or memory saturated with a huge batch), graphs add little. Their territory is decode with small kernels.

"--enforce-eager gives more stable results." It gives slower results. The stability it appears to give is that it avoids graph capture bugs on new hardware (for example a newly supported architecture). It is a temporary patch, not a production configuration.

Capturing too many buckets “just in case”. Each bucket adds capture time and HBM. Capturing 30 sizes when in production you only see 4 means paying cold start and memory for graphs that are never replayed. Tune to the real distribution.

Confusing utilisation with efficiency. nvidia-smi at 100% “utilisation” only says there is a kernel running, not that the SM is full of useful work. A low-occupancy kernel keeps “utilisation” high while wasting the SM. nvidia-smi utilisation is a coarse thermometer; to know whether the silicon is performing you need nsys/DCGM and to look at real occupancy and gaps.

Optimising the silicon before memory. If decode is limited by HBM bandwidth (large batch, large model), fighting with graphs and occupancy is polishing what is not the bottleneck. The right order: measure the regime (memory / compute / launch) and attack the one in charge.

Conclusion

Intuition says that a GPU generating slow tokens is “working hard”. Almost never: in decode it is waiting for orders. The 132 SMs cook a tiny token in an instant and then sit staring at the CPU, which can only call out one order every 5-10 µs and has hundreds to call out per token. That bottleneck, neither power nor memory but launch, is invisible on any dashboard that looks at “GPU utilisation”, and it is the real reason why --enforce-eager performs at half speed. CUDA graphs solve it with a simple idea: stop calling out order by order and hand over the whole sheet for the night, so the silicon runs without pauses. And there is an uncomfortable truth that reorders the optimisation priority: the better you batch, the more launch-bound you become, because batching kills the memory bottleneck and exposes the launch one. That is why graphs and batching are not separate optimisations: they are the same lever seen from two sides. The chef who learns not to call out every dish is what finally makes the kitchen run as fast as the hobs always could.

References

vLLM, CUDA Graphs (diseño, modos FULL/PIECEWISE, captura): https://docs.vllm.ai/en/stable/design/cuda_graphs/.
vLLM, Inside vLLM: Anatomy of a High-Throughput LLM Inference System: https://blog.vllm.ai/2025/09/05/anatomy-of-vllm.html.
NVIDIA, Getting Started with CUDA Graphs: https://developer.nvidia.com/blog/cuda-graphs/.
NVIDIA, Achieved Occupancy (ocupación de SM): https://archive.docs.nvidia.com/gameworks/content/developertools/desktop/analysis/report/cudaexperiments/kernellevel/achievedoccupancy.htm.
PyTorch, torch.compile y CUDA Graphs para inferencia LLM: https://docs.vllm.ai/en/stable/design/cuda_graphs/.
Understanding the Overheads of Launching CUDA Kernels (ICPP 2019): https://www.hpcs.cs.tsukuba.ac.jp/icpp2019/data/posters/Poster17-abst.pdf.

Cuda-Graphs on lo0 — Tech Blog

Speeding up model cold start: from minutes to seconds

TL;DR

The analogy: opening the kitchen in the morning

The five line items of cold start

Speeding up item 4: loading the weights

safetensors versus pickle: the format rules

Tensorizer: streaming tensor by tensor straight to the GPU

Run:ai Model Streamer: reading and copying at the same time

The maths of the weights: $t = W/B$ and the saving from overlapping

The storage tier

Speeding up item 5: everything that is not weights

Caching the torch.compile/Inductor compilation

Bounding the CUDA graph capture

Warm pools: not paying for item 5 on the critical path

Sleep mode: the shortcut that skips all five items

An example cold start, item by item

The economics of scale-to-zero

Applied to the generic 4×H100 cluster

Traps and things that are not what they seem

Conclusion

See also

References

The roofline flips: why optimising small models is a different performance game

TL;DR

The analogy: the pantry and the waiter

The bare mechanism: what the roofline says

The point of the title: why it flips in SLMs

The maths that matter: a 3B on a 4090

Implications per optimisation

Applied to on-premise hardware

What we have not covered

See also

References

The chef who calls out every order: SMs, CUDA streams and CUDA graphs, or why the GPU gets bored generating tokens

TL;DR

Where you are: the silicon, below the kernel launch

The analogy: the chef who calls out every order

The mechanism: SM, warps and occupancy

The mechanism: streams, the order queue

The maths that matters: when the GPU is left waiting

vLLM’s CUDA graphs, specifically

The 10 knobs worth touching

Knob 1 — Measure whether decode is launch-bound

Knob 2 — Do not use --enforce-eager in production

Knob 3 — Capture buckets (cudagraph_capture_sizes)

Knob 4 — CUDA graph mode

Knob 5 — torch.compile

Knob 6 — Batch size: fill the hobs

Knob 7 — Do not break stream overlap

Knob 8 — Persistence mode + locked clocks

Knob 9 — Fused kernels (FlashAttention, FP8 kernels)

Knob 10 — Accept the capture cost in the cold start

Summary table

How this connects with the rest of the stack

Traps and things that are not what they look like

Conclusion

See also

References

Knob 2 — Do not use `--enforce-eager` in production

Knob 3 — Capture buckets (`cudagraph_capture_sizes`)

Knob 5 — `torch.compile`