The chef who calls out every order: SMs, CUDA streams and CUDA graphs, or why the GPU gets bored generating tokens

· By José Manuel Román
Contents

This closes the “outside the API” pair. The previous post moved the weights from disk into HBM; here we look at what happens once they are inside, in the silicon that executes them. It is the floor below the kernel launch that the NUMA post mentioned without opening: who launches those kernels, how, and why in decode the GPU spends more time waiting for orders than computing.

TL;DR

An H100 has ~132 streaming multiprocessors (SMs), the “hobs” that execute the compute, and occupancy measures how many warps (groups of 32 threads) it has active to hide latency. But the decode bottleneck is rarely the power of those SMs. Each decode step launches hundreds of tiny kernels (several projections per layer × ~80 layers), and each kernel launch costs 5-10 µs of serial CPU. Since decode kernels are small (small batch, a single token), the GPU finishes them before the CPU calls out the next one: bubbles appear and the GPU sits bored waiting for orders. That regime is called launch-bound, and it is the deep reason, not power, not memory, why --enforce-eager delivers 54 tok/s where with optimisations you reach 89-140. The solution is CUDA graphs: record the entire kernel sequence once and replay it as a single submission, removing the per-launch overhead (~28% of the latency per iteration). vLLM captures ~102 graphs at startup and pads the real batch to the nearest bucket so it can replay a graph with fixed shapes. This post explains SM, occupancy, streams, launch overhead with the maths, CUDA graphs, the 10 knobs, and the trap that this capture is the second half of the cold start from the previous post. With scepticism about what actually moves the needle. On the generic 4×H100 SXM cluster.

Where you are: the silicon, below the kernel launch

The vertical stack · you are in the siliconEngine · vLLM (batching, sampling, scheduler)Host · CPU launches kernels (NUMA post)YOU ARE HERE · CUDA: streams, kernels, CUDA graphsthe order queue that reaches the siliconSMs · 132 hobs execute the warpsHBM · 3.35 TB/s — the weights the SMs readthe post's question: are the SMs computing, or waiting for orders?

The analogy: the chef who calls out every order

Last scene in the restaurant of the series. The kitchen is set up, the pantry is stocked (the previous post). Now the food has to be plated. The hobs are the SMs: 132 stations cooking in parallel. The head chef is the CPU: he calls out the orders, each kernel launch is a shout of “one matrix multiplication, coming up!”. The cooks (the SMs) execute what the chef calls out.

In prefill, processing the whole prompt, each order is a huge dish: a giant matmul over hundreds of tokens at once. The chef calls out an order and the hobs take a good while to produce it. The chef has plenty of time to call out the next one. The hobs are flat out: compute-bound.

In decode, generating one token at a time, each order is minuscule: a matmul over a single token. The hob finishes it in an instant… and sits staring at the chef waiting for the next one. But the chef can only call out one order every 5-10 µs, and there are hundreds of orders per token. The hobs, lightning fast, get bored between shouts. The restaurant is not slow because the cooks are bad: it is slow because the chef does not call out fast enough. That is the launch-bound regime.

The solution is not more hobs or faster cooks. It is to stop calling out order by order. If the chef prints the whole night’s sequence on a single sheet and hands it to the line (“do this, in this order, without waiting for me”), the hobs run without pauses. That is a CUDA graph: record the kernel sequence once and replay it in one go, without the CPU calling out each one. And --enforce-eager is exactly the opposite: forcing the chef to call out order by order, all night long.

The mechanism: SM, warps and occupancy

An H100 SXM has ~132 SMs. Each SM executes threads in groups of 32 called warps, and can have several warps “in flight” at once. Occupancy is the fraction of active warps against the maximum the SM supports. What is the point of having many active warps? To hide latency: while one warp waits for data from HBM (hundreds of cycles), the SM executes another ready warp. With few warps, the SM runs out of anyone to give a turn to and stalls.

But, and this is key, occupancy is a necessary, not sufficient condition, and it only matters if the SM has work to do. In decode, the typical problem is not low occupancy inside a kernel: it is that between kernels the SM has nothing, because the CPU has not launched the next one yet. Raising the occupancy of a kernel that lasts 8 µs does not help if the GPU spends 6 µs waiting for it to be launched.

The mechanism: streams, the order queue

A CUDA stream is a queue of operations that the GPU executes in order. Operations in the same stream are sequential; operations in different streams can overlap. That is what allows, for instance, copying data H2D in one stream while another stream computes, the compute/copy overlap. vLLM uses streams to overlap work, but a stream on its own does not remove the cost of launching each kernel: it only decides ordering and parallelism. The launch cost is still there, order by order, until the graphs come in.

The maths that matters: when the GPU is left waiting

The number that governs everything: a kernel launch costs 5-10 µs of CPU, serially. Take a Llama-70B with ~80 layers. Each layer, without fusion, launches on the order of ~10 kernels (Q/K/V projections, attention, output projection, the two or three MLP matmuls, the normalisations, RoPE and so on). That gives:

$$ N_{\text{kernels}} \approx 80 \text{ layers} \times 10 \approx 800 \text{ launches per token} $$

At 5 µs per launch, serially:

$$ T_{\text{launch}} \approx 800 \times 5\,\mu s = 4{.}0 \text{ ms per token} $$

Those 4 ms are CPU calling out orders only, not counting how long the SMs take to cook. If the GPU could compute instantaneously, the launch ceiling would be ~250 tok/s, and with synchronisation points between kernels, worse. Now compare with the memory ceiling of decode: each token reads the 140 GB of weights once from HBM:

$$ T_{\text{mem}} = \frac{140 \text{ GB}}{3{.}35 \text{ TB/s}} \approx 42 \text{ ms} \;\Rightarrow\; \approx 24 \text{ tok/s (one sequence, no batch)} $$

Here is the subtlety almost nobody has in mind. For a single sequence, decode is memory-bound at ~24 tok/s, and the 4 ms of launch fit inside the 42 ms of reading: the launch hides. But batching changes everything. When serving a batch of B sequences, the weights are read once and serve all B, so the memory cost per token amortises and falls. The GPU stops being memory-bound… and what was underneath emerges: the launch cost, which does not amortise with batch because you have to launch the same kernel sequence regardless. The result: the better you batch, the more launch-bound you become, and the more CUDA graphs pay off. That is why the raw measurement confirms it, --enforce-eager gives 54 tok/s where graphs give 89, and up to 8× in configurations where decode is very small and launch dominates completely.

Eager vs CUDA graph on the GPU timelineEager (order by order)kernel↑ gaps = GPU waiting for the CPU to launchCUDA graph (whole sheet)no gaps: a single submission, replaytime → same kernels, same GPU; the graph removes the launch bubbles

vLLM’s CUDA graphs, specifically

vLLM does not capture a single graph: it captures ~102 at startup, on the order of 51 piecewise (for the mixed prefill+decode steps) and 51 full (for pure decode). Each one is recorded for a fixed batch size (a bucket: 1, 2, 4, 8… up to a maximum). In service the real batch almost never falls exactly on a bucket, so vLLM pads with zeros up to the next bucket up, replays that graph, and trims the output to the real size. That is the price of graphs: they need static shapes, and padding is what makes them static.

This has two consequences that show up in the knobs and the traps:

Capture costs time and memory. Recording 102 graphs at startup adds seconds to the cold start, the second half of the startup that the previous post left pending, and consumes HBM (each graph retains its buffers). The FULL_AND_PIECEWISE mode (the default) is the fastest in service but the one that asks for the most memory and the longest capture; FULL_DECODE_ONLY saves both in exchange for not accelerating the mixed steps.

Padding wastes some compute. Padding a batch of 33 up to the bucket of 64 computes 31 phantom sequences. It is a small cost against what removing the launch overhead saves, but it exists, and it grows if the buckets are badly chosen.

The 10 knobs worth touching

Knob 1 — Measure whether decode is launch-bound

Before touching anything: is the GPU computing or waiting? With nsys (Nsight Systems) you can see the gaps between kernels on the timeline; if there are gaps in decode, it is launch-bound and graphs will help. If the GPU is at 100% with no gaps, the bottleneck is something else (memory or compute) and graphs will not work miracles. nvidia-smi dmon showing low utilisation in decode but poor TPS is the cheap signal.

Knob 2 — Do not use --enforce-eager in production

--enforce-eager disables CUDA graphs. It is a debugging tool (to isolate which kernel fails), not a production one. Leaving it on “because it used to start up” throws away 26-50% of decode throughput. If it is in your production command, take it out and measure.

Knob 3 — Capture buckets (cudagraph_capture_sizes)

Which batch sizes to capture. Buckets too far apart make padding expensive; too many of them make capture slow and eat HBM. Tuning them to the real distribution of batch sizes you see in production is the fine adjustment, but only after measuring that distribution.

Knob 4 — CUDA graph mode

FULL_AND_PIECEWISE (default, fastest, more memory/capture), FULL_DECODE_ONLY (saves memory and capture, ideal for pure decode pods in disaggregated serving), PIECEWISE, or NONE (= eager). The right mode depends on whether the pod does pure or mixed decode.

Knob 5 — torch.compile

vLLM leans on torch.compile to fuse and optimise kernels before capturing them in graphs. Fewer kernels (fusion) = fewer launches = less dependence on the graph and better decode even in eager. The compilation level is a knob, with its startup time cost.

Knob 6 — Batch size: fill the hobs

Memory-bound decode amortises with batching (as we saw in continuous batching): read the weights once for B sequences. More batch = more SM occupancy and more memory amortisation. The limit is set by the HBM available for the KV cache. It is the knob that moves aggregate throughput most.

Knob 7 — Do not break stream overlap

vLLM overlaps compute and copy with streams. Patching the code to “simplify” can serialise what was overlapped. If you do not know why there are several streams, do not collapse them.

Knob 8 — Persistence mode + locked clocks

nvidia-smi -pm 1 keeps the driver resident (avoiding reinitialisations that add launch latency). Locking clocks to the boost frequency stops the GPU dropping P-state between tiny decode kernels and paying the ramp-up latency. It is the same anti-jitter spirit as the NUMA post, applied to the GPU.

Knob 9 — Fused kernels (FlashAttention, FP8 kernels)

Fewer kernels = fewer orders to call out. FlashAttention fuses attention into one kernel instead of several; fused FP8 kernels reduce the count. Fusion attacks the problem at the root: it does not speed up the launch, it removes launches.

Knob 10 — Accept the capture cost in the cold start

Graph capture adds seconds to startup. In a pod that lives for hours, it amortises easily. In one that scales up and down every minute, that cost is paid over and over; there, FULL_DECODE_ONLY (shorter capture) or accepting slightly less throughput can be worth it. It is the same warm-vs-elastic tension as in the cold start.

Summary table

#KnobWhat it attacksRisk / cost
1nsys / dmonknowing whether it is launch-boundnone; do it first
2remove --enforce-eagergraphs disabledit was for debugging; reactivate if a bug returns
3capture bucketsexpensive padding / slow capturerequires measuring the real distribution
4graph modememory and captureless coverage on mixed steps
5torch.compileunfused kernelsstartup time
6batch sizeoccupancy + memoryHBM for KV cache
7streamsbroken overlapdo not touch if not understood
8persistence + clocksjitter / P-stateselectrical consumption
9fused kernelsnumber of launcheskernel compatibility
10capture vs cold startslower startupless throughput if trimmed

How this connects with the rest of the stack

With the cold start. CUDA graph capture is the second half of the startup that the previous post opened: loading weights + capturing graphs = the complete cold start.

With continuous batching. Continuous batching is what makes decode launch-bound (it amortises memory and leaves the launch exposed), and that is why graphs and batching reinforce each other.

With the KV cache. The KV cache decides how much batch fits in HBM, and the batch decides SM occupancy and how much launch overhead matters. Everything is coupled through memory.

With the interconnect. In TP, between the compute kernels there are all-reduces (NVLink/NCCL) that also get launched and synchronised. vLLM’s custom all-reduce is integrated into the same graph so the sequence is not broken by a CPU synchronisation.

With NUMA. Who launches the kernels is the CPU from the host post; if that thread suffers jitter or lands on the wrong socket, launch overhead gets worse. Graphs reduce the dependence on that thread, which is another reason they help.

With disaggregated serving. The pure decode pods of disaggregated serving are the ideal case for FULL_DECODE_ONLY: they maximise the graph benefit exactly in the most launch-bound phase.

Traps and things that are not what they look like

“Raising occupancy will fix slow decode.” Not if the problem is launch-bound. Occupancy matters inside a kernel that has work; if the GPU is idle between kernels waiting for the CPU, more occupancy does not touch that bubble. Measure before optimising what is not the bottleneck.

“CUDA graphs always speed things up.” They speed things up when decode is launch-bound. If the GPU is already at 100% (compute-bound in prefill, or memory saturated with a huge batch), graphs add little. Their territory is decode with small kernels.

"--enforce-eager gives more stable results." It gives slower results. The stability it appears to give is that it avoids graph capture bugs on new hardware (for example a newly supported architecture). It is a temporary patch, not a production configuration.

Capturing too many buckets “just in case”. Each bucket adds capture time and HBM. Capturing 30 sizes when in production you only see 4 means paying cold start and memory for graphs that are never replayed. Tune to the real distribution.

Confusing utilisation with efficiency. nvidia-smi at 100% “utilisation” only says there is a kernel running, not that the SM is full of useful work. A low-occupancy kernel keeps “utilisation” high while wasting the SM. nvidia-smi utilisation is a coarse thermometer; to know whether the silicon is performing you need nsys/DCGM and to look at real occupancy and gaps.

Optimising the silicon before memory. If decode is limited by HBM bandwidth (large batch, large model), fighting with graphs and occupancy is polishing what is not the bottleneck. The right order: measure the regime (memory / compute / launch) and attack the one in charge.

Conclusion

Intuition says that a GPU generating slow tokens is “working hard”. Almost never: in decode it is waiting for orders. The 132 SMs cook a tiny token in an instant and then sit staring at the CPU, which can only call out one order every 5-10 µs and has hundreds to call out per token. That bottleneck, neither power nor memory but launch, is invisible on any dashboard that looks at “GPU utilisation”, and it is the real reason why --enforce-eager performs at half speed. CUDA graphs solve it with a simple idea: stop calling out order by order and hand over the whole sheet for the night, so the silicon runs without pauses. And there is an uncomfortable truth that reorders the optimisation priority: the better you batch, the more launch-bound you become, because batching kills the memory bottleneck and exposes the launch one. That is why graphs and batching are not separate optimisations: they are the same lever seen from two sides. The chef who learns not to call out every dish is what finally makes the kitchen run as fast as the hobs always could.

See also

References