The roofline flips: why optimising small models is a different performance game

Contents

This post is the anchor of a mini-series on inference performance in small models (SLMs). Almost every optimisation post on the blog, KV cache, decode, quantisation, was written with a 70B in mind. Here I argue that when the model shrinks by an order of magnitude, the roofline changes regime and several of those intuitions flip. This is not a nuance: it is a different game.

TL;DR

Autoregressive decode in a large LLM is memory-bandwidth-bound: at every step you have to move all the model weights from HBM to the registers of the SMs, and that dominates over the arithmetic operations. The GPU spends its time waiting for bytes, not computing. That single sentence, that decode “waits on HBM”, is the root of half the optimisations on this blog. In a small model (SLM, say 0.5B–7B) the sentence stops being true in the simple form we used to tell it. At batch 1 you are still memory-bound with respect to the hardware, yes, but the forward pass is so cheap (moving 6 GB at 1 TB/s takes ~6 ms, not 70 ms) that the fixed per-step costs, kernel launches, Python scheduler overhead, the sampler, host↔device copies, the synchronize calls, stop being noise and start eating 20-30 % of the time. The bottleneck moves from HBM to orchestration. The concrete, quantitative consequences: (1) CUDA graphs and cutting scheduler overhead pay off more in SLMs than in large models; (2) weight quantisation gives less latency improvement at batch 1 in SLMs, because proportionally there are fewer weights to move relative to activations, KV cache and fixed overhead; (3) batching has more headroom because you cross the ridge point late; (4) the KV cache can dominate relative memory. All of this comes out of a single model, the roofline, applied with numerical honesty.

The analogy: the pantry and the waiter

A kitchen with two very different services.

Tasting menu service, one enormous slow dish (the 70B LLM). Every dish involves heavy ingredients that the assistant has to fetch from the pantry at the back, several times, carrying boxes. The chef, by contrast, assembles the dish in a moment: the slow part is fetching the ingredients, not cooking them. If you want the service to go faster, you do not hire a more skilful chef: you widen the corridor to the pantry, or make each trip bring more boxes. The pantry is HBM; the trip is memory bandwidth; cooking is compute. The big dish is bound by the pantry.

Tapas service, tiny plates (the SLM). Now each tapa has two ingredients and is assembled in a second. The trip to the pantry per tapa is very short. But a cost appears that was negligible for the big dish: the waiter. For every tapa, the waiter has to walk to the kitchen, pick up the plate, carry it to the bar, walk back, take the order, call it out. That back and forth is fixed: it costs the same for one tapa as for the enormous dish. When the tapa is assembled in a second, the waiter, not the pantry, is the bottleneck. Shortening the corridor to the pantry (widening HBM, quantising the weights) barely improves the service any more; what improves it is the waiter chaining several orders together without going back to the kitchen each time (CUDA graphs) or serving several tables in one pass (batching).

The roofline is the tool that says, with numbers, at what point the waiter dominates over the pantry. That boundary is the ridge point, and the joke in the title is that in SLMs we cross the regime much earlier than large-model intuition led us to believe.

The bare mechanism: what the roofline says

The roofline model (Williams, Waterman and Patterson, 2009) starts from a single quantity: arithmetic intensity, which is how many operations you do per byte moved from memory.

$$\text{AI} = \frac{\text{FLOPs}}{\text{bytes moved from memory}} \quad [\text{FLOP/byte}]$$

The hardware has two ceilings: compute (peak FLOPS) and memory (peak bandwidth × AI). Achievable performance is the minimum of the two:

$$\text{Perf} = \min\big(\text{peak FLOPS},\; \text{BW} \times \text{AI}\big)$$

Where the two lines cross is the ridge point, the AI above which you stop being limited by memory and start being limited by compute:

$$\text{AI}_{\text{ridge}} = \frac{\text{peak FLOPS}}{\text{peak BW}}$$

If your kernel has an AI below the ridge, you are memory-bound (the GPU waits for bytes). Above it, compute-bound (the GPU computes flat out and memory is in surplus). What matters is that the ridge point is a property of the hardware, not of the model. Let us look at the numbers, approximate, and I flag them as such because marketing figures mix dense and sparse, different dtypes and unrealistic thermal conditions.

Generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). Per GPU, ~989 TFLOPS BF16 dense (~1,979 TFLOPS FP8 dense; the figure with sparsity is double that and almost never applies to LLM inference). HBM3 ~3.35 TB/s. The ridge in BF16:

$$\text{AI}_{\text{ridge}}^{\text{H100,BF16}} \approx \frac{989 \times 10^{12}}{3.35 \times 10^{12}} \approx 295 \ \text{FLOP/byte}$$

In FP8 the ridge rises to ~590 FLOP/byte (twice the FLOPS against the same BW). Careful: these are datasheet peak figures; in practice a real kernel rarely gets past 70-80 % of either ceiling.

RTX 4090 (24 GB, Ada Lovelace). ~330 TFLOPS FP16 with FP16 accumulation via tensor cores (the “660 TOPS” figure that circulates is with sparsity), and ~1 TB/s of GDDR6X. The ridge:

$$\text{AI}_{\text{ridge}}^{\text{4090,FP16}} \approx \frac{330 \times 10^{12}}{1.0 \times 10^{12}} \approx 330 \ \text{FLOP/byte}$$

Curiously, the same order as the H100 in BF16: the 4090 has less BW but also fewer FLOPS, and the ratio comes out similar. The ridge sits around 300 FLOP/byte in both cases. Hold on to that number.

And where does decode fall? In decode at batch 1, each weight is loaded once from HBM and used for a single multiply-accumulate (one token, one row of activation). The AI of the decode GEMM at batch 1 is on the order of AI ≈ 1-2 FLOP/byte (each weight byte takes part in ~2 FLOP). With batch B, the same weight loaded once serves B rows of activation, so the AI scales roughly linearly:

$$\text{AI}_{\text{decode}}(B) \approx 2B \ \text{FLOP/byte} \quad (\text{for the weight GEMM part})$$

You cross the ridge when 2B ≈ 300, that is B ≈ 150 as an order of magnitude (in practice earlier, because of attention and overheads, but that is the frame). The clean conclusion: decode at low batch is always deeply memory-bound, nowhere near the ridge. That is why we say “decode waits on HBM”, and why quantising weights (moving fewer bytes) speeds up decode in a large model almost linearly. So far, this is the standard story from the large-model posts.

The point of the title: why it flips in SLMs

The classic roofline has a blind spot that does not matter in large models and is everything in small ones: it only models the work inside the kernel. It assumes the only time is bytes/BW or FLOPs/FLOPS. But a real decode step is not just the GEMM. It is a sequence of dozens of kernels (QKV projections, attention, the two MLP layers, normalisations, residuals, the logits head, sampling) and, around each one, there is a fixed orchestration cost:

Kernel launches (kernel launch): each cudaLaunchKernel costs on the order of 5-10 µs of CPU/driver overhead, regardless of kernel size. A decode forward with ~30-60 kernels launched sequentially drags ~0.3-0.6 ms just in launching.
Python scheduler overhead: the vLLM scheduler loop prepares metadata, decides which requests enter the step, builds the input tensors. In pure Python this is hundreds of µs to a couple of ms per step, especially at low concurrency where it is not amortised.
Sampling and post-processing: applying temperature, top-p, penalties, the argmax/multinomial, copying the token back. Another block of hundreds of µs.
Synchronisations and host↔device copies: every synchronize or small copy adds latency that is neither FLOPs nor HBM bytes.

Let us call the sum of all of that T_fixed, the per-step cost independent of model size, on the order of 1-3 ms in an unoptimised Python stack. The real time of a step is now:

$$T_{\text{step}} \approx \underbrace{\frac{\text{weight bytes}}{\text{BW}}}_{T_{\text{HBM}} \text{ (memory-bound)}} + \; T_{\text{fixed}}$$

In a 70B BF16, moving ~140 GB at 3.35 TB/s takes ~42 ms of T_HBM. Against that, a T_fixed of 1-3 ms is noise (2-7 %). The classic roofline is right: the model is memory-bound, full stop. But in a 3B BF16, T_HBM drops to a few ms, and suddenly T_fixed is the same order as T_HBM. The bottleneck stops being the pantry and becomes the waiter. This is the flip in the title, and four counterintuitive consequences follow from it:

(a) At batch 1 you are still memory-bound with respect to the hardware. The AI has not changed: it is still ~2 FLOP/byte, below the ridge. Anyone reading only the roofline will conclude “memory-bound, quantise the weights”. That is true but incomplete: the roofline does not see T_fixed.

(b) Fixed costs become a huge fraction of the step. This is the central point. In the 70B, T_fixed / T_step ≈ 5 %. In the 3B it can be 20-30 %. The effective bottleneck of the 3B is half HBM, half orchestration.

(c) That is why CUDA graphs and cutting scheduler overhead pay off MORE in SLMs. A CUDA graph captures the whole kernel sequence of the step and relaunches it with a single cudaGraphLaunch, removing almost all of the per-kernel launch overhead and much of the per-iteration Python scheduler work. In the 70B, shaving 0.5 ms off a 42 ms step is a +1 % you barely notice. In the 3B, shaving those same 0.5 ms off a ~7 ms step is +7 %, and if you take away nearly all of T_fixed you can gain 20-30 %. The same optimisation, a different prize, because the denominator changed.

(d) Weight quantisation gives LESS latency improvement at batch 1 in SLMs. This is the most counterintuitive one. In the 70B, T_HBM is almost the whole step; going from BF16 to INT4 quadruples the effective weight bandwidth and almost quadruples decode speed. In the 3B, T_HBM is only part of the step (the rest is T_fixed + attention + KV). By Amdahl’s law, if the weights are 60 % of the step and you speed them up 4×, the total step improves only 1/(0.4 + 0.6/4) = 1.8×, not 4×. And proportionally there are fewer weights to move relative to activations, KV cache and fixed overhead. Aggressive quantisation in SLMs does help, yes, but not through raw latency at batch 1, where it gives diminishing returns, rather through capacity and concurrency (we will see that at the end).

(e) The KV cache can dominate relative memory. With 6 GB of weights (3B BF16), a single long-context session can approach that order of magnitude in KV cache. In a 70B (140 GB of weights) the KV is proportionally small until high concurrency. In SLMs the VRAM balance tips towards KV much earlier (the detail is in KV cache), and that changes which memory optimisation is the lever.

The maths that matter: a 3B on a 4090

Let us do the whole calculation, which is where the flip shows up without rhetoric.

Model: 3B parameters, BF16 → 2 bytes/param → ~6 GB of weights. Hardware: RTX 4090, BW ≈ 1 TB/s.

Memory-bound ceiling for decode (batch 1). Each token requires loading the 6 GB once:

$$T_{\text{HBM}} = \frac{6 \times 10^{9} \ \text{bytes}}{1 \times 10^{12} \ \text{bytes/s}} = 6 \times 10^{-3}\ \text{s} = 6\ \text{ms/token}$$ $$\text{Ceiling} = \frac{1}{6\ \text{ms}} \approx 166\ \text{tok/s}$$

That is the theoretical memory-bound ceiling: 166 tok/s, assuming that moving the weights is the only cost. The classic roofline would stop here and say “166 tok/s, go get more BW or quantise”.

Now the fixed overhead. Let us put T_fixed ≈ 2 ms/step (a reasonable value for a Python scheduler + ~40 kernels launched + sampling, without CUDA graphs). The real step:

$$T_{\text{step}} = T_{\text{HBM}} + T_{\text{fixed}} = 6 + 2 = 8\ \text{ms} \;\Rightarrow\; \frac{1}{8\ \text{ms}} = 125\ \text{tok/s}$$

The overhead has eaten 41 tok/s out of the theoretical 166: T_fixed is 25 % of the step (2 of 8 ms). Compare with the 70B: T_HBM ≈ 42 ms, T_step ≈ 44 ms, T_fixed is 4.5 %. Same absolute overhead, relative impact 5-6× larger in the SLM.

What happens if you apply CUDA graphs and take away, say, 1.5 of the 2 ms of T_fixed:

$$T_{\text{step}}^{\text{graphs}} = 6 + 0.5 = 6.5\ \text{ms} \;\Rightarrow\; 154\ \text{tok/s}$$

From 125 to 154 tok/s: +23 % from orchestration alone, without touching the model or the memory hardware. In the 70B the same intervention would have gone from 44 to 42.5 ms, +3.5 %. There it is, in two numbers: “a different game”.

What happens if you quantise the weights to INT4 (1.5 GB instead of 6 GB), with T_fixed still at 2 ms:

$$T_{\text{HBM}}^{\text{INT4}} = \frac{1.5 \times 10^{9}}{1 \times 10^{12}} = 1.5\ \text{ms};\quad T_{\text{step}} = 1.5 + 2 = 3.5\ \text{ms} \;\Rightarrow\; 285\ \text{tok/s}$$

The 4× weight quantisation did not give 4× in latency: it went from 125 to 285 tok/s, a 2.3×, because the 2 ms of T_fixed now dominates (it is 57 % of the step). In the 70B, quantising to INT4 gives almost the full 4× because T_fixed is still noise. The same quantisation delivers twice the speedup on the large model as on the small one, at batch 1. And if you also apply CUDA graphs on top of INT4 (T_fixed → 0.5 ms): 1.5 + 0.5 = 2 ms → 500 tok/s. The order of the optimisations matters: in SLMs, attacking T_fixed first unlocks the rest.

Configuration (3B, 4090, batch 1)	T_HBM	T_fixed	T_step	tok/s	vs. baseline
BF16, no graphs (baseline)	6.0 ms	2.0 ms	8.0 ms	125	1.00×
BF16 + CUDA graphs	6.0 ms	0.5 ms	6.5 ms	154	1.23×
INT4, no graphs	1.5 ms	2.0 ms	3.5 ms	285	2.28×
INT4 + CUDA graphs	1.5 ms	0.5 ms	2.0 ms	500	4.00×

(Illustrative figures with T_fixed rounded; the point is the pattern, not the decimal. The real T_fixed depends on the stack, the PyTorch/CUDA version and whether there is tensor parallelism. Measure it in your own setup before believing any row.)

Implications per optimisation

With the model in hand, the blog’s levers reorder themselves once the regime changes.

Batching: far more headroom in SLMs. Remember that you cross the ridge at B ≈ ridge/2 ≈ 150 as an order of magnitude. In a large model, VRAM runs out long before you saturate compute (weights + KV do not let you reach batch 150). In an SLM the weights take up little space, so you can fit large batches in VRAM and stay memory-bound over a much wider range: the T_HBM of the weights is amortised across the B requests (you load them once, they serve B), so aggregate throughput per GPU rises almost linearly with B until very high values. This is exactly the opposite of the 70B’s fear of saturating compute. In SLMs, batching is the throughput lever par excellence because you saturate compute late; the batch grid search in vLLM has a much wider plateau of good behaviour. Careful: batching improves throughput, not per-request latency; for single-stream latency the prize is in T_fixed.

Speculative decoding: a different crossover point. Speculative wins when verifying γ tokens is “almost free” because you are memory-bound. In an SLM the target is already cheap, so the draft has to be tiny for c = T_draft/T_target to stay small, and the draft’s own T_fixed (launching its kernels) bites harder. The crossover to compute-bound with batching also arrives earlier in absolute terms of tok/s served. The variant that fits best here avoids a separate draft: self-speculative / early-exit reuses early layers of the model itself and saves the T_fixed of orchestrating two models.

Quantisation: it helps for capacity, not for latency at batch 1. As the table showed, INT4 in an SLM at batch 1 gives diminishing returns in latency. Its real prize in SLMs is capacity: weights 4× smaller free VRAM for more KV cache → more concurrency, and it is at high concurrency (aggregate throughput) that saving bytes pays off again. Aggressive sub-4-bit and ternary quantisation takes this to the extreme: in SLMs it makes sense above all for fitting more sessions per GPU, not for lowering the latency of a single one. And it is worth remembering (see quantisation) that at batch 1 the dequantize adds compute work which, in a regime already grazed by T_fixed, is not always free.

Architecture: fine-grained MoE changes which bytes you move. A fine-grained device-native MoE activates few parameters per token, so T_HBM drops relative to a dense model of the same total size, but the T_fixed fraction rises even further, and the router adds its own fixed overhead. It is the SLM regime taken to its limit: almost the whole game is played in orchestration.

Scheduler and CUDA graphs first. The operational conclusion, inverted with respect to the large-model posts: in SLMs, before touching the model, kill T_fixed. CUDA graphs (see SMs, streams and graphs), a vLLM scheduler with its Python part minimised or compiled, and kernel persistence are the first-order levers. In a 70B they would be marginal polish; in a 3B they are half the available speedup.

Applied to on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace). This is the scenario where the flip is most visible, because the 4090 has ~1 TB/s (a third of the H100) but T_fixed is the same in absolute terms. A 3B BF16 without CUDA graphs leaves ~125 tok/s on the table when the memory-bound ceiling is 166; enabling graphs and cleaning up the scheduler recovers most of that. The 4090 has plenty of VRAM room for an SLM, so the bottleneck is almost never total memory but orchestration and, at high concurrency, the KV cache. Rule of thumb: on a 4090 with an SLM, profile the per-step overhead first (Nsight Systems on the gap between kernels) before you quantise.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). The H100 has 3.35 TB/s, so the T_HBM of an SLM is even smaller (a 3B FP8 is ~3 GB → ~0.9 ms) and T_fixed dominates even earlier: a badly orchestrated SLM on an H100 can spend more time in the Python scheduler than moving weights. Serving a single single-stream SLM on an H100 is close to a waste; the right mode is aggressive batching (you saturate compute late, so you push large batches through and per-GPU throughput takes off) or multiplexing many SLMs/sessions per GPU via MPS/MIG. This connects with capacity planning: for SLMs the capacity calculation is governed by concurrency and KV cache, not by the weights. And with the dilemma of one big versus N small: replicating SLMs makes sense precisely because each replica saturates compute late and TP brings nothing (the model already fits; TP would only add communication T_fixed).

What we have not covered

The exact measured T_fixed, kernel by kernel, with Nsight Systems: how much is launch, how much scheduler, how much sampling. That is the content of the next post in the series.
torch.compile / partial captures: alternatives and complements to CUDA graphs when there is dynamic control flow.
The prefill regime in SLMs: prefill is compute-bound even in small models (it processes many tokens at once, high AI), so its roofline is the opposite of decode’s; see prefill.
Attention and KV as the second term of T_HBM: here we have folded them in implicitly; the fine-grained breakdown of attention (which scales with sequence length, not with the weights) deserves its own treatment.

References

Williams, S., Waterman, A., Patterson, D. Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM, 52(4), 2009. https://doi.org/10.1145/1498765.1498785
Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference. arXiv:2503.08311, 2025. https://arxiv.org/abs/2503.08311
Databricks. LLM Inference Performance Engineering: Best Practices. https://www.databricks.com/blog/llm-inference-performance-engineering-best-practices
NVIDIA. NVIDIA H100 Tensor Core GPU Datasheet. https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor-core-gpu-datasheet
NVIDIA. GeForce RTX 4090 — product specifications (Ada Lovelace tensor core figures; treat as approximate, they mix dense/sparse).
Yuan, Z. et al. LLM Inference Unveiled: Survey and Roofline Model Insights. arXiv:2402.16363, 2024 — application of the roofline specifically to LLM inference.