The floor next door: NUMA, hugepages and CPU isolation, or why your GPU waits on the kernel

Contents

Second post in the “below the engine” series. The first opened up the cable between GPUs (NVLink/NCCL). This one goes down to the host: the cores, the memory and the kernel that surround those GPUs and that, badly configured, leave them waiting. The third will explain how Kubernetes automates all of this; here is the raw layer, the one you have to understand before delegating it.

TL;DR

A node with 4×H100 SXM is, physically, a two-socket server = two NUMA domains. Each socket has its own cores, its own memory channels and PCIe lanes towards half of the GPUs and NICs. Inference is not only GPU: the host does work on the hot path of every token, launching CUDA kernels, sampling the next token, tokenising, moving pinned buffers between host and GPU, running the NCCL threads. If those threads and their memory land on the socket that is not local to the GPU, every access crosses the inter-socket link (UPI/Infinity Fabric): 2-3× slower and with p99 spikes. There are three kernel levers that decide the latency tail and that almost nobody touches: locality (NUMA affinity: CPU, memory, GPU and NIC on the same “floor”), page tables (hugepages: a few large pages instead of millions of small ones, plus pinned memory for DMA), and jitter (CPU isolation with isolcpus/nohz_full/rcu_nocbs plus IRQ affinity, so that the kernel does not interrupt the thread launching the next decode kernel). This post explains the mechanism, gives the 10 real knobs, and connects with the interconnect and latency-bound decode. With scepticism about which knobs move the needle in inference and which are cargo cult inherited from low-latency trading.

Where you are: the host below the cable

The analogy: the two-floor office

Picture a consultancy in a building with two floors. On each floor there are desks (CPU cores), an archive with the files (the memory of that socket) and a loading bay connecting to the outside (the PCIe lanes towards the GPUs and NICs of that socket). An analyst works fast while everything they need is on their floor: they reach out and take the file from the archive next to them.

The problem starts when the analyst is on floor 1 but their file is in the archive on floor 2. Every time they need it, they take the lift. The work “functions”, but every lookup costs a trip. And if on top of that the loading bay through which their materials arrive (their GPU) is on the other floor, every delivery crosses the building. This is NUMA: local access (same floor) is fast; remote access (another floor, via the inter-socket link) is 2-3× slower.

And there are two more ways to ruin that analyst even when they are on the right floor:

Interrupting them constantly. Every few minutes, a tannoy announcement, a colleague with a question, a fire alarm test. Every interruption breaks their concentration right when they were about to deliver. This is kernel jitter: the scheduler tick, device IRQs, RCU callbacks, interrupting the host thread right when it was about to launch the next GPU kernel. CPU isolation is putting them in an office with a “do not disturb” sign.
Giving them an index of a thousand tiny tabs. If to find each file they have to search an index with a million minuscule entries, they lose time in the search. If the index has a few large entries, they find it instantly. These are hugepages: pages of 2 MB or 1 GB instead of 4 KB reduce the pressure on the TLB (the cache of the page index).

The thesis: the GPU is expensive and fast, but it spends a surprising fraction of decode waiting on the host. If the host is on the wrong floor, interrupted, and searching a giant index, the GPU, the 30,000 € resource, waits. The three levers in this post exist so that it does not wait.

The mechanism: what the host does on the token path

It is tempting to think that in inference “the GPU does everything”. It is not true. For every token, the host (CPU) does, at a minimum:

Launch the CUDA kernels for each operation. The GPU does not decide what to execute; the host keeps putting kernels in its queue. In decode, where each kernel is short, the host has to run ahead feeding the queue; if the host thread stalls, the GPU runs out of work: a bubble.
Sample the next token (argmax/top-p/top-k over the logits), which comes back from device to host.
Tokenise the input and detokenise the output.
Move pinned buffers (page-locked) between host and GPU by DMA: prompts, logits, and in offload configurations, part of the KV cache.
Run the NCCL threads that coordinate the collectives of the previous post.

All of that is CPU and host memory work. And all of it suffers if:

The process runs on the socket that is not local to its GPU → every DMA and every memory access crosses the inter-socket link.
The kernel interrupts the threads → bubbles in the GPU queue.
The memory is not pinned or uses small pages → page faults, TLB misses, and worse: if the memory of the DMA transfer is not pinned, the driver makes an intermediate copy.

The map: `nvidia-smi topo -m`

Everything starts by looking at the map. nvidia-smi topo -m shows, for each GPU, which NUMA node and which cores it is local to, and through what kind of path it talks to each NIC and to each other GPU:

        GPU0  GPU1  GPU2  GPU3  NIC0  CPU Affinity  NUMA Affinity
GPU0     X    NV18  NV18  NV18  PIX   0-31,64-95    0
GPU1    NV18   X    NV18  NV18  SYS   0-31,64-95    0
GPU2    NV18  NV18   X    NV18  SYS   32-63,96-127  1
GPU3    NV18  NV18  NV18   X    SYS   32-63,96-127  1

Read it like this: GPU0 and GPU1 are local to NUMA node 0 (cores 0-31, 64-95); GPU2 and GPU3 to NUMA node 1. NV18 between GPUs = 18 NVLink links (the good stuff, from the previous post). In the NIC column: PIX = a single PCIe switch in between (optimal for GPUDirect RDMA); SYS = the path crosses the inter-socket link (the worst). The rule: the process serving on GPU0/1 must be pinned to cores 0-31/64-95 and to the memory of node 0; and if it also uses RDMA, you want the NIC that is on PIX with its GPU.

The maths that matter: a kernel tick is a GPU bubble

The figure worth internalising is not the bandwidth one, it is the jitter one. In decode, the host launches many short kernels per token. If the host thread that launches them is preempted by the scheduler (a timer tick, an IRQ, an RCU callback) for $t_{\text{stall}}$, and the GPU drains its queue in that time, a bubble appears: the GPU stops.

Put numbers on it. A typical timer tick or the handling of an IRQ costs of the order of tens of microseconds of diversion. If a decode kernel lasts around 50-100 µs and the queue carries 2-3 kernels in flight, a host stall of 50-100 µs drains the queue and the GPU sits idle until the host resumes. Multiply by the interrupt frequency of a non-isolated kernel (the default tick is 250-1000 Hz, plus network and disk IRQs): the p99/p999 tail of TTFT and of inter-token time fills up with these episodes.

$$ \text{jitter}_{p99} \approx f_{\text{interrupts}} \times t_{\text{stall}} \times \mathbb{1}[\text{GPU queue drained}] $$

The intuition: in average throughput it is barely noticeable (the bubbles average out), but in the tail, which is what an SLO measures, kernel jitter is a first-order contributor. That is why CPU isolation, which was born in low-latency trading, makes sense in LLM decode: it is the same physics, a critical thread that cannot afford the kernel stopping it.

And the NUMA cost, in parallel: a remote memory access (another floor) has latency around 1.5-2× the local one and half the bandwidth. For the pinned buffers moved by DMA on every step, and for the vLLM scheduler structures that live on the host, that penalty is paid token by token.

The three levers, one by one

Locality (NUMA): getting everything on the same floor

The goal is that the inference process using GPU0/1 has its cores, its memory and (if applicable) its NIC on NUMA node 0. Raw, without Kubernetes:

# Pin process to node 0 (cores and memory) to serve on GPU0/1
numactl --cpunodebind=0 --membind=0 \
        vllm serve meta-llama/Llama-3-70B --tensor-parallel-size 2

--membind=0 is the key: it forces all the process memory to be allocated on node 0. Without --membind, the kernel can place pages on node 1 under pressure, and you start paying for the lift without knowing it.

Page tables (hugepages): a few large pages and pinned memory

Two different things under the same umbrella. First, hugepages reduce TLB pressure for the large host buffers (pinned, KV offload). Second, pinned memory (page-locked) is what allows direct DMA with no intermediate copy. The silent trap is transparent hugepages (THP): their background compaction causes latency spikes, exactly what you do not want.

Jitter (CPU isolation): the office with “do not disturb”

Three kernel boot parameters, coordinated:

isolcpus=2-31,66-95     # take these cores out of the scheduler's balancing
nohz_full=2-31,66-95    # tickless: no timer tick if there is only 1 runnable thread
rcu_nocbs=2-31,66-95    # offload RCU callbacks to housekeeping cores

isolcpus sets the cores aside; you have to pin the inference threads there (the non-isolated cores, 0-1, are left for the system). nohz_full removes the periodic tick (it only works if there is a single runnable thread on the core). rcu_nocbs takes RCU work off those cores. And separately, IRQ affinity: moving device interrupts away from the inference cores.

The 10 knobs worth touching

Ordered by impact/frequency. Almost all of them are sysctls, kernel boot parameters or numactl. The low-level reference is Rigtorp’s low-latency guide and Red Hat’s real-time docs.

Knob 1 — `nvidia-smi topo -m`: look at the map before touching anything

Just as in the interconnect post: the map first. Which GPU is local to which NUMA node and to which cores, and what path (PIX/PHB/SYS) there is to each NIC. Without this, any pinning is blind. Half of the “inference has latency spikes” problems are processes running on the wrong socket without anyone having looked.

Knob 2 — `numactl --cpunodebind --membind`: pin to the local node

numactl --cpunodebind=0 --membind=0 <process>   # cores AND memory on node 0
numactl --hardware                                 # see nodes, distances, free memory

The --membind is what really matters: without it, the memory scatters. It is the highest-impact knob on sustained throughput.

Knob 3 — `kernel.numa_balancing=0`: turn off automatic migration

sysctl -w kernel.numa_balancing=0

The kernel’s automatic NUMA balancing migrates pages between nodes trying to “bring them closer”, but that background work causes jitter and, with the memory already pinned by knob 2, it adds nothing. On dedicated inference nodes, turn it off.

Knob 4 — Explicit hugepages (1 GB) for host buffers

# Kernel boot, for KV offload / large pinned buffers
default_hugepagesz=1G hugepagesz=1G hugepages=32

grep Huge /proc/meminfo     # verify the reservation

Useful when there is host memory on the hot path (vLLM with --cpu-offload-gb, or large pinned buffers). If your deployment does not touch host memory in the hot path, explicit hugepages add little, so do not set them out of cargo cult.

Knob 5 — THP on `madvise` or `never`: avoid the compaction spikes

echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
echo never   > /sys/kernel/mm/transparent_hugepage/defrag

Transparent hugepages on always save TLB but their compaction triggers unpredictable latency. For tail-sensitive workloads, madvise (only where the app asks for it) or never is what is recommended. It is one of the few knobs with clear consensus: THP always is bad for latency.

Knob 6 — `isolcpus`: set the inference cores aside from the scheduler

isolcpus=2-31,66-95

Takes those cores out of the scheduler’s load balancing; the system (kernel threads, daemons) stays on the non-isolated ones. You have to pin the inference threads explicitly to the isolated cores (via numactl/taskset or, on K8s, the CPU Manager of the next post). Isolating without pinning is useless.

Knob 7 — `nohz_full` + `rcu_nocbs`: tickless and no RCU on the critical cores

nohz_full=2-31,66-95 rcu_nocbs=2-31,66-95

Removes the periodic timer tick and the RCU callbacks from the inference cores. Two warnings from practice: nohz_full only eliminates the tick if there is a single runnable thread on the core (if you pin two threads there, the tick comes back); and nohz_full is not compatible with the intel_pstate driver in some configurations, which has to be validated, not assumed.

Knob 8 — IRQ affinity: interrupts, away from the inference cores

systemctl stop irqbalance      # or configure it to respect isolcpus
# move a device's IRQs to the housekeeping cores (0-1)
echo 3 > /proc/irq/<N>/smp_affinity     # mask for cores 0-1

A network or disk IRQ landing on an inference core is a direct interruption of the thread feeding the GPU. Move them to the housekeeping cores. (irqbalance can respect isolcpus automatically if it is configured to.)

Knob 9 — CPU governor `performance` + C-states

cpupower frequency-set -g performance
# stop idle cores entering deep C-states (wakeup latency)
cpupower idle-set -D 0      # or limit the C-state depth

With the powersave/ondemand governor, a core that was idle takes time to ramp up its frequency: wakeup latency exactly when work arrives. performance keeps it flat out. On servers dedicated to inference, the energy saving does not compensate for the latency tail.

Knob 10 — Memory locking + `swappiness=0`

sysctl -w vm.swappiness=0        # do not evict inference pages to swap
# and in the app / container: ulimit -l unlimited (memlock) for pinned memory

An inference page that the kernel decides to swap to disk is a page fault of milliseconds on the hot path. swappiness=0 and adequate memlock limits (so the driver can pin memory) close that door. On K8s, this translates into Guaranteed QoS and memory limits, the bridge to the next post.

Summary table

#	Knob	Mechanism	What it attacks
1	`nvidia-smi topo -m`	diagnosis	see GPU–NUMA–NIC affinity
2	`numactl --cpunodebind --membind`	pinning	locality (the biggest lever)
3	`kernel.numa_balancing=0`	sysctl	jitter from page migration
4	explicit 1G hugepages	boot param	TLB on host buffers
5	THP `madvise`/`never`	sysfs	compaction spikes
6	`isolcpus`	boot param	scheduler off the critical cores
7	`nohz_full`+`rcu_nocbs`	boot param	tick + RCU jitter
8	IRQ affinity	`/proc/irq`	device interrupts
9	`performance` governor	cpupower	frequency wakeup latency
10	`swappiness=0` + memlock	sysctl/ulimit	page faults on the hot path

How it connects with the rest of the stack

With the interconnect (previous post). The NCCL host threads want cores local to their GPU; and for multi-node, the RDMA NIC must be on the PIX path with its GPU (knob 1). A GPUDirect RDMA with the NIC under the other socket loses half its advantage. NUMA and NVLink are the same story seen from the host and from the cable.

With vLLM and decode. Decode is latency-bound: the host thread feeding the kernel queue is exactly the one CPU isolation protects. And vLLM’s --cpu-offload-gb puts host memory on the hot path, where NUMA locality plus hugepages (knobs 2, 4) go from “nice” to “critical”. Continuous batching helps here too: more tokens per iteration amortise both the collective latency and the fixed cost of the host launches.

With Kubernetes (next post). Everything in this post is done by hand (numactl, taskset, boot parameters). In production it is not done by hand: the kubelet automates it with CPU Manager, Memory Manager and Topology Manager. The next post is exactly how this is declared so that every vLLM pod is born pinned to the right NUMA node, with no scripts.

With observability. The p99 spikes from jitter or from remote access are visible: in DCGM, low GPU utilisation with a full queue (bubbles); in system metrics, inter-socket traffic and CPU migrations. GPU observability with DCGM is where you diagnose a “the GPU is at 60 % and I do not know why” that many times is the host waiting.

With capacity planning. Reserving cores for the system (housekeeping) and dedicating the rest to inference changes the calculation of how many pods/replicas fit per node. Capacity planning has to count those reserved cores, not assume that all 128 vCPUs are available for serving.

Traps and things that are not what they look like

Cargo cult from low-latency trading. Many isolcpus/nohz_full guides come from HFT, where the last microsecond is squeezed out. In LLM inference, CPU isolation does help on the decode tail, but do not expect a miracle: if your bottleneck is HBM bandwidth or the interconnect, isolating cores does not move the needle. Measure first; apply it where the host is the limit.

Isolating without pinning. isolcpus takes the cores out of the scheduler, but if you do not pin the inference threads there, those cores stay empty and inference runs on the housekeeping ones, worse than before. Isolating and pinning always go together.

Forgotten --membind. Pinning cores but not memory (--cpunodebind without --membind) lets the pages scatter to the other node under pressure. Memory pinning is the half that gets forgotten most and the one that pays off most.

THP always “because it saves TLB”. It saves TLB and throws in latency spikes from compaction. For workloads with a tail SLO, it is a bad trade. madvise/never.

nohz_full with two threads on the core. Tickless only works with a single runnable thread per core. If you pin two inference threads to the same isolated core, the tick comes back and you have complicated the kernel boot for nothing.

Assuming the topology instead of reading it. Different servers wire GPUs and NICs to different sockets. nvidia-smi topo -m and numactl --hardware are the truth; the manufacturer’s diagram is indicative. Read it on every node model.

Conclusion

The GPU is the expensive resource, but in decode it spends a surprising part of its time waiting on the host: waiting for it to launch the next kernel, to sample, to move a buffer. If that host is on the wrong floor (remote NUMA), interrupted (kernel jitter) or searching a giant index (4 KB pages), the GPU is left with an empty queue and p99 blows up, without any API dashboard saying why. Of the ten knobs, the first (reading the map with nvidia-smi topo -m) and the second (pinning cores and memory to the local node with --membind) solve most of it; CPU isolation (isolcpus/nohz_full/IRQ affinity) is the second layer, the one that trims the decode tail, and it makes sense where the host is the limit, not as a ritual. The idea that reorders the intuition: inference is not “all GPU”; it is a dance between GPU and host, and the host dances better close by, uninterrupted, and with few large pages. The next post shows how Kubernetes choreographs that dance for every pod without a single hand-written script.

References

Erik Rigtorp, Low Latency Tuning Guide (isolcpus, nohz_full, IRQ affinity, THP): https://rigtorp.se/low-latency-guide/.
Red Hat, Usage, constraints and implications of isolcpus=, nohz_full= and rcu_nocbs=: https://access.redhat.com/articles/3720611.
NVIDIA, nvidia-smi topo -m and the GPU–NUMA–NIC affinity matrix (CUDA docs / Developer Forums): https://forums.developer.nvidia.com/t/nvidia-smi-topo-m-revisited/216584.
Chaim Rand, The Crucial Role of NUMA Awareness in High-Performance Deep Learning: https://chaimrand.medium.com/the-crucial-role-of-numa-awareness-in-high-performance-deep-learning-99ae3e8eb49a.
SUSE Labs, CPU Isolation – nohz_full (part 3): https://www.suse.com/c/cpu-isolation-nohz_full-part-3/.
Linux kernel, Automatic NUMA Balancing and Transparent Hugepage Support (Documentation/admin-guide): https://docs.kernel.org/admin-guide/mm/transhuge.html.