Numa on lo0 — Tech Blog

The kitchen door the maître d' never looked at: network NUMA, Cilium eBPF and DRANET, the fourth leg of pinning

Sat, 06 Jun 2026 12:00:00 +0200

Fourth instalment, a coda, of “under the engine”. The series closed with three legs of locality: the cable between GPUs, the host by hand and the declarative orchestration of the kubelet. But the maître d’ of the last post seated the group looking at CPU, memory and GPU, and never asked which door the dishes come through. That door is the NIC. Here is the fourth leg.

TL;DR

The Topology Manager admits a pod under single-numa-node if its CPUs, its memory and its GPU fit on the same NUMA node. The NIC is not part of that count: the kubelet has no Hint Provider for the network card. On an inference node with the network at 200/400 Gb/s, the case of disaggregated serving, where the KV-cache travels over RDMA between the prefill pool and the decode pool, a NIC on the wrong socket makes every packet cross the UPI/QPI, exactly the “remote NUMA” the series fights on the compute side, but coming in through the network door. And there is a second front: the softirq (NET_RX) that processes the datapath runs on the CPU that services the NIC’s IRQ; if that CPU is one of the cores isolcpus/reserved-cpus gave exclusively to vLLM, the softirq steals cycles from it and injects jitter into the p99 tail. Cilium eBPF replaces two RKE2 pieces, kube-proxy (with eBPF/XDP load balancing) and the default CNI Canal (with a native datapath), and its own tuning guide tells you to kill irqbalance and pin the NIC IRQs: a fourth list to align alongside isolcpus and reserved-cpus. The 2026 state of the art closes the gap from above: netkit (kernel ≥6.8, zero namespace overhead), BIG TCP (192k super-packets for 100Gb/s+), host-routing (iptables bypass), and above all DRA/DRANET, the network driver that finally co-schedules NUMA-local GPU and NIC on the same PCIe root, enabling GPUDirect RDMA with +59.6% bus bandwidth in all_gather and +58.1% in all_reduce. On a generic RKE2 cluster with 4×H100 SXM nodes.

Where you are: the network plane the trilogy did not open

The analogy: the door the dishes come through

Go back to the restaurant of the previous post. The maître d’, the Topology Manager, seated the party of eight at a single table (one NUMA node) because the diners (CPUs), the cutlery (memory) and the reserved bottle (the GPU) all fitted. A perfect table. But the maître d’ never looked at where the kitchen pass is: the door every dish comes in and goes out through.

That door is the NIC. The prompt comes in there, the tokens go out there, and, in disaggregated serving, the KV-cache that the prefill pool sends to the decode pool circulates there. If the table is in the left-hand room (socket 0) but the kitchen pass is in the right-hand one (socket 1), every dish crosses the entire restaurant (the UPI/QPI), over and over, however impeccably the table is laid. The diner does not notice the perfect table: they notice the dish arriving late and cold.

And there is a finer detail: the waiter who crosses the room with the dishes (the softirq that processes the packets) is one of the seated diners. If the maître d’ gave them a chair of their own so they could eat in peace (a core isolated with isolcpus for vLLM) but the restaurant also has them waiting on the far door, that diner does not eat: they spend the whole dinner crossing the room. The jitter shows up exactly where you thought you had bought calm.

The trilogy levelled three legs of the table: the cable, the host and the orchestration. The fourth, which door the dishes come through and who carries them, is levelled by no kubelet manager. Until 2026.

The gap: why the Topology Manager does not look at the NIC

The mechanism of post 3 is a coordinator (Topology Manager) that consults three Hint Providers: CPU Manager, Memory Manager and Device Manager (the GPU plugin). Each says on which NUMA node it can satisfy its part; the coordinator computes the intersection and admits or rejects.

The problem is one of census: the classic NIC is not a Device Manager “device”. A standard Ethernet/InfiniBand card is managed by the CNI and the kernel, it is not requested in the pod’s resources: like nvidia.com/gpu, and therefore it emits no NUMA hint. The Topology Manager aligns CPU+memory+GPU and leaves the NIC wherever the hardware put it, which may be the other socket. The maître d’ has three assistants and is missing the fourth: the one who knows which door the dishes come through.

This did not matter when a node’s network was 10/25 Gb/s and the bottleneck was elsewhere. It matters now, with two workloads that saturate the node’s network:

Disaggregated serving. The KV-cache that travels between the prefill pool and the decode pool moves over RDMA. These are large transfers, sensitive to latency and bandwidth, which in multi-node setups go out through the NIC.
Multi-node NCCL collectives. When tensor/pipeline parallelism crosses the node boundary, the all-reduce/all-gather operations no longer go over NVLink but over GPUDirect RDMA on the NIC.

In both, where the NIC sits relative to the GPU and to the pod’s cores decides performance. And the kubelet, on its own, does not coordinate that.

The network datapath under NUMA: IRQ, softirq and DMA

To see why NIC locality carries weight, you have to look at the path of an incoming packet:

Three kernel facts the analogy compresses:

The IRQ has affinity. Each NIC queue raises an interrupt that the kernel services on a specific CPU (/proc/irq/<n>/smp_affinity). The heavy processing is deferred to a softirq (NET_RX/NET_TX), which runs on that same CPU. If irqbalance is left loose, it migrates them non-deterministically, poison for the p99.
The softirq competes with the pod. If the IRQ lands on a core that isolcpus reserved for vLLM, that queue’s NET_RX steals cycles from the model. The signal in /proc/softirqs: a NET_RX column that spikes on a single CPU. It is the same jitter from post 2, coming in through the network.
DMA has a NUMA origin. The NIC writes the packet by DMA into the RAM of the socket of its PCIe root. If the consumer (the pod’s thread) is on the other socket, it reads across the UPI/QPI. RFS (Receive Flow Steering) tries to bring the processing to the consumer’s CPU, but it cannot teleport the NIC to the other socket.

One number, with its caveat

Take a 2-socket node, a 400 Gb/s = 50 GB/s NIC on the PCIe root of socket 0, and a decode pod pinned to socket 1. If the NIC saturates, those ~50 GB/s of receive traffic cross the UPI towards socket 1. A UPI 2.0 link runs at around 20–40 GB/s per direction and link depending on generation; even with several links, 50 GB/s of network traffic going against the grain eats a far from negligible fraction of the inter-socket budget, the same budget already contended by the pod’s remote memory accesses and, if there is multi-node traffic, the KV-cache of the disaggregation. I am not giving a firm “X% degradation” because it depends on CPU generation, number of UPI links, MTU and traffic pattern; without that methodology, any exact figure is marketing.

What is measured with public methodology is the aggregate effect of aligning GPU and NIC: the DRANET project reports +59.6% bus bandwidth in all_gather and +58.1% in all_reduce (NCCL collectives) when the assigned NIC is NUMA-local to the GPU rather than not. That is the size of the gap the Topology Manager was leaving open.

What RKE2’s Cilium eBPF replaces (and why it touches this story)

RKE2 ships Canal (Flannel + Calico) by default as the CNI and kube-proxy (iptables/IPVS rules) for Service balancing. Switching to Cilium (cni: cilium in /etc/rancher/rke2/config.yaml) replaces both pieces with an eBPF datapath:

RKE2 piece	What it does	What Cilium eBPF puts in
`kube-proxy` (iptables/IPVS)	Service balancing	LB in eBPF; with `kubeProxyReplacement=true`, and acceleration in XDP (driver layer)
Canal (Flannel+Calico)	VXLAN overlay + NetworkPolicy	native datapath (`routingMode=native`), L3/L4 and L7 NetworkPolicy in eBPF
veth per pod	namespace interface pair	netkit (kernel ≥6.8): namespace overhead ~0
host iptables traversal	netfilter hooks	eBPF host-routing: bypasses iptables and the upper part of the stack

Up to here this is pure networking and it does not touch the kubelet’s resource managers: Cilium does not assign exclusive CPUs nor emit compute NUMA hints. The ten knobs of post 3 stay identical whether you run Canal or Cilium.

But Cilium does enter the fourth leg through two doors. The first: its own tuning guide recommends, literally, “kill irqbalance and pin the NIC IRQs to specific CPUs for maximum workload isolation”, along with the tuned network-latency profile, the performance governor and CONFIG_PREEMPT_NONE. In other words: the eBPF datapath only really performs if you coordinate IRQ affinity, and that affinity has to point at the housekeeping cores (reserved-cpus), never at the isolated ones. So a fourth list appears that must be kept consistent with isolcpus and reserved-cpus:

isolcpus = 2-31,34-63 # cores exclusive to vLLM (host, post 2)
reserved-cpus = 0-1,32-33 # kubelet housekeeping (post 3)
IRQ affinity = 0-1,32-33 # NIC IRQs → housekeeping ONLY (this post)
 # never 2-31: there the softirq would steal from the model

The second door: netkit + host-routing + BIG TCP reduce how many times the packet crosses the stack and the namespace, which cushions (does not eliminate) the cost of the NUMA crossing. BIG TCP assembles super-packets of up to 192k (against 64k) for 100Gb/s+; fewer stack traversals means less softirq work on the core, and therefore less pressure on the inter-socket budget. It is the continuous batching analogy applied to the network stack: amortising a fixed cost over larger batches.

Cilium performance profile (state 1.19, kernel ≥6.8)

# Helm, recommended performance profile (summary of the tuning guide)
helm install cilium cilium/cilium --version 1.19.4 \
 --namespace kube-system \
 --set routingMode=native \
 --set bpf.datapathMode=netkit \  # namespace overhead ~0 (kernel >=6.8)
 --set bpf.masquerade=true \
 --set kubeProxyReplacement=true \  # replaces RKE2's kube-proxy
 --set enableIPv4BIGTCP=true \  # 192k super-packets (mlx5/ice NIC)
 --set enableIPv6BIGTCP=true \
 --set bpf.distributedLRU.enabled=true \# per-CPU BPF maps: less spinlock contention
 --set bpf.mapDynamicSizeRatio=0.08 \
 --set bpfClockProbe=true

# Verification inside a Cilium pod:
cilium status --verbose | grep -E "Device Mode|Host Routing|BIG TCP|XDP"
# Device Mode: netkit · Host Routing: BPF · IPv4 BIG TCP: enabled · XDP Acceleration: Native

Sceptical caveat: netkit and BIG TCP are beta and require kernel ≥6.8 and specific NICs (mlx4/mlx5/ice). They are not in-place: they change datapath fundamentals and force you to restart pods or, better, to apply them by per-node config only on new nodes. For an ENS cluster in production, that is a maintenance window, not a blind helm upgrade.

The 2026 state of the art: DRA and DRANET, the maître d’ who finally looks at the door

What closes the gap at the root is not Cilium, it is the admission mechanism the kubelet did not have for the NIC: Dynamic Resource Allocation (DRA), beta since Kubernetes 1.32 and with advances in every release up to 1.36 (May 2026). DRA generalises the “devices” model beyond the GPU: a driver discovers the hardware, publishes ResourceSlices with its attributes, including the NUMA topology and the PCIe root, and the scheduler resolves ResourceClaims that can demand affinity between devices.

DRANET (a kubernetes-sigs project) is the network DRA driver. It discovers the NICs (including the RDMA-capable ones), advertises them as ResourceSlices, and via NRI injects them into the pod’s namespace, compatible with whatever CNI you already have, Cilium included. The key piece for this story: combined with the NVIDIA GPU DRA driver, it allows co-scheduling GPU and NIC that share a PCIe root (the relationship NVIDIA calls NODE), which is exactly the condition for GPUDirect RDMA. The maître d’ finally has a fourth assistant: “is there a NIC NUMA-local to this GPU?”.

The ResourceClaimTemplate uses CEL selectors to request exactly that alignment:

# Request an RDMA NIC NUMA-local to the assigned GPU (illustrative DRANET/DRA schema)
apiVersion: resource.k8s.io/v1beta1
kind: ResourceClaimTemplate
metadata:
 name: gpu-nic-numa-aligned
spec:
 spec:
 devices:
 requests:
 - name: rdma-nic
 deviceClassName: dra.net  # NICs published by DRANET
 constraints:
 - requests: ["rdma-nic"]
 matchAttribute: "dra.net/pcieRoot" # same PCIe root as the GPU
 # → enables GPUDirect RDMA over a NUMA-local path

Why it matters for inference, not for abstract “AI training”: in disaggregated serving, RDMA is what moves the KV-cache between the prefill pool and the decode pool with the latency that TTFT demands; and in multi-node, GPUDirect RDMA replaces NVLink as the medium of the collective. Aligning GPU+NIC on the same PCIe root is what turns an “RDMA that works” into an “RDMA that performs”, the +60% bus bandwidth from DRANET.

Status and caveats: DRA is beta (gates to enable by hand), DRANET is young (a SIG project, still evolving) and the managed offering exists mostly in cloud (GKE managed DRANET in preview, AKS for RDMA). For on-premise ENS it is a path, not a finished product: the value today is understanding that the fourth leg already has a standard OSS mechanism, and starting to pilot it on a lab node, not putting it into critical production this quarter.

How it connects with the rest of the stack

With the host (post 2). The NIC’s IRQ affinity is a third list to marry with isolcpus and reserved-cpus. The IRQs go to housekeeping; the isolated cores, untouched. Uncoordinating them lets in through the network door the jitter that isolcpus threw out through the compute one.

With the orchestration (post 3). DRA is the natural extension of the Topology Manager: the same “admit only if it fits on the NUMA node” principle carried over to the NIC. Where the Device Manager left the network out of the census, DRANET puts it in.

With the interconnect (post 1). Inside the node NVLink rules; on crossing the node boundary, GPUDirect RDMA over the NIC is the medium of the collective. The kubelet’s NUMA policy guarantees that GPU and CPUs share a socket; DRANET adds that the NIC does too, and only then does the RDMA take the short path.

With disaggregated serving. The prefill→decode KV-cache is the traffic that punishes a badly placed NIC hardest. The fourth leg is what stops separating prefill and decode from being paid for in transfer latency.

With capacity planning. Sizing gains a dimension: “GPUs per node and cores per NUMA node” is not enough; you have to count how many NICs NUMA-local to a GPU the chassis has. A node with 4 GPUs and a single NIC on socket 0 has two GPUs “far from the door”.

With observability. What confirms that the fourth leg is properly set is not an application dashboard: it is /proc/softirqs (is NET_RX concentrated on housekeeping?), nvidia-smi topo -m (is the GPU↔NIC relationship NODE/PHB?) and the NIC counters. It fits with GPU observability with DCGM: the GPU “at 60% for no reason” may be the host waiting for packets that cross the socket.

Traps and things that are not what they look like

Believing that switching to Cilium “already optimises the network”. Cilium eBPF replaces kube-proxy and Canal and performs better out of the box, but the default deployment prioritises compatibility, not performance. Without irqbalance disabled, without IRQs pinned to housekeeping and without netkit/host-routing, you leave most of the improvement on the table. Cilium’s docs say so; plenty of people do not read the tuning guide.

Pinning the NIC IRQs to isolated cores. The symmetric mistake to knob 6 of post 3: if you set IRQ affinity over isolcpus, the NET_RX softirq steals cycles from vLLM on exactly the cores you isolated so nobody would disturb it. The IRQs go to reserved-cpus, always.

Assuming the Topology Manager already aligns the NIC. It does not: the classic NIC is not a Hint Provider. If you need NIC↔GPU locality, today the mechanism is DRA/DRANET, not a kubelet policy. Waiting for single-numa-node to solve it is waiting for something that is not in its design.

Putting DRA/DRANET into ENS production this quarter. It is beta and young. The sensible move is to pilot it on a lab node, measure all_reduce/all_gather with and without alignment, and decide with data. The +60% figure is from a specific environment; reproduce it in yours before promising it.

BIG TCP / netkit without reading the requirements. Kernel ≥6.8, mlx4/mlx5/ice NICs, no tunnel or encryption for BIG TCP, and nothing in-place: it forces pod restarts or per-node config. On a cluster with IPsec or with unsupported NICs, part of this does not apply. Check cilium status --verbose before taking for granted that it is active.

Confusing the eBPF datapath (kernel) with the Cilium agent (pod). cilium-agent is a Burstable DaemonSet that should live on housekeeping (covered by system-reserved). But datapath processing runs in softirq, governed by the host’s IRQ affinity, not by reserved-cpus. They are two different things; pinning the pod well does not pin the softirq.

Conclusion

The “under the engine” series chased one idea: performance that looks like an engine problem (vLLM is slow) or a model problem (quantisation) is, far too often, a locality problem one layer down. The trilogy covered three: the cable (NVLink not used), the host (remote NUMA, jitter) and the orchestration (pinning that never happened). The fourth was missing: the network. The Topology Manager seats the pod at a perfect NUMA table and never asks which door the dishes come through nor who carries them. On a node at 25 Gb/s it made no difference; on one at 400 Gb/s with the KV-cache crossing over RDMA, that door decides TTFT and the collective’s bandwidth. Cilium eBPF replaces kube-proxy and Canal with a datapath that performs, if you coordinate IRQ affinity with isolcpus/reserved-cpus, a fourth list to align, and DRA/DRANET finally provides the census that was missing: co-scheduling NUMA-local GPU and NIC on the same PCIe root, with an improvement of a size (+60% NCCL bus bandwidth) that measures how big the gap was. Going down a level is not snobbery: it is that the root cause lived, once again, one layer below where the dashboard looks.

References

Cilium, Tuning Guide (netkit, host-routing, BIG TCP, XDP, pin the IRQs and kill irqbalance): https://docs.cilium.io/en/stable/operations/performance/tuning/.
Cilium 1.19 (February 2026), Cilium at Ten Years — hardening of encryption, policies and observability: https://www.infoq.com/news/2026/02/cilium-119/.
Isovalent, Cilium 1.18 (IPv6, encrypted overlay, ingress bandwidth, policy perf): https://isovalent.com/blog/post/cilium-1-18/.
RKE2, Network Options (Canal by default; Cilium with kube-proxy replacement): https://docs.rke2.io/networking/basic_network_options.
Kubernetes, Dynamic Resource Allocation: https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/.
Kubernetes blog, v1.36: More Drivers, New Features, and the Next Era of DRA (May 2026): https://kubernetes.io/blog/2026/05/07/kubernetes-v1-36-dra-136-updates/.
DRANET (kubernetes-sigs), the network DRA driver and the paper The Kubernetes Network Driver Model (+59.6% all_gather / +58.1% all_reduce): https://github.com/kubernetes-sigs/dranet.
AKS Engineering, Optimizing RDMA performance for AI workloads on AKS with DRANET (April 2026): https://blog.aks.azure.com/2026/04/01/dranet-rdma-optimization-for-ai-on-aks.
Linux network tuning — IRQ affinity, RSS/RPS/RFS and NUMA softirq: https://andreaskaris.github.io/blog/networking/rss-irq-affinity-and-rps/.

The maître d' who only seats you if you all fit at one table: CPU, Memory and Topology Manager on RKE2

Sat, 06 Jun 2026 08:00:00 +0200

The close of the “under the engine” series. We saw the cable between GPUs and the host: NUMA, hugepages and CPU isolation done by hand. Here is the piece that makes it declarative and at scale: how the RKE2 kubelet pins each vLLM pod to the right NUMA node without a single script.

TL;DR

Pinning NUMA with numactl/isolcpus/taskset, the subject of the previous post, does not scale to a cluster where pods come and go and there are dozens of nodes. The kubelet automates it with three components that work as Hint Providers for a central coordinator, the Topology Manager: the CPU Manager (assigns exclusive CPUs to containers of Guaranteed pods with whole CPUs), the Memory Manager (NUMA-local memory and hugepages) and the Device Manager/GPU plugin (which knows which GPU is on which NUMA node). With the single-numa-node policy, the Topology Manager only admits the pod if its CPUs, its memory and its GPU fit in the same NUMA domain; if they do not fit, it rejects the pod, strict admission, like the maître d’ who will not seat a party of eight unless there is a table for eight. On RKE2 all of this is configured with kubelet-arg in /etc/rancher/rke2/config.yaml. This post explains the mechanism, gives the 10 knobs and takes apart the gotchas that break pinning silently: the cpu_manager_state file you have to delete when changing policy, the QoS that has to be exactly Guaranteed, and the reserved-cpus that must match the host’s isolcpus. On a generic RKE2 cluster with 4×H100 SXM nodes.

Where you are: the orchestration that materialises the host

The analogy: the maître d’ of a restaurant whose tables cannot be pushed together

A restaurant has tables of different sizes and, house rule, tables are never pushed together. A party of eight arrives. The maître d’ looks for a single table where all eight fit. If there is one, he seats them; if only tables for four are left, he does not accept them. He would rather turn the booking away than seat the group split across two separate tables, because he knows a split dinner goes badly.

That maître d’ is the Topology Manager under the single-numa-node policy. The “party” is an inference pod asking for CPUs, memory and a GPU. The “table” is a NUMA node. The maître d’ asks three assistants (are there free CPUs on any node? the CPU Manager; is there free memory? the Memory Manager; is there a free GPU? the Device Manager) and only admits the pod if all three resources fit on the same node. If not, he rejects it (the pod ends up Failed with TopologyAffinityError), and the scheduler will try another node.

The difference from the previous post: there you were the maître d’, seating each process by hand with numactl. Here the maître d’ is the kubelet, and it does it for every pod, on every node, automatically, rejecting whatever does not fit. That is what turns artisanal pinning into a declarative property of the cluster.

The mechanism: Hint Providers and the coordinator

The Topology Manager does not assign resources; it coordinates the ones that do. The flow, when a Guaranteed pod arrives at a node:

The three managers are Hint Providers: each one tells the Topology Manager on which NUMA node(s) it could satisfy its part. The Topology Manager computes the intersection and, depending on the policy, decides:

none (default): no coordination; each manager does its own thing without aligning. No NUMA guarantee.
best-effort: it tries to align on one node; if it cannot, it admits anyway (on whichever node). Better than nothing, no guarantee.
restricted: if it fails to align, it rejects the pod. Strict, but it allows multi-node affinity if the intersection provides it.
single-numa-node: it requires everything to fit on a single NUMA node, or it rejects. The strictest one, and the one that genuinely guarantees the locality of the previous post.

And two preconditions without which none of this kicks in:

The pod has to be QoS Guaranteed: requests == limits on CPU and memory, and whole CPUs (not 500m). Only then does the CPU Manager assign exclusive CPUs.
The CPU Manager has to be on the static policy (not none).

Without those two, the Topology Manager has nothing to align and pinning does not happen, even with the policy in place. That is gotcha no. 1.

How it is configured on RKE2

RKE2 passes arguments to the kubelet with the kubelet-arg key in /etc/rancher/rke2/config.yaml. The reference configuration for GPU inference nodes:

# /etc/rancher/rke2/config.yaml (on every agent node with GPUs)
kubelet-arg:
 - "cpu-manager-policy=static"
 - "topology-manager-policy=single-numa-node"
 - "topology-manager-scope=pod"
 - "memory-manager-policy=Static"
 - "reserved-cpus=0-1,64-65" # housekeeping; must match the host
 - "system-reserved=memory=8Gi"
 - "kube-reserved=memory=4Gi"
 - "reserved-memory=0:memory=4Gi;1:memory=4Gi" # required by Memory Manager Static
node-label:
 - "fibercli.local/numa-pinned=true"

After deploying it: systemctl restart rke2-agent. Critical gotcha: if the node has already run with cpu-manager-policy=none, there is a state file /var/lib/kubelet/cpu_manager_state that pins the old policy; changing the arg without deleting that file makes the kubelet fail to start or ignore the new policy. You have to stop the agent, rm /var/lib/kubelet/cpu_manager_state, and start again. (The same applies to memory_manager_state.)

And the vLLM pod, to be eligible, Guaranteed with whole CPUs:

resources:
 requests:
 cpu: "16" # whole, not a weird fractional "16000m"
 memory: "200Gi"
 nvidia.com/gpu: "2" # TP=2 → 2 GPUs on the same NUMA node
 hugepages-1Gi: "16Gi"
 limits:
 cpu: "16" # == requests → Guaranteed QoS
 memory: "200Gi"
 nvidia.com/gpu: "2"
 hugepages-1Gi: "16Gi"

With this, on a node with the config above, the kubelet assigns 16 exclusive CPUs from the NUMA node where the 2 requested GPUs live, its local memory and the hugepages, or it rejects the pod if they do not fit together. The artisanal pinning of the previous post, now declarative.

The 10 knobs to reach for

Ordered by dependency (the first ones are a precondition for the later ones). The canonical reference is the Kubernetes Topology Manager documentation and the RKE2 configuration.

Knob 1 — `cpu-manager-policy=static`: the foundation

Without this there are no exclusive CPUs and nothing else kicks in.

kubelet-arg: [ "cpu-manager-policy=static" ]

Gotcha: changing it requires deleting /var/lib/kubelet/cpu_manager_state and restarting the kubelet, or startup fails. It is cause no. 1 of “I set the policy and it does not pin”.

Knob 2 — `Guaranteed` QoS + whole CPUs: the pod’s precondition

This is not node config, it is pod config, but without it knob 1 does nothing for that pod. requests == limits on CPU and memory, and whole CPUs. A cpu: 500m or a requests != limits downgrades the pod to Burstable and it loses the pinning. Plenty of people set the node policy and forget the pod’s QoS.

Knob 3 — `topology-manager-policy=single-numa-node`: strict admission

kubelet-arg: [ "topology-manager-policy=single-numa-node" ]

The strict maître d’. For GPU inference it is the right policy: it guarantees that CPU+memory+GPU share a node. best-effort guarantees nothing (it admits misaligned); restricted allows multi-node affinity. Start with single-numa-node and drop to restricted only if you have admission problems.

Knob 4 — `topology-manager-scope=pod`: group the whole pod

kubelet-arg: [ "topology-manager-scope=pod" ]

With container scope (the default), each container is aligned separately; with pod scope, the whole pod goes to the same node. For a vLLM pod with sidecars (metrics, proxy), pod scope stops the sidecar from dragging the main container onto another node. Recommended for inference.

Knob 5 — `memory-manager-policy=Static` + `reserved-memory`: NUMA-local memory

kubelet-arg:
 - "memory-manager-policy=Static"
 - "reserved-memory=0:memory=4Gi;1:memory=4Gi"

Memory Manager Static forces cpuset.mems so that the pod’s memory comes out of the right node (and so do the hugepages). It requires declaring reserved-memory per node, or the kubelet does not start. It is the declarative equivalent of the previous post’s --membind.

Knob 6 — `reserved-cpus`: the housekeeping cores (must match `isolcpus`)

kubelet-arg: [ "reserved-cpus=0-1,64-65" ]

It reserves cores for the system and the daemons; the rest are left for exclusive pods. Key point of the series: these reserved-cpus must be the same cores you left out of isolcpus on the host (previous post). If the host isolates 2-31 but RKE2 reserves 0-3, there is a mismatch: isolated cores that the kubelet assigns to pods without them actually being quiet. Coordinate the two layers.

Knob 7 — GPU plugin with NUMA topology (NVIDIA GPU Operator)

The Device Manager can only give a correct NUMA hint if the GPU plugin exposes which node each GPU is on. The NVIDIA device plugin / GPU Operator does this, but you have to verify that the topology information arrives (in some versions it requires flags). Without a GPU hint, the Topology Manager aligns CPU and memory but not the GPU, and GPU locality is precisely the one that matters most.

Knob 8 — hugepages as a pod resource

resources:
 limits:
 hugepages-1Gi: "16Gi" # the node must have them pre-reserved (post 2, knob 4)

The hugepages you reserved at host boot (previous post) are requested as a resource. The Memory Manager assigns them NUMA-local. If you request them without having reserved them on the node, the pod does not get scheduled.

Knob 9 — `system-reserved` / `kube-reserved`: do not oversubscribe

kubelet-arg:
 - "system-reserved=cpu=500m,memory=8Gi"
 - "kube-reserved=cpu=500m,memory=4Gi"

It reserves resources for the system and the K8s components so the node does not run out of air under load. Badly calibrated, either the node chokes (too little reserved) or you waste capacity (too much). It must be consistent with reserved-cpus.

Knob 10 — Labels + taints: so vLLM lands here and nothing else does

# GPU node: taint to repel anything that does not need a GPU
node-taint: [ "nvidia.com/gpu=present:NoSchedule" ]
node-label: [ "fibercli.local/numa-pinned=true" ]

Keep the NUMA-pinned nodes for inference and push out whatever does not need it (databases, the Langfuse backend, runners). A ClickHouse stealing memory bandwidth from a carefully pinned vLLM pod throws away all the work of the previous nine knobs. Workload isolation is the closing move.

Summary table

#	Knob	Where	Function
1	`cpu-manager-policy=static`	kubelet-arg	exclusive CPUs (foundation)
2	`Guaranteed` QoS + whole CPUs	pod spec	pinning precondition
3	`topology-manager-policy=single-numa-node`	kubelet-arg	strict NUMA admission
4	`topology-manager-scope=pod`	kubelet-arg	group the whole pod
5	`memory-manager-policy=Static`	kubelet-arg	NUMA-local memory/hugepages
6	`reserved-cpus`	kubelet-arg	housekeeping (match isolcpus)
7	GPU plugin with topology	GPU Operator	NUMA hint for the GPU
8	`hugepages-1Gi`	pod spec	hugepages as a resource
9	`system/kube-reserved`	kubelet-arg	do not oversubscribe
10	taints + labels	node config	isolate GPU workloads

Verifying that the pinning really happened

Do not trust that the config “is in place”. Check:

# Is the active policy the one you set?
cat /var/lib/kubelet/cpu_manager_state | jq .policyName # "static"

# Which exclusive CPUs does the container have?
kubectl exec <pod> -- cat /sys/fs/cgroup/cpuset.cpus.effective

# Inside the pod: is the assigned GPU local to those cores?
kubectl exec <pod> -- nvidia-smi topo -m

# Were there topology rejections?
kubectl describe pod <pod> | grep -i TopologyAffinityError

A pod in Failed with TopologyAffinityError is not a bug: it is the maître d’ doing his job. That node did not have a table where CPU+memory+GPU fitted together. The answer is to review the sizing of the pod or the node, not to relax the policy casually.

How it connects with the rest of the stack

With the host (previous post). This post is the declarative automation of that one. cpu-manager-policy=static materialises the taskset; memory-manager-policy=Static materialises the --membind; reserved-cpus must match isolcpus. The two layers are one single decision seen from two places: the host executes it, the kubelet declares it. Uncoordinating them (isolcpus 2-31 vs reserved-cpus 0-3) breaks both.

With the interconnect (post 1). The Topology Manager pins the right GPU to the pod, but if you ask for 2 GPUs for TP=2, you will want those two to share NVLink. The NUMA policy guarantees they are on the same socket; that they are NVLink-connected is guaranteed by the baseboard hardware (post 1, knob 1). The two things together are what makes TP=2 perform.

With autoscaling. When KEDA scales vLLM pods, each new replica goes through Topology Manager admission. If the node has no free “table”, the pod stays pending. Pod autoscaling and node autoscaling (cluster-autoscaler) have to account for NUMA granularity, not just aggregate CPU/memory.

With capacity planning. Sizing changes: it is not “128 vCPU per machine”, it is “128 minus the reserved-cpus, in blocks that fit per NUMA node”. A machine with 2 sockets × 64 cores does not serve a pod asking for 80 cores under single-numa-node: they do not fit at one table. Planning has to reason per NUMA node, not per machine.

With service coexistence. The knob 10 taint is what keeps Langfuse, databases and runners off the inference nodes. Without that boundary, all the fine-grained pinning gets eaten by a noisy neighbour. Observability goes on its own nodes; inference, pinned, on its own.

Traps and things that are not what they seem

Policy in place, QoS forgotten. The most common mistake: cpu-manager-policy=static on the node but the pod is Burstable (requests != limits or fractional CPU). Pinning does not happen and nobody warns you. Guaranteed QoS with whole CPUs is a necessary condition.

Fossilised cpu_manager_state. Changing policy without deleting /var/lib/kubelet/cpu_manager_state (and memory_manager_state) makes the kubelet fail or ignore the change. Stop agent → delete file → start.

reserved-cpus ≠ isolcpus. If the host isolates some cores and RKE2 reserves others, the managers assign cores to pods that are not really quiet, or they leave isolated cores idle. The two lists have to be consistent. It is the coordination failure between the previous post and this one.

GPU plugin without NUMA topology. If the device plugin does not expose each GPU’s NUMA node, the Topology Manager aligns CPU and memory but leaves the GPU to chance, and GPU locality is the one that weighs most. Verify that the GPU Operator publishes the topology.

single-numa-node that rejects too much. If the pods ask for more resources than fit on one node (for example, more cores than a socket has), rejection is constant. The answer is not to drop to best-effort (which silences the problem by serving misaligned), but to size the pod so it fits at one table, or to accept restricted with full knowledge of what you are doing.

Believing best-effort “is nearly the same”. best-effort admits the pod even if it fails to align: it gives you a false sense of NUMA-awareness while you serve from the wrong socket. For inference with a tail SLO, single-numa-node or restricted; best-effort only if the alternative is scheduling nothing at all.

Conclusion

The previous post pinned by hand; this one does it at scale and with a guarantee that artisanal numactl did not give: strict admission. The kubelet, via CPU Manager, Memory Manager and Topology Manager, acts as a maître d’ who only seats the pod if its CPUs, its memory and its GPU fit at the same NUMA table, and who rejects what does not fit instead of serving a split dinner. Of the ten knobs, the first two, cpu-manager-policy=static and Guaranteed QoS with whole CPUs, are the precondition without which the other eight do nothing, and they are exactly the ones most often forgotten; the rest tune the policy, the memory, the hugepages and the coexistence. The thread that closes the series: the inference performance that looked like an engine problem (vLLM is slow) or a model problem (quantisation) is, far too often, a cable problem (NVLink not used), a host problem (remote NUMA, jitter) or an orchestration problem (pinning that never happened because the QoS was wrong). Going down a level is not infrastructure snobbery: it is where the root causes live that no application-layer dashboard is ever going to point out for you.

References

Kubernetes, Control Topology Management Policies on a node: https://kubernetes.io/docs/tasks/administer-cluster/topology-manager/.
Kubernetes, Control CPU Management Policies on the Node: https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/.
Kubernetes, Control Memory Management Policies on a Node: https://kubernetes.io/docs/tasks/administer-cluster/memory-manager/.
RKE2, Configuration Options (kubelet-arg in config.yaml): https://docs.rke2.io/install/configuration.
RKE2, Advanced Options and Configuration: https://docs.rke2.io/advanced.
rancher/rke2, discussion #3034 CPU Management Policies for RKE2 (the cpu_manager_state gotcha): https://github.com/rancher/rke2/discussions/3034.

The floor next door: NUMA, hugepages and CPU isolation, or why your GPU waits on the kernel

Sat, 06 Jun 2026 07:30:00 +0200

Second post in the “below the engine” series. The first opened up the cable between GPUs (NVLink/NCCL). This one goes down to the host: the cores, the memory and the kernel that surround those GPUs and that, badly configured, leave them waiting. The third will explain how Kubernetes automates all of this; here is the raw layer, the one you have to understand before delegating it.

TL;DR

A node with 4×H100 SXM is, physically, a two-socket server = two NUMA domains. Each socket has its own cores, its own memory channels and PCIe lanes towards half of the GPUs and NICs. Inference is not only GPU: the host does work on the hot path of every token, launching CUDA kernels, sampling the next token, tokenising, moving pinned buffers between host and GPU, running the NCCL threads. If those threads and their memory land on the socket that is not local to the GPU, every access crosses the inter-socket link (UPI/Infinity Fabric): 2-3× slower and with p99 spikes. There are three kernel levers that decide the latency tail and that almost nobody touches: locality (NUMA affinity: CPU, memory, GPU and NIC on the same “floor”), page tables (hugepages: a few large pages instead of millions of small ones, plus pinned memory for DMA), and jitter (CPU isolation with isolcpus/nohz_full/rcu_nocbs plus IRQ affinity, so that the kernel does not interrupt the thread launching the next decode kernel). This post explains the mechanism, gives the 10 real knobs, and connects with the interconnect and latency-bound decode. With scepticism about which knobs move the needle in inference and which are cargo cult inherited from low-latency trading.

Where you are: the host below the cable

The analogy: the two-floor office

Picture a consultancy in a building with two floors. On each floor there are desks (CPU cores), an archive with the files (the memory of that socket) and a loading bay connecting to the outside (the PCIe lanes towards the GPUs and NICs of that socket). An analyst works fast while everything they need is on their floor: they reach out and take the file from the archive next to them.

The problem starts when the analyst is on floor 1 but their file is in the archive on floor 2. Every time they need it, they take the lift. The work “functions”, but every lookup costs a trip. And if on top of that the loading bay through which their materials arrive (their GPU) is on the other floor, every delivery crosses the building. This is NUMA: local access (same floor) is fast; remote access (another floor, via the inter-socket link) is 2-3× slower.

And there are two more ways to ruin that analyst even when they are on the right floor:

Interrupting them constantly. Every few minutes, a tannoy announcement, a colleague with a question, a fire alarm test. Every interruption breaks their concentration right when they were about to deliver. This is kernel jitter: the scheduler tick, device IRQs, RCU callbacks, interrupting the host thread right when it was about to launch the next GPU kernel. CPU isolation is putting them in an office with a “do not disturb” sign.
Giving them an index of a thousand tiny tabs. If to find each file they have to search an index with a million minuscule entries, they lose time in the search. If the index has a few large entries, they find it instantly. These are hugepages: pages of 2 MB or 1 GB instead of 4 KB reduce the pressure on the TLB (the cache of the page index).

The thesis: the GPU is expensive and fast, but it spends a surprising fraction of decode waiting on the host. If the host is on the wrong floor, interrupted, and searching a giant index, the GPU, the 30,000 € resource, waits. The three levers in this post exist so that it does not wait.

The mechanism: what the host does on the token path

It is tempting to think that in inference “the GPU does everything”. It is not true. For every token, the host (CPU) does, at a minimum:

Launch the CUDA kernels for each operation. The GPU does not decide what to execute; the host keeps putting kernels in its queue. In decode, where each kernel is short, the host has to run ahead feeding the queue; if the host thread stalls, the GPU runs out of work: a bubble.
Sample the next token (argmax/top-p/top-k over the logits), which comes back from device to host.
Tokenise the input and detokenise the output.
Move pinned buffers (page-locked) between host and GPU by DMA: prompts, logits, and in offload configurations, part of the KV cache.
Run the NCCL threads that coordinate the collectives of the previous post.

All of that is CPU and host memory work. And all of it suffers if:

The process runs on the socket that is not local to its GPU → every DMA and every memory access crosses the inter-socket link.
The kernel interrupts the threads → bubbles in the GPU queue.
The memory is not pinned or uses small pages → page faults, TLB misses, and worse: if the memory of the DMA transfer is not pinned, the driver makes an intermediate copy.

The map: `nvidia-smi topo -m`

Everything starts by looking at the map. nvidia-smi topo -m shows, for each GPU, which NUMA node and which cores it is local to, and through what kind of path it talks to each NIC and to each other GPU:

 GPU0 GPU1 GPU2 GPU3 NIC0 CPU Affinity NUMA Affinity
GPU0 X NV18 NV18 NV18 PIX 0-31,64-95 0
GPU1 NV18 X NV18 NV18 SYS 0-31,64-95 0
GPU2 NV18 NV18 X NV18 SYS 32-63,96-127 1
GPU3 NV18 NV18 NV18 X SYS 32-63,96-127 1

Read it like this: GPU0 and GPU1 are local to NUMA node 0 (cores 0-31, 64-95); GPU2 and GPU3 to NUMA node 1. NV18 between GPUs = 18 NVLink links (the good stuff, from the previous post). In the NIC column: PIX = a single PCIe switch in between (optimal for GPUDirect RDMA); SYS = the path crosses the inter-socket link (the worst). The rule: the process serving on GPU0/1 must be pinned to cores 0-31/64-95 and to the memory of node 0; and if it also uses RDMA, you want the NIC that is on PIX with its GPU.

The maths that matter: a kernel tick is a GPU bubble

The figure worth internalising is not the bandwidth one, it is the jitter one. In decode, the host launches many short kernels per token. If the host thread that launches them is preempted by the scheduler (a timer tick, an IRQ, an RCU callback) for $t_{\text{stall}}$, and the GPU drains its queue in that time, a bubble appears: the GPU stops.

Put numbers on it. A typical timer tick or the handling of an IRQ costs of the order of tens of microseconds of diversion. If a decode kernel lasts around 50-100 µs and the queue carries 2-3 kernels in flight, a host stall of 50-100 µs drains the queue and the GPU sits idle until the host resumes. Multiply by the interrupt frequency of a non-isolated kernel (the default tick is 250-1000 Hz, plus network and disk IRQs): the p99/p999 tail of TTFT and of inter-token time fills up with these episodes.

$$ \text{jitter}_{p99} \approx f_{\text{interrupts}} \times t_{\text{stall}} \times \mathbb{1}[\text{GPU queue drained}] $$

The intuition: in average throughput it is barely noticeable (the bubbles average out), but in the tail, which is what an SLO measures, kernel jitter is a first-order contributor. That is why CPU isolation, which was born in low-latency trading, makes sense in LLM decode: it is the same physics, a critical thread that cannot afford the kernel stopping it.

And the NUMA cost, in parallel: a remote memory access (another floor) has latency around 1.5-2× the local one and half the bandwidth. For the pinned buffers moved by DMA on every step, and for the vLLM scheduler structures that live on the host, that penalty is paid token by token.

The three levers, one by one

Locality (NUMA): getting everything on the same floor

The goal is that the inference process using GPU0/1 has its cores, its memory and (if applicable) its NIC on NUMA node 0. Raw, without Kubernetes:

# Pin process to node 0 (cores and memory) to serve on GPU0/1
numactl --cpunodebind=0 --membind=0 \
 vllm serve meta-llama/Llama-3-70B --tensor-parallel-size 2

--membind=0 is the key: it forces all the process memory to be allocated on node 0. Without --membind, the kernel can place pages on node 1 under pressure, and you start paying for the lift without knowing it.

Page tables (hugepages): a few large pages and pinned memory

Two different things under the same umbrella. First, hugepages reduce TLB pressure for the large host buffers (pinned, KV offload). Second, pinned memory (page-locked) is what allows direct DMA with no intermediate copy. The silent trap is transparent hugepages (THP): their background compaction causes latency spikes, exactly what you do not want.

Jitter (CPU isolation): the office with “do not disturb”

Three kernel boot parameters, coordinated:

isolcpus=2-31,66-95 # take these cores out of the scheduler's balancing
nohz_full=2-31,66-95 # tickless: no timer tick if there is only 1 runnable thread
rcu_nocbs=2-31,66-95 # offload RCU callbacks to housekeeping cores

isolcpus sets the cores aside; you have to pin the inference threads there (the non-isolated cores, 0-1, are left for the system). nohz_full removes the periodic tick (it only works if there is a single runnable thread on the core). rcu_nocbs takes RCU work off those cores. And separately, IRQ affinity: moving device interrupts away from the inference cores.

The 10 knobs worth touching

Ordered by impact/frequency. Almost all of them are sysctls, kernel boot parameters or numactl. The low-level reference is Rigtorp’s low-latency guide and Red Hat’s real-time docs.

Knob 1 — `nvidia-smi topo -m`: look at the map before touching anything

Just as in the interconnect post: the map first. Which GPU is local to which NUMA node and to which cores, and what path (PIX/PHB/SYS) there is to each NIC. Without this, any pinning is blind. Half of the “inference has latency spikes” problems are processes running on the wrong socket without anyone having looked.

Knob 2 — `numactl --cpunodebind --membind`: pin to the local node

numactl --cpunodebind=0 --membind=0 <process> # cores AND memory on node 0
numactl --hardware # see nodes, distances, free memory

The --membind is what really matters: without it, the memory scatters. It is the highest-impact knob on sustained throughput.

Knob 3 — `kernel.numa_balancing=0`: turn off automatic migration

sysctl -w kernel.numa_balancing=0

The kernel’s automatic NUMA balancing migrates pages between nodes trying to “bring them closer”, but that background work causes jitter and, with the memory already pinned by knob 2, it adds nothing. On dedicated inference nodes, turn it off.

Knob 4 — Explicit hugepages (1 GB) for host buffers

# Kernel boot, for KV offload / large pinned buffers
default_hugepagesz=1G hugepagesz=1G hugepages=32

grep Huge /proc/meminfo # verify the reservation

Useful when there is host memory on the hot path (vLLM with --cpu-offload-gb, or large pinned buffers). If your deployment does not touch host memory in the hot path, explicit hugepages add little, so do not set them out of cargo cult.

Knob 5 — THP on `madvise` or `never`: avoid the compaction spikes

echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

Transparent hugepages on always save TLB but their compaction triggers unpredictable latency. For tail-sensitive workloads, madvise (only where the app asks for it) or never is what is recommended. It is one of the few knobs with clear consensus: THP always is bad for latency.

Knob 6 — `isolcpus`: set the inference cores aside from the scheduler

isolcpus=2-31,66-95

Takes those cores out of the scheduler’s load balancing; the system (kernel threads, daemons) stays on the non-isolated ones. You have to pin the inference threads explicitly to the isolated cores (via numactl/taskset or, on K8s, the CPU Manager of the next post). Isolating without pinning is useless.

Knob 7 — `nohz_full` + `rcu_nocbs`: tickless and no RCU on the critical cores

nohz_full=2-31,66-95 rcu_nocbs=2-31,66-95

Removes the periodic timer tick and the RCU callbacks from the inference cores. Two warnings from practice: nohz_full only eliminates the tick if there is a single runnable thread on the core (if you pin two threads there, the tick comes back); and nohz_full is not compatible with the intel_pstate driver in some configurations, which has to be validated, not assumed.

Knob 8 — IRQ affinity: interrupts, away from the inference cores

systemctl stop irqbalance # or configure it to respect isolcpus
# move a device's IRQs to the housekeeping cores (0-1)
echo 3 > /proc/irq/<N>/smp_affinity # mask for cores 0-1

A network or disk IRQ landing on an inference core is a direct interruption of the thread feeding the GPU. Move them to the housekeeping cores. (irqbalance can respect isolcpus automatically if it is configured to.)

Knob 9 — CPU governor `performance` + C-states

cpupower frequency-set -g performance
# stop idle cores entering deep C-states (wakeup latency)
cpupower idle-set -D 0 # or limit the C-state depth

With the powersave/ondemand governor, a core that was idle takes time to ramp up its frequency: wakeup latency exactly when work arrives. performance keeps it flat out. On servers dedicated to inference, the energy saving does not compensate for the latency tail.

Knob 10 — Memory locking + `swappiness=0`

sysctl -w vm.swappiness=0 # do not evict inference pages to swap
# and in the app / container: ulimit -l unlimited (memlock) for pinned memory

An inference page that the kernel decides to swap to disk is a page fault of milliseconds on the hot path. swappiness=0 and adequate memlock limits (so the driver can pin memory) close that door. On K8s, this translates into Guaranteed QoS and memory limits, the bridge to the next post.

Summary table

#	Knob	Mechanism	What it attacks
1	`nvidia-smi topo -m`	diagnosis	see GPU–NUMA–NIC affinity
2	`numactl --cpunodebind --membind`	pinning	locality (the biggest lever)
3	`kernel.numa_balancing=0`	sysctl	jitter from page migration
4	explicit 1G hugepages	boot param	TLB on host buffers
5	THP `madvise`/`never`	sysfs	compaction spikes
6	`isolcpus`	boot param	scheduler off the critical cores
7	`nohz_full`+`rcu_nocbs`	boot param	tick + RCU jitter
8	IRQ affinity	`/proc/irq`	device interrupts
9	`performance` governor	cpupower	frequency wakeup latency
10	`swappiness=0` + memlock	sysctl/ulimit	page faults on the hot path

How it connects with the rest of the stack

With the interconnect (previous post). The NCCL host threads want cores local to their GPU; and for multi-node, the RDMA NIC must be on the PIX path with its GPU (knob 1). A GPUDirect RDMA with the NIC under the other socket loses half its advantage. NUMA and NVLink are the same story seen from the host and from the cable.

With vLLM and decode. Decode is latency-bound: the host thread feeding the kernel queue is exactly the one CPU isolation protects. And vLLM’s --cpu-offload-gb puts host memory on the hot path, where NUMA locality plus hugepages (knobs 2, 4) go from “nice” to “critical”. Continuous batching helps here too: more tokens per iteration amortise both the collective latency and the fixed cost of the host launches.

With Kubernetes (next post). Everything in this post is done by hand (numactl, taskset, boot parameters). In production it is not done by hand: the kubelet automates it with CPU Manager, Memory Manager and Topology Manager. The next post is exactly how this is declared so that every vLLM pod is born pinned to the right NUMA node, with no scripts.

With observability. The p99 spikes from jitter or from remote access are visible: in DCGM, low GPU utilisation with a full queue (bubbles); in system metrics, inter-socket traffic and CPU migrations. GPU observability with DCGM is where you diagnose a “the GPU is at 60 % and I do not know why” that many times is the host waiting.

With capacity planning. Reserving cores for the system (housekeeping) and dedicating the rest to inference changes the calculation of how many pods/replicas fit per node. Capacity planning has to count those reserved cores, not assume that all 128 vCPUs are available for serving.

Traps and things that are not what they look like

Cargo cult from low-latency trading. Many isolcpus/nohz_full guides come from HFT, where the last microsecond is squeezed out. In LLM inference, CPU isolation does help on the decode tail, but do not expect a miracle: if your bottleneck is HBM bandwidth or the interconnect, isolating cores does not move the needle. Measure first; apply it where the host is the limit.

Isolating without pinning. isolcpus takes the cores out of the scheduler, but if you do not pin the inference threads there, those cores stay empty and inference runs on the housekeeping ones, worse than before. Isolating and pinning always go together.

Forgotten --membind. Pinning cores but not memory (--cpunodebind without --membind) lets the pages scatter to the other node under pressure. Memory pinning is the half that gets forgotten most and the one that pays off most.

THP always “because it saves TLB”. It saves TLB and throws in latency spikes from compaction. For workloads with a tail SLO, it is a bad trade. madvise/never.

nohz_full with two threads on the core. Tickless only works with a single runnable thread per core. If you pin two inference threads to the same isolated core, the tick comes back and you have complicated the kernel boot for nothing.

Assuming the topology instead of reading it. Different servers wire GPUs and NICs to different sockets. nvidia-smi topo -m and numactl --hardware are the truth; the manufacturer’s diagram is indicative. Read it on every node model.

Conclusion

The GPU is the expensive resource, but in decode it spends a surprising part of its time waiting on the host: waiting for it to launch the next kernel, to sample, to move a buffer. If that host is on the wrong floor (remote NUMA), interrupted (kernel jitter) or searching a giant index (4 KB pages), the GPU is left with an empty queue and p99 blows up, without any API dashboard saying why. Of the ten knobs, the first (reading the map with nvidia-smi topo -m) and the second (pinning cores and memory to the local node with --membind) solve most of it; CPU isolation (isolcpus/nohz_full/IRQ affinity) is the second layer, the one that trims the decode tail, and it makes sense where the host is the limit, not as a ritual. The idea that reorders the intuition: inference is not “all GPU”; it is a dance between GPU and host, and the host dances better close by, uninterrupted, and with few large pages. The next post shows how Kubernetes choreographs that dance for every pod without a single hand-written script.

References

Erik Rigtorp, Low Latency Tuning Guide (isolcpus, nohz_full, IRQ affinity, THP): https://rigtorp.se/low-latency-guide/.
Red Hat, Usage, constraints and implications of isolcpus=, nohz_full= and rcu_nocbs=: https://access.redhat.com/articles/3720611.
NVIDIA, nvidia-smi topo -m and the GPU–NUMA–NIC affinity matrix (CUDA docs / Developer Forums): https://forums.developer.nvidia.com/t/nvidia-smi-topo-m-revisited/216584.
Chaim Rand, The Crucial Role of NUMA Awareness in High-Performance Deep Learning: https://chaimrand.medium.com/the-crucial-role-of-numa-awareness-in-high-performance-deep-learning-99ae3e8eb49a.
SUSE Labs, CPU Isolation – nohz_full (part 3): https://www.suse.com/c/cpu-isolation-nohz_full-part-3/.
Linux kernel, Automatic NUMA Balancing and Transparent Hugepage Support (Documentation/admin-guide): https://docs.kernel.org/admin-guide/mm/transhuge.html.

Numa on lo0 — Tech Blog

The kitchen door the maître d' never looked at: network NUMA, Cilium eBPF and DRANET, the fourth leg of pinning

TL;DR

Where you are: the network plane the trilogy did not open

The analogy: the door the dishes come through

The gap: why the Topology Manager does not look at the NIC

The network datapath under NUMA: IRQ, softirq and DMA

One number, with its caveat

What RKE2’s Cilium eBPF replaces (and why it touches this story)

Cilium performance profile (state 1.19, kernel ≥6.8)

The 2026 state of the art: DRA and DRANET, the maître d’ who finally looks at the door

How it connects with the rest of the stack

Traps and things that are not what they look like

Conclusion

See also

References

The maître d' who only seats you if you all fit at one table: CPU, Memory and Topology Manager on RKE2

TL;DR

Where you are: the orchestration that materialises the host

The analogy: the maître d’ of a restaurant whose tables cannot be pushed together

The mechanism: Hint Providers and the coordinator

How it is configured on RKE2

The 10 knobs to reach for

Knob 1 — cpu-manager-policy=static: the foundation

Knob 2 — Guaranteed QoS + whole CPUs: the pod’s precondition

Knob 3 — topology-manager-policy=single-numa-node: strict admission

Knob 4 — topology-manager-scope=pod: group the whole pod

Knob 5 — memory-manager-policy=Static + reserved-memory: NUMA-local memory

Knob 6 — reserved-cpus: the housekeeping cores (must match isolcpus)

Knob 7 — GPU plugin with NUMA topology (NVIDIA GPU Operator)

Knob 8 — hugepages as a pod resource

Knob 9 — system-reserved / kube-reserved: do not oversubscribe

Knob 10 — Labels + taints: so vLLM lands here and nothing else does

Summary table

Verifying that the pinning really happened

How it connects with the rest of the stack

Traps and things that are not what they seem

Conclusion

See also

References

The floor next door: NUMA, hugepages and CPU isolation, or why your GPU waits on the kernel

TL;DR

Where you are: the host below the cable

The analogy: the two-floor office

The mechanism: what the host does on the token path

The map: nvidia-smi topo -m

The maths that matter: a kernel tick is a GPU bubble

The three levers, one by one

Locality (NUMA): getting everything on the same floor

Page tables (hugepages): a few large pages and pinned memory

Jitter (CPU isolation): the office with “do not disturb”

The 10 knobs worth touching

Knob 1 — nvidia-smi topo -m: look at the map before touching anything

Knob 2 — numactl --cpunodebind --membind: pin to the local node

Knob 3 — kernel.numa_balancing=0: turn off automatic migration

Knob 4 — Explicit hugepages (1 GB) for host buffers

Knob 5 — THP on madvise or never: avoid the compaction spikes

Knob 6 — isolcpus: set the inference cores aside from the scheduler

Knob 7 — nohz_full + rcu_nocbs: tickless and no RCU on the critical cores

Knob 8 — IRQ affinity: interrupts, away from the inference cores

Knob 9 — CPU governor performance + C-states

Knob 10 — Memory locking + swappiness=0

Summary table

How it connects with the rest of the stack

Traps and things that are not what they look like

Conclusion

See also

References

Knob 1 — `cpu-manager-policy=static`: the foundation

Knob 2 — `Guaranteed` QoS + whole CPUs: the pod’s precondition

Knob 3 — `topology-manager-policy=single-numa-node`: strict admission

Knob 4 — `topology-manager-scope=pod`: group the whole pod

Knob 5 — `memory-manager-policy=Static` + `reserved-memory`: NUMA-local memory

Knob 6 — `reserved-cpus`: the housekeeping cores (must match `isolcpus`)

Knob 9 — `system-reserved` / `kube-reserved`: do not oversubscribe

The map: `nvidia-smi topo -m`

Knob 1 — `nvidia-smi topo -m`: look at the map before touching anything

Knob 2 — `numactl --cpunodebind --membind`: pin to the local node

Knob 3 — `kernel.numa_balancing=0`: turn off automatic migration

Knob 5 — THP on `madvise` or `never`: avoid the compaction spikes

Knob 6 — `isolcpus`: set the inference cores aside from the scheduler

Knob 7 — `nohz_full` + `rcu_nocbs`: tickless and no RCU on the critical cores

Knob 9 — CPU governor `performance` + C-states

Knob 10 — Memory locking + `swappiness=0`