The kitchen door the maître d' never looked at: network NUMA, Cilium eBPF and DRANET, the fourth leg of pinning

· By José Manuel Román
Contents

Fourth instalment, a coda, of “under the engine”. The series closed with three legs of locality: the cable between GPUs, the host by hand and the declarative orchestration of the kubelet. But the maître d’ of the last post seated the group looking at CPU, memory and GPU, and never asked which door the dishes come through. That door is the NIC. Here is the fourth leg.

TL;DR

The Topology Manager admits a pod under single-numa-node if its CPUs, its memory and its GPU fit on the same NUMA node. The NIC is not part of that count: the kubelet has no Hint Provider for the network card. On an inference node with the network at 200/400 Gb/s, the case of disaggregated serving, where the KV-cache travels over RDMA between the prefill pool and the decode pool, a NIC on the wrong socket makes every packet cross the UPI/QPI, exactly the “remote NUMA” the series fights on the compute side, but coming in through the network door. And there is a second front: the softirq (NET_RX) that processes the datapath runs on the CPU that services the NIC’s IRQ; if that CPU is one of the cores isolcpus/reserved-cpus gave exclusively to vLLM, the softirq steals cycles from it and injects jitter into the p99 tail. Cilium eBPF replaces two RKE2 pieces, kube-proxy (with eBPF/XDP load balancing) and the default CNI Canal (with a native datapath), and its own tuning guide tells you to kill irqbalance and pin the NIC IRQs: a fourth list to align alongside isolcpus and reserved-cpus. The 2026 state of the art closes the gap from above: netkit (kernel ≥6.8, zero namespace overhead), BIG TCP (192k super-packets for 100Gb/s+), host-routing (iptables bypass), and above all DRA/DRANET, the network driver that finally co-schedules NUMA-local GPU and NIC on the same PCIe root, enabling GPUDirect RDMA with +59.6% bus bandwidth in all_gather and +58.1% in all_reduce. On a generic RKE2 cluster with 4×H100 SXM nodes.

Where you are: the network plane the trilogy did not open

The vertical stack · the fourth leg of localityEngine · vLLM pod (TP, batching, KV-cache)Orchestration · kubelet: CPU/Mem/Topology Mgr (post 3)YOU ARE HERE · network plane: Cilium eBPF + DRA/NICNIC NUMA locality · IRQ · GPUDirect RDMAHost · NUMA, hugepages, isolcpus (post 2)CUDA + NCCL + NVLink (post 1)Hardware · 2 sockets, 4×H100 SXM, 400 Gb/s NICCPU+memory+GPU are pinned by the kubelet; the NIC, until 2026, was pinned by nobody

The analogy: the door the dishes come through

Go back to the restaurant of the previous post. The maître d’, the Topology Manager, seated the party of eight at a single table (one NUMA node) because the diners (CPUs), the cutlery (memory) and the reserved bottle (the GPU) all fitted. A perfect table. But the maître d’ never looked at where the kitchen pass is: the door every dish comes in and goes out through.

That door is the NIC. The prompt comes in there, the tokens go out there, and, in disaggregated serving, the KV-cache that the prefill pool sends to the decode pool circulates there. If the table is in the left-hand room (socket 0) but the kitchen pass is in the right-hand one (socket 1), every dish crosses the entire restaurant (the UPI/QPI), over and over, however impeccably the table is laid. The diner does not notice the perfect table: they notice the dish arriving late and cold.

And there is a finer detail: the waiter who crosses the room with the dishes (the softirq that processes the packets) is one of the seated diners. If the maître d’ gave them a chair of their own so they could eat in peace (a core isolated with isolcpus for vLLM) but the restaurant also has them waiting on the far door, that diner does not eat: they spend the whole dinner crossing the room. The jitter shows up exactly where you thought you had bought calm.

The trilogy levelled three legs of the table: the cable, the host and the orchestration. The fourth, which door the dishes come through and who carries them, is levelled by no kubelet manager. Until 2026.

The gap: why the Topology Manager does not look at the NIC

The mechanism of post 3 is a coordinator (Topology Manager) that consults three Hint Providers: CPU Manager, Memory Manager and Device Manager (the GPU plugin). Each says on which NUMA node it can satisfy its part; the coordinator computes the intersection and admits or rejects.

The problem is one of census: the classic NIC is not a Device Manager “device”. A standard Ethernet/InfiniBand card is managed by the CNI and the kernel, it is not requested in the pod’s resources: like nvidia.com/gpu, and therefore it emits no NUMA hint. The Topology Manager aligns CPU+memory+GPU and leaves the NIC wherever the hardware put it, which may be the other socket. The maître d’ has three assistants and is missing the fourth: the one who knows which door the dishes come through.

This did not matter when a node’s network was 10/25 Gb/s and the bottleneck was elsewhere. It matters now, with two workloads that saturate the node’s network:

In both, where the NIC sits relative to the GPU and to the pod’s cores decides performance. And the kubelet, on its own, does not coordinate that.

The network datapath under NUMA: IRQ, softirq and DMA

To see why NIC locality carries weight, you have to look at the path of an incoming packet:

Socket 0 (NIC here) vs Socket 1 (vLLM pod here): the crossing you do not seeNUMA node 0 · PCIe root with the NICNIC 400 Gb/smulti-queue, RSSIRQ + softirqNET_RX on core 0..RAM node 0packet DMANUMA node 1 · the kubelet pinned the pod herevLLM pod + GPUexclusive CPUs node 1every packet crosses the UPI/QPI+latency, +load on the inter-socket linkThe Topology Manager did its job on node 1; the NIC stayed on node 0. Nobody aligned the two.

Three kernel facts the analogy compresses:

  1. The IRQ has affinity. Each NIC queue raises an interrupt that the kernel services on a specific CPU (/proc/irq/<n>/smp_affinity). The heavy processing is deferred to a softirq (NET_RX/NET_TX), which runs on that same CPU. If irqbalance is left loose, it migrates them non-deterministically, poison for the p99.
  2. The softirq competes with the pod. If the IRQ lands on a core that isolcpus reserved for vLLM, that queue’s NET_RX steals cycles from the model. The signal in /proc/softirqs: a NET_RX column that spikes on a single CPU. It is the same jitter from post 2, coming in through the network.
  3. DMA has a NUMA origin. The NIC writes the packet by DMA into the RAM of the socket of its PCIe root. If the consumer (the pod’s thread) is on the other socket, it reads across the UPI/QPI. RFS (Receive Flow Steering) tries to bring the processing to the consumer’s CPU, but it cannot teleport the NIC to the other socket.

One number, with its caveat

Take a 2-socket node, a 400 Gb/s = 50 GB/s NIC on the PCIe root of socket 0, and a decode pod pinned to socket 1. If the NIC saturates, those ~50 GB/s of receive traffic cross the UPI towards socket 1. A UPI 2.0 link runs at around 20–40 GB/s per direction and link depending on generation; even with several links, 50 GB/s of network traffic going against the grain eats a far from negligible fraction of the inter-socket budget, the same budget already contended by the pod’s remote memory accesses and, if there is multi-node traffic, the KV-cache of the disaggregation. I am not giving a firm “X% degradation” because it depends on CPU generation, number of UPI links, MTU and traffic pattern; without that methodology, any exact figure is marketing.

What is measured with public methodology is the aggregate effect of aligning GPU and NIC: the DRANET project reports +59.6% bus bandwidth in all_gather and +58.1% in all_reduce (NCCL collectives) when the assigned NIC is NUMA-local to the GPU rather than not. That is the size of the gap the Topology Manager was leaving open.

What RKE2’s Cilium eBPF replaces (and why it touches this story)

RKE2 ships Canal (Flannel + Calico) by default as the CNI and kube-proxy (iptables/IPVS rules) for Service balancing. Switching to Cilium (cni: cilium in /etc/rancher/rke2/config.yaml) replaces both pieces with an eBPF datapath:

RKE2 pieceWhat it doesWhat Cilium eBPF puts in
kube-proxy (iptables/IPVS)Service balancingLB in eBPF; with kubeProxyReplacement=true, and acceleration in XDP (driver layer)
Canal (Flannel+Calico)VXLAN overlay + NetworkPolicynative datapath (routingMode=native), L3/L4 and L7 NetworkPolicy in eBPF
veth per podnamespace interface pairnetkit (kernel ≥6.8): namespace overhead ~0
host iptables traversalnetfilter hookseBPF host-routing: bypasses iptables and the upper part of the stack

Up to here this is pure networking and it does not touch the kubelet’s resource managers: Cilium does not assign exclusive CPUs nor emit compute NUMA hints. The ten knobs of post 3 stay identical whether you run Canal or Cilium.

But Cilium does enter the fourth leg through two doors. The first: its own tuning guide recommends, literally, “kill irqbalance and pin the NIC IRQs to specific CPUs for maximum workload isolation”, along with the tuned network-latency profile, the performance governor and CONFIG_PREEMPT_NONE. In other words: the eBPF datapath only really performs if you coordinate IRQ affinity, and that affinity has to point at the housekeeping cores (reserved-cpus), never at the isolated ones. So a fourth list appears that must be kept consistent with isolcpus and reserved-cpus:

isolcpus      = 2-31,34-63      # cores exclusive to vLLM (host, post 2)
reserved-cpus = 0-1,32-33       # kubelet housekeeping (post 3)
IRQ affinity  = 0-1,32-33       # NIC IRQs → housekeeping ONLY (this post)
                                  # never 2-31: there the softirq would steal from the model

The second door: netkit + host-routing + BIG TCP reduce how many times the packet crosses the stack and the namespace, which cushions (does not eliminate) the cost of the NUMA crossing. BIG TCP assembles super-packets of up to 192k (against 64k) for 100Gb/s+; fewer stack traversals means less softirq work on the core, and therefore less pressure on the inter-socket budget. It is the continuous batching analogy applied to the network stack: amortising a fixed cost over larger batches.

Cilium performance profile (state 1.19, kernel ≥6.8)

# Helm, recommended performance profile (summary of the tuning guide)
helm install cilium cilium/cilium --version 1.19.4 \
  --namespace kube-system \
  --set routingMode=native \
  --set bpf.datapathMode=netkit \        # namespace overhead ~0 (kernel >=6.8)
  --set bpf.masquerade=true \
  --set kubeProxyReplacement=true \      # replaces RKE2's kube-proxy
  --set enableIPv4BIGTCP=true \          # 192k super-packets (mlx5/ice NIC)
  --set enableIPv6BIGTCP=true \
  --set bpf.distributedLRU.enabled=true \# per-CPU BPF maps: less spinlock contention
  --set bpf.mapDynamicSizeRatio=0.08 \
  --set bpfClockProbe=true

# Verification inside a Cilium pod:
cilium status --verbose | grep -E "Device Mode|Host Routing|BIG TCP|XDP"
# Device Mode: netkit · Host Routing: BPF · IPv4 BIG TCP: enabled · XDP Acceleration: Native

Sceptical caveat: netkit and BIG TCP are beta and require kernel ≥6.8 and specific NICs (mlx4/mlx5/ice). They are not in-place: they change datapath fundamentals and force you to restart pods or, better, to apply them by per-node config only on new nodes. For an ENS cluster in production, that is a maintenance window, not a blind helm upgrade.

The 2026 state of the art: DRA and DRANET, the maître d’ who finally looks at the door

What closes the gap at the root is not Cilium, it is the admission mechanism the kubelet did not have for the NIC: Dynamic Resource Allocation (DRA), beta since Kubernetes 1.32 and with advances in every release up to 1.36 (May 2026). DRA generalises the “devices” model beyond the GPU: a driver discovers the hardware, publishes ResourceSlices with its attributes, including the NUMA topology and the PCIe root, and the scheduler resolves ResourceClaims that can demand affinity between devices.

DRANET (a kubernetes-sigs project) is the network DRA driver. It discovers the NICs (including the RDMA-capable ones), advertises them as ResourceSlices, and via NRI injects them into the pod’s namespace, compatible with whatever CNI you already have, Cilium included. The key piece for this story: combined with the NVIDIA GPU DRA driver, it allows co-scheduling GPU and NIC that share a PCIe root (the relationship NVIDIA calls NODE), which is exactly the condition for GPUDirect RDMA. The maître d’ finally has a fourth assistant: “is there a NIC NUMA-local to this GPU?”.

The ResourceClaimTemplate uses CEL selectors to request exactly that alignment:

# Request an RDMA NIC NUMA-local to the assigned GPU (illustrative DRANET/DRA schema)
apiVersion: resource.k8s.io/v1beta1
kind: ResourceClaimTemplate
metadata:
  name: gpu-nic-numa-aligned
spec:
  spec:
    devices:
      requests:
        - name: rdma-nic
          deviceClassName: dra.net          # NICs published by DRANET
      constraints:
        - requests: ["rdma-nic"]
          matchAttribute: "dra.net/pcieRoot"  # same PCIe root as the GPU
          # → enables GPUDirect RDMA over a NUMA-local path

Why it matters for inference, not for abstract “AI training”: in disaggregated serving, RDMA is what moves the KV-cache between the prefill pool and the decode pool with the latency that TTFT demands; and in multi-node, GPUDirect RDMA replaces NVLink as the medium of the collective. Aligning GPU+NIC on the same PCIe root is what turns an “RDMA that works” into an “RDMA that performs”, the +60% bus bandwidth from DRANET.

Status and caveats: DRA is beta (gates to enable by hand), DRANET is young (a SIG project, still evolving) and the managed offering exists mostly in cloud (GKE managed DRANET in preview, AKS for RDMA). For on-premise ENS it is a path, not a finished product: the value today is understanding that the fourth leg already has a standard OSS mechanism, and starting to pilot it on a lab node, not putting it into critical production this quarter.

How it connects with the rest of the stack

With the host (post 2). The NIC’s IRQ affinity is a third list to marry with isolcpus and reserved-cpus. The IRQs go to housekeeping; the isolated cores, untouched. Uncoordinating them lets in through the network door the jitter that isolcpus threw out through the compute one.

With the orchestration (post 3). DRA is the natural extension of the Topology Manager: the same “admit only if it fits on the NUMA node” principle carried over to the NIC. Where the Device Manager left the network out of the census, DRANET puts it in.

With the interconnect (post 1). Inside the node NVLink rules; on crossing the node boundary, GPUDirect RDMA over the NIC is the medium of the collective. The kubelet’s NUMA policy guarantees that GPU and CPUs share a socket; DRANET adds that the NIC does too, and only then does the RDMA take the short path.

With disaggregated serving. The prefill→decode KV-cache is the traffic that punishes a badly placed NIC hardest. The fourth leg is what stops separating prefill and decode from being paid for in transfer latency.

With capacity planning. Sizing gains a dimension: “GPUs per node and cores per NUMA node” is not enough; you have to count how many NICs NUMA-local to a GPU the chassis has. A node with 4 GPUs and a single NIC on socket 0 has two GPUs “far from the door”.

With observability. What confirms that the fourth leg is properly set is not an application dashboard: it is /proc/softirqs (is NET_RX concentrated on housekeeping?), nvidia-smi topo -m (is the GPU↔NIC relationship NODE/PHB?) and the NIC counters. It fits with GPU observability with DCGM: the GPU “at 60% for no reason” may be the host waiting for packets that cross the socket.

Traps and things that are not what they look like

Believing that switching to Cilium “already optimises the network”. Cilium eBPF replaces kube-proxy and Canal and performs better out of the box, but the default deployment prioritises compatibility, not performance. Without irqbalance disabled, without IRQs pinned to housekeeping and without netkit/host-routing, you leave most of the improvement on the table. Cilium’s docs say so; plenty of people do not read the tuning guide.

Pinning the NIC IRQs to isolated cores. The symmetric mistake to knob 6 of post 3: if you set IRQ affinity over isolcpus, the NET_RX softirq steals cycles from vLLM on exactly the cores you isolated so nobody would disturb it. The IRQs go to reserved-cpus, always.

Assuming the Topology Manager already aligns the NIC. It does not: the classic NIC is not a Hint Provider. If you need NIC↔GPU locality, today the mechanism is DRA/DRANET, not a kubelet policy. Waiting for single-numa-node to solve it is waiting for something that is not in its design.

Putting DRA/DRANET into ENS production this quarter. It is beta and young. The sensible move is to pilot it on a lab node, measure all_reduce/all_gather with and without alignment, and decide with data. The +60% figure is from a specific environment; reproduce it in yours before promising it.

BIG TCP / netkit without reading the requirements. Kernel ≥6.8, mlx4/mlx5/ice NICs, no tunnel or encryption for BIG TCP, and nothing in-place: it forces pod restarts or per-node config. On a cluster with IPsec or with unsupported NICs, part of this does not apply. Check cilium status --verbose before taking for granted that it is active.

Confusing the eBPF datapath (kernel) with the Cilium agent (pod). cilium-agent is a Burstable DaemonSet that should live on housekeeping (covered by system-reserved). But datapath processing runs in softirq, governed by the host’s IRQ affinity, not by reserved-cpus. They are two different things; pinning the pod well does not pin the softirq.

Conclusion

The “under the engine” series chased one idea: performance that looks like an engine problem (vLLM is slow) or a model problem (quantisation) is, far too often, a locality problem one layer down. The trilogy covered three: the cable (NVLink not used), the host (remote NUMA, jitter) and the orchestration (pinning that never happened). The fourth was missing: the network. The Topology Manager seats the pod at a perfect NUMA table and never asks which door the dishes come through nor who carries them. On a node at 25 Gb/s it made no difference; on one at 400 Gb/s with the KV-cache crossing over RDMA, that door decides TTFT and the collective’s bandwidth. Cilium eBPF replaces kube-proxy and Canal with a datapath that performs, if you coordinate IRQ affinity with isolcpus/reserved-cpus, a fourth list to align, and DRA/DRANET finally provides the census that was missing: co-scheduling NUMA-local GPU and NIC on the same PCIe root, with an improvement of a size (+60% NCCL bus bandwidth) that measures how big the gap was. Going down a level is not snobbery: it is that the root cause lived, once again, one layer below where the dashboard looks.

See also

References