Rke2 on lo0 — Tech Blog

Runbook: caging the AI agent — bubblewrap on the client, Tetragon on the cluster

Tue, 09 Jun 2026 17:00:00 +0200

Operational companion to The contractor with the master key. That post explains the why and the where: the threat model, the five isolation families, which domain uses each. This one is the how, with commands. If you have not read it, read it first: here I take for granted what the blast radius is, why bwrap runs without root and what Tetragon watches. The procedure comes in two independent tracks, client and cluster, because, as the sibling post argues, the control extrapolates but the primitive gets rewritten.

TL;DR

Two reproducible procedures. Client (workstation): install ai-jail (it wraps bubblewrap), generate the per-project .ai-jail, audit with --dry-run, fix the allowlists with --bootstrap, use --lockdown for anything you do not trust, and leave the agent without git push permission. Cluster (RKE2 with Cilium + Tetragon): put the pod baseline in place (unprivileged securityContext, seccomp: RuntimeDefault, default-deny NetworkPolicy), put the untrusted agent’s pod in a microVM with runtimeClassName: kata, and deploy the Tetragon TracingPolicy files in two phases: observe with action: Post to raise the baseline, then promote to action: Sigkill on tcp_connect (egress) and security_file_open (secret paths). The golden rule of the Tetragon phase: adopt first, block later; never put a Sigkill into production without having seen the events in observation mode first.

The flow of the two tracks

Track A — Client (the developer’s workstation)

A0 — Install ai-jail and bubblewrap

ai-jail wraps the sandbox; on Linux it needs bubblewrap separately, on macOS it needs no extra dependency.

# ai-jail (macOS and Linux)
brew tap akitaonrails/tap && brew install ai-jail
# or, with cargo:
cargo install ai-jail
# or, with mise:
mise use -g ubi:akitaonrails/ai-jail

# bubblewrap on Linux (pick your distro)
sudo pacman -S bubblewrap # Arch
sudo apt install bubblewrap # Debian / Ubuntu
sudo dnf install bubblewrap # Fedora

Check the binary is there and that bwrap runs without root (it must not ask for sudo):

ai-jail --version
bwrap --ro-bind / / --unshare-all echo "bwrap ok without root"

If bwrap fails asking for privileges, your kernel has unprivileged user namespaces disabled; enable them (sysctl kernel.unprivileged_userns_clone=1 on older Debian/Ubuntu) before continuing.

A1 — The per-project .ai-jail file

On the first launch inside the project, ai-jail creates a .ai-jail (TOML) that is committable to the repo: any colleague who clones inherits the same policy.

cd ~/Projects/mi-app
ai-jail claude # creates .ai-jail and launches Claude Code inside the sandbox

The generated file:

# .ai-jail — sandbox configuration (commit it to the repo)
command = ["claude"]
rw_maps = [] # extra writable directories
ro_maps = [] # extra read-only directories

Before trusting the sandbox, audit it. --dry-run --verbose prints every mount point, every isolation flag and the full bwrap command, without executing anything:

ai-jail --dry-run --verbose claude

Read the output and confirm three things: that $HOME is mounted as a tmpfs (not the real one), that ~/.ssh, ~/.aws and ~/.gnupg do not appear among the mounts, and that the only writable directory is the project one. If you need an extra directory:

ai-jail --rw-map ~/Projects/shared-lib claude # extra, writable
ai-jail --map /opt/datasets claude # extra, read-only

Other agents, same binary:

ai-jail codex
ai-jail opencode
ai-jail bash # bare shell to debug the sandbox
ai-jail -- python script.py # any command

A2 — Permission allowlists with –bootstrap

--bootstrap generates each agent’s permission configuration, with sensible allow/deny/ask, and takes a backup before overwriting:

ai-jail --bootstrap

What it produces, in summary:

Agent	File	Base policy
Claude Code	`~/.claude/settings.json`	allow: `git status/diff/log`, `ls`, `grep`, `cargo`, `npm`, `python`, `docker compose` · ask: `git push`, `rm`, `docker run` · deny: `rm -rf`, `sudo`, `chmod 777`, `git push --force`
Codex	`~/.codex/config.toml`	`approval_policy = "on-request"`
OpenCode	`~/.config/opencode/opencode.json`	permissions for `bash`, `edit`, `write`

The operational key: git push is under ask, not allow, and git push --force under deny. The agent can commit, branch and rebase locally as much as it wants; none of that touches the remote. (If you use the Claude Code /sandbox, also set "allowUnsandboxedCommands": false to close the dangerouslyDisableSandbox escape hatch, which is opt-out out of the box.)

A3 — Lockdown for anything you do not trust

To audit third-party code or run an agent over a project you do not know, --lockdown goes further: project mounted read-only, GPU/Docker/display disabled, --rw-map/--map ignored, $HOME a pure tmpfs with no host dotfiles, network cut with --unshare-net and the environment wiped with --clearenv.

ai-jail --lockdown bash

It is the most restrictive sandbox possible short of a VM. Use it as your mental default for anything that is not your own code on your own machine.

A4 — The safety net: git with no push

It is not a flag, it is a property of the environment that changes the risk arithmetic. If the project is in git with a remote, and the agent does not have push permission, the worst case, that it corrupts every file in the project, is reverted with:

git checkout . # back to the last commit
# and if it touched .git (unlikely): delete the dir and re-clone

The remote was never touched. Sandbox for the filesystem + git for the code + manual push is already a reasonable level for daily use: ai-jail protects your data and the system, git protects the code, and the decision to publish is still yours.

Track B — Cluster (RKE2 with Cilium + Tetragon)

The untrusted agent, or the inference that executes generated code, runs as a pod. The same principle as on the client, different primitives. We assume a generic RKE2 cluster with Cilium as CNI and Tetragon already deployed (the eBPF agent DaemonSet on every node).

B0 — The pod baseline

Before any eBPF, the standard kit. Unprivileged securityContext, read-only root, default seccomp:

apiVersion: v1
kind: Pod
metadata:
 name: ai-agent
 namespace: agentes
 labels:
 app: ai-agent
spec:
 securityContext:
 runAsNonRoot: true
 runAsUser: 10001
 seccompProfile:
 type: RuntimeDefault
 containers:
 - name: agent
 image: registry.interno/ai-agent:pinned
 securityContext:
 allowPrivilegeEscalation: false
 readOnlyRootFilesystem: true
 capabilities:
 drop: ["ALL"]
 volumeMounts:
 - { name: work, mountPath: /work }  # the only writable one
 volumes:
 - name: work
 emptyDir: {}

And the default egress cut, the cluster twin of --unshare-net. A default-deny egress NetworkPolicy in the namespace, opening only DNS and the bare essentials:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: default-deny-egress
 namespace: agentes
spec:
 podSelector: {}
 policyTypes: ["Egress"]
 egress:
 - to:
 - namespaceSelector:
 matchLabels: { kubernetes.io/metadata.name: kube-system }
 ports:
 - { protocol: UDP, port: 53 }
 - { protocol: TCP, port: 53 }

B1 — Kata RuntimeClass: the untrusted pod in its own microVM

For genuinely untrusted code, take it out of the shared kernel. With Kata deployed there is a RuntimeClass:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
 name: kata
handler: kata

And the pod asks for it with one line, runtimeClassName: kata, running in its own microVM with a dedicated kernel instead of sharing the node’s:

spec:
 runtimeClassName: kata  # ← the pod runs in a microVM, not on the node's kernel
 # ...the rest the same as B0

It is the cluster twin of isolation by construction: a kernel exploit inside the pod does not reach the node.

B2 — Tetragon, observation phase (Post)

Now the layer that separates a platform with runtime visibility from one without. Observe first, never kill from the outset. A TracingPolicyNamespaced, scoped to the namespace and to the agent’s label, that reports (does not kill) three things: process executions, network connections and opens of sensitive paths. action: Post only emits the event.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: agente-observa
 namespace: agentes
spec:
 podSelector:
 matchLabels:
 app: ai-agent
 kprobes:
 # --- outbound connections ---
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchActions:
 - action: Post
 # --- opens of sensitive files ---
 - call: "security_file_open"
 syscall: false
 args:
 - index: 0
 type: "file"
 selectors:
 - matchArgs:
 - index: 0
 operator: "Prefix"
 values:
 - "/var/run/secrets"
 - "/work/.git/config"
 matchActions:
 - action: Post

(Process executions need no kprobe: Tetragon emits process_exec/process_exit natively.) Deploy it and watch the events live from the node’s Tetragon pod:

kubectl apply -f agente-observa.yaml
# readable events, filtered by namespace:
kubectl exec -n kube-system ds/tetragon -c tetragon -- \
 tetra getevents -o compact --namespace agentes

Leave this running for a typical working day of the agent. Note which destinations it really connects to (your internal registry, your HF mirror, your vLLM endpoint) and which paths it opens. That is your baseline: the list of what is legitimate. Without this step, a Sigkill kills good work and generates an availability incident, precisely what ENS asks you to avoid.

B3 — Tetragon, enforcement phase (Sigkill)

With the baseline in hand, promote to blocking. Two rules. The first: kill any connection whose destination is not on the allowlist — NotDAddr inverts the match: it fires for everything that is not those networks. The second: kill any attempt to open a secrets path.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: agente-enforce
 namespace: agentes
spec:
 podSelector:
 matchLabels:
 app: ai-agent
 kprobes:
 # --- egress: kill everything that is NOT on the allowlist ---
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchArgs:
 - index: 0
 operator: "NotDAddr"
 values:
 - "127.0.0.1"
 - "10.0.0.0/8" # internal cluster network
 - "172.16.10.20" # internal registry (example)
 matchActions:
 - action: Sigkill
 # --- reading secrets: kill the process ---
 - call: "security_file_open"
 syscall: false
 args:
 - index: 0
 type: "file"
 selectors:
 - matchArgs:
 - index: 0
 operator: "Prefix"
 values:
 - "/var/run/secrets/kubernetes.io/serviceaccount/token"
 - "/work/.ssh"
 matchActions:
 - action: Sigkill

kubectl apply -f agente-enforce.yaml

Now the agent can do whatever it likes inside the pod, but the instant it tries to connect to a disallowed destination or read the service account token, Tetragon kills it in the kernel, before the packet leaves or the read returns bytes. It is the cluster twin of the curl blocklist and the unmounted ~/.ssh, but applied at runtime and over any binary, not just the ones you know about.

Operational warning. Enforcement with Sigkill requires a recent kernel with eBPF support for the action (5.10+ is safe). Deploy agente-enforce in a test namespace first, and keep agente-observa active in parallel: if the block fires, the Post event tells you exactly what triggered it. Adopt first, block later.

The client ↔ cluster equivalence table

The same vector, the two primitives. This is “extrapolating the technology” made explicit:

Threat vector	Client (workstation)	Cluster (RKE2)
`$HOME` / writable root	`$HOME` as ephemeral tmpfs (`bwrap`)	`readOnlyRootFilesystem: true` + `emptyDir`
Arbitrary egress	`curl`/`wget` blocklist · `--unshare-net`	default-deny NetworkPolicy + Tetragon `NotDAddr`→`Sigkill`
Reading secrets	`~/.ssh`/`~/.aws`/`~/.gnupg` not mounted	secrets outside the pod + Tetragon `security_file_open`→`Sigkill`
Kernel escape	Landlock (2nd VFS barrier)	`runtimeClassName: kata` (microVM, own kernel)
No escape hatch	process inside `bwrap`, no way out	no `privileged`, `drop ALL`, `allowPrivilegeEscalation:false`
Damage to the code	git remote with no `push` → `git checkout .`	GitOps + PR review, the agent does not apply to `main`
Visibility	`--dry-run --verbose` (static, pre-run)	Tetragon `tetra getevents` (dynamic, at runtime)

Checklist of gotchas

Do not put in a Sigkill without passing through Post. The observation baseline is not optional: it is what separates “blocking a C2” from “killing your own fine-tuning job”.
The .ai-jail gets committed; secrets do not. The TOML is policy, not credentials. Check you are not putting paths holding sensitive data into rw_maps.
readOnlyRootFilesystem breaks apps that write to /tmp. Mount an emptyDir at /tmp as well as the work one.
A NetworkPolicy with no DNS rule leaves the pod blind. Open port 53 to kube-system or nothing resolves.
Kata is not free. It adds startup latency and not every workload with special devices (GPU passthrough) fits; reserve it for the untrusted, not for everything.
The Claude Code /sandbox does not cover MCP or hooks unless you enable sandbox-runtime. If your agent uses MCP servers, assume they run with full permissions until you do.
NotDAddr with literal IPs ages badly. Document the allowlist and review it when the registry or the inference endpoint changes; consider stable internal CIDRs instead of loose IPs.

References

ai-jail (Fabio Akita), GPL-3.0: https://github.com/akitaonrails/ai-jail
bubblewrap: https://github.com/containers/bubblewrap
Landlock LSM: https://landlock.io
Tetragon — TracingPolicy: https://tetragon.io/docs/concepts/tracing-policy/
Tetragon — enforcement (Sigkill/Override): https://tetragon.io/docs/concepts/enforcement/
Kata Containers — Kubernetes RuntimeClass: https://katacontainers.io
Kubernetes — Pod Security & seccomp: https://kubernetes.io/docs/tutorials/security/seccomp/
Kubernetes — Network Policies: https://kubernetes.io/docs/concepts/services-networking/network-policies/
Cilium: https://cilium.io

The kitchen door the maître d' never looked at: network NUMA, Cilium eBPF and DRANET, the fourth leg of pinning

Sat, 06 Jun 2026 12:00:00 +0200

Fourth instalment, a coda, of “under the engine”. The series closed with three legs of locality: the cable between GPUs, the host by hand and the declarative orchestration of the kubelet. But the maître d’ of the last post seated the group looking at CPU, memory and GPU, and never asked which door the dishes come through. That door is the NIC. Here is the fourth leg.

TL;DR

The Topology Manager admits a pod under single-numa-node if its CPUs, its memory and its GPU fit on the same NUMA node. The NIC is not part of that count: the kubelet has no Hint Provider for the network card. On an inference node with the network at 200/400 Gb/s, the case of disaggregated serving, where the KV-cache travels over RDMA between the prefill pool and the decode pool, a NIC on the wrong socket makes every packet cross the UPI/QPI, exactly the “remote NUMA” the series fights on the compute side, but coming in through the network door. And there is a second front: the softirq (NET_RX) that processes the datapath runs on the CPU that services the NIC’s IRQ; if that CPU is one of the cores isolcpus/reserved-cpus gave exclusively to vLLM, the softirq steals cycles from it and injects jitter into the p99 tail. Cilium eBPF replaces two RKE2 pieces, kube-proxy (with eBPF/XDP load balancing) and the default CNI Canal (with a native datapath), and its own tuning guide tells you to kill irqbalance and pin the NIC IRQs: a fourth list to align alongside isolcpus and reserved-cpus. The 2026 state of the art closes the gap from above: netkit (kernel ≥6.8, zero namespace overhead), BIG TCP (192k super-packets for 100Gb/s+), host-routing (iptables bypass), and above all DRA/DRANET, the network driver that finally co-schedules NUMA-local GPU and NIC on the same PCIe root, enabling GPUDirect RDMA with +59.6% bus bandwidth in all_gather and +58.1% in all_reduce. On a generic RKE2 cluster with 4×H100 SXM nodes.

Where you are: the network plane the trilogy did not open

The analogy: the door the dishes come through

Go back to the restaurant of the previous post. The maître d’, the Topology Manager, seated the party of eight at a single table (one NUMA node) because the diners (CPUs), the cutlery (memory) and the reserved bottle (the GPU) all fitted. A perfect table. But the maître d’ never looked at where the kitchen pass is: the door every dish comes in and goes out through.

That door is the NIC. The prompt comes in there, the tokens go out there, and, in disaggregated serving, the KV-cache that the prefill pool sends to the decode pool circulates there. If the table is in the left-hand room (socket 0) but the kitchen pass is in the right-hand one (socket 1), every dish crosses the entire restaurant (the UPI/QPI), over and over, however impeccably the table is laid. The diner does not notice the perfect table: they notice the dish arriving late and cold.

And there is a finer detail: the waiter who crosses the room with the dishes (the softirq that processes the packets) is one of the seated diners. If the maître d’ gave them a chair of their own so they could eat in peace (a core isolated with isolcpus for vLLM) but the restaurant also has them waiting on the far door, that diner does not eat: they spend the whole dinner crossing the room. The jitter shows up exactly where you thought you had bought calm.

The trilogy levelled three legs of the table: the cable, the host and the orchestration. The fourth, which door the dishes come through and who carries them, is levelled by no kubelet manager. Until 2026.

The gap: why the Topology Manager does not look at the NIC

The mechanism of post 3 is a coordinator (Topology Manager) that consults three Hint Providers: CPU Manager, Memory Manager and Device Manager (the GPU plugin). Each says on which NUMA node it can satisfy its part; the coordinator computes the intersection and admits or rejects.

The problem is one of census: the classic NIC is not a Device Manager “device”. A standard Ethernet/InfiniBand card is managed by the CNI and the kernel, it is not requested in the pod’s resources: like nvidia.com/gpu, and therefore it emits no NUMA hint. The Topology Manager aligns CPU+memory+GPU and leaves the NIC wherever the hardware put it, which may be the other socket. The maître d’ has three assistants and is missing the fourth: the one who knows which door the dishes come through.

This did not matter when a node’s network was 10/25 Gb/s and the bottleneck was elsewhere. It matters now, with two workloads that saturate the node’s network:

Disaggregated serving. The KV-cache that travels between the prefill pool and the decode pool moves over RDMA. These are large transfers, sensitive to latency and bandwidth, which in multi-node setups go out through the NIC.
Multi-node NCCL collectives. When tensor/pipeline parallelism crosses the node boundary, the all-reduce/all-gather operations no longer go over NVLink but over GPUDirect RDMA on the NIC.

In both, where the NIC sits relative to the GPU and to the pod’s cores decides performance. And the kubelet, on its own, does not coordinate that.

The network datapath under NUMA: IRQ, softirq and DMA

To see why NIC locality carries weight, you have to look at the path of an incoming packet:

Three kernel facts the analogy compresses:

The IRQ has affinity. Each NIC queue raises an interrupt that the kernel services on a specific CPU (/proc/irq/<n>/smp_affinity). The heavy processing is deferred to a softirq (NET_RX/NET_TX), which runs on that same CPU. If irqbalance is left loose, it migrates them non-deterministically, poison for the p99.
The softirq competes with the pod. If the IRQ lands on a core that isolcpus reserved for vLLM, that queue’s NET_RX steals cycles from the model. The signal in /proc/softirqs: a NET_RX column that spikes on a single CPU. It is the same jitter from post 2, coming in through the network.
DMA has a NUMA origin. The NIC writes the packet by DMA into the RAM of the socket of its PCIe root. If the consumer (the pod’s thread) is on the other socket, it reads across the UPI/QPI. RFS (Receive Flow Steering) tries to bring the processing to the consumer’s CPU, but it cannot teleport the NIC to the other socket.

One number, with its caveat

Take a 2-socket node, a 400 Gb/s = 50 GB/s NIC on the PCIe root of socket 0, and a decode pod pinned to socket 1. If the NIC saturates, those ~50 GB/s of receive traffic cross the UPI towards socket 1. A UPI 2.0 link runs at around 20–40 GB/s per direction and link depending on generation; even with several links, 50 GB/s of network traffic going against the grain eats a far from negligible fraction of the inter-socket budget, the same budget already contended by the pod’s remote memory accesses and, if there is multi-node traffic, the KV-cache of the disaggregation. I am not giving a firm “X% degradation” because it depends on CPU generation, number of UPI links, MTU and traffic pattern; without that methodology, any exact figure is marketing.

What is measured with public methodology is the aggregate effect of aligning GPU and NIC: the DRANET project reports +59.6% bus bandwidth in all_gather and +58.1% in all_reduce (NCCL collectives) when the assigned NIC is NUMA-local to the GPU rather than not. That is the size of the gap the Topology Manager was leaving open.

What RKE2’s Cilium eBPF replaces (and why it touches this story)

RKE2 ships Canal (Flannel + Calico) by default as the CNI and kube-proxy (iptables/IPVS rules) for Service balancing. Switching to Cilium (cni: cilium in /etc/rancher/rke2/config.yaml) replaces both pieces with an eBPF datapath:

RKE2 piece	What it does	What Cilium eBPF puts in
`kube-proxy` (iptables/IPVS)	Service balancing	LB in eBPF; with `kubeProxyReplacement=true`, and acceleration in XDP (driver layer)
Canal (Flannel+Calico)	VXLAN overlay + NetworkPolicy	native datapath (`routingMode=native`), L3/L4 and L7 NetworkPolicy in eBPF
veth per pod	namespace interface pair	netkit (kernel ≥6.8): namespace overhead ~0
host iptables traversal	netfilter hooks	eBPF host-routing: bypasses iptables and the upper part of the stack

Up to here this is pure networking and it does not touch the kubelet’s resource managers: Cilium does not assign exclusive CPUs nor emit compute NUMA hints. The ten knobs of post 3 stay identical whether you run Canal or Cilium.

But Cilium does enter the fourth leg through two doors. The first: its own tuning guide recommends, literally, “kill irqbalance and pin the NIC IRQs to specific CPUs for maximum workload isolation”, along with the tuned network-latency profile, the performance governor and CONFIG_PREEMPT_NONE. In other words: the eBPF datapath only really performs if you coordinate IRQ affinity, and that affinity has to point at the housekeeping cores (reserved-cpus), never at the isolated ones. So a fourth list appears that must be kept consistent with isolcpus and reserved-cpus:

isolcpus = 2-31,34-63 # cores exclusive to vLLM (host, post 2)
reserved-cpus = 0-1,32-33 # kubelet housekeeping (post 3)
IRQ affinity = 0-1,32-33 # NIC IRQs → housekeeping ONLY (this post)
 # never 2-31: there the softirq would steal from the model

The second door: netkit + host-routing + BIG TCP reduce how many times the packet crosses the stack and the namespace, which cushions (does not eliminate) the cost of the NUMA crossing. BIG TCP assembles super-packets of up to 192k (against 64k) for 100Gb/s+; fewer stack traversals means less softirq work on the core, and therefore less pressure on the inter-socket budget. It is the continuous batching analogy applied to the network stack: amortising a fixed cost over larger batches.

Cilium performance profile (state 1.19, kernel ≥6.8)

# Helm, recommended performance profile (summary of the tuning guide)
helm install cilium cilium/cilium --version 1.19.4 \
 --namespace kube-system \
 --set routingMode=native \
 --set bpf.datapathMode=netkit \  # namespace overhead ~0 (kernel >=6.8)
 --set bpf.masquerade=true \
 --set kubeProxyReplacement=true \  # replaces RKE2's kube-proxy
 --set enableIPv4BIGTCP=true \  # 192k super-packets (mlx5/ice NIC)
 --set enableIPv6BIGTCP=true \
 --set bpf.distributedLRU.enabled=true \# per-CPU BPF maps: less spinlock contention
 --set bpf.mapDynamicSizeRatio=0.08 \
 --set bpfClockProbe=true

# Verification inside a Cilium pod:
cilium status --verbose | grep -E "Device Mode|Host Routing|BIG TCP|XDP"
# Device Mode: netkit · Host Routing: BPF · IPv4 BIG TCP: enabled · XDP Acceleration: Native

Sceptical caveat: netkit and BIG TCP are beta and require kernel ≥6.8 and specific NICs (mlx4/mlx5/ice). They are not in-place: they change datapath fundamentals and force you to restart pods or, better, to apply them by per-node config only on new nodes. For an ENS cluster in production, that is a maintenance window, not a blind helm upgrade.

The 2026 state of the art: DRA and DRANET, the maître d’ who finally looks at the door

What closes the gap at the root is not Cilium, it is the admission mechanism the kubelet did not have for the NIC: Dynamic Resource Allocation (DRA), beta since Kubernetes 1.32 and with advances in every release up to 1.36 (May 2026). DRA generalises the “devices” model beyond the GPU: a driver discovers the hardware, publishes ResourceSlices with its attributes, including the NUMA topology and the PCIe root, and the scheduler resolves ResourceClaims that can demand affinity between devices.

DRANET (a kubernetes-sigs project) is the network DRA driver. It discovers the NICs (including the RDMA-capable ones), advertises them as ResourceSlices, and via NRI injects them into the pod’s namespace, compatible with whatever CNI you already have, Cilium included. The key piece for this story: combined with the NVIDIA GPU DRA driver, it allows co-scheduling GPU and NIC that share a PCIe root (the relationship NVIDIA calls NODE), which is exactly the condition for GPUDirect RDMA. The maître d’ finally has a fourth assistant: “is there a NIC NUMA-local to this GPU?”.

The ResourceClaimTemplate uses CEL selectors to request exactly that alignment:

# Request an RDMA NIC NUMA-local to the assigned GPU (illustrative DRANET/DRA schema)
apiVersion: resource.k8s.io/v1beta1
kind: ResourceClaimTemplate
metadata:
 name: gpu-nic-numa-aligned
spec:
 spec:
 devices:
 requests:
 - name: rdma-nic
 deviceClassName: dra.net  # NICs published by DRANET
 constraints:
 - requests: ["rdma-nic"]
 matchAttribute: "dra.net/pcieRoot" # same PCIe root as the GPU
 # → enables GPUDirect RDMA over a NUMA-local path

Why it matters for inference, not for abstract “AI training”: in disaggregated serving, RDMA is what moves the KV-cache between the prefill pool and the decode pool with the latency that TTFT demands; and in multi-node, GPUDirect RDMA replaces NVLink as the medium of the collective. Aligning GPU+NIC on the same PCIe root is what turns an “RDMA that works” into an “RDMA that performs”, the +60% bus bandwidth from DRANET.

Status and caveats: DRA is beta (gates to enable by hand), DRANET is young (a SIG project, still evolving) and the managed offering exists mostly in cloud (GKE managed DRANET in preview, AKS for RDMA). For on-premise ENS it is a path, not a finished product: the value today is understanding that the fourth leg already has a standard OSS mechanism, and starting to pilot it on a lab node, not putting it into critical production this quarter.

How it connects with the rest of the stack

With the host (post 2). The NIC’s IRQ affinity is a third list to marry with isolcpus and reserved-cpus. The IRQs go to housekeeping; the isolated cores, untouched. Uncoordinating them lets in through the network door the jitter that isolcpus threw out through the compute one.

With the orchestration (post 3). DRA is the natural extension of the Topology Manager: the same “admit only if it fits on the NUMA node” principle carried over to the NIC. Where the Device Manager left the network out of the census, DRANET puts it in.

With the interconnect (post 1). Inside the node NVLink rules; on crossing the node boundary, GPUDirect RDMA over the NIC is the medium of the collective. The kubelet’s NUMA policy guarantees that GPU and CPUs share a socket; DRANET adds that the NIC does too, and only then does the RDMA take the short path.

With disaggregated serving. The prefill→decode KV-cache is the traffic that punishes a badly placed NIC hardest. The fourth leg is what stops separating prefill and decode from being paid for in transfer latency.

With capacity planning. Sizing gains a dimension: “GPUs per node and cores per NUMA node” is not enough; you have to count how many NICs NUMA-local to a GPU the chassis has. A node with 4 GPUs and a single NIC on socket 0 has two GPUs “far from the door”.

With observability. What confirms that the fourth leg is properly set is not an application dashboard: it is /proc/softirqs (is NET_RX concentrated on housekeeping?), nvidia-smi topo -m (is the GPU↔NIC relationship NODE/PHB?) and the NIC counters. It fits with GPU observability with DCGM: the GPU “at 60% for no reason” may be the host waiting for packets that cross the socket.

Traps and things that are not what they look like

Believing that switching to Cilium “already optimises the network”. Cilium eBPF replaces kube-proxy and Canal and performs better out of the box, but the default deployment prioritises compatibility, not performance. Without irqbalance disabled, without IRQs pinned to housekeeping and without netkit/host-routing, you leave most of the improvement on the table. Cilium’s docs say so; plenty of people do not read the tuning guide.

Pinning the NIC IRQs to isolated cores. The symmetric mistake to knob 6 of post 3: if you set IRQ affinity over isolcpus, the NET_RX softirq steals cycles from vLLM on exactly the cores you isolated so nobody would disturb it. The IRQs go to reserved-cpus, always.

Assuming the Topology Manager already aligns the NIC. It does not: the classic NIC is not a Hint Provider. If you need NIC↔GPU locality, today the mechanism is DRA/DRANET, not a kubelet policy. Waiting for single-numa-node to solve it is waiting for something that is not in its design.

Putting DRA/DRANET into ENS production this quarter. It is beta and young. The sensible move is to pilot it on a lab node, measure all_reduce/all_gather with and without alignment, and decide with data. The +60% figure is from a specific environment; reproduce it in yours before promising it.

BIG TCP / netkit without reading the requirements. Kernel ≥6.8, mlx4/mlx5/ice NICs, no tunnel or encryption for BIG TCP, and nothing in-place: it forces pod restarts or per-node config. On a cluster with IPsec or with unsupported NICs, part of this does not apply. Check cilium status --verbose before taking for granted that it is active.

Confusing the eBPF datapath (kernel) with the Cilium agent (pod). cilium-agent is a Burstable DaemonSet that should live on housekeeping (covered by system-reserved). But datapath processing runs in softirq, governed by the host’s IRQ affinity, not by reserved-cpus. They are two different things; pinning the pod well does not pin the softirq.

Conclusion

The “under the engine” series chased one idea: performance that looks like an engine problem (vLLM is slow) or a model problem (quantisation) is, far too often, a locality problem one layer down. The trilogy covered three: the cable (NVLink not used), the host (remote NUMA, jitter) and the orchestration (pinning that never happened). The fourth was missing: the network. The Topology Manager seats the pod at a perfect NUMA table and never asks which door the dishes come through nor who carries them. On a node at 25 Gb/s it made no difference; on one at 400 Gb/s with the KV-cache crossing over RDMA, that door decides TTFT and the collective’s bandwidth. Cilium eBPF replaces kube-proxy and Canal with a datapath that performs, if you coordinate IRQ affinity with isolcpus/reserved-cpus, a fourth list to align, and DRA/DRANET finally provides the census that was missing: co-scheduling NUMA-local GPU and NIC on the same PCIe root, with an improvement of a size (+60% NCCL bus bandwidth) that measures how big the gap was. Going down a level is not snobbery: it is that the root cause lived, once again, one layer below where the dashboard looks.

References

Cilium, Tuning Guide (netkit, host-routing, BIG TCP, XDP, pin the IRQs and kill irqbalance): https://docs.cilium.io/en/stable/operations/performance/tuning/.
Cilium 1.19 (February 2026), Cilium at Ten Years — hardening of encryption, policies and observability: https://www.infoq.com/news/2026/02/cilium-119/.
Isovalent, Cilium 1.18 (IPv6, encrypted overlay, ingress bandwidth, policy perf): https://isovalent.com/blog/post/cilium-1-18/.
RKE2, Network Options (Canal by default; Cilium with kube-proxy replacement): https://docs.rke2.io/networking/basic_network_options.
Kubernetes, Dynamic Resource Allocation: https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/.
Kubernetes blog, v1.36: More Drivers, New Features, and the Next Era of DRA (May 2026): https://kubernetes.io/blog/2026/05/07/kubernetes-v1-36-dra-136-updates/.
DRANET (kubernetes-sigs), the network DRA driver and the paper The Kubernetes Network Driver Model (+59.6% all_gather / +58.1% all_reduce): https://github.com/kubernetes-sigs/dranet.
AKS Engineering, Optimizing RDMA performance for AI workloads on AKS with DRANET (April 2026): https://blog.aks.azure.com/2026/04/01/dranet-rdma-optimization-for-ai-on-aks.
Linux network tuning — IRQ affinity, RSS/RPS/RFS and NUMA softirq: https://andreaskaris.github.io/blog/networking/rss-irq-affinity-and-rps/.

The maître d' who only seats you if you all fit at one table: CPU, Memory and Topology Manager on RKE2

Sat, 06 Jun 2026 08:00:00 +0200

The close of the “under the engine” series. We saw the cable between GPUs and the host: NUMA, hugepages and CPU isolation done by hand. Here is the piece that makes it declarative and at scale: how the RKE2 kubelet pins each vLLM pod to the right NUMA node without a single script.

TL;DR

Pinning NUMA with numactl/isolcpus/taskset, the subject of the previous post, does not scale to a cluster where pods come and go and there are dozens of nodes. The kubelet automates it with three components that work as Hint Providers for a central coordinator, the Topology Manager: the CPU Manager (assigns exclusive CPUs to containers of Guaranteed pods with whole CPUs), the Memory Manager (NUMA-local memory and hugepages) and the Device Manager/GPU plugin (which knows which GPU is on which NUMA node). With the single-numa-node policy, the Topology Manager only admits the pod if its CPUs, its memory and its GPU fit in the same NUMA domain; if they do not fit, it rejects the pod, strict admission, like the maître d’ who will not seat a party of eight unless there is a table for eight. On RKE2 all of this is configured with kubelet-arg in /etc/rancher/rke2/config.yaml. This post explains the mechanism, gives the 10 knobs and takes apart the gotchas that break pinning silently: the cpu_manager_state file you have to delete when changing policy, the QoS that has to be exactly Guaranteed, and the reserved-cpus that must match the host’s isolcpus. On a generic RKE2 cluster with 4×H100 SXM nodes.

Where you are: the orchestration that materialises the host

The analogy: the maître d’ of a restaurant whose tables cannot be pushed together

A restaurant has tables of different sizes and, house rule, tables are never pushed together. A party of eight arrives. The maître d’ looks for a single table where all eight fit. If there is one, he seats them; if only tables for four are left, he does not accept them. He would rather turn the booking away than seat the group split across two separate tables, because he knows a split dinner goes badly.

That maître d’ is the Topology Manager under the single-numa-node policy. The “party” is an inference pod asking for CPUs, memory and a GPU. The “table” is a NUMA node. The maître d’ asks three assistants (are there free CPUs on any node? the CPU Manager; is there free memory? the Memory Manager; is there a free GPU? the Device Manager) and only admits the pod if all three resources fit on the same node. If not, he rejects it (the pod ends up Failed with TopologyAffinityError), and the scheduler will try another node.

The difference from the previous post: there you were the maître d’, seating each process by hand with numactl. Here the maître d’ is the kubelet, and it does it for every pod, on every node, automatically, rejecting whatever does not fit. That is what turns artisanal pinning into a declarative property of the cluster.

The mechanism: Hint Providers and the coordinator

The Topology Manager does not assign resources; it coordinates the ones that do. The flow, when a Guaranteed pod arrives at a node:

The three managers are Hint Providers: each one tells the Topology Manager on which NUMA node(s) it could satisfy its part. The Topology Manager computes the intersection and, depending on the policy, decides:

none (default): no coordination; each manager does its own thing without aligning. No NUMA guarantee.
best-effort: it tries to align on one node; if it cannot, it admits anyway (on whichever node). Better than nothing, no guarantee.
restricted: if it fails to align, it rejects the pod. Strict, but it allows multi-node affinity if the intersection provides it.
single-numa-node: it requires everything to fit on a single NUMA node, or it rejects. The strictest one, and the one that genuinely guarantees the locality of the previous post.

And two preconditions without which none of this kicks in:

The pod has to be QoS Guaranteed: requests == limits on CPU and memory, and whole CPUs (not 500m). Only then does the CPU Manager assign exclusive CPUs.
The CPU Manager has to be on the static policy (not none).

Without those two, the Topology Manager has nothing to align and pinning does not happen, even with the policy in place. That is gotcha no. 1.

How it is configured on RKE2

RKE2 passes arguments to the kubelet with the kubelet-arg key in /etc/rancher/rke2/config.yaml. The reference configuration for GPU inference nodes:

# /etc/rancher/rke2/config.yaml (on every agent node with GPUs)
kubelet-arg:
 - "cpu-manager-policy=static"
 - "topology-manager-policy=single-numa-node"
 - "topology-manager-scope=pod"
 - "memory-manager-policy=Static"
 - "reserved-cpus=0-1,64-65" # housekeeping; must match the host
 - "system-reserved=memory=8Gi"
 - "kube-reserved=memory=4Gi"
 - "reserved-memory=0:memory=4Gi;1:memory=4Gi" # required by Memory Manager Static
node-label:
 - "fibercli.local/numa-pinned=true"

After deploying it: systemctl restart rke2-agent. Critical gotcha: if the node has already run with cpu-manager-policy=none, there is a state file /var/lib/kubelet/cpu_manager_state that pins the old policy; changing the arg without deleting that file makes the kubelet fail to start or ignore the new policy. You have to stop the agent, rm /var/lib/kubelet/cpu_manager_state, and start again. (The same applies to memory_manager_state.)

And the vLLM pod, to be eligible, Guaranteed with whole CPUs:

resources:
 requests:
 cpu: "16" # whole, not a weird fractional "16000m"
 memory: "200Gi"
 nvidia.com/gpu: "2" # TP=2 → 2 GPUs on the same NUMA node
 hugepages-1Gi: "16Gi"
 limits:
 cpu: "16" # == requests → Guaranteed QoS
 memory: "200Gi"
 nvidia.com/gpu: "2"
 hugepages-1Gi: "16Gi"

With this, on a node with the config above, the kubelet assigns 16 exclusive CPUs from the NUMA node where the 2 requested GPUs live, its local memory and the hugepages, or it rejects the pod if they do not fit together. The artisanal pinning of the previous post, now declarative.

The 10 knobs to reach for

Ordered by dependency (the first ones are a precondition for the later ones). The canonical reference is the Kubernetes Topology Manager documentation and the RKE2 configuration.

Knob 1 — `cpu-manager-policy=static`: the foundation

Without this there are no exclusive CPUs and nothing else kicks in.

kubelet-arg: [ "cpu-manager-policy=static" ]

Gotcha: changing it requires deleting /var/lib/kubelet/cpu_manager_state and restarting the kubelet, or startup fails. It is cause no. 1 of “I set the policy and it does not pin”.

Knob 2 — `Guaranteed` QoS + whole CPUs: the pod’s precondition

This is not node config, it is pod config, but without it knob 1 does nothing for that pod. requests == limits on CPU and memory, and whole CPUs. A cpu: 500m or a requests != limits downgrades the pod to Burstable and it loses the pinning. Plenty of people set the node policy and forget the pod’s QoS.

Knob 3 — `topology-manager-policy=single-numa-node`: strict admission

kubelet-arg: [ "topology-manager-policy=single-numa-node" ]

The strict maître d’. For GPU inference it is the right policy: it guarantees that CPU+memory+GPU share a node. best-effort guarantees nothing (it admits misaligned); restricted allows multi-node affinity. Start with single-numa-node and drop to restricted only if you have admission problems.

Knob 4 — `topology-manager-scope=pod`: group the whole pod

kubelet-arg: [ "topology-manager-scope=pod" ]

With container scope (the default), each container is aligned separately; with pod scope, the whole pod goes to the same node. For a vLLM pod with sidecars (metrics, proxy), pod scope stops the sidecar from dragging the main container onto another node. Recommended for inference.

Knob 5 — `memory-manager-policy=Static` + `reserved-memory`: NUMA-local memory

kubelet-arg:
 - "memory-manager-policy=Static"
 - "reserved-memory=0:memory=4Gi;1:memory=4Gi"

Memory Manager Static forces cpuset.mems so that the pod’s memory comes out of the right node (and so do the hugepages). It requires declaring reserved-memory per node, or the kubelet does not start. It is the declarative equivalent of the previous post’s --membind.

Knob 6 — `reserved-cpus`: the housekeeping cores (must match `isolcpus`)

kubelet-arg: [ "reserved-cpus=0-1,64-65" ]

It reserves cores for the system and the daemons; the rest are left for exclusive pods. Key point of the series: these reserved-cpus must be the same cores you left out of isolcpus on the host (previous post). If the host isolates 2-31 but RKE2 reserves 0-3, there is a mismatch: isolated cores that the kubelet assigns to pods without them actually being quiet. Coordinate the two layers.

Knob 7 — GPU plugin with NUMA topology (NVIDIA GPU Operator)

The Device Manager can only give a correct NUMA hint if the GPU plugin exposes which node each GPU is on. The NVIDIA device plugin / GPU Operator does this, but you have to verify that the topology information arrives (in some versions it requires flags). Without a GPU hint, the Topology Manager aligns CPU and memory but not the GPU, and GPU locality is precisely the one that matters most.

Knob 8 — hugepages as a pod resource

resources:
 limits:
 hugepages-1Gi: "16Gi" # the node must have them pre-reserved (post 2, knob 4)

The hugepages you reserved at host boot (previous post) are requested as a resource. The Memory Manager assigns them NUMA-local. If you request them without having reserved them on the node, the pod does not get scheduled.

Knob 9 — `system-reserved` / `kube-reserved`: do not oversubscribe

kubelet-arg:
 - "system-reserved=cpu=500m,memory=8Gi"
 - "kube-reserved=cpu=500m,memory=4Gi"

It reserves resources for the system and the K8s components so the node does not run out of air under load. Badly calibrated, either the node chokes (too little reserved) or you waste capacity (too much). It must be consistent with reserved-cpus.

Knob 10 — Labels + taints: so vLLM lands here and nothing else does

# GPU node: taint to repel anything that does not need a GPU
node-taint: [ "nvidia.com/gpu=present:NoSchedule" ]
node-label: [ "fibercli.local/numa-pinned=true" ]

Keep the NUMA-pinned nodes for inference and push out whatever does not need it (databases, the Langfuse backend, runners). A ClickHouse stealing memory bandwidth from a carefully pinned vLLM pod throws away all the work of the previous nine knobs. Workload isolation is the closing move.

Summary table

#	Knob	Where	Function
1	`cpu-manager-policy=static`	kubelet-arg	exclusive CPUs (foundation)
2	`Guaranteed` QoS + whole CPUs	pod spec	pinning precondition
3	`topology-manager-policy=single-numa-node`	kubelet-arg	strict NUMA admission
4	`topology-manager-scope=pod`	kubelet-arg	group the whole pod
5	`memory-manager-policy=Static`	kubelet-arg	NUMA-local memory/hugepages
6	`reserved-cpus`	kubelet-arg	housekeeping (match isolcpus)
7	GPU plugin with topology	GPU Operator	NUMA hint for the GPU
8	`hugepages-1Gi`	pod spec	hugepages as a resource
9	`system/kube-reserved`	kubelet-arg	do not oversubscribe
10	taints + labels	node config	isolate GPU workloads

Verifying that the pinning really happened

Do not trust that the config “is in place”. Check:

# Is the active policy the one you set?
cat /var/lib/kubelet/cpu_manager_state | jq .policyName # "static"

# Which exclusive CPUs does the container have?
kubectl exec <pod> -- cat /sys/fs/cgroup/cpuset.cpus.effective

# Inside the pod: is the assigned GPU local to those cores?
kubectl exec <pod> -- nvidia-smi topo -m

# Were there topology rejections?
kubectl describe pod <pod> | grep -i TopologyAffinityError

A pod in Failed with TopologyAffinityError is not a bug: it is the maître d’ doing his job. That node did not have a table where CPU+memory+GPU fitted together. The answer is to review the sizing of the pod or the node, not to relax the policy casually.

How it connects with the rest of the stack

With the host (previous post). This post is the declarative automation of that one. cpu-manager-policy=static materialises the taskset; memory-manager-policy=Static materialises the --membind; reserved-cpus must match isolcpus. The two layers are one single decision seen from two places: the host executes it, the kubelet declares it. Uncoordinating them (isolcpus 2-31 vs reserved-cpus 0-3) breaks both.

With the interconnect (post 1). The Topology Manager pins the right GPU to the pod, but if you ask for 2 GPUs for TP=2, you will want those two to share NVLink. The NUMA policy guarantees they are on the same socket; that they are NVLink-connected is guaranteed by the baseboard hardware (post 1, knob 1). The two things together are what makes TP=2 perform.

With autoscaling. When KEDA scales vLLM pods, each new replica goes through Topology Manager admission. If the node has no free “table”, the pod stays pending. Pod autoscaling and node autoscaling (cluster-autoscaler) have to account for NUMA granularity, not just aggregate CPU/memory.

With capacity planning. Sizing changes: it is not “128 vCPU per machine”, it is “128 minus the reserved-cpus, in blocks that fit per NUMA node”. A machine with 2 sockets × 64 cores does not serve a pod asking for 80 cores under single-numa-node: they do not fit at one table. Planning has to reason per NUMA node, not per machine.

With service coexistence. The knob 10 taint is what keeps Langfuse, databases and runners off the inference nodes. Without that boundary, all the fine-grained pinning gets eaten by a noisy neighbour. Observability goes on its own nodes; inference, pinned, on its own.

Traps and things that are not what they seem

Policy in place, QoS forgotten. The most common mistake: cpu-manager-policy=static on the node but the pod is Burstable (requests != limits or fractional CPU). Pinning does not happen and nobody warns you. Guaranteed QoS with whole CPUs is a necessary condition.

Fossilised cpu_manager_state. Changing policy without deleting /var/lib/kubelet/cpu_manager_state (and memory_manager_state) makes the kubelet fail or ignore the change. Stop agent → delete file → start.

reserved-cpus ≠ isolcpus. If the host isolates some cores and RKE2 reserves others, the managers assign cores to pods that are not really quiet, or they leave isolated cores idle. The two lists have to be consistent. It is the coordination failure between the previous post and this one.

GPU plugin without NUMA topology. If the device plugin does not expose each GPU’s NUMA node, the Topology Manager aligns CPU and memory but leaves the GPU to chance, and GPU locality is the one that weighs most. Verify that the GPU Operator publishes the topology.

single-numa-node that rejects too much. If the pods ask for more resources than fit on one node (for example, more cores than a socket has), rejection is constant. The answer is not to drop to best-effort (which silences the problem by serving misaligned), but to size the pod so it fits at one table, or to accept restricted with full knowledge of what you are doing.

Believing best-effort “is nearly the same”. best-effort admits the pod even if it fails to align: it gives you a false sense of NUMA-awareness while you serve from the wrong socket. For inference with a tail SLO, single-numa-node or restricted; best-effort only if the alternative is scheduling nothing at all.

Conclusion

The previous post pinned by hand; this one does it at scale and with a guarantee that artisanal numactl did not give: strict admission. The kubelet, via CPU Manager, Memory Manager and Topology Manager, acts as a maître d’ who only seats the pod if its CPUs, its memory and its GPU fit at the same NUMA table, and who rejects what does not fit instead of serving a split dinner. Of the ten knobs, the first two, cpu-manager-policy=static and Guaranteed QoS with whole CPUs, are the precondition without which the other eight do nothing, and they are exactly the ones most often forgotten; the rest tune the policy, the memory, the hugepages and the coexistence. The thread that closes the series: the inference performance that looked like an engine problem (vLLM is slow) or a model problem (quantisation) is, far too often, a cable problem (NVLink not used), a host problem (remote NUMA, jitter) or an orchestration problem (pinning that never happened because the QoS was wrong). Going down a level is not infrastructure snobbery: it is where the root causes live that no application-layer dashboard is ever going to point out for you.

References

Kubernetes, Control Topology Management Policies on a node: https://kubernetes.io/docs/tasks/administer-cluster/topology-manager/.
Kubernetes, Control CPU Management Policies on the Node: https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/.
Kubernetes, Control Memory Management Policies on a Node: https://kubernetes.io/docs/tasks/administer-cluster/memory-manager/.
RKE2, Configuration Options (kubelet-arg in config.yaml): https://docs.rke2.io/install/configuration.
RKE2, Advanced Options and Configuration: https://docs.rke2.io/advanced.
rancher/rke2, discussion #3034 CPU Management Policies for RKE2 (the cpu_manager_state gotcha): https://github.com/rancher/rke2/discussions/3034.

Kubernetes with Cilium BGP: services reachable without Ingress

Sun, 08 Mar 2026 00:00:00 +0000

One of the advantages of using Cilium as the CNI in Kubernetes is its native BGP support. It lets you advertise ClusterIPs and LoadBalancer IPs directly to the LAN router, making services reachable without needing Ingress or NodePort.

The problem

In a standard Kubernetes cluster, pods and services live on internal networks that are not reachable from outside the cluster. Getting to them requires NodePort, Ingress or an external LoadBalancer.

With Cilium BGP, the pod CIDRs and service CIDRs are advertised over BGP to the upstream router, making the whole cluster network routable from the LAN.

Work in progress — full configurations for RKE2 + CRS327 coming soon.

Rke2 on lo0 — Tech Blog

Runbook: caging the AI agent — bubblewrap on the client, Tetragon on the cluster

TL;DR

The flow of the two tracks

Track A — Client (the developer’s workstation)

A0 — Install ai-jail and bubblewrap

A1 — The per-project .ai-jail file

A2 — Permission allowlists with –bootstrap

A3 — Lockdown for anything you do not trust

A4 — The safety net: git with no push

Track B — Cluster (RKE2 with Cilium + Tetragon)

B0 — The pod baseline

B1 — Kata RuntimeClass: the untrusted pod in its own microVM

B2 — Tetragon, observation phase (Post)

B3 — Tetragon, enforcement phase (Sigkill)

The client ↔ cluster equivalence table

Checklist of gotchas

See also

References

The kitchen door the maître d' never looked at: network NUMA, Cilium eBPF and DRANET, the fourth leg of pinning

TL;DR

Where you are: the network plane the trilogy did not open

The analogy: the door the dishes come through

The gap: why the Topology Manager does not look at the NIC

The network datapath under NUMA: IRQ, softirq and DMA

One number, with its caveat

What RKE2’s Cilium eBPF replaces (and why it touches this story)

Cilium performance profile (state 1.19, kernel ≥6.8)

The 2026 state of the art: DRA and DRANET, the maître d’ who finally looks at the door

How it connects with the rest of the stack

Traps and things that are not what they look like

Conclusion

See also

References

The maître d' who only seats you if you all fit at one table: CPU, Memory and Topology Manager on RKE2

TL;DR

Where you are: the orchestration that materialises the host

The analogy: the maître d’ of a restaurant whose tables cannot be pushed together

The mechanism: Hint Providers and the coordinator

How it is configured on RKE2

The 10 knobs to reach for

Knob 1 — cpu-manager-policy=static: the foundation

Knob 2 — Guaranteed QoS + whole CPUs: the pod’s precondition

Knob 3 — topology-manager-policy=single-numa-node: strict admission

Knob 4 — topology-manager-scope=pod: group the whole pod

Knob 5 — memory-manager-policy=Static + reserved-memory: NUMA-local memory

Knob 6 — reserved-cpus: the housekeeping cores (must match isolcpus)

Knob 7 — GPU plugin with NUMA topology (NVIDIA GPU Operator)

Knob 8 — hugepages as a pod resource

Knob 9 — system-reserved / kube-reserved: do not oversubscribe

Knob 10 — Labels + taints: so vLLM lands here and nothing else does

Summary table

Verifying that the pinning really happened

How it connects with the rest of the stack

Traps and things that are not what they seem

Conclusion

See also

References

Kubernetes with Cilium BGP: services reachable without Ingress

The problem

Knob 1 — `cpu-manager-policy=static`: the foundation

Knob 2 — `Guaranteed` QoS + whole CPUs: the pod’s precondition

Knob 3 — `topology-manager-policy=single-numa-node`: strict admission

Knob 4 — `topology-manager-scope=pod`: group the whole pod

Knob 5 — `memory-manager-policy=Static` + `reserved-memory`: NUMA-local memory

Knob 6 — `reserved-cpus`: the housekeeping cores (must match `isolcpus`)

Knob 9 — `system-reserved` / `kube-reserved`: do not oversubscribe