The maître d' who only seats you if you all fit at one table: CPU, Memory and Topology Manager on RKE2

· By José Manuel Román
Contents

The close of the “under the engine” series. We saw the cable between GPUs and the host: NUMA, hugepages and CPU isolation done by hand. Here is the piece that makes it declarative and at scale: how the RKE2 kubelet pins each vLLM pod to the right NUMA node without a single script.

TL;DR

Pinning NUMA with numactl/isolcpus/taskset, the subject of the previous post, does not scale to a cluster where pods come and go and there are dozens of nodes. The kubelet automates it with three components that work as Hint Providers for a central coordinator, the Topology Manager: the CPU Manager (assigns exclusive CPUs to containers of Guaranteed pods with whole CPUs), the Memory Manager (NUMA-local memory and hugepages) and the Device Manager/GPU plugin (which knows which GPU is on which NUMA node). With the single-numa-node policy, the Topology Manager only admits the pod if its CPUs, its memory and its GPU fit in the same NUMA domain; if they do not fit, it rejects the pod, strict admission, like the maître d’ who will not seat a party of eight unless there is a table for eight. On RKE2 all of this is configured with kubelet-arg in /etc/rancher/rke2/config.yaml. This post explains the mechanism, gives the 10 knobs and takes apart the gotchas that break pinning silently: the cpu_manager_state file you have to delete when changing policy, the QoS that has to be exactly Guaranteed, and the reserved-cpus that must match the host’s isolcpus. On a generic RKE2 cluster with 4×H100 SXM nodes.

Where you are: the orchestration that materialises the host

The vertical stack · you are at orchestrationEngine · vLLM pod (TP, batching)YOU ARE HERE · kubelet: CPU/Memory/Topology Mgrdeclarative pinning + NUMA admissionHost · NUMA, hugepages, isolcpus (post 2)CUDA + NCCL + NVLink (post 1)Hardware · 2 sockets, 4×H100 SXMthe kubelet turns declarative intent into the raw pinning below

The analogy: the maître d’ of a restaurant whose tables cannot be pushed together

A restaurant has tables of different sizes and, house rule, tables are never pushed together. A party of eight arrives. The maître d’ looks for a single table where all eight fit. If there is one, he seats them; if only tables for four are left, he does not accept them. He would rather turn the booking away than seat the group split across two separate tables, because he knows a split dinner goes badly.

That maître d’ is the Topology Manager under the single-numa-node policy. The “party” is an inference pod asking for CPUs, memory and a GPU. The “table” is a NUMA node. The maître d’ asks three assistants (are there free CPUs on any node? the CPU Manager; is there free memory? the Memory Manager; is there a free GPU? the Device Manager) and only admits the pod if all three resources fit on the same node. If not, he rejects it (the pod ends up Failed with TopologyAffinityError), and the scheduler will try another node.

The difference from the previous post: there you were the maître d’, seating each process by hand with numactl. Here the maître d’ is the kubelet, and it does it for every pod, on every node, automatically, rejecting whatever does not fit. That is what turns artisanal pinning into a declarative property of the cluster.

The mechanism: Hint Providers and the coordinator

The Topology Manager does not assign resources; it coordinates the ones that do. The flow, when a Guaranteed pod arrives at a node:

Admission of a Guaranteed pod · single-numa-nodeTopology Managercoordinator · admits/rejectsCPU Managerstatic: exclusive CPUshint: cores on node X?Memory ManagerStatic: cpuset.memshint: RAM/hugepages?Device ManagerNVIDIA GPU pluginhint: GPU on node X?NUMA hintNUMA hintNUMA hintDo the three hints meet on 1 node?YES → admitNO → reject

The three managers are Hint Providers: each one tells the Topology Manager on which NUMA node(s) it could satisfy its part. The Topology Manager computes the intersection and, depending on the policy, decides:

  • none (default): no coordination; each manager does its own thing without aligning. No NUMA guarantee.
  • best-effort: it tries to align on one node; if it cannot, it admits anyway (on whichever node). Better than nothing, no guarantee.
  • restricted: if it fails to align, it rejects the pod. Strict, but it allows multi-node affinity if the intersection provides it.
  • single-numa-node: it requires everything to fit on a single NUMA node, or it rejects. The strictest one, and the one that genuinely guarantees the locality of the previous post.

And two preconditions without which none of this kicks in:

  1. The pod has to be QoS Guaranteed: requests == limits on CPU and memory, and whole CPUs (not 500m). Only then does the CPU Manager assign exclusive CPUs.
  2. The CPU Manager has to be on the static policy (not none).

Without those two, the Topology Manager has nothing to align and pinning does not happen, even with the policy in place. That is gotcha no. 1.

How it is configured on RKE2

RKE2 passes arguments to the kubelet with the kubelet-arg key in /etc/rancher/rke2/config.yaml. The reference configuration for GPU inference nodes:

# /etc/rancher/rke2/config.yaml  (on every agent node with GPUs)
kubelet-arg:
  - "cpu-manager-policy=static"
  - "topology-manager-policy=single-numa-node"
  - "topology-manager-scope=pod"
  - "memory-manager-policy=Static"
  - "reserved-cpus=0-1,64-65"            # housekeeping; must match the host
  - "system-reserved=memory=8Gi"
  - "kube-reserved=memory=4Gi"
  - "reserved-memory=0:memory=4Gi;1:memory=4Gi"   # required by Memory Manager Static
node-label:
  - "fibercli.local/numa-pinned=true"

After deploying it: systemctl restart rke2-agent. Critical gotcha: if the node has already run with cpu-manager-policy=none, there is a state file /var/lib/kubelet/cpu_manager_state that pins the old policy; changing the arg without deleting that file makes the kubelet fail to start or ignore the new policy. You have to stop the agent, rm /var/lib/kubelet/cpu_manager_state, and start again. (The same applies to memory_manager_state.)

And the vLLM pod, to be eligible, Guaranteed with whole CPUs:

resources:
  requests:
    cpu: "16"            # whole, not a weird fractional "16000m"
    memory: "200Gi"
    nvidia.com/gpu: "2"  # TP=2 → 2 GPUs on the same NUMA node
    hugepages-1Gi: "16Gi"
  limits:
    cpu: "16"            # == requests  → Guaranteed QoS
    memory: "200Gi"
    nvidia.com/gpu: "2"
    hugepages-1Gi: "16Gi"

With this, on a node with the config above, the kubelet assigns 16 exclusive CPUs from the NUMA node where the 2 requested GPUs live, its local memory and the hugepages, or it rejects the pod if they do not fit together. The artisanal pinning of the previous post, now declarative.

The 10 knobs to reach for

Ordered by dependency (the first ones are a precondition for the later ones). The canonical reference is the Kubernetes Topology Manager documentation and the RKE2 configuration.

Knob 1 — cpu-manager-policy=static: the foundation

Without this there are no exclusive CPUs and nothing else kicks in.

kubelet-arg: [ "cpu-manager-policy=static" ]

Gotcha: changing it requires deleting /var/lib/kubelet/cpu_manager_state and restarting the kubelet, or startup fails. It is cause no. 1 of “I set the policy and it does not pin”.

Knob 2 — Guaranteed QoS + whole CPUs: the pod’s precondition

This is not node config, it is pod config, but without it knob 1 does nothing for that pod. requests == limits on CPU and memory, and whole CPUs. A cpu: 500m or a requests != limits downgrades the pod to Burstable and it loses the pinning. Plenty of people set the node policy and forget the pod’s QoS.

Knob 3 — topology-manager-policy=single-numa-node: strict admission

kubelet-arg: [ "topology-manager-policy=single-numa-node" ]

The strict maître d’. For GPU inference it is the right policy: it guarantees that CPU+memory+GPU share a node. best-effort guarantees nothing (it admits misaligned); restricted allows multi-node affinity. Start with single-numa-node and drop to restricted only if you have admission problems.

Knob 4 — topology-manager-scope=pod: group the whole pod

kubelet-arg: [ "topology-manager-scope=pod" ]

With container scope (the default), each container is aligned separately; with pod scope, the whole pod goes to the same node. For a vLLM pod with sidecars (metrics, proxy), pod scope stops the sidecar from dragging the main container onto another node. Recommended for inference.

Knob 5 — memory-manager-policy=Static + reserved-memory: NUMA-local memory

kubelet-arg:
  - "memory-manager-policy=Static"
  - "reserved-memory=0:memory=4Gi;1:memory=4Gi"

Memory Manager Static forces cpuset.mems so that the pod’s memory comes out of the right node (and so do the hugepages). It requires declaring reserved-memory per node, or the kubelet does not start. It is the declarative equivalent of the previous post’s --membind.

Knob 6 — reserved-cpus: the housekeeping cores (must match isolcpus)

kubelet-arg: [ "reserved-cpus=0-1,64-65" ]

It reserves cores for the system and the daemons; the rest are left for exclusive pods. Key point of the series: these reserved-cpus must be the same cores you left out of isolcpus on the host (previous post). If the host isolates 2-31 but RKE2 reserves 0-3, there is a mismatch: isolated cores that the kubelet assigns to pods without them actually being quiet. Coordinate the two layers.

Knob 7 — GPU plugin with NUMA topology (NVIDIA GPU Operator)

The Device Manager can only give a correct NUMA hint if the GPU plugin exposes which node each GPU is on. The NVIDIA device plugin / GPU Operator does this, but you have to verify that the topology information arrives (in some versions it requires flags). Without a GPU hint, the Topology Manager aligns CPU and memory but not the GPU, and GPU locality is precisely the one that matters most.

Knob 8 — hugepages as a pod resource

resources:
  limits:
    hugepages-1Gi: "16Gi"   # the node must have them pre-reserved (post 2, knob 4)

The hugepages you reserved at host boot (previous post) are requested as a resource. The Memory Manager assigns them NUMA-local. If you request them without having reserved them on the node, the pod does not get scheduled.

Knob 9 — system-reserved / kube-reserved: do not oversubscribe

kubelet-arg:
  - "system-reserved=cpu=500m,memory=8Gi"
  - "kube-reserved=cpu=500m,memory=4Gi"

It reserves resources for the system and the K8s components so the node does not run out of air under load. Badly calibrated, either the node chokes (too little reserved) or you waste capacity (too much). It must be consistent with reserved-cpus.

Knob 10 — Labels + taints: so vLLM lands here and nothing else does

# GPU node: taint to repel anything that does not need a GPU
node-taint: [ "nvidia.com/gpu=present:NoSchedule" ]
node-label: [ "fibercli.local/numa-pinned=true" ]

Keep the NUMA-pinned nodes for inference and push out whatever does not need it (databases, the Langfuse backend, runners). A ClickHouse stealing memory bandwidth from a carefully pinned vLLM pod throws away all the work of the previous nine knobs. Workload isolation is the closing move.

Summary table

#KnobWhereFunction
1cpu-manager-policy=statickubelet-argexclusive CPUs (foundation)
2Guaranteed QoS + whole CPUspod specpinning precondition
3topology-manager-policy=single-numa-nodekubelet-argstrict NUMA admission
4topology-manager-scope=podkubelet-arggroup the whole pod
5memory-manager-policy=Statickubelet-argNUMA-local memory/hugepages
6reserved-cpuskubelet-arghousekeeping (match isolcpus)
7GPU plugin with topologyGPU OperatorNUMA hint for the GPU
8hugepages-1Gipod spechugepages as a resource
9system/kube-reservedkubelet-argdo not oversubscribe
10taints + labelsnode configisolate GPU workloads

Verifying that the pinning really happened

Do not trust that the config “is in place”. Check:

# Is the active policy the one you set?
cat /var/lib/kubelet/cpu_manager_state | jq .policyName    # "static"

# Which exclusive CPUs does the container have?
kubectl exec <pod> -- cat /sys/fs/cgroup/cpuset.cpus.effective

# Inside the pod: is the assigned GPU local to those cores?
kubectl exec <pod> -- nvidia-smi topo -m

# Were there topology rejections?
kubectl describe pod <pod> | grep -i TopologyAffinityError

A pod in Failed with TopologyAffinityError is not a bug: it is the maître d’ doing his job. That node did not have a table where CPU+memory+GPU fitted together. The answer is to review the sizing of the pod or the node, not to relax the policy casually.

How it connects with the rest of the stack

With the host (previous post). This post is the declarative automation of that one. cpu-manager-policy=static materialises the taskset; memory-manager-policy=Static materialises the --membind; reserved-cpus must match isolcpus. The two layers are one single decision seen from two places: the host executes it, the kubelet declares it. Uncoordinating them (isolcpus 2-31 vs reserved-cpus 0-3) breaks both.

With the interconnect (post 1). The Topology Manager pins the right GPU to the pod, but if you ask for 2 GPUs for TP=2, you will want those two to share NVLink. The NUMA policy guarantees they are on the same socket; that they are NVLink-connected is guaranteed by the baseboard hardware (post 1, knob 1). The two things together are what makes TP=2 perform.

With autoscaling. When KEDA scales vLLM pods, each new replica goes through Topology Manager admission. If the node has no free “table”, the pod stays pending. Pod autoscaling and node autoscaling (cluster-autoscaler) have to account for NUMA granularity, not just aggregate CPU/memory.

With capacity planning. Sizing changes: it is not “128 vCPU per machine”, it is “128 minus the reserved-cpus, in blocks that fit per NUMA node”. A machine with 2 sockets × 64 cores does not serve a pod asking for 80 cores under single-numa-node: they do not fit at one table. Planning has to reason per NUMA node, not per machine.

With service coexistence. The knob 10 taint is what keeps Langfuse, databases and runners off the inference nodes. Without that boundary, all the fine-grained pinning gets eaten by a noisy neighbour. Observability goes on its own nodes; inference, pinned, on its own.

Traps and things that are not what they seem

Policy in place, QoS forgotten. The most common mistake: cpu-manager-policy=static on the node but the pod is Burstable (requests != limits or fractional CPU). Pinning does not happen and nobody warns you. Guaranteed QoS with whole CPUs is a necessary condition.

Fossilised cpu_manager_state. Changing policy without deleting /var/lib/kubelet/cpu_manager_state (and memory_manager_state) makes the kubelet fail or ignore the change. Stop agent → delete file → start.

reserved-cpus ≠ isolcpus. If the host isolates some cores and RKE2 reserves others, the managers assign cores to pods that are not really quiet, or they leave isolated cores idle. The two lists have to be consistent. It is the coordination failure between the previous post and this one.

GPU plugin without NUMA topology. If the device plugin does not expose each GPU’s NUMA node, the Topology Manager aligns CPU and memory but leaves the GPU to chance, and GPU locality is the one that weighs most. Verify that the GPU Operator publishes the topology.

single-numa-node that rejects too much. If the pods ask for more resources than fit on one node (for example, more cores than a socket has), rejection is constant. The answer is not to drop to best-effort (which silences the problem by serving misaligned), but to size the pod so it fits at one table, or to accept restricted with full knowledge of what you are doing.

Believing best-effort “is nearly the same”. best-effort admits the pod even if it fails to align: it gives you a false sense of NUMA-awareness while you serve from the wrong socket. For inference with a tail SLO, single-numa-node or restricted; best-effort only if the alternative is scheduling nothing at all.

Conclusion

The previous post pinned by hand; this one does it at scale and with a guarantee that artisanal numactl did not give: strict admission. The kubelet, via CPU Manager, Memory Manager and Topology Manager, acts as a maître d’ who only seats the pod if its CPUs, its memory and its GPU fit at the same NUMA table, and who rejects what does not fit instead of serving a split dinner. Of the ten knobs, the first two, cpu-manager-policy=static and Guaranteed QoS with whole CPUs, are the precondition without which the other eight do nothing, and they are exactly the ones most often forgotten; the rest tune the policy, the memory, the hugepages and the coexistence. The thread that closes the series: the inference performance that looked like an engine problem (vLLM is slow) or a model problem (quantisation) is, far too often, a cable problem (NVLink not used), a host problem (remote NUMA, jitter) or an orchestration problem (pinning that never happened because the QoS was wrong). Going down a level is not infrastructure snobbery: it is where the root causes live that no application-layer dashboard is ever going to point out for you.

See also

References