Gpudirect on lo0 — Tech Blog

Storage in the AI era (2/4): performance

Mon, 22 Jun 2026 11:00:00 +0200

In the first article we drew the map: the memory-storage hierarchy, the media technologies and the software stack that dominates GPU clusters. This second article goes into the property that justifies all that complexity: performance. The question an architect has to answer is not “how many GB/s does my array deliver?”, but “can I get my GPUs working at 95 % instead of 40 %?”. That difference is, literally, half the compute bill.

The metric that really matters: accelerator utilisation

It is tempting to measure AI storage with the classic metrics, sequential throughput, random IOPS, latency, and we do need them. But none of them captures what the business pays for. The integrating metric is accelerator utilisation: what fraction of the time the GPU spends computing rather than waiting for data.

Most real ML pipelines run at between 30 % and 60 % GPU utilisation. Industry surveys are damning: only around 7 % of AI/ML teams manage to exceed 85 % at peak, and poor forecasting or weak autoscaling can leave GPUs idle 70-85 % of the time. A job running at 30 % utilisation is paying for 70 % of its compute capacity and getting nothing for it.

We can formalise this. If $ t_{\text{cmp}} $ is the compute time per batch and $ t_{\text{io}} $ the time the GPU spends blocked waiting for I/O that fails to overlap with compute, effective utilisation is:

$$U = \frac{t_{\text{cmp}}}{t_{\text{cmp}} + t_{\text{io}}}$$

The goal of all AI storage engineering is to drive $ t_{\text{io}} $ towards zero, whether by increasing bandwidth, reducing latency or overlapping I/O with compute through prefetching and pipelining. And there is an aggravating factor: the faster the GPU, the harder it is to feed. Moving from V100 to H100 cut the per-batch compute time of the 3D U-Net model by 76 %, turning a bandwidth-sensitive workload into a latency-sensitive one. Every accelerator generation raises the bar for storage.

This change of regime, from bandwidth-limited to latency-limited, has a design consequence worth internalising. When a workload is bandwidth-limited, the fix is to add transfer capacity: more NVMe, more lanes, more nodes. When it becomes latency-limited, that recipe stops working; what matters then is response time per operation, which is attacked with cache, with intelligent prefetching and with data paths that remove hops (GPUDirect, for instance). Diagnosing correctly which of the two regimes each workload sits in is the first task of any optimisation: throwing bandwidth at a latency-limited workload is spending money without moving utilisation.

MLPerf Storage: the benchmark that does measure the right thing

For years there was no neutral way to compare AI storage platforms. MLPerf Storage, from MLCommons, has provided one. Its design insight is not to measure GB/s in the abstract but to simulate the accelerators’ think time so as to generate a realistic I/O pattern, and to require those simulated accelerators to sustain a minimum utilisation level. In other words, it measures exactly what matters: how many accelerators a platform can keep fed without their utilisation dropping below the threshold.

v1.0 (September 2024) used the 3D U-Net, ResNet-50 and CosmoFlow models, simulating A100 and H100, with utilisation thresholds of 90 % for the first two and 70 % for CosmoFlow. v2.0 (August 2025) marked a leap: more than 200 results from 26 organisations, including Alluxio, DDN, Hammerspace, HPE, IBM, KIOXIA, Micron, Oracle, Samsung and WDC, systems able to sustain roughly twice as many accelerators as in v1.0, and, above all, a new checkpointing workload (which we will come back to).

Some v2.0 results help calibrate the state of the art:

System	Throughput	Accelerators	GPU utilisation
Volumez	1.079 TB/s	—	92.21 %
Hammerspace (5 nodes)	420.8 GB/s	140	>94.7 %
Hammerspace (3 nodes)	253.1 GB/s	84	>94.7 %
Hammerspace (1 node)	85.6 GB/s	28	>94.7 %
Alluxio	24.14 GiB/s	128	99.57 %

The Hammerspace figure illustrates a property an architect should demand: linear scaling. From 1 to 5 nodes throughput grows from 85.6 to 420.8 GB/s while keeping utilisation above 94.7 % with a minute coefficient of variation (0.08-0.14 %). Linear scaling is what separates an AI platform from a fast NAS.

The numbers from commercial platforms

Outside the benchmark, platforms certified for DGX SuperPOD publish aggregate figures worth keeping as an order-of-magnitude reference:

WEKA WEKApod Nitro: 720 GB/s read and 186 GB/s write per configuration, 18 million IOPS, with ConnectX-8 NICs at 800 Gb/s. Per node, 70 GB/s read and 40 GB/s write; the minimum 8-node configuration delivers 560 GB/s read. A single entry-level system covers the I/O demand of a 1,152-GPU GB200 scalable unit.
DDN AI400X2: more than 90 GB/s and 3 million IOPS per appliance; the Turbo variant saturates close to 100 GB/s (800 Gbps) per DGX B200. DDN’s new platform claims more than 1 TB/s read per appliance with per-rack scaling, and has fed NVIDIA’s Eos supercomputer at 4 TB/s.
VAST Data: multiple TB/s from a single mount point via NFS multipathing, RDMA over NFS and GPUDirect, managing exabytes in a single multiprotocol namespace.
Pure Storage: certified for DGX SuperPOD with GB200 and GB300.

These figures only mean something in relation to the cluster they feed. A GB200 NVL72 moves 130 TB/s of internal GPU-to-GPU communication; storage does not compete with that, but it does have to sustain data loading and checkpointing without strangling 72 GPUs that behave as one.

The I/O pattern of training and GPUDirect Storage

Understanding the I/O pattern is the key to neither over- nor under-sizing. Training combines several patterns: large sequential reads of the dataset, small random reads during shuffling between epochs, and massive periodic writes during checkpointing. Each stresses storage differently.

The traditional data path goes through the CPU: from storage to a bounce buffer in host memory and from there to GPU memory. Every byte of every checkpoint and every weight load pays the latency and CPU overhead of that double hop. GPUDirect Storage (GDS) cuts it out: it enables direct DMA between GPU memory and NVMe through the cuFile user-space library and the nvidia-fs kernel module, which intercepts POSIX calls and redirects them through the DMA engine. The result is zero-copy, with direct transfers exceeding 40 GB/s and, above all, without stealing cycles from the CPU.

There is a nuance that naive deployments miss: GDS is optimised for large, aligned, sequential transfers. For workloads dominated by small files or heavily random access, the traditional path with CPU caching can be more efficient. GDS is not a magic “more performance” switch; it is a tool for a specific pattern. Designing the data pipeline, meaning dataset format, shard size and number of DataLoader workers, matters as much as picking the array.

The data pipeline deserves a pause, because it is where performance leaks most quietly. Training does not read the dataset just any way: it walks it in batches, shuffles it between epochs to avoid ordering bias, and often applies transformations (image decoding, tokenisation, augmentation) on the CPU before handing the tensor to the GPU. Each of those steps can become the bottleneck. Too few DataLoader workers leaves the GPU waiting; a dataset format that forces millions of small files open sinks metadata performance; a badly designed shuffle turns efficient sequential reads into costly random accesses. Common practice in serious workloads is to pack the dataset into large shards (formats such as WebDataset, TFRecord or Parquet) precisely so that the read pattern is large and sequential, the terrain where GDS and parallel file systems perform best. The mental rule: the fastest storage in the world will not save a badly designed data pipeline.

The real cost of checkpointing (when there is training)

Checkpointing is the most demanding write workload in the AI world, but it needs to be placed properly: it is a training problem. A pure inference factory barely suffers from it, since its heavy writes are different ones, KV-cache, vector indexes, telemetry. We cover it here because many installations combine some fine-tuning or occasional training with serving, and because it illustrates better than any other case how asynchrony beats brute force. If your platform is for inference, you can read this section as context and jump to the next one, which is the one that really concerns you.

That said, most people’s intuition about checkpointing is wrong on two counts: size and frequency.

Size. A checkpoint is not just the weights. The rule of thumb is around 16 bytes per parameter, because the bulk is optimiser state. The figures from the MLPerf Storage v2.0 checkpointing workload, based on Meta’s Llama 3 Herd, make it plain:

Model	Processes	Checkpoint size	Of which, optimiser state
8B	8	105 GB	90 GB
70B	64	912 GB	—
405B	512	5.29 TB	—
1T	1024	15 TB	13.2 TB

Frequency. At scale, hardware failures are constant (we develop this in the article on availability). Meta’s model for a 16,000-accelerator cluster predicts a failure every few hours. To lose less than 5 % of progress between failures, you need on the order of 20 checkpoints in that interval. That translates into aggressive cadences:

16,000 accelerators: around 155 checkpoints a day, one every 9.3 minutes.
100,000 accelerators: around 967 checkpoints a day, one every 1.5 minutes.

And here is where the bandwidth problem appears. A synchronous checkpoint halts training while it writes. If $ S_{\text{ckpt}} $ is the checkpoint size and $ t_{\text{win}} $ the acceptable time window for writing it while keeping overhead below 5 %, the required bandwidth is:

$$B_{\text{req}} = \frac{S_{\text{ckpt}}}{t_{\text{win}}}$$

For a 1T model on 100,000 accelerators, the window falls to around 4.4 seconds. Writing 15 TB in that time demands on the order of 3.6 TB/s across the cluster, or some 200 GB/s sustained over the whole job. On checkpoints alone, that scenario generates more than 14 PB of writes a day. This is not a marginal workload: it is one of the most demanding write workloads in existence.

It is worth seeing the calculation laid out, because it reveals where the lever is. Take the 405B model, whose checkpoint weighs 5.29 TB. If policy requires writing it while keeping overhead below 5 % at a cadence where the acceptable window is, say, 30 seconds, the synchronous bandwidth needed is around 176 GB/s sustained, within reach of a high-end array, but only one dedicated to that task. If instead of 30 seconds the window contracts to 5 (a larger cluster, more frequent failures), the requirement jumps above 1 TB/s. Sensitivity to the window is brutal: the difference between a viable design and an impossible one lies in how time is managed, not in how many drives are bought. That is where asynchrony changes the rules.

The solution is not just more bandwidth; it is asynchrony. Asynchronous checkpointing decouples the snapshot (a fast GPU→CPU copy) from persistence (a background write, overlapped with compute). PyTorch’s Distributed Async Checkpointing cuts effective checkpoint time by 10 to 20 times; in one IBM example a 7B model brought its down time down from 148.8 to 6.3 seconds, a 23.6-fold improvement. Techniques such as CheckFreq pipeline the snapshot and persistence with training, allowing a checkpoint every 14-19 iterations. This is why vendors like VAST argue the relevant metric is not GB/s but checkpoint overlap: what fraction of the checkpoint overlaps with compute. Around 10 % overlap is usually enough.

The design lesson is clear: sizing training storage for the peak write rate of synchronous checkpointing is expensive and often unnecessary if the software stack gets asynchrony right. But trusting asynchrony without measuring overlap is playing roulette with weeks of compute.

Performance in an inference factory

Here is the heart of the matter for anyone running an inference factory. In an inference factory storage is not measured in epochs or GPU-hours but in the experience the client perceives, and that experience condenses into two metrics: TTFT (time to first token, the time until the first token of the answer) and inter-token latency (the rate at which the following tokens arrive). At fleet scale, the metric that pays the bills is throughput in tokens per second per GPU, because it determines how many requests each accelerator serves. Storage touches all three, in ways a training-centric design does not anticipate.

Prefill, decode and their disaggregation

Inference has two phases with opposite profiles. Prefill processes the entire input prompt at once, generates the key-value pairs of the KV-cache and is a compute-intensive phase; it dominates TTFT. Decode generates tokens one by one, reusing that cache, is memory-bandwidth-intensive and dominates inter-token latency. Because their bottlenecks differ, the 2025-2026 trend is to disaggregate them: run prefill and decode on separate GPU pools and transfer the KV-cache between them. That transfer, which may travel over NVLink, over the network or through a shared storage layer, makes KV-cache movement a performance-critical operation. A factory that disaggregates prefill and decode needs a very high bandwidth, low latency KV-cache path between both pools; storage (or the DPU that governs it) stops being a passive actor and enters the critical path of every request.

The KV-cache as a performance layer

The central problem is the size of the KV-cache with long contexts: it overflows HBM, which is scarce and expensive. The options are to recompute it on every request, ruinously expensive in compute and lethal for TTFT, or to offload it to a slower layer and reuse it. Two approaches compete here. Offload to CXL memory offers gains of more than 5 times against SSD or RDMA-based caching, according to CXL supply chain vendors, and allows batches up to 30 % larger while holding latency targets, which raises throughput and GPU utilisation. Offload to NVMe, standardised by NVIDIA in 2026 with its ICMS platform on BlueField-4, is slower but much cheaper and more capacious, ideal for pod-scale KV-cache.

The big performance lever here is KV-cache reuse (prefix caching). Many requests share prefixes: the same system prompt, the same context document, the same prior conversation. If the KV-cache of those prefixes is kept in a fast storage layer and reused instead of recomputed, prefill shortens drastically and TTFT collapses. This turns a low-latency storage layer into a direct multiplier of the factory’s throughput: the more useful cache fits and the faster it is served, the less compute is wasted recalculating what has already been calculated. Sizing this layer well, in capacity, bandwidth, latency and eviction policy, is one of the most profitable performance decisions in the whole factory.

Model load time and cold start

There is a storage cost specific to inference that training ignores: starting a replica. When autoscaling decides to bring up a new instance of a model, because demand is rising, because a replica has failed or because a version is being rotated, the weights have to be read from the registry and loaded into HBM. For a model of tens or hundreds of GB, that cold start can run from seconds to minutes depending on registry bandwidth and the load path. And during that time the new GPU serves no requests while the system is, perhaps, saturated. A slow model registry translates directly into missed SLAs at peak. The mitigations, keeping weights on hot local flash, using fast-loading formats, streaming weights as they are needed, or GPUDirect Storage to load straight into HBM, are storage decisions that determine the factory’s real elasticity.

RAG performance and vector search

If the factory serves RAG, there is one more link in the critical path of every request: retrieval. Before the model generates anything, the system searches a vector database for the relevant fragments, and that similarity search, over indexes that can run to TB, adds to TTFT. Vector search performance depends on keeping the indexes on low-latency flash and on an efficient random read pattern; an index that does not fit in memory and is served from slow storage adds latency to every request. Beyond that, precomputed KV-caches of the retrieved documents speed up prefill, because the model reuses keys and values already calculated and only encodes the user’s query. An inference factory with RAG therefore has two latency-sensitive read workloads coexisting, the vector index and the KV-cache, competing for the same fast flash and needing to be sized together.

For an inference architect, all this introduces a storage plane with a profile of its own: latency-sensitive random reads (indexes, reusable KV-cache prefixes), high-turnover ephemeral writes (new KV-cache), large, sporadic but urgent reads (model loading) and an append-only telemetry flow. It is not sized in the least like the storage of a training cluster.

How to measure your own: beyond the spec sheet

Vendor and MLPerf figures are useful as a reference, but real performance depends on the specific workload, and an architect must measure rather than assume. The methodology that works starts from three principles.

The first is to measure accelerator utilisation, not storage in the abstract. An array can deliver 500 GB/s in a synthetic test and still leave the GPUs at 50 % because the real pattern is small random files that the sequential test does not reproduce. The right question is always: what fraction of the time is my GPU computing, with this workload, this dataset format and this pipeline?

The second is to reproduce the real I/O pattern, not a convenient one. That means testing with the same dataset format, the same shard size, the same number of workers and the same checkpointing cadence that will be used in production. MLPerf Storage’s insight was precisely to simulate accelerator think time in order to generate a realistic pattern; replicating that rigour at your own scale is what separates a useful measurement from a marketing number.

The third is to look for scaling, not peak. A platform that delivers a very high peak in a small configuration but flattens out as nodes are added is no use for growth. Tests should be run at several scales to verify that throughput grows roughly linearly and that accelerator utilisation holds, as the Hammerspace example in MLPerf v2.0 showed. A low coefficient of variation across configurations matters as much as the peak.

The cost of getting it wrong

It is worth putting a number on the problem. The price of an H100 in the cloud in mid-2026 moves across a wide range by provider, with a market median around 2.3-3.1 USD per GPU-hour. Multiply that by thousands of GPUs and by the weeks a training run lasts, and 40 % utilisation instead of 90 % translates into millions of euros of wasted compute. That is why more than 50 % of organisations report data or storage bottlenecks limiting their AI in 2026, and why storage bandwidth has emerged as a hard ceiling on scaling, comparable to power and cooling: when there is data starvation, adding compute gives diminishing returns.

The operational recommendation that keeps coming up is to size the I/O pipeline for 300 Gbps or more in serious workloads, and to measure, not assume, accelerator utilisation as a health indicator. Optimised storage can deliver up to 5 times the throughput of standard S3 access over HTTP: more than 100 GB/s aggregate against some 20 GB/s.

The physical layer: networks and PCIe Gen6

All of the above rests on a physical layer that has also advanced. NVIDIA’s ConnectX-8 NIC combines PCIe Gen6 with 800 GbE or 800 Gb InfiniBand XDR, designed for Blackwell systems. InfiniBand XDR offers 800 Gbps per direction (double NDR, which remains the usual storage fabric), and the Quantum-X800 switch adds in-network compute that accelerates collective operations. The transition to XDR is accelerating with the Blackwell deployments of 2025-2026. The mental rule is simple: storage must match the network, and the network must match the GPU. One slow link defines the performance of the whole chain.

It is worth translating this into concrete numbers. An 800 Gb/s NIC delivers, at best, around 100 GB/s to a node. If the array can serve more than the NIC absorbs, the network is the bottleneck; if the array serves less, storage is. Balance is found by sizing each layer so none strangles the next, and by verifying it with end-to-end measurements rather than the isolated specifications of each component. It is common to find installations with an excellent array and an undersized fabric, or the reverse, performing far below the sum of their parts. The physical layer also imposes its geography: network topology (how many hops there are between the compute node and the array, whether they share a switch or cross the cluster spine) affects latency as much as the array itself, and an AI storage design that ignores network topology is incomplete.

Three questions before sizing

All this complexity can be organised into three questions an architect should answer before buying anything. What is the dominant I/O pattern of my workload, large sequential, small random, bursty writes, and therefore which metric limits me? Is my workload bandwidth-limited or latency-limited, and consequently where do I invest? And what is my checkpointing budget, in size, cadence and, above all, what fraction can I overlap with compute? Whoever answers these three with measured data, rather than intuitions, sizes well; whoever skips them ends up with idle GPUs or with an oversized, very expensive array that is never put to use. AI storage performance is, at bottom, an exercise in fit: putting the right capacity where the workload needs it, no more and no less.

Takeaways

AI storage performance is not measured in GB/s but in accelerator utilisation, and that utilisation depends on three things: enough bandwidth, low latency and, above all, overlapping I/O with compute. MLPerf Storage v2.0 has finally given a common language for comparing platforms through that lens. Checkpointing is the most demanding write workload and is mastered with asynchrony, not hardware alone. Long-context inference adds a new storage layer around the KV-cache. And the cost of getting it wrong is measured in idle GPUs, which is the most expensive line in the budget.

Performance, however, is worth nothing if the data is neither protected nor available. The third article takes on the security of this new storage layer.

Sources

MLCommons, MLPerf Storage v2.0 Results — https://mlcommons.org/2025/08/mlperf-storage-v2-0-results/
MLCommons, Announcing the MLPerf Storage v2.0 Checkpointing Workload — https://mlcommons.org/2025/08/storage-2-checkpointing/
MLCommons, MLPerf Storage v1.0 Benchmark Results — https://mlcommons.org/2024/09/mlperf-storage-v1-0-benchmark-results/
Hammerspace, MLPerf Storage v2.0 Benchmark Results — https://hammerspace.com/hammerspace-mlperf-storage-v2-0-benchmark-results/
Alluxio, Strong Performance in MLPerf Storage v2.0 — https://www.alluxio.io/blog/alluxio-demonstrates-strong-performance-in-mlperf-storage-v2-0-benchmarks
NVIDIA, GPUDirect Storage Overview Guide — https://docs.nvidia.com/gpudirect-storage/overview-guide/index.html
Spheron, GPU Direct Storage Guide 2026 — https://www.spheron.network/blog/gpu-direct-storage-nvme-ai-training-inference-guide/
PyTorch, Reducing Checkpointing Times with Async Checkpointing — https://pytorch.org/blog/reducing-checkpointing-times/
VAST Data, Optimizing Checkpoint Bandwidth for LLM Training — https://www.vastdata.com/blog/optimizing-checkpoint-bandwidth-for-llm-training
Astera Labs, How CXL Transforms RAG and KV Cache Performance — https://www.asteralabs.com/breaking-through-the-memory-wall-how-cxl-transforms-rag-and-kv-cache-performance/
MinIO, AI Storage Architecture: Overcoming the Bottleneck in 2026 — https://www.min.io/blog/ai-storage-architecture-bottleneck-2026
WEKA, WEKApod Nitro reference architecture — https://www.weka.io/resources/reference-architecture/nvidia-gb200-nvl72-systems-reference-architecture-for-cloud-partners/
Blocks & Files, Storage vendors rally behind Nvidia at GTC 2025 — https://blocksandfiles.com/2025/03/18/nvidia-storage-announcements/
ServeTheHome, NVIDIA ConnectX-8 SuperNIC PCIe Gen6 800G — https://www.servethehome.com/nvidia-connectx-8-supernic-pcie-gen6-800g-nic-detailed/
SpendArk, GPU Cloud Pricing 2026 — https://spendark.com/blog/machine-learning-cloud-cost/

The corridors and the security guard: PCIe topology, GPUDirect P2P and ACS

Mon, 08 Jun 2026 06:00:00 +0200

This continues the below the engine series. The NVLink and NCCL post explained the shared bench over which the GPUs pass data to each other at 450 GB/s. But that bench only connects GPUs to each other. Everything else, disk, network, the host, travels over another bus, PCIe, and through its corridors. The cold start already brushed against this with GPUDirect Storage; this post opens up the complete floor plan of the corridors and the guard who watches them.

TL;DR

On a 4×H100 SXM node, the GPUs talk over NVLink (450 GB/s per direction, around 7× PCIe), and for the all-reduce of tensor parallel that is the path. But PCIe does not go away: it is where everything else comes in. The weights come up from the NVMe over PCIe (the cold start), the data from another node arrives via the NIC over PCIe (RDMA), and a KV that moves between nodes travels over PCIe. GPUDirect is the family that lets those bytes go straight from the device into the HBM without bouncing through host RAM: P2P (GPU↔GPU), RDMA (GPU↔NIC) and Storage (GPU↔NVMe). The obstacle is a guard called ACS (Access Control Services): a PCIe security feature that by default forces peer-to-peer traffic up to the root complex for inspection, which destroys the direct path and inserts a detour through the CPU. The IOMMU (VT-d) does something similar if it is not in passthrough mode. Disabling them gives performance; keeping them gives isolation and virtualisation, and that is a real decision in an ENS environment. This post explains the topology (nvidia-smi topo -m), GPUDirect, why ACS and IOMMU break P2P with numbers, the 10 knobs and the trap of removing the guard without knowing what it was watching. On the generic 4×H100 SXM cluster.

Where you are: the corridors, not the bench

Picture the kitchen as a building. The elite cooking stations, the GPUs, are in a room with a huge central bench (NVLink/NVSwitch) over which they pass ingredients at full speed without standing up. That bench is for them and them alone.

But the building has more to it: the pantry (the NVMe storage), the loading door (the network, the NIC) and reception (the CPU and its RAM). To reach any of those, the stations do not use the central bench: they use the corridors of the building, the PCIe bus. And here the character of the post appears: at the entrance of each corridor there is a security guard (ACS) who, by default, does not let two stations hand each other something directly along the corridor: it forces them to take the parcel up to reception to be checked, and only then does it come down to its destination. It is safe, but it is an absurd detour when the two stations are right next to each other. GPUDirect is the permission to skip that detour; ACS and IOMMU are the reasons why, often, you cannot.

The topology of a node: two buses, not one

The most common mistake is to think that a node has “a bus”. There are (at least) two, and they do different things:

NVLink / NVSwitch — the high-speed GPU↔GPU mesh. On H100 SXM, 18 links × 50 GB/s = 900 GB/s bidirectional between any two GPUs, with NVSwitch giving an all-to-all with no contention (NVLink, NVIDIA). It is the shared bench.
PCIe Gen5 — the general I/O bus. An x16 link gives 128 GB/s bidirectional (around 64 per direction) (H100 product brief). It connects each GPU with the CPU, the RAM, the NICs and the NVMes. It is the corridor.

The difference is 7×: NVLink moves in one second what PCIe takes seven to move. That is why intra-node tensor parallel goes over NVLink and nobody argues about it. PCIe matters for the other things: pulling weights off disk, receiving from the network, moving KV between nodes.

The tool for seeing this is nvidia-smi topo -m, which prints a matrix of how each pair is connected (NVIDIA forum):

 GPU0 GPU1 GPU2 GPU3 NIC0 CPU Affinity NUMA
GPU0 X NV18 NV18 NV18 PXB 0-47 0
GPU1 NV18 X NV18 NV18 PXB 0-47 0
GPU2 NV18 NV18 X NV18 SYS 48-95 1
GPU3 NV18 NV18 NV18 X SYS 48-95 1

The legend is what matters: NV18 = 18 NVLink links (the bench); PXB = crosses PCIe switches but not the host; PHB = goes through the host bridge; NODE = same NUMA, crossing PCIe; SYS = crosses the inter-socket interconnect (the worst case, it traverses NUMA). The fact that GPU0↔NIC0 is PXB and GPU2↔NIC0 is SYS tells you exactly which GPU should handle that NIC’s traffic, GPU 0, without crossing NUMA. This links directly with the NUMA post and the network NUMA one: PCIe affinity is NUMA affinity.

GPUDirect: skipping reception

Without GPUDirect, moving a piece of data from the NIC (or the NVMe) to the GPU takes a mandatory detour: device → host RAM → GPU. That bounce through RAM consumes CPU bandwidth, spends copies and adds latency. GPUDirect removes the bounce by letting the data go straight from the device into the HBM. Three flavours:

GPUDirect P2P — GPU↔GPU over PCIe (when there is no NVLink between them, or for traffic that does not use the bench).
GPUDirect RDMA — GPU↔NIC: the network card writes directly into the HBM. It is what makes efficient multi-node viable (NCCL over InfiniBand/RoCE).
GPUDirect Storage (GDS) — GPU↔NVMe: the disk writes straight into the HBM, with no host buffer. It is the lever of the cold start.

On an SXM node, the GPU↔GPU traffic of tensor parallel does not use P2P over PCIe: it uses NVLink. That is why GPUDirect matters above all at the edges of the node: the network (RDMA, for multi-node) and the disk (GDS, for startup). That is where ACS does damage.

The guard: why ACS and IOMMU break P2P

ACS (Access Control Services) is a PCIe security feature designed for virtualisation and isolation: it guarantees that one device cannot read from or write to another directly without the root complex mediating. To achieve that, it forces peer-to-peer transactions up to the root complex and back down (GDS best practices, NVIDIA). It is exactly the opposite of what GPUDirect wants: the direct path stops being direct.

The IOMMU (VT-d on Intel, the equivalent on AMD) translates addresses and isolates devices. If it is active and not in passthrough mode, it also redirects P2P traffic through the root complex, with the same effect: performance through the floor or, in extreme cases, hangs (NCCL troubleshooting).

Summarised without detours (Giraud, debugging P2P): ACS forces the trip through the root for security checks; IOMMU forces it for isolation and virtualisation. Both break the goal of P2P (direct communication with no intermediaries) and add overhead. If you do not need that security/virtualisation on that path, disabling them recovers the performance. The operational recipe for maximum GPUDirect performance: ACS off on the switches along the path and IOMMU in passthrough (iommu=pt) or disabled.

The maths that matter: how much the detour costs

Take a KV SWAP of 5 GB (preemption from the scheduler that sends KV to the host, or a transfer between nodes in disaggregated serving):

Path	Effective BW	Time for 5 GB
NVLink (GPU↔GPU intra-node)	~450 GB/s	~11 ms
Direct PCIe Gen5 x16 (P2P, ACS off)	~55 GB/s	~91 ms
PCIe via root complex (ACS on)	~25-30 GB/s*	~170-200 ms

* The detour through the root does not just “add latency”: it saturates the host bridge bandwidth, contends with other traffic and, depending on the topology, can fall well below the direct path. The figure is an order of magnitude, to show the size of the problem, not a benchmark.

The reading: on the path that does use PCIe (network, disk, swap), having ACS on can double or triple the time. And if that time is on the critical path, a cold start, a preemption swap, an inter-node all-reduce, it shows up in the latency the user sees. What disabling ACS does not fix: traffic that already went over NVLink (intra-node TP). There, ACS is irrelevant.

The real tension: performance vs isolation (and ENS)

Here the post gets serious, because the “disable ACS and IOMMU” recipe has a cost that in a regulated environment is not free. ACS and IOMMU exist for a reason: to isolate devices. On a dedicated bare-metal node for inference, with no virtualisation or multi-tenancy, you are not isolating anything that matters and disabling them is reasonable. But:

If you do GPU passthrough to VMs or use containers with strong isolation, the IOMMU is necessary, not optional.
In a multi-tenant scenario where several workloads share a node, ACS provides a guarantee that one device does not snoop on another.
Under ENS (see technical controls), workload isolation and access traceability can be requirements; disabling bus isolation to gain 80 ms is a decision you have to justify and document, not a silent tweak.

The design way out, when you need both: keep the isolation where compliance requires it and design so that the hot path does not depend on P2P over PCIe. Concretely, on an SXM node, the bulk of the critical traffic (TP) already goes over NVLink and ACS does not affect it. For the network, dedicate one NIC per GPU on its own PCIe switch (PXB) and use GPUDirect RDMA only on the path you control. For the disk, cache weights on local NVMe. That way you do not pay the choice between performance and isolation: you avoid it on the path that matters.

The 10 knobs

#	Knob	What it controls	Cost / risk
1	ACS off (switches on the path)	P2P detour through root	you lose bus isolation
2	`iommu=pt` / off	P2P redirection through root	breaks VM passthrough if off
3	`nvidia-smi topo -m`	audit the real topology	— (always worth it)
4	`p2pBandwidthLatencyTest`	measure P2P for real	— (verify before assuming)
5	NIC placement	same PCIe switch as the GPU	SYS if it crosses NUMA
6	NVMe placement	NUMA-local to the GPU	H2D crossing UPI
7	`nvidia-peermem` (GDR)	enables RDMA into HBM	correct driver/kernel
8	PCIe gen/lanes (x16)	width of the corridor	GPU silently at x8
9	relaxed ordering / ASPM	PCIe latency and power	jitter if misconfigured
10	persistence mode	avoids re-init of the path	idle GPU still paid for

How it connects with the rest of the stack

With NVLink and NCCL. The NVLink post covers the GPU↔GPU bench; this one covers the other bus, the one connecting to disk, network and host. They are complementary: ACS affects PCIe, not NVLink.

With the cold start. The GPUDirect Storage of the disk→HBM post is GPUDirect over the storage path; ACS on strangles it just as it strangles P2P.

With NUMA. The PCIe affinity of topo -m is the NUMA affinity of the host post; placing NIC and NVMe on the correct NUMA avoids the SYS path.

With the network. NIC placement and GPUDirect RDMA is the subject of the network NUMA post; the same principle of “get the CPU out of the way”.

With PagedAttention and the scheduler. The preemption SWAP (scheduler) moves KV blocks over PCIe; that is why V1 prefers RECOMPUTE and why this bus matters.

With disaggregated serving. Transferring KV between pools in disaggregated serving travels PCIe→NIC→PCIe; ACS and placement decide whether it is viable.

With ENS. Bus isolation is a technical control; see ENS/42001/AI Act controls.

Traps and things that are not what they look like

“Disable ACS everywhere, it goes faster.” On a dedicated node, fine. On one with virtualisation, multi-tenancy or isolation requirements (ENS), you are removing a security control. The correct decision is per path and documented, not global and silent.

“NVLink and PCIe are the same bus, more or less.” No. They are two buses with a 7× difference and different purposes. TP goes over NVLink; disk, network and host go over PCIe. Confusing them leads to “optimising” ACS for traffic that does not even go through PCIe.

“P2P just works, there is nothing to check.” P2P is silently disabled with ACS/IOMMU active, and many distros enable them by default. Check with p2pBandwidthLatencyTest and nvidia-smi topo -m; do not assume you have the direct path just because the GPUs are on the same node.

“IOMMU off always, for performance.” If you do GPU passthrough to virtual machines, the IOMMU is mandatory; disabling it breaks the passthrough. The correct mode is usually passthrough (iommu=pt): it keeps the necessary mapping without penalising P2P.

“More PCIe lanes = faster GPU.” PCIe is the I/O path, not the compute path. A GPU at x8 instead of x16 takes longer to load and to communicate over PCIe, but generates tokens at the same speed once the weights are inside. The damage from x8 is in the cold start and in multi-node, not in decode throughput.

“GPUDirect fixes any I/O bottleneck.” GPUDirect removes the bounce through the CPU; if your bottleneck is the device itself (saturated NVMe, NIC at full tilt) or the topology (a SYS path crossing NUMA), GPUDirect does not touch it. Measure where the bottleneck is first.

Conclusion

This whole series has been going down floors looking for where the time goes, and this one reaches the wiring of the building. Intuition treats the node as a homogeneous box where “the GPUs talk to everything”; the reality is that there are two buses with opposite purposes, an elite bench for the GPUs (NVLink) and some service corridors for everything else (PCIe), and a security guard in the corridors who, with the best of intentions, makes every parcel go up to reception before it is delivered. GPUDirect is the permission for direct delivery; ACS and IOMMU are the legitimate reasons why sometimes you are not given it. The lesson is not “disable the guard”: it is understanding which path is critical (almost never the one you think) and what the guard was watching before sending it home. On a dedicated node, the direct path is almost free and worth taking. On one that shares workloads or lives under ENS, bus isolation is a control you sacrifice with justification or do not sacrifice at all. Good design does not choose between performance and isolation blindly: it puts the critical traffic on the bench that needs no guard, and leaves the corridors for what can wait.

References

NVIDIA, GPUDirect Storage Best Practices Guide (ACS, IOMMU, paths): https://docs.nvidia.com/gpudirect-storage/best-practices-guide/index.html.
NVIDIA, NCCL Troubleshooting (IOMMU/VT-d and P2P): https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html.
M. Giraud, Multi-GPU (NVIDIA) P2P capabilities and debugging tips: https://morgangiraud.medium.com/multi-gpu-nvidia-p2p-capabilities-and-debugging-tips-fb7597b4e2b5.
NVIDIA, H100 Product Brief (PCIe Gen5, NVLink 900 GB/s): https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/h100/PB-11773-001_v01.pdf.
NVIDIA, GPUDirect RDMA documentation: https://docs.nvidia.com/cuda/gpudirect-rdma/index.html.

The kitchen door the maître d' never looked at: network NUMA, Cilium eBPF and DRANET, the fourth leg of pinning

Sat, 06 Jun 2026 12:00:00 +0200

Fourth instalment, a coda, of “under the engine”. The series closed with three legs of locality: the cable between GPUs, the host by hand and the declarative orchestration of the kubelet. But the maître d’ of the last post seated the group looking at CPU, memory and GPU, and never asked which door the dishes come through. That door is the NIC. Here is the fourth leg.

TL;DR

The Topology Manager admits a pod under single-numa-node if its CPUs, its memory and its GPU fit on the same NUMA node. The NIC is not part of that count: the kubelet has no Hint Provider for the network card. On an inference node with the network at 200/400 Gb/s, the case of disaggregated serving, where the KV-cache travels over RDMA between the prefill pool and the decode pool, a NIC on the wrong socket makes every packet cross the UPI/QPI, exactly the “remote NUMA” the series fights on the compute side, but coming in through the network door. And there is a second front: the softirq (NET_RX) that processes the datapath runs on the CPU that services the NIC’s IRQ; if that CPU is one of the cores isolcpus/reserved-cpus gave exclusively to vLLM, the softirq steals cycles from it and injects jitter into the p99 tail. Cilium eBPF replaces two RKE2 pieces, kube-proxy (with eBPF/XDP load balancing) and the default CNI Canal (with a native datapath), and its own tuning guide tells you to kill irqbalance and pin the NIC IRQs: a fourth list to align alongside isolcpus and reserved-cpus. The 2026 state of the art closes the gap from above: netkit (kernel ≥6.8, zero namespace overhead), BIG TCP (192k super-packets for 100Gb/s+), host-routing (iptables bypass), and above all DRA/DRANET, the network driver that finally co-schedules NUMA-local GPU and NIC on the same PCIe root, enabling GPUDirect RDMA with +59.6% bus bandwidth in all_gather and +58.1% in all_reduce. On a generic RKE2 cluster with 4×H100 SXM nodes.

Where you are: the network plane the trilogy did not open

The analogy: the door the dishes come through

Go back to the restaurant of the previous post. The maître d’, the Topology Manager, seated the party of eight at a single table (one NUMA node) because the diners (CPUs), the cutlery (memory) and the reserved bottle (the GPU) all fitted. A perfect table. But the maître d’ never looked at where the kitchen pass is: the door every dish comes in and goes out through.

That door is the NIC. The prompt comes in there, the tokens go out there, and, in disaggregated serving, the KV-cache that the prefill pool sends to the decode pool circulates there. If the table is in the left-hand room (socket 0) but the kitchen pass is in the right-hand one (socket 1), every dish crosses the entire restaurant (the UPI/QPI), over and over, however impeccably the table is laid. The diner does not notice the perfect table: they notice the dish arriving late and cold.

And there is a finer detail: the waiter who crosses the room with the dishes (the softirq that processes the packets) is one of the seated diners. If the maître d’ gave them a chair of their own so they could eat in peace (a core isolated with isolcpus for vLLM) but the restaurant also has them waiting on the far door, that diner does not eat: they spend the whole dinner crossing the room. The jitter shows up exactly where you thought you had bought calm.

The trilogy levelled three legs of the table: the cable, the host and the orchestration. The fourth, which door the dishes come through and who carries them, is levelled by no kubelet manager. Until 2026.

The gap: why the Topology Manager does not look at the NIC

The mechanism of post 3 is a coordinator (Topology Manager) that consults three Hint Providers: CPU Manager, Memory Manager and Device Manager (the GPU plugin). Each says on which NUMA node it can satisfy its part; the coordinator computes the intersection and admits or rejects.

The problem is one of census: the classic NIC is not a Device Manager “device”. A standard Ethernet/InfiniBand card is managed by the CNI and the kernel, it is not requested in the pod’s resources: like nvidia.com/gpu, and therefore it emits no NUMA hint. The Topology Manager aligns CPU+memory+GPU and leaves the NIC wherever the hardware put it, which may be the other socket. The maître d’ has three assistants and is missing the fourth: the one who knows which door the dishes come through.

This did not matter when a node’s network was 10/25 Gb/s and the bottleneck was elsewhere. It matters now, with two workloads that saturate the node’s network:

Disaggregated serving. The KV-cache that travels between the prefill pool and the decode pool moves over RDMA. These are large transfers, sensitive to latency and bandwidth, which in multi-node setups go out through the NIC.
Multi-node NCCL collectives. When tensor/pipeline parallelism crosses the node boundary, the all-reduce/all-gather operations no longer go over NVLink but over GPUDirect RDMA on the NIC.

In both, where the NIC sits relative to the GPU and to the pod’s cores decides performance. And the kubelet, on its own, does not coordinate that.

The network datapath under NUMA: IRQ, softirq and DMA

To see why NIC locality carries weight, you have to look at the path of an incoming packet:

Three kernel facts the analogy compresses:

The IRQ has affinity. Each NIC queue raises an interrupt that the kernel services on a specific CPU (/proc/irq/<n>/smp_affinity). The heavy processing is deferred to a softirq (NET_RX/NET_TX), which runs on that same CPU. If irqbalance is left loose, it migrates them non-deterministically, poison for the p99.
The softirq competes with the pod. If the IRQ lands on a core that isolcpus reserved for vLLM, that queue’s NET_RX steals cycles from the model. The signal in /proc/softirqs: a NET_RX column that spikes on a single CPU. It is the same jitter from post 2, coming in through the network.
DMA has a NUMA origin. The NIC writes the packet by DMA into the RAM of the socket of its PCIe root. If the consumer (the pod’s thread) is on the other socket, it reads across the UPI/QPI. RFS (Receive Flow Steering) tries to bring the processing to the consumer’s CPU, but it cannot teleport the NIC to the other socket.

One number, with its caveat

Take a 2-socket node, a 400 Gb/s = 50 GB/s NIC on the PCIe root of socket 0, and a decode pod pinned to socket 1. If the NIC saturates, those ~50 GB/s of receive traffic cross the UPI towards socket 1. A UPI 2.0 link runs at around 20–40 GB/s per direction and link depending on generation; even with several links, 50 GB/s of network traffic going against the grain eats a far from negligible fraction of the inter-socket budget, the same budget already contended by the pod’s remote memory accesses and, if there is multi-node traffic, the KV-cache of the disaggregation. I am not giving a firm “X% degradation” because it depends on CPU generation, number of UPI links, MTU and traffic pattern; without that methodology, any exact figure is marketing.

What is measured with public methodology is the aggregate effect of aligning GPU and NIC: the DRANET project reports +59.6% bus bandwidth in all_gather and +58.1% in all_reduce (NCCL collectives) when the assigned NIC is NUMA-local to the GPU rather than not. That is the size of the gap the Topology Manager was leaving open.

What RKE2’s Cilium eBPF replaces (and why it touches this story)

RKE2 ships Canal (Flannel + Calico) by default as the CNI and kube-proxy (iptables/IPVS rules) for Service balancing. Switching to Cilium (cni: cilium in /etc/rancher/rke2/config.yaml) replaces both pieces with an eBPF datapath:

RKE2 piece	What it does	What Cilium eBPF puts in
`kube-proxy` (iptables/IPVS)	Service balancing	LB in eBPF; with `kubeProxyReplacement=true`, and acceleration in XDP (driver layer)
Canal (Flannel+Calico)	VXLAN overlay + NetworkPolicy	native datapath (`routingMode=native`), L3/L4 and L7 NetworkPolicy in eBPF
veth per pod	namespace interface pair	netkit (kernel ≥6.8): namespace overhead ~0
host iptables traversal	netfilter hooks	eBPF host-routing: bypasses iptables and the upper part of the stack

Up to here this is pure networking and it does not touch the kubelet’s resource managers: Cilium does not assign exclusive CPUs nor emit compute NUMA hints. The ten knobs of post 3 stay identical whether you run Canal or Cilium.

But Cilium does enter the fourth leg through two doors. The first: its own tuning guide recommends, literally, “kill irqbalance and pin the NIC IRQs to specific CPUs for maximum workload isolation”, along with the tuned network-latency profile, the performance governor and CONFIG_PREEMPT_NONE. In other words: the eBPF datapath only really performs if you coordinate IRQ affinity, and that affinity has to point at the housekeeping cores (reserved-cpus), never at the isolated ones. So a fourth list appears that must be kept consistent with isolcpus and reserved-cpus:

isolcpus = 2-31,34-63 # cores exclusive to vLLM (host, post 2)
reserved-cpus = 0-1,32-33 # kubelet housekeeping (post 3)
IRQ affinity = 0-1,32-33 # NIC IRQs → housekeeping ONLY (this post)
 # never 2-31: there the softirq would steal from the model

The second door: netkit + host-routing + BIG TCP reduce how many times the packet crosses the stack and the namespace, which cushions (does not eliminate) the cost of the NUMA crossing. BIG TCP assembles super-packets of up to 192k (against 64k) for 100Gb/s+; fewer stack traversals means less softirq work on the core, and therefore less pressure on the inter-socket budget. It is the continuous batching analogy applied to the network stack: amortising a fixed cost over larger batches.

Cilium performance profile (state 1.19, kernel ≥6.8)

# Helm, recommended performance profile (summary of the tuning guide)
helm install cilium cilium/cilium --version 1.19.4 \
 --namespace kube-system \
 --set routingMode=native \
 --set bpf.datapathMode=netkit \  # namespace overhead ~0 (kernel >=6.8)
 --set bpf.masquerade=true \
 --set kubeProxyReplacement=true \  # replaces RKE2's kube-proxy
 --set enableIPv4BIGTCP=true \  # 192k super-packets (mlx5/ice NIC)
 --set enableIPv6BIGTCP=true \
 --set bpf.distributedLRU.enabled=true \# per-CPU BPF maps: less spinlock contention
 --set bpf.mapDynamicSizeRatio=0.08 \
 --set bpfClockProbe=true

# Verification inside a Cilium pod:
cilium status --verbose | grep -E "Device Mode|Host Routing|BIG TCP|XDP"
# Device Mode: netkit · Host Routing: BPF · IPv4 BIG TCP: enabled · XDP Acceleration: Native

Sceptical caveat: netkit and BIG TCP are beta and require kernel ≥6.8 and specific NICs (mlx4/mlx5/ice). They are not in-place: they change datapath fundamentals and force you to restart pods or, better, to apply them by per-node config only on new nodes. For an ENS cluster in production, that is a maintenance window, not a blind helm upgrade.

The 2026 state of the art: DRA and DRANET, the maître d’ who finally looks at the door

What closes the gap at the root is not Cilium, it is the admission mechanism the kubelet did not have for the NIC: Dynamic Resource Allocation (DRA), beta since Kubernetes 1.32 and with advances in every release up to 1.36 (May 2026). DRA generalises the “devices” model beyond the GPU: a driver discovers the hardware, publishes ResourceSlices with its attributes, including the NUMA topology and the PCIe root, and the scheduler resolves ResourceClaims that can demand affinity between devices.

DRANET (a kubernetes-sigs project) is the network DRA driver. It discovers the NICs (including the RDMA-capable ones), advertises them as ResourceSlices, and via NRI injects them into the pod’s namespace, compatible with whatever CNI you already have, Cilium included. The key piece for this story: combined with the NVIDIA GPU DRA driver, it allows co-scheduling GPU and NIC that share a PCIe root (the relationship NVIDIA calls NODE), which is exactly the condition for GPUDirect RDMA. The maître d’ finally has a fourth assistant: “is there a NIC NUMA-local to this GPU?”.

The ResourceClaimTemplate uses CEL selectors to request exactly that alignment:

# Request an RDMA NIC NUMA-local to the assigned GPU (illustrative DRANET/DRA schema)
apiVersion: resource.k8s.io/v1beta1
kind: ResourceClaimTemplate
metadata:
 name: gpu-nic-numa-aligned
spec:
 spec:
 devices:
 requests:
 - name: rdma-nic
 deviceClassName: dra.net  # NICs published by DRANET
 constraints:
 - requests: ["rdma-nic"]
 matchAttribute: "dra.net/pcieRoot" # same PCIe root as the GPU
 # → enables GPUDirect RDMA over a NUMA-local path

Why it matters for inference, not for abstract “AI training”: in disaggregated serving, RDMA is what moves the KV-cache between the prefill pool and the decode pool with the latency that TTFT demands; and in multi-node, GPUDirect RDMA replaces NVLink as the medium of the collective. Aligning GPU+NIC on the same PCIe root is what turns an “RDMA that works” into an “RDMA that performs”, the +60% bus bandwidth from DRANET.

Status and caveats: DRA is beta (gates to enable by hand), DRANET is young (a SIG project, still evolving) and the managed offering exists mostly in cloud (GKE managed DRANET in preview, AKS for RDMA). For on-premise ENS it is a path, not a finished product: the value today is understanding that the fourth leg already has a standard OSS mechanism, and starting to pilot it on a lab node, not putting it into critical production this quarter.

How it connects with the rest of the stack

With the host (post 2). The NIC’s IRQ affinity is a third list to marry with isolcpus and reserved-cpus. The IRQs go to housekeeping; the isolated cores, untouched. Uncoordinating them lets in through the network door the jitter that isolcpus threw out through the compute one.

With the orchestration (post 3). DRA is the natural extension of the Topology Manager: the same “admit only if it fits on the NUMA node” principle carried over to the NIC. Where the Device Manager left the network out of the census, DRANET puts it in.

With the interconnect (post 1). Inside the node NVLink rules; on crossing the node boundary, GPUDirect RDMA over the NIC is the medium of the collective. The kubelet’s NUMA policy guarantees that GPU and CPUs share a socket; DRANET adds that the NIC does too, and only then does the RDMA take the short path.

With disaggregated serving. The prefill→decode KV-cache is the traffic that punishes a badly placed NIC hardest. The fourth leg is what stops separating prefill and decode from being paid for in transfer latency.

With capacity planning. Sizing gains a dimension: “GPUs per node and cores per NUMA node” is not enough; you have to count how many NICs NUMA-local to a GPU the chassis has. A node with 4 GPUs and a single NIC on socket 0 has two GPUs “far from the door”.

With observability. What confirms that the fourth leg is properly set is not an application dashboard: it is /proc/softirqs (is NET_RX concentrated on housekeeping?), nvidia-smi topo -m (is the GPU↔NIC relationship NODE/PHB?) and the NIC counters. It fits with GPU observability with DCGM: the GPU “at 60% for no reason” may be the host waiting for packets that cross the socket.

Traps and things that are not what they look like

Believing that switching to Cilium “already optimises the network”. Cilium eBPF replaces kube-proxy and Canal and performs better out of the box, but the default deployment prioritises compatibility, not performance. Without irqbalance disabled, without IRQs pinned to housekeeping and without netkit/host-routing, you leave most of the improvement on the table. Cilium’s docs say so; plenty of people do not read the tuning guide.

Pinning the NIC IRQs to isolated cores. The symmetric mistake to knob 6 of post 3: if you set IRQ affinity over isolcpus, the NET_RX softirq steals cycles from vLLM on exactly the cores you isolated so nobody would disturb it. The IRQs go to reserved-cpus, always.

Assuming the Topology Manager already aligns the NIC. It does not: the classic NIC is not a Hint Provider. If you need NIC↔GPU locality, today the mechanism is DRA/DRANET, not a kubelet policy. Waiting for single-numa-node to solve it is waiting for something that is not in its design.

Putting DRA/DRANET into ENS production this quarter. It is beta and young. The sensible move is to pilot it on a lab node, measure all_reduce/all_gather with and without alignment, and decide with data. The +60% figure is from a specific environment; reproduce it in yours before promising it.

BIG TCP / netkit without reading the requirements. Kernel ≥6.8, mlx4/mlx5/ice NICs, no tunnel or encryption for BIG TCP, and nothing in-place: it forces pod restarts or per-node config. On a cluster with IPsec or with unsupported NICs, part of this does not apply. Check cilium status --verbose before taking for granted that it is active.

Confusing the eBPF datapath (kernel) with the Cilium agent (pod). cilium-agent is a Burstable DaemonSet that should live on housekeeping (covered by system-reserved). But datapath processing runs in softirq, governed by the host’s IRQ affinity, not by reserved-cpus. They are two different things; pinning the pod well does not pin the softirq.

Conclusion

The “under the engine” series chased one idea: performance that looks like an engine problem (vLLM is slow) or a model problem (quantisation) is, far too often, a locality problem one layer down. The trilogy covered three: the cable (NVLink not used), the host (remote NUMA, jitter) and the orchestration (pinning that never happened). The fourth was missing: the network. The Topology Manager seats the pod at a perfect NUMA table and never asks which door the dishes come through nor who carries them. On a node at 25 Gb/s it made no difference; on one at 400 Gb/s with the KV-cache crossing over RDMA, that door decides TTFT and the collective’s bandwidth. Cilium eBPF replaces kube-proxy and Canal with a datapath that performs, if you coordinate IRQ affinity with isolcpus/reserved-cpus, a fourth list to align, and DRA/DRANET finally provides the census that was missing: co-scheduling NUMA-local GPU and NIC on the same PCIe root, with an improvement of a size (+60% NCCL bus bandwidth) that measures how big the gap was. Going down a level is not snobbery: it is that the root cause lived, once again, one layer below where the dashboard looks.

References

Cilium, Tuning Guide (netkit, host-routing, BIG TCP, XDP, pin the IRQs and kill irqbalance): https://docs.cilium.io/en/stable/operations/performance/tuning/.
Cilium 1.19 (February 2026), Cilium at Ten Years — hardening of encryption, policies and observability: https://www.infoq.com/news/2026/02/cilium-119/.
Isovalent, Cilium 1.18 (IPv6, encrypted overlay, ingress bandwidth, policy perf): https://isovalent.com/blog/post/cilium-1-18/.
RKE2, Network Options (Canal by default; Cilium with kube-proxy replacement): https://docs.rke2.io/networking/basic_network_options.
Kubernetes, Dynamic Resource Allocation: https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/.
Kubernetes blog, v1.36: More Drivers, New Features, and the Next Era of DRA (May 2026): https://kubernetes.io/blog/2026/05/07/kubernetes-v1-36-dra-136-updates/.
DRANET (kubernetes-sigs), the network DRA driver and the paper The Kubernetes Network Driver Model (+59.6% all_gather / +58.1% all_reduce): https://github.com/kubernetes-sigs/dranet.
AKS Engineering, Optimizing RDMA performance for AI workloads on AKS with DRANET (April 2026): https://blog.aks.azure.com/2026/04/01/dranet-rdma-optimization-for-ai-on-aks.
Linux network tuning — IRQ affinity, RSS/RPS/RFS and NUMA softirq: https://andreaskaris.github.io/blog/networking/rss-irq-affinity-and-rps/.

Gpudirect on lo0 — Tech Blog

Storage in the AI era (2/4): performance

The metric that really matters: accelerator utilisation

MLPerf Storage: the benchmark that does measure the right thing

The numbers from commercial platforms

The I/O pattern of training and GPUDirect Storage

The real cost of checkpointing (when there is training)

Performance in an inference factory

Prefill, decode and their disaggregation

The KV-cache as a performance layer

Model load time and cold start

RAG performance and vector search

How to measure your own: beyond the spec sheet

The cost of getting it wrong

The physical layer: networks and PCIe Gen6

Three questions before sizing

Takeaways

See also

Sources

The corridors and the security guard: PCIe topology, GPUDirect P2P and ACS

TL;DR

Where you are: the corridors, not the bench

The topology of a node: two buses, not one

GPUDirect: skipping reception

The guard: why ACS and IOMMU break P2P

The maths that matter: how much the detour costs

The real tension: performance vs isolation (and ENS)

The 10 knobs

How it connects with the rest of the stack

Traps and things that are not what they look like

Conclusion

See also

References

The kitchen door the maître d' never looked at: network NUMA, Cilium eBPF and DRANET, the fourth leg of pinning

TL;DR

Where you are: the network plane the trilogy did not open

The analogy: the door the dishes come through

The gap: why the Topology Manager does not look at the NIC

The network datapath under NUMA: IRQ, softirq and DMA

One number, with its caveat

What RKE2’s Cilium eBPF replaces (and why it touches this story)

Cilium performance profile (state 1.19, kernel ≥6.8)

The 2026 state of the art: DRA and DRANET, the maître d’ who finally looks at the door

How it connects with the rest of the stack

Traps and things that are not what they look like

Conclusion

See also

References