The corridors and the security guard: PCIe topology, GPUDirect P2P and ACS

This continues the below the engine series. The NVLink and NCCL post explained the shared bench over which the GPUs pass data to each other at 450 GB/s. But that bench only connects GPUs to each other. Everything else, disk, network, the host, travels over another bus, PCIe, and through its corridors. The cold start already brushed against this with GPUDirect Storage; this post opens up the complete floor plan of the corridors and the guard who watches them.

TL;DR

On a 4×H100 SXM node, the GPUs talk over NVLink (450 GB/s per direction, around 7× PCIe), and for the all-reduce of tensor parallel that is the path. But PCIe does not go away: it is where everything else comes in. The weights come up from the NVMe over PCIe (the cold start), the data from another node arrives via the NIC over PCIe (RDMA), and a KV that moves between nodes travels over PCIe. GPUDirect is the family that lets those bytes go straight from the device into the HBM without bouncing through host RAM: P2P (GPU↔GPU), RDMA (GPU↔NIC) and Storage (GPU↔NVMe). The obstacle is a guard called ACS (Access Control Services): a PCIe security feature that by default forces peer-to-peer traffic up to the root complex for inspection, which destroys the direct path and inserts a detour through the CPU. The IOMMU (VT-d) does something similar if it is not in passthrough mode. Disabling them gives performance; keeping them gives isolation and virtualisation, and that is a real decision in an ENS environment. This post explains the topology (nvidia-smi topo -m), GPUDirect, why ACS and IOMMU break P2P with numbers, the 10 knobs and the trap of removing the guard without knowing what it was watching. On the generic 4×H100 SXM cluster.

Where you are: the corridors, not the bench

Picture the kitchen as a building. The elite cooking stations, the GPUs, are in a room with a huge central bench (NVLink/NVSwitch) over which they pass ingredients at full speed without standing up. That bench is for them and them alone.

But the building has more to it: the pantry (the NVMe storage), the loading door (the network, the NIC) and reception (the CPU and its RAM). To reach any of those, the stations do not use the central bench: they use the corridors of the building, the PCIe bus. And here the character of the post appears: at the entrance of each corridor there is a security guard (ACS) who, by default, does not let two stations hand each other something directly along the corridor: it forces them to take the parcel up to reception to be checked, and only then does it come down to its destination. It is safe, but it is an absurd detour when the two stations are right next to each other. GPUDirect is the permission to skip that detour; ACS and IOMMU are the reasons why, often, you cannot.

The topology of a node: two buses, not one

The most common mistake is to think that a node has “a bus”. There are (at least) two, and they do different things:

• NVLink / NVSwitch — the high-speed GPU↔GPU mesh. On H100 SXM, 18 links × 50 GB/s = 900 GB/s bidirectional between any two GPUs, with NVSwitch giving an all-to-all with no contention (NVLink, NVIDIA). It is the shared bench.
• PCIe Gen5 — the general I/O bus. An x16 link gives 128 GB/s bidirectional (around 64 per direction) (H100 product brief). It connects each GPU with the CPU, the RAM, the NICs and the NVMes. It is the corridor.

The difference is 7×: NVLink moves in one second what PCIe takes seven to move. That is why intra-node tensor parallel goes over NVLink and nobody argues about it. PCIe matters for the other things: pulling weights off disk, receiving from the network, moving KV between nodes.

The tool for seeing this is nvidia-smi topo -m, which prints a matrix of how each pair is connected (NVIDIA forum):

        GPU0   GPU1   GPU2   GPU3   NIC0    CPU Affinity  NUMA
GPU0     X     NV18   NV18   NV18   PXB     0-47          0
GPU1    NV18    X     NV18   NV18   PXB     0-47          0
GPU2    NV18   NV18    X     NV18   SYS     48-95         1
GPU3    NV18   NV18   NV18    X     SYS     48-95         1

The legend is what matters: NV18 = 18 NVLink links (the bench); PXB = crosses PCIe switches but not the host; PHB = goes through the host bridge; NODE = same NUMA, crossing PCIe; SYS = crosses the inter-socket interconnect (the worst case, it traverses NUMA). The fact that GPU0↔NIC0 is PXB and GPU2↔NIC0 is SYS tells you exactly which GPU should handle that NIC’s traffic, GPU 0, without crossing NUMA. This links directly with the NUMA post and the network NUMA one: PCIe affinity is NUMA affinity.

GPUDirect: skipping reception

Without GPUDirect, moving a piece of data from the NIC (or the NVMe) to the GPU takes a mandatory detour: device → host RAM → GPU. That bounce through RAM consumes CPU bandwidth, spends copies and adds latency. GPUDirect removes the bounce by letting the data go straight from the device into the HBM. Three flavours:

• GPUDirect P2P — GPU↔GPU over PCIe (when there is no NVLink between them, or for traffic that does not use the bench).
• GPUDirect RDMA — GPU↔NIC: the network card writes directly into the HBM. It is what makes efficient multi-node viable (NCCL over InfiniBand/RoCE).
• GPUDirect Storage (GDS) — GPU↔NVMe: the disk writes straight into the HBM, with no host buffer. It is the lever of the cold start.

On an SXM node, the GPU↔GPU traffic of tensor parallel does not use P2P over PCIe: it uses NVLink. That is why GPUDirect matters above all at the edges of the node: the network (RDMA, for multi-node) and the disk (GDS, for startup). That is where ACS does damage.

The guard: why ACS and IOMMU break P2P

ACS (Access Control Services) is a PCIe security feature designed for virtualisation and isolation: it guarantees that one device cannot read from or write to another directly without the root complex mediating. To achieve that, it forces peer-to-peer transactions up to the root complex and back down (GDS best practices, NVIDIA). It is exactly the opposite of what GPUDirect wants: the direct path stops being direct.

The IOMMU (VT-d on Intel, the equivalent on AMD) translates addresses and isolates devices. If it is active and not in passthrough mode, it also redirects P2P traffic through the root complex, with the same effect: performance through the floor or, in extreme cases, hangs (NCCL troubleshooting).

Summarised without detours (Giraud, debugging P2P): ACS forces the trip through the root for security checks; IOMMU forces it for isolation and virtualisation. Both break the goal of P2P (direct communication with no intermediaries) and add overhead. If you do not need that security/virtualisation on that path, disabling them recovers the performance. The operational recipe for maximum GPUDirect performance: ACS off on the switches along the path and IOMMU in passthrough (iommu=pt) or disabled.

The maths that matter: how much the detour costs

Take a KV SWAP of 5 GB (preemption from the scheduler that sends KV to the host, or a transfer between nodes in disaggregated serving):

* The detour through the root does not just “add latency”: it saturates the host bridge bandwidth, contends with other traffic and, depending on the topology, can fall well below the direct path. The figure is an order of magnitude, to show the size of the problem, not a benchmark.

The reading: on the path that does use PCIe (network, disk, swap), having ACS on can double or triple the time. And if that time is on the critical path, a cold start, a preemption swap, an inter-node all-reduce, it shows up in the latency the user sees. What disabling ACS does not fix: traffic that already went over NVLink (intra-node TP). There, ACS is irrelevant.

The real tension: performance vs isolation (and ENS)

Here the post gets serious, because the “disable ACS and IOMMU” recipe has a cost that in a regulated environment is not free. ACS and IOMMU exist for a reason: to isolate devices. On a dedicated bare-metal node for inference, with no virtualisation or multi-tenancy, you are not isolating anything that matters and disabling them is reasonable. But:

• If you do GPU passthrough to VMs or use containers with strong isolation, the IOMMU is necessary, not optional.
• In a multi-tenant scenario where several workloads share a node, ACS provides a guarantee that one device does not snoop on another.
• Under ENS (see technical controls), workload isolation and access traceability can be requirements; disabling bus isolation to gain 80 ms is a decision you have to justify and document, not a silent tweak.

The design way out, when you need both: keep the isolation where compliance requires it and design so that the hot path does not depend on P2P over PCIe. Concretely, on an SXM node, the bulk of the critical traffic (TP) already goes over NVLink and ACS does not affect it. For the network, dedicate one NIC per GPU on its own PCIe switch (PXB) and use GPUDirect RDMA only on the path you control. For the disk, cache weights on local NVMe. That way you do not pay the choice between performance and isolation: you avoid it on the path that matters.

The 10 knobs

How it connects with the rest of the stack

With NVLink and NCCL. The NVLink post covers the GPU↔GPU bench; this one covers the other bus, the one connecting to disk, network and host. They are complementary: ACS affects PCIe, not NVLink.

With the cold start. The GPUDirect Storage of the disk→HBM post is GPUDirect over the storage path; ACS on strangles it just as it strangles P2P.

With NUMA. The PCIe affinity of topo -m is the NUMA affinity of the host post; placing NIC and NVMe on the correct NUMA avoids the SYS path.

With the network. NIC placement and GPUDirect RDMA is the subject of the network NUMA post; the same principle of “get the CPU out of the way”.

With PagedAttention and the scheduler. The preemption SWAP (scheduler) moves KV blocks over PCIe; that is why V1 prefers RECOMPUTE and why this bus matters.

With disaggregated serving. Transferring KV between pools in disaggregated serving travels PCIe→NIC→PCIe; ACS and placement decide whether it is viable.

With ENS. Bus isolation is a technical control; see ENS/42001/AI Act controls.

Traps and things that are not what they look like

“Disable ACS everywhere, it goes faster.” On a dedicated node, fine. On one with virtualisation, multi-tenancy or isolation requirements (ENS), you are removing a security control. The correct decision is per path and documented, not global and silent.

“NVLink and PCIe are the same bus, more or less.” No. They are two buses with a 7× difference and different purposes. TP goes over NVLink; disk, network and host go over PCIe. Confusing them leads to “optimising” ACS for traffic that does not even go through PCIe.

“P2P just works, there is nothing to check.” P2P is silently disabled with ACS/IOMMU active, and many distros enable them by default. Check with p2pBandwidthLatencyTest and nvidia-smi topo -m; do not assume you have the direct path just because the GPUs are on the same node.

“IOMMU off always, for performance.” If you do GPU passthrough to virtual machines, the IOMMU is mandatory; disabling it breaks the passthrough. The correct mode is usually passthrough (iommu=pt): it keeps the necessary mapping without penalising P2P.

“More PCIe lanes = faster GPU.” PCIe is the I/O path, not the compute path. A GPU at x8 instead of x16 takes longer to load and to communicate over PCIe, but generates tokens at the same speed once the weights are inside. The damage from x8 is in the cold start and in multi-node, not in decode throughput.

“GPUDirect fixes any I/O bottleneck.” GPUDirect removes the bounce through the CPU; if your bottleneck is the device itself (saturated NVMe, NIC at full tilt) or the topology (a SYS path crossing NUMA), GPUDirect does not touch it. Measure where the bottleneck is first.

Conclusion

This whole series has been going down floors looking for where the time goes, and this one reaches the wiring of the building. Intuition treats the node as a homogeneous box where “the GPUs talk to everything”; the reality is that there are two buses with opposite purposes, an elite bench for the GPUs (NVLink) and some service corridors for everything else (PCIe), and a security guard in the corridors who, with the best of intentions, makes every parcel go up to reception before it is delivered. GPUDirect is the permission for direct delivery; ACS and IOMMU are the legitimate reasons why sometimes you are not given it. The lesson is not “disable the guard”: it is understanding which path is critical (almost never the one you think) and what the guard was watching before sending it home. On a dedicated node, the direct path is almost free and worth taking. On one that shares workloads or lives under ENS, bus isolation is a control you sacrifice with justification or do not sacrifice at all. Good design does not choose between performance and isolation blindly: it puts the critical traffic on the bench that needs no guard, and leaves the corridors for what can wait.

See also

• The shared bench: NVLink, NVSwitch and NCCL — the GPU↔GPU bus that ACS does not touch; complementary to this post.
• From disk to HBM: cold start — GPUDirect Storage over the NVMe path, strangled by ACS just like P2P.
• NUMA, hugepages and CPU isolation — PCIe affinity is NUMA affinity; placing NIC and NVMe on the right socket.
• Network NUMA, Cilium eBPF and DRANET — NIC placement and GPUDirect RDMA, the same principle of getting the CPU out of the way.
• PagedAttention and the block manager — the KV that travels over PCIe when a SWAP happens.
• The pass: vLLM’s scheduler step — why V1 prefers RECOMPUTE to SWAP (it avoids the trip over PCIe).
• Disaggregated serving: prefill and decode separated — transferring KV between nodes goes PCIe→NIC→PCIe.
• Technical controls ENS / ISO 42001 / EU AI Act — bus isolation as a security control to be justified.

References

• NVIDIA, GPUDirect Storage Best Practices Guide (ACS, IOMMU, paths): https://docs.nvidia.com/gpudirect-storage/best-practices-guide/index.html.
• NVIDIA, NCCL Troubleshooting (IOMMU/VT-d and P2P): https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html.
• M. Giraud, Multi-GPU (NVIDIA) P2P capabilities and debugging tips: https://morgangiraud.medium.com/multi-gpu-nvidia-p2p-capabilities-and-debugging-tips-fb7597b4e2b5.
• NVIDIA, H100 Product Brief (PCIe Gen5, NVLink 900 GB/s): https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/h100/PB-11773-001_v01.pdf.
• NVIDIA, GPUDirect RDMA documentation: https://docs.nvidia.com/cuda/gpudirect-rdma/index.html.