Nvlink on lo0 — Tech Blog

The corridors and the security guard: PCIe topology, GPUDirect P2P and ACS

Mon, 08 Jun 2026 06:00:00 +0200

This continues the below the engine series. The NVLink and NCCL post explained the shared bench over which the GPUs pass data to each other at 450 GB/s. But that bench only connects GPUs to each other. Everything else, disk, network, the host, travels over another bus, PCIe, and through its corridors. The cold start already brushed against this with GPUDirect Storage; this post opens up the complete floor plan of the corridors and the guard who watches them.

TL;DR

On a 4×H100 SXM node, the GPUs talk over NVLink (450 GB/s per direction, around 7× PCIe), and for the all-reduce of tensor parallel that is the path. But PCIe does not go away: it is where everything else comes in. The weights come up from the NVMe over PCIe (the cold start), the data from another node arrives via the NIC over PCIe (RDMA), and a KV that moves between nodes travels over PCIe. GPUDirect is the family that lets those bytes go straight from the device into the HBM without bouncing through host RAM: P2P (GPU↔GPU), RDMA (GPU↔NIC) and Storage (GPU↔NVMe). The obstacle is a guard called ACS (Access Control Services): a PCIe security feature that by default forces peer-to-peer traffic up to the root complex for inspection, which destroys the direct path and inserts a detour through the CPU. The IOMMU (VT-d) does something similar if it is not in passthrough mode. Disabling them gives performance; keeping them gives isolation and virtualisation, and that is a real decision in an ENS environment. This post explains the topology (nvidia-smi topo -m), GPUDirect, why ACS and IOMMU break P2P with numbers, the 10 knobs and the trap of removing the guard without knowing what it was watching. On the generic 4×H100 SXM cluster.

Where you are: the corridors, not the bench

Picture the kitchen as a building. The elite cooking stations, the GPUs, are in a room with a huge central bench (NVLink/NVSwitch) over which they pass ingredients at full speed without standing up. That bench is for them and them alone.

But the building has more to it: the pantry (the NVMe storage), the loading door (the network, the NIC) and reception (the CPU and its RAM). To reach any of those, the stations do not use the central bench: they use the corridors of the building, the PCIe bus. And here the character of the post appears: at the entrance of each corridor there is a security guard (ACS) who, by default, does not let two stations hand each other something directly along the corridor: it forces them to take the parcel up to reception to be checked, and only then does it come down to its destination. It is safe, but it is an absurd detour when the two stations are right next to each other. GPUDirect is the permission to skip that detour; ACS and IOMMU are the reasons why, often, you cannot.

The topology of a node: two buses, not one

The most common mistake is to think that a node has “a bus”. There are (at least) two, and they do different things:

NVLink / NVSwitch — the high-speed GPU↔GPU mesh. On H100 SXM, 18 links × 50 GB/s = 900 GB/s bidirectional between any two GPUs, with NVSwitch giving an all-to-all with no contention (NVLink, NVIDIA). It is the shared bench.
PCIe Gen5 — the general I/O bus. An x16 link gives 128 GB/s bidirectional (around 64 per direction) (H100 product brief). It connects each GPU with the CPU, the RAM, the NICs and the NVMes. It is the corridor.

The difference is 7×: NVLink moves in one second what PCIe takes seven to move. That is why intra-node tensor parallel goes over NVLink and nobody argues about it. PCIe matters for the other things: pulling weights off disk, receiving from the network, moving KV between nodes.

The tool for seeing this is nvidia-smi topo -m, which prints a matrix of how each pair is connected (NVIDIA forum):

 GPU0 GPU1 GPU2 GPU3 NIC0 CPU Affinity NUMA
GPU0 X NV18 NV18 NV18 PXB 0-47 0
GPU1 NV18 X NV18 NV18 PXB 0-47 0
GPU2 NV18 NV18 X NV18 SYS 48-95 1
GPU3 NV18 NV18 NV18 X SYS 48-95 1

The legend is what matters: NV18 = 18 NVLink links (the bench); PXB = crosses PCIe switches but not the host; PHB = goes through the host bridge; NODE = same NUMA, crossing PCIe; SYS = crosses the inter-socket interconnect (the worst case, it traverses NUMA). The fact that GPU0↔NIC0 is PXB and GPU2↔NIC0 is SYS tells you exactly which GPU should handle that NIC’s traffic, GPU 0, without crossing NUMA. This links directly with the NUMA post and the network NUMA one: PCIe affinity is NUMA affinity.

GPUDirect: skipping reception

Without GPUDirect, moving a piece of data from the NIC (or the NVMe) to the GPU takes a mandatory detour: device → host RAM → GPU. That bounce through RAM consumes CPU bandwidth, spends copies and adds latency. GPUDirect removes the bounce by letting the data go straight from the device into the HBM. Three flavours:

GPUDirect P2P — GPU↔GPU over PCIe (when there is no NVLink between them, or for traffic that does not use the bench).
GPUDirect RDMA — GPU↔NIC: the network card writes directly into the HBM. It is what makes efficient multi-node viable (NCCL over InfiniBand/RoCE).
GPUDirect Storage (GDS) — GPU↔NVMe: the disk writes straight into the HBM, with no host buffer. It is the lever of the cold start.

On an SXM node, the GPU↔GPU traffic of tensor parallel does not use P2P over PCIe: it uses NVLink. That is why GPUDirect matters above all at the edges of the node: the network (RDMA, for multi-node) and the disk (GDS, for startup). That is where ACS does damage.

The guard: why ACS and IOMMU break P2P

ACS (Access Control Services) is a PCIe security feature designed for virtualisation and isolation: it guarantees that one device cannot read from or write to another directly without the root complex mediating. To achieve that, it forces peer-to-peer transactions up to the root complex and back down (GDS best practices, NVIDIA). It is exactly the opposite of what GPUDirect wants: the direct path stops being direct.

The IOMMU (VT-d on Intel, the equivalent on AMD) translates addresses and isolates devices. If it is active and not in passthrough mode, it also redirects P2P traffic through the root complex, with the same effect: performance through the floor or, in extreme cases, hangs (NCCL troubleshooting).

Summarised without detours (Giraud, debugging P2P): ACS forces the trip through the root for security checks; IOMMU forces it for isolation and virtualisation. Both break the goal of P2P (direct communication with no intermediaries) and add overhead. If you do not need that security/virtualisation on that path, disabling them recovers the performance. The operational recipe for maximum GPUDirect performance: ACS off on the switches along the path and IOMMU in passthrough (iommu=pt) or disabled.

The maths that matter: how much the detour costs

Take a KV SWAP of 5 GB (preemption from the scheduler that sends KV to the host, or a transfer between nodes in disaggregated serving):

Path	Effective BW	Time for 5 GB
NVLink (GPU↔GPU intra-node)	~450 GB/s	~11 ms
Direct PCIe Gen5 x16 (P2P, ACS off)	~55 GB/s	~91 ms
PCIe via root complex (ACS on)	~25-30 GB/s*	~170-200 ms

* The detour through the root does not just “add latency”: it saturates the host bridge bandwidth, contends with other traffic and, depending on the topology, can fall well below the direct path. The figure is an order of magnitude, to show the size of the problem, not a benchmark.

The reading: on the path that does use PCIe (network, disk, swap), having ACS on can double or triple the time. And if that time is on the critical path, a cold start, a preemption swap, an inter-node all-reduce, it shows up in the latency the user sees. What disabling ACS does not fix: traffic that already went over NVLink (intra-node TP). There, ACS is irrelevant.

The real tension: performance vs isolation (and ENS)

Here the post gets serious, because the “disable ACS and IOMMU” recipe has a cost that in a regulated environment is not free. ACS and IOMMU exist for a reason: to isolate devices. On a dedicated bare-metal node for inference, with no virtualisation or multi-tenancy, you are not isolating anything that matters and disabling them is reasonable. But:

If you do GPU passthrough to VMs or use containers with strong isolation, the IOMMU is necessary, not optional.
In a multi-tenant scenario where several workloads share a node, ACS provides a guarantee that one device does not snoop on another.
Under ENS (see technical controls), workload isolation and access traceability can be requirements; disabling bus isolation to gain 80 ms is a decision you have to justify and document, not a silent tweak.

The design way out, when you need both: keep the isolation where compliance requires it and design so that the hot path does not depend on P2P over PCIe. Concretely, on an SXM node, the bulk of the critical traffic (TP) already goes over NVLink and ACS does not affect it. For the network, dedicate one NIC per GPU on its own PCIe switch (PXB) and use GPUDirect RDMA only on the path you control. For the disk, cache weights on local NVMe. That way you do not pay the choice between performance and isolation: you avoid it on the path that matters.

The 10 knobs

#	Knob	What it controls	Cost / risk
1	ACS off (switches on the path)	P2P detour through root	you lose bus isolation
2	`iommu=pt` / off	P2P redirection through root	breaks VM passthrough if off
3	`nvidia-smi topo -m`	audit the real topology	— (always worth it)
4	`p2pBandwidthLatencyTest`	measure P2P for real	— (verify before assuming)
5	NIC placement	same PCIe switch as the GPU	SYS if it crosses NUMA
6	NVMe placement	NUMA-local to the GPU	H2D crossing UPI
7	`nvidia-peermem` (GDR)	enables RDMA into HBM	correct driver/kernel
8	PCIe gen/lanes (x16)	width of the corridor	GPU silently at x8
9	relaxed ordering / ASPM	PCIe latency and power	jitter if misconfigured
10	persistence mode	avoids re-init of the path	idle GPU still paid for

How it connects with the rest of the stack

With NVLink and NCCL. The NVLink post covers the GPU↔GPU bench; this one covers the other bus, the one connecting to disk, network and host. They are complementary: ACS affects PCIe, not NVLink.

With the cold start. The GPUDirect Storage of the disk→HBM post is GPUDirect over the storage path; ACS on strangles it just as it strangles P2P.

With NUMA. The PCIe affinity of topo -m is the NUMA affinity of the host post; placing NIC and NVMe on the correct NUMA avoids the SYS path.

With the network. NIC placement and GPUDirect RDMA is the subject of the network NUMA post; the same principle of “get the CPU out of the way”.

With PagedAttention and the scheduler. The preemption SWAP (scheduler) moves KV blocks over PCIe; that is why V1 prefers RECOMPUTE and why this bus matters.

With disaggregated serving. Transferring KV between pools in disaggregated serving travels PCIe→NIC→PCIe; ACS and placement decide whether it is viable.

With ENS. Bus isolation is a technical control; see ENS/42001/AI Act controls.

Traps and things that are not what they look like

“Disable ACS everywhere, it goes faster.” On a dedicated node, fine. On one with virtualisation, multi-tenancy or isolation requirements (ENS), you are removing a security control. The correct decision is per path and documented, not global and silent.

“NVLink and PCIe are the same bus, more or less.” No. They are two buses with a 7× difference and different purposes. TP goes over NVLink; disk, network and host go over PCIe. Confusing them leads to “optimising” ACS for traffic that does not even go through PCIe.

“P2P just works, there is nothing to check.” P2P is silently disabled with ACS/IOMMU active, and many distros enable them by default. Check with p2pBandwidthLatencyTest and nvidia-smi topo -m; do not assume you have the direct path just because the GPUs are on the same node.

“IOMMU off always, for performance.” If you do GPU passthrough to virtual machines, the IOMMU is mandatory; disabling it breaks the passthrough. The correct mode is usually passthrough (iommu=pt): it keeps the necessary mapping without penalising P2P.

“More PCIe lanes = faster GPU.” PCIe is the I/O path, not the compute path. A GPU at x8 instead of x16 takes longer to load and to communicate over PCIe, but generates tokens at the same speed once the weights are inside. The damage from x8 is in the cold start and in multi-node, not in decode throughput.

“GPUDirect fixes any I/O bottleneck.” GPUDirect removes the bounce through the CPU; if your bottleneck is the device itself (saturated NVMe, NIC at full tilt) or the topology (a SYS path crossing NUMA), GPUDirect does not touch it. Measure where the bottleneck is first.

Conclusion

This whole series has been going down floors looking for where the time goes, and this one reaches the wiring of the building. Intuition treats the node as a homogeneous box where “the GPUs talk to everything”; the reality is that there are two buses with opposite purposes, an elite bench for the GPUs (NVLink) and some service corridors for everything else (PCIe), and a security guard in the corridors who, with the best of intentions, makes every parcel go up to reception before it is delivered. GPUDirect is the permission for direct delivery; ACS and IOMMU are the legitimate reasons why sometimes you are not given it. The lesson is not “disable the guard”: it is understanding which path is critical (almost never the one you think) and what the guard was watching before sending it home. On a dedicated node, the direct path is almost free and worth taking. On one that shares workloads or lives under ENS, bus isolation is a control you sacrifice with justification or do not sacrifice at all. Good design does not choose between performance and isolation blindly: it puts the critical traffic on the bench that needs no guard, and leaves the corridors for what can wait.

References

NVIDIA, GPUDirect Storage Best Practices Guide (ACS, IOMMU, paths): https://docs.nvidia.com/gpudirect-storage/best-practices-guide/index.html.
NVIDIA, NCCL Troubleshooting (IOMMU/VT-d and P2P): https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html.
M. Giraud, Multi-GPU (NVIDIA) P2P capabilities and debugging tips: https://morgangiraud.medium.com/multi-gpu-nvidia-p2p-capabilities-and-debugging-tips-fb7597b4e2b5.
NVIDIA, H100 Product Brief (PCIe Gen5, NVLink 900 GB/s): https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/h100/PB-11773-001_v01.pdf.
NVIDIA, GPUDirect RDMA documentation: https://docs.nvidia.com/cuda/gpudirect-rdma/index.html.

The shared bench: NVLink, NVSwitch and NCCL, the cable every token crosses in tensor parallel

Sat, 06 Jun 2026 07:00:00 +0200

This post goes one floor below the engine. In the seven-layer inference stack and in one big replica or many small ones the decision was how many GPUs and how to split the model; here we explain the cable that makes that split work, or that strangles it. It is the first of a mini-series “below the engine”: interconnect (this one) → kernel and NUMA → Kubernetes resource managers.

TL;DR

Tensor parallelism (TP) does not split a model into four chunks that run on their own. It spreads each layer across the GPUs, but after attention and after the MLP the GPUs have to add up their partial results with an all-reduce. On a Llama-70B with 80 layers, that is around 160 all-reduces per generated token. That all-reduce travels over the interconnect, so the interconnect sits on the critical path of every token, not in the background plumbing. On an HGX H100 baseboard, the 8 GPUs all talk to each other at 900 GB/s bidirectional via four NVSwitch chips; without NVSwitch/NVLink, that same traffic falls back to the CPU over PCIe and loses an order of magnitude. NCCL is the library that decides how each collective is done (ring, tree, or NVLS = NVLink-SHARP, which offloads the summation to the switch itself). And there is an asymmetry almost nobody keeps in mind: decode is latency-bound (tiny messages, 16 KB) and prefill is bandwidth-bound (huge batched activations), which is why “more NVLink bandwidth” speeds up prefill but barely touches token-by-token decode. This post explains the mechanism, gives the 10 real NCCL/driver knobs worth touching, and connects with vLLM’s custom all-reduce, disaggregated serving and GPU observability. With scepticism about which levers move the needle.

Where you are: the floor below the engine

The analogy: four mechanics and a single bench

Four mechanics assemble one and the same car engine. It is not that each builds their own engine in parallel, that would be four cars (four replicas of the model, a different strategy). Here they build a single one, at the same time, splitting the parts: one does the pistons, another the cylinder head, another the crankshaft. The problem is that the parts fit into each other: before going on, all four have to put together what they have and check that it matches. That “put together and check” happens dozens of times during the build.

There are two ways to organise the workshop:

A single large bench, everyone around it (NVSwitch). Each mechanic reaches out and hands their part directly to any other, all at once, without standing up. It is instant and simultaneous. This is NVLink + NVSwitch: the GPUs form an all-to-all where anyone talks to anyone at 900 GB/s at the same time.
Four separate workshops with a courier (PCIe via CPU). Every part a mechanic wants to hand to another goes into a box, goes down to reception (host memory, via CPU), and from there up to the destination workshop. Slower, and serialised by reception. This is what happens when there is no NVLink: inter-GPU traffic falls back to PCIe and bounces through the CPU, around 14× slower than NVLink.

The thesis of the post follows on its own: tensor parallelism only makes sense if the mechanics share the bench. As soon as the “put together and check” (the all-reduce) has to go through reception, splitting the work costs more than it saves. That is why, on a serious platform, TP does not cross the NVLink boundary: TP=4 or 8 inside the baseboard where there is NVSwitch, and above that you replicate or use pipeline, never stretch TP over PCIe or over the network. When each option makes sense is covered in one big replica or many small ones; here we explain why the cable dictates that decision.

The mechanism: what an all-reduce really is and why there are 160 per token

Tensor parallelism splits the weight matrices by columns/rows across the $N$ GPUs. Each GPU computes a portion of the layer output. But the next operation needs the complete output, so it has to be recombined. That recombination is a collective operation: an all-reduce, which adds the partial tensors of all the GPUs element by element and leaves the result identical on all of them.

In a standard transformer block there are two synchronisation points per layer:

After the output projection of attention (the o_proj that recombines the split heads).
After the second matrix of the MLP (the down_proj that recombines the split feed-forward).

$$ \text{all-reduces per token} = 2 \times L_{\text{layers}} $$

For a Llama-70B ($L = 80$): $2 \times 80 = 160$ all-reduces per generated token. Not per request, not per sequence: per token. Multiply by decode throughput and you understand why the interconnect is not background infrastructure but a hot path.

How the all-reduce is done: ring, tree, NVLS

NCCL does not have a single way of doing an all-reduce; it picks an algorithm according to topology and message size:

Ring. The GPUs form a ring; each one passes a chunk to its neighbour, adds, and rotates. It takes $2(N-1)$ steps. It is bandwidth-optimal for large messages: the cost of moving the data is $\frac{2(N-1)}{N} \times M$ bytes over the link, almost independent of $N$. The downside: $2(N-1)$ latency hops, bad for small messages.
Tree. Tree reduction: $\log N$ levels. Better latency for small messages and many nodes, worse bandwidth utilisation.
NVLS (NVLink SHARP). The Hopper trick: the summation is not done by the GPUs, it is done by the NVSwitch. The switch has reduction units; the GPUs send their tensors, the switch adds them in flight and returns the result. It takes work away from the GPUs (frees up SMs) and reduces hops. Available only with a 3rd-generation NVSwitch (NVLink4) plus Hopper or newer.

The mental rule: decode (tiny messages) wants latency → tree/LL or vLLM’s custom kernel; prefill (huge messages) wants bandwidth → ring/NVLS. That is why there is no global “optimal NCCL_ALGO”; it depends on which phase you are looking at.

The maths that matter: why decode and prefill stress the cable the other way round

Here is the asymmetry almost everyone skips. The size of the tensor that gets all-reduced in each layer is, approximately:

$$ M \approx B \times S \times h \times 2\ \text{bytes (BF16)} $$

where $B$ = batch, $S$ = tokens processed in this forward pass, $h$ = hidden size.

In decode, you generate 1 token per sequence per iteration. For a single sequence ($B \times S = 1$) and $h = 8192$ (Llama-70B):

$$ M_{\text{decode}} \approx 1 \times 8192 \times 2 = 16\ \text{KB per all-reduce} $$

16 KB is tiny. At 900 GB/s, moving 16 KB takes around 18 nanoseconds of pure transfer, but the real cost is dominated by the launch latency of the collective (synchronisation, kernel launch), of the order of single-digit microseconds. With 160 all-reduces per token:

$$ t_{\text{comms/token}} \approx 160 \times (5\text{–}10\,\mu s) \approx 0.8\text{–}1.6\ \text{ms} $$

That is the communication floor per token, independent of bandwidth. An uncomfortable and counter-intuitive implication: buying more NVLink bandwidth does not speed up token-by-token decode of a single sequence. What helps in decode is lowering the latency per collective (LL protocol, vLLM’s custom all-reduce, NVLS to remove hops) and batching (raising $B$ amortises the fixed latency over more tokens, the deep reason continuous batching exists, covered in continuous batching).

In prefill, you process the whole prompt at once: $S$ can be thousands of tokens, and with batching $B \times S$ reaches tens of thousands. There:

$$ M_{\text{prefill}} \approx 8000 \times 8192 \times 2 \approx 131\ \text{MB per all-reduce} $$

131 MB does stress the bandwidth. At 900 GB/s (NVSwitch) the ring all-reduce moves $\frac{2 \cdot 3}{4} \times 131 \approx 196$ MB effective in around 0.22 ms; over PCIe (around 64 GB/s aggregate, bouncing through the CPU) it would be around 3 ms and serialised. Here the cable is the bottleneck and NVLS/bandwidth rule.

Summary in one line: prefill is bandwidth-bound, decode is latency-bound. Any interconnect tuning that does not say which phase it helps is noise.

The hardware: NVLink 4 and NVSwitch on the HGX baseboard

On the generic reference cluster, 4×H100 SXM inside an HGX baseboard, the concrete figures:

Each H100 SXM5 has 18 NVLink 4 links, each one 50 GB/s bidirectional ⇒ 900 GB/s bidirectional aggregate per GPU. That is >14× the bandwidth of a PCIe Gen4 x16 (around 64 GB/s bidir).
On an 8-GPU HGX H100 baseboard, the 18 links of each GPU are spread against four 3rd-generation NVSwitch chips (grouped 5+4+4+5). The result is all-to-all: any GPU talks to any other at 900 GB/s simultaneously, without going through CPU or PCIe.
A 4-GPU baseboard is half a board: same principle, via NVSwitch. Design key: if your 4 H100s are connected by NVSwitch, you have real all-to-all; if they are on different boards connected by PCIe (some “4×PCIe” configurations), you do not have NVLink between all of them and TP=4 suffers. Verify it, do not assume it.

The 10 knobs worth touching

Almost all of them are NCCL environment variables (injected into the inference engine process) or driver settings. Ordered by impact/frequency in an on-premise deployment. The canonical detail is in the NCCL env vars docs.

Knob 1 — `NCCL_DEBUG` + topology dump: see what is happening before touching anything

Do not optimise blind: first confirm which topology and algorithms NCCL picked. This tells you whether it really is using NVLink or whether, silently, it fell back to PCIe/SHM, the number one failure and the most expensive one.

NCCL_DEBUG=INFO # prints topology, rings/trees built, transport chosen
NCCL_DEBUG_SUBSYS=GRAPH,TUNING,NET # narrows it down to what matters
# look in the log for: "via NVLink" / "via P2P" (good) vs "via SHM" / "via PCI" (bad)

If you see via SHM or via PCI between GPUs that should have NVLink, you have a topology problem (PCIe ACS enabled, IOMMU, GPUs on different boards) and no other knob fixes it. This is knob 1 for a reason: half of the “NVLink is slow” cases are “NVLink is not being used”.

Knob 2 — `NCCL_ALGO`: ring vs tree vs NVLS

Forces or excludes algorithms. By default NCCL picks according to size, and it usually gets it right; touch it only with a measurement in front of you.

NCCL_ALGO=NVLS,Tree,Ring # order of preference
NCCL_ALGO=^Ring # exclude Ring (^ prefix)

Rule: prefill/training (bandwidth) ⇒ Ring/NVLS; decode (latency) ⇒ Tree or, better, vLLM’s custom kernel (knob 10/stack). In most inference workloads, leaving it on auto and validating with knob 1 is the right thing; forcing it “just in case” usually makes things worse.

Knob 3 — `NCCL_PROTO`: LL / LL128 / Simple

The protocol controls the latency/bandwidth trade-off at low level:

NCCL_PROTO=Simple # maximum bandwidth, more latency (large messages)
NCCL_PROTO=LL # low-latency, half-bandwidth (tiny messages: decode)
NCCL_PROTO=LL128 # compromise, default on platforms that support it

LL (low-latency) uses flags instead of barriers and wins on the 16 KB messages of decode; Simple wins on the 131 MB of prefill. The default LL,LL128,Simple lets NCCL choose by size, which again is normally the best option.

Knob 4 — `NCCL_NVLS_ENABLE`: offload the summation to the NVSwitch

NVLink SHARP (NVLS) makes the switch do the reduction, freeing up GPU SMs:

NCCL_NVLS_ENABLE=1 # default: ON where there is an NVLink4+ NVSwitch (Hopper)

An important sceptical caveat: NVLS requires NVSwitch (3rd gen, NVLink4). On a node with NVLink through direct GPU-to-GPU bridges (no switch) or on 4×PCIe, NVLS is not available and this knob does nothing. Before “enabling it”, confirm with knob 1 that your topology has a switch. Where it does apply, its biggest advantage is freeing SMs for compute, which is relevant when comms and kernels compete (knob 5).

Knob 5 — `NCCL_MIN_NCHANNELS` / `NCCL_MAX_NCHANNELS`: how many SMs communication steals

Each NCCL “channel” consumes GPU SMs to move data. More channels = more collective bandwidth, but fewer SMs for the inference kernel. It is a split of a fixed resource.

NCCL_MIN_NCHANNELS=4
NCCL_MAX_NCHANNELS=16 # raising it helps prefill (bandwidth); steals SMs from decode

In decode, where the GPU is under-used on compute but tied to latency, trimming channels rarely hurts and sometimes helps; in prefill, more channels squeeze out the bandwidth. A measurement knob, not a faith knob.

Knob 6 — `NCCL_BUFFSIZE`: the buffer size per channel

NCCL_BUFFSIZE=8388608 # 8 MB (default 4 MB); larger buffers → better BW on large messages

Raising it helps bandwidth-bound prefill at the cost of memory per channel. For workloads dominated by small messages (pure decode), the default is plenty.

Knob 7 — `NCCL_P2P_LEVEL` / `NCCL_P2P_DISABLE`: guarantee P2P over NVLink

P2P is what lets one GPU read another’s memory directly over NVLink without going through the host. If it is disabled or degraded, traffic falls back to SHM/PCIe.

NCCL_P2P_LEVEL=NVL # use P2P up to the NVLink level
# NCCL_P2P_DISABLE=1 ← only as a workaround if P2P HANGS (multi-NUMA PCIe, certain Blackwell)

Watch out for the trap: NCCL_P2P_DISABLE=1 and --disable-custom-all-reduce are recommended as a patch when vLLM hangs on PCIe-only multi-NUMA topologies. It is a robustness patch that sacrifices performance: use it if it hangs, never “by default”.

Knob 8 — GPUDirect RDMA for multi-node: `NCCL_NET_GDR_LEVEL`

When TP fits in one node, this does not apply. When you have to cross nodes (huge model, pipeline parallel between baseboards), GPUDirect RDMA lets the GPU talk to the NIC without bouncing through host memory:

NCCL_NET_GDR_LEVEL=PHB # enables GDR according to GPU–NIC proximity on the PCIe bus

Without GDR, every inter-node hop adds a host copy. With InfiniBand/RoCE plus GDR, the KV or the activations travel GPU→NIC→network→NIC→GPU. It is the basis of serious multi-node work and of mixed environments.

Knob 9 — `NCCL_IB_HCA` / `NCCL_SOCKET_IFNAME`: pin the right NIC

The most common and silent multi-node mistake: NCCL picks the management NIC (1 GbE) instead of the fabric one (InfiniBand/100 GbE). Result: collectives at a snail’s pace with no visible error.

NCCL_SOCKET_IFNAME=eth0 # control interface (bootstrap)
NCCL_IB_HCA=mlx5_0,mlx5_1 # the real InfiniBand HCAs of the fabric
NCCL_IB_GID_INDEX=3 # correct GID for RoCE v2

Pin them explicitly. “Auto” gets it right on clean clusters and fails as soon as there is more than one NIC.

Knob 10 — Driver: persistence mode, clocks and NVLink error counters

Below NCCL, the driver has levers and, above all, telemetry you have to look at:

nvidia-smi -pm 1 # persistence mode: avoids driver re-init (latency/jitter)
nvidia-smi nvlink --status # are all 18 links active and at full speed?
nvidia-smi nvlink -e # error/CRC counters per link
nvidia-smi -q -d ECC # memory errors that degrade silently

An NVLink link that negotiates at half speed or accumulates CRC errors degrades the all-reduce without raising any error: the system “works”, it is just slower. These counters are the difference between diagnosing in five minutes and chasing a ghost for days. They are integrated in DCGM (knob/stack: observability).

Summary table

#	Knob	Variable / command	Phase it helps
1	Topology diagnosis	`NCCL_DEBUG=INFO` + `SUBSYS=GRAPH`	always, first
2	Collective algorithm	`NCCL_ALGO` (NVLS/Tree/Ring)	depends on phase; auto usually wins
3	Protocol	`NCCL_PROTO` (LL/LL128/Simple)	LL=decode, Simple=prefill
4	NVLink SHARP	`NCCL_NVLS_ENABLE=1`	prefill; frees SMs (requires NVSwitch)
5	Channels (SMs)	`NCCL_MIN/MAX_NCHANNELS`	+prefill bandwidth / −SM theft in decode
6	Buffer	`NCCL_BUFFSIZE`	bandwidth-bound prefill
7	NVLink P2P	`NCCL_P2P_LEVEL=NVL`	critical; disable only if it hangs
8	GPUDirect RDMA	`NCCL_NET_GDR_LEVEL`	multi-node
9	Fabric NIC	`NCCL_IB_HCA`/`SOCKET_IFNAME`	multi-node (avoids mgmt NIC)
10	Driver + telemetry	`nvidia-smi -pm 1` / `nvlink -e`	jitter + silent diagnosis

How it connects with the rest of the stack

The interconnect is not an island; it touches almost every layer above.

With vLLM, the custom all-reduce. vLLM does not always use NCCL: for the tiny messages of decode (world_size==2 or a fully-connected NVLink topology, below a certain max_size) it uses its own all-reduce kernel that beats NCCL on latency, exactly the decode bottleneck we saw in the maths. It falls back to NCCL for large messages and for topologies without NVLink (where its custom kernel “adds little over NCCL”). The flag --disable-custom-all-reduce / VLLM_DISABLE_CUSTOM_ALL_REDUCE turns it off; it is the patch for hangs on multi-NUMA PCIe. Translation: the most effective decode latency knob is sometimes not an NCCL one, it is choosing well between vLLM’s custom kernel and NCCL.

With TP vs replicas. Everything in one big replica or many small ones rests on this: high TP is only viable inside the NVLink domain. The boundary of “TP=4 or 4 replicas at TP=1?” is drawn by the cable: crossing NVLink with TP means paying for the all-reduce at PCIe prices.

With disaggregated serving. In disaggregated prefill/decode, the KV cache generated in the prefill pool has to travel to the decode pool. That transfer is another consumer of the interconnect (NVLink intra-node, GPUDirect RDMA inter-node) and competes with the all-reduces. Designing the disaggregation without counting the cost of transferring the KV is the classic trap.

With MoE. Mixture-of-Experts models add expert parallelism: an all-to-all (not an all-reduce) that routes each token to its expert, possibly on another GPU. It is a different communication pattern and heavier on bandwidth; MoE in inference lives or dies by the same cable, with an even more demanding collective.

With GPU observability. The NVLink counters (nvidia-smi nvlink -e, TX/RX bytes per link, CRC errors) and NVSwitch utilisation are exposed via DCGM and land in Prometheus/Grafana. The question “is the interconnect healthy and saturated?” is answered there, alongside the rest of GPU observability with DCGM. A slow all-reduce shows up sooner in an NVLink error counter than in API latency.

With capacity planning. The inference sizing that assumes “TP=4 scales almost linearly” only holds inside NVLink. Outside it, scaling efficiency collapses and the capacity plan lies. The cable is a parameter of the capacity model, not a detail.

Traps and things that are not what they look like

“More NVLink bandwidth = faster decode.” False for a single sequence. Decode is latency-bound; bandwidth is barely touched by 16 KB messages. What speeds up decode is batching (amortising the fixed latency) and lowering latency per collective (LL, custom kernel, NVLS). Bandwidth rules in prefill.

“I have 4 H100s, so I have NVLink between all four.” Not necessarily. There are configurations where the GPUs are on different boards joined by PCIe, or with NVLink bridges only in pairs. Confirm it with nvidia-smi nvlink --status and knob 1 before planning TP=4. A TP=4 over P2P-over-PCIe performs far worse than the brochure says.

Forcing NCCL_ALGO/NCCL_PROTO “to go faster”. NCCL picks well by size in most cases. Forcing an algorithm without measuring usually makes one of the two phases worse. The correct sequence is: knob 1 (see what it is doing) → measure → touch only if there is evidence.

Disabling P2P/custom all-reduce by default. They are robustness patches for broken topologies (multi-NUMA PCIe, certain Blackwell). Leaving them on “for stability” on a node with healthy NVLink throws performance in the bin.

Stretching TP over the network. TP=8 crossing two nodes over InfiniBand because “there is bandwidth” ignores that the per-layer all-reduce now pays network latency ×160 per token. To cross nodes, pipeline parallel (which communicates once per micro-batch, not per layer) almost always wins. The communication pattern, not just the bandwidth, decides.

Ignoring the NVLink error counters. A degraded link does not raise an exception: the system works, it is just slow. Without watching nvlink -e and ECC, you chase a performance ghost that a counter would have pointed out in five minutes.

Conclusion

Tensor parallelism sells a simple promise, split the model, multiply the VRAM, serve models that do not fit in one GPU, but the small print is that every layer forces the GPUs to come together and add up, twice, dozens of times per token. That all-reduce is the real hidden protagonist of performance, and it lives in the cable: NVLink does it on the shared bench of the NVSwitch at 900 GB/s, or PCIe drags it through the CPU’s reception desk 14× slower. Of the ten knobs, the first one, looking with NCCL_DEBUG at what is really happening, solves half the problems, because half of the “NVLink is slow” cases are “NVLink is not being used”. The rest are refinements that only mean something if you know which phase you are in: prefill wants bandwidth (NVLS, Simple, channels, buffer), decode wants latency (LL, vLLM’s custom kernel, batching). And above all, an idea that reorders the intuition: in on-premise inference, the interconnect is not plumbing you install and forget, it is a hot path, a capacity parameter and, when it degrades silently, the root cause that no API dashboard will point out to you unless you look at the counters of the cable itself.

References

NVIDIA, NVIDIA Hopper Architecture In-Depth (NVLink 4, 900 GB/s): https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/.
NVIDIA, Introducing NVIDIA HGX H100 (4× NVSwitch, all-to-all): https://developer.nvidia.com/blog/introducing-nvidia-hgx-h100-an-accelerated-server-platform-for-ai-and-high-performance-computing/.
NVIDIA, NCCL Environment Variables (all the knobs in this post): https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html.
NVIDIA, The NVLink-Network Switch (Hot Chips 2022, NVLink SHARP): https://hc34.hotchips.org/.
vLLM, Why does vLLM use a custom all-reduce method? (discussion #6159) and custom_all_reduce.py: https://github.com/vllm-project/vllm/discussions/6159.
NVIDIA, NCCL Multi-Node NVLink Tuning Guide: https://docs.nvidia.com/multi-node-nvlink-systems/multi-node-tuning-guide/nccl.html.

Anatomy of the twelve DCGM and five vLLM metrics: analogies, documented anomalies and real cases 2024-2026

Tue, 02 Jun 2026 04:00:00 +0200

This post goes deeper into the list of metrics presented in GPU observability for LLM inference. There each metric got its G/A/R threshold and PromQL query; here each one gets its explanatory analogy and the anomaly documented in production with a referenced public case. It is the post worth having open when an alert fires and you still do not know what to do with it; the next post on runbooks translates each anomaly into concrete action.

TL;DR

The twelve DCGM metrics (compute, memory, thermal-power, health) and the five vLLM engine metrics (concurrency, KV pool, SLO latencies) covered in the previous post paint the cluster’s cockpit, but a list without context does not teach you to diagnose. Each metric has a recurring anomalous pattern documented in the public literature (academic papers, GitHub issues, OEM KBs, operator blogs) that the veteran operator recognises instantly and the junior one does not. This post develops each metric with an analogy of its own that pins down which question it answers, and with the statistically relevant anomaly backed by figures from documented incidents. Three examples of the calibre involved: Meta published that during the training of Llama 3 405B on 16,384 H100s there were 419 unplanned failures in 54 days, one every 3 hours, with GPU + HBM3 accounting for 47 % of the total; the paper Story of Two GPUs (arXiv 2503.11901) quantifies that the H100 has 3.2× worse MTBE from uncorrectable ECC than the A100, attributable to the higher density of HBM3; issue vllm#16300 documents that on a cluster of 8×A100 80 GB TP=8 delivers worse throughput than TP=4 because NVLink saturation kills the partition speedup. The KBs Dell 000220508 and Lenovo HT514380 formalise the recurring HW Power Brake case in H100 racks overcommitted at PDU level. Issue vllm#25677 showed chunked prefill 10× slower than without it on Qwen3-30B-A3B (bad calibration of max_num_batched_tokens). Issue vllm#11912 documents a TPOT regression from 15.7 ms to 25.7 ms crossing version 0.6.4. Each case includes a verifiable URL. The operational rule: when an alert arrives, look first at the anomalous pattern associated with the metric that fired, before opening the request’s trace; 80 % of degradations match one of the documented patterns.

You are here: OBSERVE — the diagnosis layer

Family 1 — Compute

`DCGM_FI_PROF_SM_OCCUPANCY` — is there parallel work in the engines?

The analogy. An industrial kitchen with 32 hobs and a single chef. The metric answers “how many hobs have a pan on them right now?”. If half are empty, the kitchen is under-used: the orders come out one after another because the chef does not open up parallelism. If all of them are occupied but the chef is standing still watching a stopwatch, the hobs are on but nothing is being cooked (a pathological kernel saturating SMs without doing useful work).

The documented anomaly. The best-known trap: high SM occupancy does not imply real throughput. The article “GPU Utilization Is a Counter, Not a Cause” (Ingero, May 2026) put it in one exact sentence: “a kernel running at 5 % of peak FLOPS for 100 ms still reads 100 % on SM_ACTIVE”. In MoE workloads the effect turns pathological: overloaded experts produce the Straggler Effect (paper arXiv 2503.05066). The SMs look busy while waiting for the saturated expert, and the utilisation dashboard paints green while latency goes through the roof.

Operational implication. Do not trust sizing or autoscaling to SM occupancy alone. Always combine it with PIPE_TENSOR_ACTIVE (is there useful compute?) and DRAM_ACTIVE (is memory the bottleneck?). The normal LLM decode regime is 30–55 %, not 99 %; seeing a sustained 99 % with high TPOT is a symptom of a kernel bug or an MoE straggler.

`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE` — are the tensor cores producing?

The analogy. A factory with two lines: the manual one (CUDA cores) and the automated one (tensor cores). The metric answers “what percentage of the time is the automated line active?”. If you buy an H100 for its tensor cores and the automated line is at 5 %, you have paid for a Ferrari to do bicycle courier work.

The documented anomaly. Issue vllm#20783 (July 2025) was titled literally “Performance Anomaly: compressed-tensors shows no speedup over BF16 on H100”. The operator expected 1.5–2× with FP8 quantisation and got parity with BF16; the PIPE_TENSOR_ACTIVE metric revealed that the FP8 path was not running on the HMMA units (the tensor unit for FP16/BF16/FP8) and was falling back to CUDA cores. Issue vllm#31475 documented the parallel case on MI300X: FP8 slower than BF16 because of a regression in the ROCm path. DCGM exposes separate counters per unit (HMMA for FP16/BF16/FP8, IMMA for INT8, DMMA for TF32/FP32); if HMMA is low even though the model is BF16, the engine is not using tensor cores.

Operational implication. Check PIPE_TENSOR_ACTIVE after every change of quantisation or engine version; a supposedly neutral change may have disabled the optimised path. For prefill expect 50–80 %; for decode 15–30 % is normal (decode is memory-bound, not compute-bound). A figure below 5 % in prefill means the engine is not using tensor cores.

`DCGM_FI_PROF_DRAM_ACTIVE` — is the HBM saturated?

The analogy. A motorway with N lanes. The metric answers “what percentage of the time are all the lanes occupied moving cars?”. When the tensor cores ask for data faster than the HBM delivers it, the motorway is at 95 % and the engines wait. In decode, this is the normal regime: you walk the model weights and the KV cache through for every token.

The documented anomaly. The paper “Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference” (arXiv 2503.08311) quantifies that at contexts ≥ 128k, reading the KV cache dominates the total decode time and saturates the HBM3 (3.35 TB/s on the H100). The distinctive pattern: DRAM_ACTIVE > 80 % with PIPE_TENSOR_ACTIVE around 10–20 %. Raising the batch no longer helps: the bottleneck is not FLOPS, it is bandwidth. The useful lever is compressing the KV, see Quantisation for --kv-cache-dtype=fp8, which cuts the KV footprint by roughly 50 %.

Operational implication. If DRAM_ACTIVE > 95 % is sustained and gpu_cache_usage_perc < 70 %, something other than your engine is asking for HBM (a leak in a library, another process sharing the GPU without MIG). Investigate immediately with nvidia-smi and fuser /dev/nvidia*.

Family 2 — Memory

`DCGM_FI_DEV_FB_USED` — how much VRAM has been consumed?

The analogy. The fuel level in an aircraft’s tank in flight: the pilot needs to know how much is left and at what rate it is being consumed, not just the instantaneous figure. An H100 at 88 % FB used and stable can operate calmly; the same figure rising 2 %/min announces OOM in 7 minutes.

The documented anomaly. Issue dcgm-exporter#512 documents a surprise relevant to MIG clusters: DCGM_FI_DEV_FB_USED and DCGM_FI_DEV_FB_FREE are absent on H100 GPU instances with MIG enabled, present on A100 and B200, but an exporter bug hides them on H100-MIG. Operators who assume the dashboard covers everything discover the blind spot on the day of the first OOM. Issue dcgm-exporter#271 documents another detail: FB_USED + FB_FREE does not always add up to a constant because there is driver-reserved overhead that shows up in the delta. The original PagedAttention/vLLM paper estimated that pre-PagedAttention serving frameworks wasted 60–80 % of the KV cache to fragmentation; PagedAttention brought it down to under 4 %.

Operational implication. On H100 MIG clusters, check that DCGM_FI_DEV_FB_USED appears per instance before trusting alerts; if it is absent, monitor through nvidia-smi --query-gpu=memory.used directly. Operational rule: alert on the delta (a sustained rise), not just an absolute threshold.

`DCGM_FI_DEV_FB_FREE` — the absolute complement

The analogy. The “range remaining” indicator in a modern car: it complements the percentage with an absolute figure that is directly actionable.

The documented anomaly. When an aggressive PagedAttention pool leaves FB_FREE at small absolute values (< 2 GiB), any normal allocation of transient buffers (the activations of a large prefill) can push you into OOM. The classic pattern: a “green” percentage (87 %) but a “red” absolute (< 4 GiB free on an 80 GB H100).

Operational implication. A complementary alert with an absolute threshold: DCGM_FI_DEV_FB_FREE < 4096 (MiB). It is the safety net for the cases where the percentage misleads because the engine is configured with a very high gpu_memory_utilization.

`DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL` — does the internal bus hold up?

The analogy. An interstate motorway between four cities. Every car that crosses to do a tensor parallel all-reduce pays a toll and consumes width. When there are more cars than the motorway supports, the latency to reach the destination shoots up, even if each individual car is fast.

The documented anomaly. Issue vllm#16300 (April 2025) was titled “Performance degradation with tp=8 compared to tp=4 on 8×A100(80G)” and documented TP=8 delivering worse throughput than TP=4 on the same cluster, same model, same quantisation. Root cause: tensor parallelism requires an all-reduce after each attention and MLP block; at TP=8, the communication cost between 8 GPUs (even over NVSwitch) grows faster than the speedup of partitioning the compute. The practical rule that emerges: TP=4 + 2 replicas usually delivers better latency/throughput than TP=8 + 1 replica, except for extremely long contexts (≥128k) where you need the aggregate VRAM. Theoretical NVLink 4.0 capacity on the H100 SXM: around 450 GB/s per GPU; typical sustained TP=4 regime: 50–150 GB/s.

Operational implication. If NVLINK_BANDWIDTH_TOTAL > 90 % of capacity is sustained, this is not a problem solvable by raising parallelism. The opposite: lower TP. The metric is orthogonal to the sizing from capacity planning: the ceiling is not only VRAM/time, it is also the bus.

Family 3 — Thermal and power

`DCGM_FI_DEV_GPU_TEMP` — is the GPU breathing?

The analogy. The body temperature of an elite athlete under full effort. 36–37 °C is normal; 38 °C is sustainable stress; above 39 °C the body activates protective mechanisms (sweating, slowing down) that degrade performance. The GPU does the same: above a thermal threshold, it reduces its clock automatically. If it did not, it would break.

The documented anomaly. The H100 SXM5 with its 700 W TDP has thermal thresholds that are not entirely public (NVIDIA does not publish them exhaustively in the datasheet), but the behaviour is well known: above roughly 85 °C edge or roughly 95 °C HBM, the bit 0x40 HW_THERMAL appears in the clock throttle reasons. Operators on the NVIDIA developer forum report that with a rack inlet temperature above 27 °C, throttling is routine. The NVIDIA HGX Platform paper states that the minimum recommended air flow is > 1000 CFM/kW; densities above 30 kW/rack at 700 W TDP demand mandatory liquid cooling because forced air does not get there.

Operational implication. If GPU_TEMP > 83 °C is sustained, look first at CLOCK_THROTTLE_REASONS (bit 0x40) and at the rack inlet temperature: this is not an engine problem, it is an air flow one. For legacy air-cooled racks, consider redistributing thermal load or installing a rear-door HX.

`DCGM_FI_DEV_POWER_USAGE` — how much is it asking of the socket?

The analogy. The instantaneous draw of an industrial appliance plugged into a three-phase outlet with a sized breaker. If the washing machine starts at 9 kW and the breaker is 10 kW, you live on the edge; if the washing machine “gets along” with the breaker it is because somebody sized it consciously.

The documented anomaly. A published real measurement: an H100 SXM5 with vLLM running Llama 3.1 405B at batch=4 draws around 697 W at-wall sustained (NVIDIA TDP 700 W). Now the interesting operational lever: lowering nvidia-smi -pl from 700 W to 500 W delivers around 30 % energy savings with only around 20 % loss of throughput. A cluster of 4 nodes × 8 H100 at 700 W equals around 22 kW of GPU alone; at 500 W, around 16 kW. The difference pays an entire quarter’s electricity bill in clusters operated around the clock. A three-phase 415 VAC PDU branch at 60–80 A supports around 32 kW, roughly 4 DGX H100. Legacy 208 V does not support H100 density; reference: NVIDIA DGX SuperPOD Electrical Specifications.

Operational implication. A useful metric for three things: (1) detecting anomalously low workloads (unexpected idle), (2) computing per-tenant showback of real energy cost (not estimates), (3) alerting if the draw approaches the PDU branch limit. Keep GPU → PDU branch → breaker mapped in the CMDB.

`DCGM_FI_DEV_CLOCK_THROTTLE_REASONS` — who is stepping on the brake?

The analogy. The “limited mode” warning light on the dashboard of a modern car. When it lights up, the car reduces its performance automatically, but it does not tell you why unless you know how to read the combination of letters. The bits of the bitmap are those letters.

The documented anomaly. A public case formally acknowledged by two different OEMs: Dell KB 000220508 and Lenovo HT514380 address the same phenomenon: HW Power Brake Slowdown active (bit 0x80) on the H100 SXM. The cause is not the GPU: it is the chassis PDU sending an electrical power-brake signal because the rack branch is close to the breaker limit. The operator sees throughput down 30–50 % with no XID and no ECC, and the inference engine “is healthy”; the problem is in the electrics. The NVIDIA developer forum thread on “HW Power Brake Slowdown” corroborates the pattern. Bit 0x40 HW_THERMAL appears in badly ventilated racks; bit 0x04 SW_POWER_CAP appears if somebody left nvidia-smi -pl 500 set and nobody reverts it.

Operational implication. Any bit other than 0 or Idle (bit 0x01) sustained is an immediate alert. The recommended decoding: record the complete bitmap value in the log plus the attribute throttle.reasons.decoded=["HW_THERMAL", "HW_POWER_BRAKE"] on the OTel span. Without that, incident response does not know what to do.

Family 4 — Health (the catastrophic reports)

`DCGM_FI_DEV_XID_ERRORS` — the driver’s red codes

The analogy. The critical alarm lights in a nuclear control room. They do not climb gradually; they either appear or they do not. Each XID is a predefined code (XID 13 = graphics engine exception; XID 31 = MMU fault; XID 43 = stopped channel; XID 79 = GPU fallen off the bus; XID 95 = uncontained ECC), and each one has its documented procedure.

The documented anomaly. The most studied public case: Meta published that during the training of Llama 3 405B on 16,384 H100s over 54 days there were 419 unplanned failures, one every 3 hours at cluster scale. GPUs accounted for 148 (35 %) plus HBM3 for 72 (17 %), almost half of all failures. The paper “Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100” (arXiv 2503.11901) quantifies with a different dataset (2.1M GPU-hours) that the H100 has 3.2× worse MTBE for uncorrectable ECC than the A100. The ByteDance MegaScale paper reports that XID 79 (“GPU fallen off the bus”) co-occurs with PCIe errors in 43 % of cases. The NVIDIA developer forum documents persistent cases of XID 31 (MMU fault) that follow the GPU when it is moved to a different PCIe slot, a hardware bug in the module, not in the backplane.

Operational implication. Any increment of the counter is an immediate alert: many XIDs demand a node reset or an RMA of the GPU. The XID-by-XID distinction matters: XID 13/43 is usually a software bug if it coincides with a recent change; XID 31/48/79/94/95 is usually hardware. Keep a canonical xid → procedure table. See the runbooks for the translation into concrete action.

`DCGM_FI_DEV_ECC_DBE_VOL_TOTAL` — the errors that corrupt data

The analogy. An accounting ledger where sometimes someone erases an entry and rewrites it (corrected single-bit ECC: note a change in the margin and carry on) and sometimes someone burns two pages at once (double-bit: the information is lost, the audit has to stop).

The documented anomaly. The paper “Characterizing GPU Resilience” quantifies for the H100 that when XID 48 (DBE) appears, the job in flight dies with 100 % probability (5 out of 5 in the dataset studied). The documented recovery: drain the node, reset, complete the row remap, equalling around 19 hours of node downtime. HBM3 density explains the worse MTBE against HBM2e: there are more cells per unit area, and therefore a higher statistical probability of degradation. In Llama 3, HBM3 caused 72 of the 419 interruptions (17 %).

Operational implication. Any value > 0 is a critical alert. The GPU must be drained immediately, removed from the scheduler, fully reset, validated for row remap with nvidia-smi -q -d ROW_REMAPPER (Pending: No), and given an extensive smoke test before returning to the pool. If the row remap uses more than 4–8 spare pages on one GPU, plan a replacement in the next window: the degradation is progressive.

`DCGM_FI_DEV_RETIRED_DBE` — the pages marked for retirement

The analogy. The tiles a museum restorer marks with yellow tape because they are damaged. They pose no immediate danger (the room stays open), but the accumulation says the floor is degrading structurally and the full replacement has to be planned.

The documented anomaly. NVIDIA documents up to 512 spare pages per HBM bank on the H100; the RETIRED_DBE counter indicates how many have been used. Operators on NVIDIA forums report that above 4–8 retired pages on a specific GPU, the frequency of XID 48 rises. The pattern: a GPU with 6 retired pages today becomes 12 in a month, a first XID 48 two months later, and a forced drain.

Operational implication. A trend metric, not an immediate alert. Record the value per GPU and review it monthly; GPUs with rising values go into the proactive replacement plan before the catastrophic failure.

The five vLLM engine metrics

`vllm:num_requests_running` — how many requests fit in the batch?

The analogy. The number of cars a toll booth lets through at once. If the barrier opens for N at a time, car N+1 waits in the queue. Saturation shows up because the line does not get shorter.

The documented anomaly. Reaching the configured --max-num-seqs and staying there is the typical symptom of a cluster below its sizing; the engine admits up to the ceiling and no further. The query vllm:num_requests_running == max_num_seqs for more than 5 minutes indicates firm saturation.

Operational implication. Combine it with num_requests_waiting: if running is at the ceiling AND waiting > 0, you have to scale. If running is at the ceiling and waiting is 0, you are in the optimal regime (cluster used to the maximum with no queue).

`vllm:num_requests_waiting` — the primary saturation indicator

The analogy. The visible queue in front of the toll booth. As long as it is empty, the system flows; the moment a sustained queue forms, drivers start arriving late at their destination, and TTFT goes through the roof.

The documented anomaly. A public case in “11-Second Time to First Token on a Healthy vLLM Server” (Medium, Ingero, 2026): a server with no XIDs, no preemption, DCGM metrics all green, but num_requests_waiting sustained above 0 and a TTFT of 11 seconds. Issue vllm#16985 documents progressive degradation in long sessions: the queue grows slowly over hours without any other indicator moving. The root cause is not hardware, it is admission control: the arrival rate exceeds the completion rate and the system does not enqueue any more, it leaves requests in WAITING. Red Hat designates it as the primary saturation metric in its tutorial “5 steps to triage vLLM performance”.

Operational implication. The primary HPA metric in KEDA, see Autoscaling LLM on Kubernetes. Typical threshold: alert if avg_over_time(vllm:num_requests_waiting[5m]) > 5. For canaries: if the queue only forms in the v2 pool, it is a regression of the new model, not cluster load.

`vllm:gpu_cache_usage_perc` — the KV cache pool

The analogy. The capacity of an events hall where each guest takes up a variable amount of space. The maître d’ admits guests up to capacity; when a new guest arrives and there is no room, he throws out the guest who has been there longest to make space for the newcomer. That is vLLM’s preempt-on-OOM.

The documented anomaly. Issue vllm#5051 “Add num_requests_preempted metric” was born exactly out of operators observing degradation with no direct metric telling them how many requests were being thrown out. The official vLLM documentation confirms: “sustained gpu_cache_usage_perc above 90 % indicates the server is approaching its KV cache limit and will begin preempting sequences” (oldest-first). The distinctive visual pattern: a sawtooth near 100 % with preemption spikes. In swap mode, the latency of the preempted request explodes because there is a host↔device PCIe copy; in recompute mode (the default in V1), the preempted request redoes its prefill from scratch, which pushes its TTFT to double or triple.

Operational implication. If gpu_cache_usage_perc > 92 % is sustained, there are two levers: lower max_num_seqs (admit less concurrency, but none of it gets preempted) or raise gpu_memory_utilization (more pool, less VRAM for transient activations, a different risk). The choice depends on the workload. The metric that is directly missing, a preempted counter, is exported from vLLM v1.0 onwards as vllm:num_preemptions_total (see issue #5051).

`vllm:time_to_first_token_seconds` — the latency the client sees

The analogy. The time from a customer walking into a restaurant to receiving the first piece of bread at the table. Too long and the customer thinks they have been forgotten, even if the main course is going to arrive perfect.

The documented anomaly. Three documented patterns of recurring TTFT spikes:

Badly calibrated chunked prefill. Issue vllm#25677 (Qwen3-30B-A3B) reported prefill 10–11× slower with chunked prefill enabled than without it. Cause: a very low max_num_batched_tokens forces small chunks that do not fill the kernels. Issue vllm#7604 documents an equivalent regression on Llama-3-70B v0.5.4. The lever: raise max_num_batched_tokens to 4096–8192 for typical prompts above 2k.
Regression between engine versions. Issue vllm#8819 documents a regression of vllm:time_to_first_token_seconds_sum between minor versions. Issue vllm#11912 reports that with a prompt of around 8000 tokens, TPOT rose from 15.7 ms → 25.7 ms from v0.6.4.post1 onwards with no config change, a regression confirmed and trackable only with the metric.
Long-context prefill blocking decodes. The “11s TTFT on healthy server” case cited above: a 30k-token prefill monopolises the GPU for several seconds and the active decodes freeze. Fix: well-calibrated chunked prefill, or disaggregated serving (see Disaggregated serving).

Operational implication. Do not alert only on the absolute P95; alert also on the v2/v1 ratio when there is a canary (histogram_quantile(0.95, ..., version="v2") / histogram_quantile(0.95, ..., version="v1") > 1.10). If TTFT grows and the queue is stable, the bottleneck is prefill: not solvable by adding replicas, but there are levers in quantisation or chunked prefill.

`vllm:time_per_output_token_seconds` — streaming smoothness

The analogy. The speed at which the waiter brings the courses one after another after the first. If the next one takes a while, the diner notices something is wrong even though the first course arrived on time.

The documented anomaly. The distinctive pattern is the abrupt step when gpu_cache_usage_perc crosses roughly 85 %: TPOT jumps from 35 ms to 80 ms in a few seconds because the engine starts competing for the HBM with its own evictions. Issue vllm#35387 documents another anomalous case: MTP (speculative decoding) causing a 76 % latency regression on Qwen3-Next-80B-A3B-Instruct-FP8. The TPOT metric caught it before any client complaints were filed.

Operational implication. The difference from TTFT: if TTFT grows and Queue Time is stable, you are prefill bound; if TPOT grows at a steady rate, there is pressure on the HBM (KV cache pool or swap enabled). A secondary alert on the TPOT SLO, but also watch the derivative: TPOT rising 1 ms every 10 minutes is a latent regression that does not break the SLO yet but will.

The operational rule: read metrics by family, not in isolation

Three anti-patterns of the novice operator

Anti-pattern 1 — alerting only on absolute thresholds. An H100 at 87 % FB is not necessarily an alarm; an H100 at 87 % rising 2 %/min is. Alerts that fire on a fixed threshold without looking at the derivative produce twice the noise and half the value. Rule: for metrics with known dynamics (KV cache, FB, queue), alert on a sustained delta, not just on the level.

Anti-pattern 2 — confusing SBE with DBE. The DCGM_FI_DEV_ECC_SBE_VOL_TOTAL counter (single-bit, correctable) grows continuously on any HBM under load; it is not an alarm, it is physics. The one that matters is DCGM_FI_DEV_ECC_DBE_VOL_TOTAL (double-bit, uncorrectable). Confusing them yields false negatives (not alerting on a real DBE) or false positives (alerting on a harmless SBE).

Anti-pattern 3 — treating SM_OCCUPANCY at 99 % as “saturated”. The LLM decode regime is memory-bound, not compute-bound; high SM occupancy with low TENSOR_ACTIVE and high DRAM_ACTIVE is normal. Sizing for “GPU at 60 %” and asking for more hardware when the cluster is saturated in HBM (not in SMs) means buying twice the GPUs without gaining throughput. Rule: always read SM_OCCUPANCY together with TENSOR_ACTIVE and DRAM_ACTIVE; on its own it means nothing.

Applied to typical on-premise hardware

For a generic cluster of 4 nodes × 4×H100 SXM 80 GB with intra-node NVLink:

DCGM Exporter per node (a GPU Operator DaemonSet) emitting every 15 s; cardinality per GPU is around 80 series. A 16-GPU cluster gives roughly 1.3k base series, around 85k samples/min with a 15 s scrape.
vLLM /metrics per inference pod; each replica emits around 50 base series. For 16 replicas, around 800 additional series, around 3k samples/min.
Prometheus retention: 30 days at high resolution plus 1 year downsampled through a Thanos sidecar or Mimir. Estimated volume: 25–35 GB/day.
Alertmanager: the 6 critical alerts from the previous post plus derived alerts (delta, v2/v1 ratio, decoded throttle bitmap).

Each metric is worth exposing also as an OTel attribute on the spans of the GenAI tracing: gpu.fb_used_pct, gpu.dram_active, gpu.throttle_reasons.decoded. That lets you correlate a slow request with the state of the GPU at that instant, without jumping between dashboards.

What we have not covered (upcoming posts)

Per-alert runbooks — the translation of each anomalous metric into concrete action (drain, reset, RMA, scaling, rollback) in the next post: Incident response runbooks.
Tail sampling for metric ↔ trace correlation — what is preserved when an alert fires, for post-mortem investigation.
Per-tenant showback combining vllm:request_success_total × gen_ai.usage.* × DCGM_FI_DEV_POWER_USAGE to bill real energy cost.
Multi-tenant fairness metrics — when a tenant hogs the KV cache pool and how to detect it.

References

Meta — Faulty Nvidia H100 GPUs and HBM3 memory caused half of failures during Llama 3 training (Tom’s Hardware, 2024). tomshardware.com
Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100. arXiv 2503.11901. https://arxiv.org/html/2503.11901v3
ByteDance — Robust LLM Training Infrastructure at ByteDance. arXiv 2509.16293. https://arxiv.org/pdf/2509.16293
Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference. arXiv 2503.08311.
Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts. arXiv 2503.05066.
NVIDIA — Analyzing Xid Errors with the Xid Catalog and Memory Error Management (docs.nvidia.com/deploy).
Dell — PowerEdge XE8640 with H100 - GPU Performance Issue HW Power Brake Slowdown - Active (KB 000220508).
Lenovo — Power brake reporting on H100 GPU (HT514380).
vLLM project — issues #5051 (preempted metric), #7604 and #25677 (chunked prefill regression), #11912 (long-prompt regression), #16300 (TP=8 worse than TP=4), #16985 (long-running degradation), #20783 (compressed-tensors no speedup), #35387 (MTP regression).
Red Hat — 5 steps to triage vLLM performance. https://developers.redhat.com/articles/2026/03/09/5-steps-triage-vllm-performance
AI21 — Go big or go OOM: the art of scaling vLLM. https://www.ai21.com/blog/scaling-vllm-without-oom/
11-Second Time to First Token on a Healthy vLLM Server (Medium, Ingero, 2026).
NVIDIA — DGX SuperPOD Electrical Specifications (docs.nvidia.com/dgx-superpod).

Sources: the full URLs are linked inline on each reference.

Nvlink on lo0 — Tech Blog

The corridors and the security guard: PCIe topology, GPUDirect P2P and ACS

TL;DR

Where you are: the corridors, not the bench

The topology of a node: two buses, not one

GPUDirect: skipping reception

The guard: why ACS and IOMMU break P2P

The maths that matter: how much the detour costs

The real tension: performance vs isolation (and ENS)

The 10 knobs

How it connects with the rest of the stack

Traps and things that are not what they look like

Conclusion

See also

References

The shared bench: NVLink, NVSwitch and NCCL, the cable every token crosses in tensor parallel

TL;DR

Where you are: the floor below the engine

The analogy: four mechanics and a single bench

The mechanism: what an all-reduce really is and why there are 160 per token

How the all-reduce is done: ring, tree, NVLS

The maths that matter: why decode and prefill stress the cable the other way round

The hardware: NVLink 4 and NVSwitch on the HGX baseboard

The 10 knobs worth touching

Knob 1 — NCCL_DEBUG + topology dump: see what is happening before touching anything

Knob 2 — NCCL_ALGO: ring vs tree vs NVLS

Knob 3 — NCCL_PROTO: LL / LL128 / Simple

Knob 4 — NCCL_NVLS_ENABLE: offload the summation to the NVSwitch

Knob 5 — NCCL_MIN_NCHANNELS / NCCL_MAX_NCHANNELS: how many SMs communication steals

Knob 6 — NCCL_BUFFSIZE: the buffer size per channel

Knob 7 — NCCL_P2P_LEVEL / NCCL_P2P_DISABLE: guarantee P2P over NVLink

Knob 8 — GPUDirect RDMA for multi-node: NCCL_NET_GDR_LEVEL

Knob 9 — NCCL_IB_HCA / NCCL_SOCKET_IFNAME: pin the right NIC

Knob 10 — Driver: persistence mode, clocks and NVLink error counters

Summary table

How it connects with the rest of the stack

Traps and things that are not what they look like

Conclusion

See also

References

Anatomy of the twelve DCGM and five vLLM metrics: analogies, documented anomalies and real cases 2024-2026

TL;DR

You are here: OBSERVE — the diagnosis layer

Family 1 — Compute

DCGM_FI_PROF_SM_OCCUPANCY — is there parallel work in the engines?

DCGM_FI_PROF_PIPE_TENSOR_ACTIVE — are the tensor cores producing?

DCGM_FI_PROF_DRAM_ACTIVE — is the HBM saturated?

Family 2 — Memory

DCGM_FI_DEV_FB_USED — how much VRAM has been consumed?

DCGM_FI_DEV_FB_FREE — the absolute complement

DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL — does the internal bus hold up?

Family 3 — Thermal and power

DCGM_FI_DEV_GPU_TEMP — is the GPU breathing?

DCGM_FI_DEV_POWER_USAGE — how much is it asking of the socket?

DCGM_FI_DEV_CLOCK_THROTTLE_REASONS — who is stepping on the brake?

Family 4 — Health (the catastrophic reports)

DCGM_FI_DEV_XID_ERRORS — the driver’s red codes

DCGM_FI_DEV_ECC_DBE_VOL_TOTAL — the errors that corrupt data

DCGM_FI_DEV_RETIRED_DBE — the pages marked for retirement

The five vLLM engine metrics

vllm:num_requests_running — how many requests fit in the batch?

vllm:num_requests_waiting — the primary saturation indicator

vllm:gpu_cache_usage_perc — the KV cache pool

vllm:time_to_first_token_seconds — the latency the client sees

vllm:time_per_output_token_seconds — streaming smoothness

The operational rule: read metrics by family, not in isolation

Three anti-patterns of the novice operator

Applied to typical on-premise hardware

What we have not covered (upcoming posts)

See also

References

Knob 1 — `NCCL_DEBUG` + topology dump: see what is happening before touching anything

Knob 2 — `NCCL_ALGO`: ring vs tree vs NVLS

Knob 3 — `NCCL_PROTO`: LL / LL128 / Simple

Knob 4 — `NCCL_NVLS_ENABLE`: offload the summation to the NVSwitch

Knob 5 — `NCCL_MIN_NCHANNELS` / `NCCL_MAX_NCHANNELS`: how many SMs communication steals

Knob 6 — `NCCL_BUFFSIZE`: the buffer size per channel

Knob 7 — `NCCL_P2P_LEVEL` / `NCCL_P2P_DISABLE`: guarantee P2P over NVLink

Knob 8 — GPUDirect RDMA for multi-node: `NCCL_NET_GDR_LEVEL`

Knob 9 — `NCCL_IB_HCA` / `NCCL_SOCKET_IFNAME`: pin the right NIC

`DCGM_FI_PROF_SM_OCCUPANCY` — is there parallel work in the engines?

`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE` — are the tensor cores producing?

`DCGM_FI_PROF_DRAM_ACTIVE` — is the HBM saturated?

`DCGM_FI_DEV_FB_USED` — how much VRAM has been consumed?

`DCGM_FI_DEV_FB_FREE` — the absolute complement

`DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL` — does the internal bus hold up?

`DCGM_FI_DEV_GPU_TEMP` — is the GPU breathing?

`DCGM_FI_DEV_POWER_USAGE` — how much is it asking of the socket?

`DCGM_FI_DEV_CLOCK_THROTTLE_REASONS` — who is stepping on the brake?

`DCGM_FI_DEV_XID_ERRORS` — the driver’s red codes

`DCGM_FI_DEV_ECC_DBE_VOL_TOTAL` — the errors that corrupt data

`DCGM_FI_DEV_RETIRED_DBE` — the pages marked for retirement

`vllm:num_requests_running` — how many requests fit in the batch?

`vllm:num_requests_waiting` — the primary saturation indicator

`vllm:gpu_cache_usage_perc` — the KV cache pool

`vllm:time_to_first_token_seconds` — the latency the client sees

`vllm:time_per_output_token_seconds` — streaming smoothness